20 years deep into a '2-year' mission: How ESA keeps Cluster flying

Growing extra instruments, reducing fuel spend and those clever, clever hacks


Space Extenders ESA's Cluster mission is heading into its third decade of operations. The Register spoke to some of the people behind the four spacecraft about how the team turned a five-year nominal lifetime into 20 years and beyond.

The four identical Cluster spacecraft, flying in a tetrahedral formation, are designed to study structures in the Earth's magnetosphere in three dimensions. The proposal for the mission was made in 1982. By 1996 Cluster was ready to be launched.

Unfortunately (as it turned out) the opportunity to take advantage of a free ride on the first flight of the new Ariane 5 rocket was impossible to resist. The launch on 4 June 1996, carrying all four satellites, ended in failure. The Ariane 5 was destroyed, and the wreckage of years of work on Cluster was scattered over the swamps.

Undeterred, ESA set about building a fifth Cluster spacecraft, using flight spares for a potential launch in 1997 before deciding that, heck, the design and development was done, and a single Cluster spacecraft wouldn't meet all the objectives, why not build more and catch the peak of the solar cycle in 2000?

Cluster (or Cluster II) would use a pair of Soyuz boosters this time rather than an Ariane 5 and the four spacecraft named Rumba, Salsa, Samba and Tango (named by Raymond Cotton, a Bristol-based civil servant from the UK) were successfully launched in 2000 from the launchpad at the Baikonur Cosmodrome.

The mission was initially planned to run for two years, and the nominal lifetime of the spacecraft was between four and five years.

The Cluster spacecraft themselves weighed in at approximately 1,200kg at launch, including 650kg of fuel (most of which was used up just after launch to get the spacecraft to their operational orbits). The cylindrical vehicles (measuring 2.9m in diameter and 1.3m high) were wrapped in six curved solar panels and used five silver-cadmium batteries to keep things ticking over while the spacecraft were in the Earth's shadow.

About those batteries

We spoke to Bruno Sousa, who became operations manager for the mission five years ago, about how the team had kept the four Cluster spacecraft flying for so long.

Unsurprisingly, the batteries eventually became a challenge.

"The batteries," he told us, "were designed to produce low electromagnetic noise, to make sure they wouldn't interfere with the science."

Certainly, for a mission concerned with measuring magnetic fields, the interference from regular powerpacks would not be welcome. As such, the manufacturers came up with something a bit special, but with a slightly limited lifespan.

"It was only supposed to last two years," recalled space plasma physicist, Philippe Escoubet, who has been on the Cluster project since 1993, "but by doing maintenance of the battery, discharging and recharging slowly, we could keep them for five years."

Delighted to see that the spacecraft could survive the plug-pulling, the gang investigated further and found logic lurking within the Cluster satellites that would stop them booting autonomously...

"They were probably designed better," said Sousa. "They lasted actually for 10 years."

"But after five years," said Escoubet, "they started to crack."

"They were cracking and leaking," Sousa explained. The team were able to detect the leakage from the thermal behaviour of the spacecraft and by tiny orbit changes. The latter could only be explained by something leaving the spacecraft "and that was fluid from the batteries," said Sousa (who had yet to join the team.)

At first the team adjusted the power load drawn from the batteries while the spacecraft were in eclipse. Escoubet recalled the introduction of a new mode to the spacecraft, where it could run on minimal power "and listen to commands from the ground".

Since everything eventually had to be turned off in this mode, other than the radio, to preserve the batteries, handling an eclipse with the heaters deactivated presented its own thermal problems.

"We need to preheat the spacecraft before they enter the eclipse, such that they don't get too cold," Sousa explained. Frozen fuel and a damaged radio amplifier would be less than ideal.

"We had a PhD doing some analysis in our thermal behaviour. He came up with some recommendations in terms of how long we needed to heat up and provided a strategy, which we implemented."

"It has worked fine since then."

However, the power situation continued to deteriorate and eventually the team were forced to fully discharge the batteries and keep them at zero capacity to avoid any further leakage or contamination.

"At 10 years," Escoubet said, "it became a mission where we did not have any more battery."

"From this point onwards," Sousa told us, "the spacecraft actually had to go through a full reset." An eclipse meant no Sun on the solar arrays, which in turn meant no power and a shutdown for Cluster.

"The first time it happened," said Sousa, "this was a rather uncontrolled shutdown... the spacecraft were not designed to be switched off."

With little choice in the matter, the team was forced to hope for the best as the spacecraft essentially died in orbit.

Power from the solar arrays eventually brought the vehicles back to life as they emerged from eclipse "and things would start to come up in some random order, which would often leave the spacecraft in an inconsistent state".

Delighted to see that the spacecraft (and more importantly, the radios) could survive the plug-pulling, the gang investigated further and found logic lurking within the Cluster satellites that would stop them booting autonomously.

"This was probably set there during development on the spacecraft so they could control the activation," explained Sousa, "and it was not meant to be used during flight.

"It was only when we started having to switch them off during eclipses that we figured out this logic was implemented there, and we could use it for our benefit."

No booting meant no uncontrolled activation.

"We would then pick up the spacecraft in the blind, start radiating commands and slowly going through the switch on procedure."

It was a rather manual process as the team became accustomed to the new way of working.

"In 2011," said Sousa, "the orbits were low. They were basically having eclipses for 11 months in a row and our orbit has 54 hours of duration for each orbit. So that meant every two days they had an eclipse and they had to do all this cycle of switching it off switching it back on."

2011 brought with it other problems (more on those later) and while the team did manage to get some science done and dump the data before the next eclipse, it was very labour-intensive.

The team slowly developed new tools to automate the process. When they began, "it would require sometimes three to four hours on each spacecraft to get them fully back reconfigured. And it meant for all four spacecraft about 15 hours of work."

Nowadays, Sousa told us, "we have an automated procedure on the ground that starts commanding the spacecraft in the blind and does the recovery of a single spacecraft in about 40 minutes." Parallel operations mean that sometimes all the spacecraft are up and running in under two hours.

Youtube Video

"Now we have been through more than 1,000 eclipses," said Escoubet, "and it has worked all the time."

"With every year we basically learned something," explained Sousa, "and we adapt the next year to do it a bit more efficient."

The 'Dirty Hack' revisited

Escoubet reminded us of another infamous near-death experience that occurred in 2011.

"We realised in 2011," he remembered, "that we could not switch on half of the payload anymore on one spacecraft." The spacecraft in question was Samba.

"It was consuming too much current, and nobody knew what it was, if it was a failure of a component or some programming issue. We tried again, and after a fraction of a second the spacecraft switched off the five instruments."

The five comprised the Wave Experiment Consortium (WEC), used for measuring the electrical and magnetic fields. Losing the payload would have a serious impact on the mission's "four-spacecraft" science.

The Register noted this event at the time: ESA's investigations had shown the five power switches locked in the "closed position", something from which there were no recovery procedures.

"We found tests that were done in 1994," recalled Escoubet, "and realised that the current consumption with the five instruments switched on at the same time would not work.

"They would have to switch on only three at a time and then the last two later on. So, we had the problem that we did not have enough current in the spacecraft to do that."

However, what the team did have was two current lines, of which only one was normally used. If that failed, then the system could switch over. "But then," explained Escoubet,"we got the idea to use both in parallel and get enough current for the five instruments to switch on."

"However, we had this problem that to switch on both powerlines in parallel you needed to send two commands. And the minimum time between these two commands could be only 39 milliseconds."

"We needed 11 milliseconds."

Time to wheel out those hacker skills.

"They decided to hack the system to use only one command but to quickly change the memory the command was using. By changing the memory, they could then switch on the back-up line."

It was indeed a "Dirty Hack" as ESA put it. The team tried it on a healthy spacecraft, and it worked. They then tested it on the spacecraft with the problem without success. After what we suspect were a few deep breaths and five or ten minutes, the team tried again and "it worked."

"It was a clever trick," said Escoubet proudly, before laughing: "maybe more clever than the thing with the batteries!"

"Dirty Hack" or not, the instruments were returned to normal operation.

Running on (near) empty

The original mission plan called for the spacecraft to use separation distances of anywhere from 600km to 20,000km depending on the needs of the science, while the highly elongated orbits could see the spacecraft 119,000km from Earth, and other times considerably closer.

Much of the fuel carried by quartet was expended in reaching their operational orbits and then, according to Sousa "a lot of it was used on the first five years to aggressively change the formations of the spacecraft."

This was all well and good with the original nominal mission length, but as the spacecraft demonstrated their longevity, the team realised that the fuel would soon be exhausted. Sousa reckoned the timeframe was 2005 to 2008.

Once again, the team adapted to cope as Sousa explained: "Instead of doing aggressive formation changes in consecutive orbits, they started doing small drifts." These would slightly change the orbits of the spacecraft in relation to each other. "We would wait about a month until the drift had accumulated long enough and then we would stop it with another small manoeuvre."

Back in 2007, the drift approach saw the Samba and Tango spacecraft achieve a separation distance of just 17km.

"Just pushing or giving a kick of one spacecraft you use maybe a few hundred grams of fuel, but if you wait long enough, it will drift on orbit for a few hours, so we can reach thousands of kilometres or also very small distances, down to three kilometres" explained Escoubet.

The science approach had changed, but the fuel consumption was reduced drastically: "at the moment, we are only spending between 100 and 300 grams of fuel per year," said Sousa.

The best estimate has 6kg of fuel left in the spacecraft (about 1 per cent of the original) although Sousa cautioned that "the uncertainty of that value is in the order of three kilos. So maybe we have three, maybe we have nine."

Enough to see the spacecraft through to their eventual re-entry as the orbits naturally decay.

The end will begin in 2024, when the first of the Cluster quartet enters the Earth's atmosphere. Another will fall in 2025 and the final two will burn up in 2026, more than a quarter of a century after launch.

Growing extra instruments, Mars Express style

As with the Beagle 2 departure camera on Mars Express, the Cluster gang "rediscovered" a camera on the spacecraft late into the mission.

Originally added to snap images of the separation of the Cluster spacecraft after launch (and maybe capture a slice of the Earth), the cameras had lain dormant for 15 years. They had been used successfully on one of the Soyuz launches, but not worked in the other due to a spacecraft reboot in the launcher.

"When I joined the mission and was learning about the spacecraft, I learned there were a couple of cameras.

"The question was, could these cameras actually photograph the earth?"

The answer was a blunt "no". The position of the cameras on the spacecraft meant they were usually pointed at the "sky" rather than directly at the Earth.

"But we looked at it carefully, we went to the user manual to see the details of the cameras, the field of view. And we did some calculations. And we figured that there were some moments in our orbit, where the earth might just pass in the field of view and we would be able to see something."

The team decided to try it out. The camera needed a little commissioning to bring it online and the team ensured it could be swiftly shut down if it consumed too much power or interfered with the rest of the payload.

Once confident the thing was actually working, pictures were snapped and stored in the onboard mass memory.

"And we were actually lucky on the first attempt; we saw on the first test pictures one of our low gain antennas."

"We then had the camera take pictures when we expected the Earth to be nearby and sure enough, we actually got it in the corner of the field of view.

"So, after that we did several campaigns to use the camera, particularly when we were not doing science, like around Eclipse period. We would switch on the camera, take some pictures and play a bit with the with the exposure until we actually got some nice shots from the South Pole upwards. We can see in some pictures we can see Australia and the tip of South Africa under the clouds."

The cameras are not used regularly: "it was purely for public relations" laughed Sousa, although we'd contend that repurposing an instrument that had been deactivated for 15 years in space sounds like a lot of engineering fun.

Epilogue

The bonus science from the long-lived mission is outstanding. Escoubet explained that the mission has now been able to observe almost two solar cycles, the approximate 11-year change in the Sun's activity. "This is very important to see trends and observe differences," he said.

"The second aspect," he added, "is that we are now doing collaboration with newer missions." The data from Cluster can be combined with other spacecraft to create multi-point observations and track global changes in the Earth's magnetic field.

The spacecraft remain healthy; Escoubet told us that the team experienced some failures just after launch, but otherwise there had been minimal degradation over the years. A transponder has failed on one spacecraft, but the backup continues to function.

However, the continuation of the mission must be justified. Escoubet told us that the team were currently preparing a case, including new science goals, for the extension to 2023 and 2025.

As for the extraordinary longevity, both Escoubet and Sousa paid tribute to the designers of the spacecraft and the support of the manufacturers. "It's a very strong design," said Escoubet, "not too much flexibility, but very strong design."

"The design and implementation of the spacecraft was outstanding," agreed Sousa, paying tribute to the integrators and the experience gained from the first four spacecraft built.

While the computers on the ground have moved on (Sousa explained that the original VAX VMS hardware was long gone, and controllers like things a little more virtualised nowadays - unexpectedly handy for today's remote working needs) the core interface remains the same, albeit augmented by modern tools.

And the spacecraft? "They are just marvellous pieces of technology", said Sousa proudly.

Cluster continues to be operated by a small team and, should funding permit, carry on performing science until the end finally comes. "We got the ESA award for teamwork excellence in 2014," recalled Escoubet happily.

We'll leave the last word to Sousa, current operations manager of a team that has stretched a two-year mission to 20 years and beyond through ingenuity and innovation:

"Never give up. Never surrender." ®

Similar topics


Other stories you might like

  • Electron-to-joule conversion formulae? Cute. Welcome to the school of hard knocks

    Shake, rattle and roll is incompatible with your PABX

    On Call There are some things they don't teach you in college, as a Register reader explains in this week's instalment of tales from the On Call coalface.

    Our reader, safely Regomised as "Col", headed up the technical support team of a PABX telecom provider and installer back in the early 1990s. PABX, or Private Automatic Branch eXchange, was the telephony backbone of many an office. A failure could be both contract and career-limiting.

    Col, however, was a professional and well versed in the ins and outs of such systems. Work was brisk and so, he told us, "I took on a university grad with all the spunk and vigour that comes with it. He knew the electron-to-joule conversion formulae et al."

    Continue reading
  • Korea's NAVER Cloud outlines global ambitions, aim to become Asia's third-biggest provider

    Alibaba is number two in much of the region, but is a bit on the nose right now

    Korean web giant NAVER has outlined its ambition to bring its cloud to the world, and to become the third-largest cloud provider in the Asia-Pacific region.

    NAVER started life as a Korean web portal, added search, won the lion's share of the market, and has kept it ever since. South Korea remains one of the very few nations in which Google does not dominate the search market.

    As NAVER grew it came to resemble Google in many ways – both in terms of the services it offers and its tendency to use its muscle to favour its own properties. NAVER also used its scale to start a cloud business: the NAVER Cloud Platform. It runs the Platform in its home market, plus Japan, Hong Kong, and Singapore. Presences in Taiwan, Vietnam and Thailand are imminent.

    Continue reading
  • Build it fast and they will come? Yeah, but they’ll only stay if you build it right

    Here’s where to start

    Sponsored Developers have never had so much choice. Every week there’s a new framework, API, or cloud service that promises to help deliver software to market faster than ever. And it’s not just tooling. Agile, continuous integration, and DevOps techniques have made teams more efficient, too. But speed brings with it increased expectations. Pressure from customers and colleagues, alongside the burden of staying current with new tooling, can lead to mistakes.

    Whether it’s a showstopping bug that slips through into production or an edge case that lies in wait for years, pressure to deliver is driving some teams to pile up technical debt and mismatched stakeholder expectations.

    What’s the solution? Well, it’s to do what we’ve always done: build on what came before. In the absence of unlimited time and budget, a low-code platform gives both experienced and new developers a suite of tools to accelerate their development. Automation in just the right places lets teams bring their unique value where it really matters, while all the standard building blocks are taken care of.

    Continue reading
  • Royal Navy will be getting autonomous machines – for donkey work humans can't be bothered with

    No robot killers 'in my lifetime' says admiral

    DSEI 2021 The British armed forces will be using robots as part of future warfare – but mostly for the "dull, dangerous and dirty" parts of military life, senior officers have said.

    At London's Defence and Security Equipment International arms fair, two senior officers in charge of digitisation and automation said the near future will be more Wall-E than Terminator – but fully automated war machines are no longer just the stuff of sci-fi.

    Brigadier John Read, the Royal Navy's deputy director of maritime capability, said in a speech the military "must automate" itself so it can "take advantage of advances in robotics, AI and machine learning."

    Continue reading
  • WTF? Microsoft makes fixing deadly OMIGOD flaws on Azure your job

    Clouds usually fix this sort of thing before bugs go public. This time it's best to assume you need to do this yourself

    Microsoft Azure users running Linux VMs in the IT giant's Azure cloud need to take action to protect themselves against the four "OMIGOD" bugs in the Open Management Infrastructure (OMI) framework, because Microsoft hasn't raced to do it for them.

    As The Register outlined in our report on this month's Patch Tuesday release, Microsoft included fixes for flaws security outfit Wiz spotted in Redmond's open-source OMI agents. Wiz named the four flaws OMIGOD because they are astonishing.

    The least severe of the flaws is rated 7/10 on the Common Vulnerability Scoring System. The worst is rated critical at 9.8/10.

    Continue reading
  • Businesses put robots to work when human workers are hard to find, argue econo-boffins

    The lure of shiny new tech isn't a motivator, although in the USA bots are used to cut costs

    Researchers have found that business adoption of robots and other forms of automation is largely driven by labor shortages.

    A study, authored by boffins from MIT and Boston University, will be published in a forthcoming print edition of The Review of Economic Studies. The authors, Daron Acemoglu and Pascual Restrepo, have both studied automation, robots and the workforce in depth, publishing numerous papers together and separately.

    "Our findings suggest that quite a bit of investment in robotics is not driven by the fact that this is the next 'amazing frontier,' but because some countries have shortages of labor, especially middle-aged labor that would be necessary for blue-collar work,” said Acemoglu in a canned statement.

    Continue reading
  • After eight years, SPEC delivers a new virtualisation benchmark

    Jumps from single-server tests to four hosts – but only for vSphere and RHV

    The Standard Performance Evaluation Corporation (SPEC) has released its first new virtualisation benchmark in eight years.

    The new SPECvirt Datacenter 2021 benchmark succeeds SPEC VIRT_SC 2013. The latter was designed to help users understand performance in the heady days of server consolidation, so required just one host. The new benchmark requires four hosts – a recognition of modern datacentre realities.

    The new tests are designed to test the combined performance of hypervisors and servers. For now, only two hypervisors are supported: VMware’s vSphere (versions 6.x and 7.x) and Red Hat Virtualisation (version 4.x). David Schmidt, chair of the SPEC Virtualization Committee, told The Register that Red Hat and VMware are paid up members of the committee, hence their inclusion. But the new benchmark can be used by other hypervisors if their vendors create an SDK. He opined that Microsoft, vendor of the Hyper-V hypervisor that has around 20 per cent market share, didn’t come to play because it’s busy working on other SPEC projects.

    Continue reading
  • Forget that Loon's balloon burst, we just fired 700TB of laser broadband between two cities, says Google

    Up to 20Gbps link sustained over the Congo in comms experiment

    Engineers at Google’s technology moonshot lab X say they used lasers to beam 700TB of internet traffic between two cities separated by the Congo River.

    The capitals of the Republic of the Congo and the Democratic Republic of Congo, Brazzaville and Kinshasa, respectively, are only 4.8 km (about three miles) apart. The denizens of Kinshasa have to pay five times more than their neighbors in Brazzaville for broadband connectivity, though. That's apparently because the fiber backbone to Kinshasa has to route more than 400 km (250 miles) around the river – no one wanted to put the cable through it.

    There's a shorter route for data to take between the cities. Instead of transmitting the information as light through networks of cables, it can be directly beamed over the river by laser.

    Continue reading
  • Apple's M1 MacBook screens are stunning – stunningly fragile and defective, that is, lawsuits allege

    Latest laptops prone to cracking, distortions, owners complain

    Aggrieved MacBook owners in two separate lawsuits claim Apple's latest laptops with its M1 chips have defective screens that break easily and malfunction.

    The complaints, both filed on Wednesday in a federal district court in San Jose, California, are each seeking class certification in the hope that the law firms involved will get a judicial blessing to represent the presumed large group of affected customers and, if victorious, to share any settlement.

    Each of the filings contends Apple's 2020-2021 MacBook line – consisting of the M1-based MacBook Air and M1-based 13" MacBook Pro – have screens that frequently fail. They say Apple knew about the alleged defect or should have known, based on its own extensive internal testing, reports from technicians, and feedback from customers.

    Continue reading
  • Microsoft's Azure Virtual Desktop now works without Active Directory – but there are caveats

    General availability of Azure AD-joined VMs

    Microsoft has declared general availability for Azure Virtual Desktop with the VMs joined to Azure AD rather than Active Directory, but the initial release has many limitations.

    Azure Virtual Desktop (AVD), once called Windows Virtual Desktop, is Microsoft's first-party VDI (Virtual Desktop Infrastructure) solution.

    Although cloud-hosted, Azure Virtual Desktop is (or was) based on Microsoft's Remote Desktop Services tech which required domain-joined PCs and therefore a connection to full Windows Active Directory (AD), either in the form of on-premises AD over a VPN, or via Azure Active Directory Domain Services (AAD DS) which is a Microsoft-managed AD server automatically linked to Azure AD. In the case that on-premises AD is used, AD Connect is also required, introducing further complexity.

    Continue reading
  • It's bizarre we're at a point where reports are written on how human rights trump AI rights

    But that's what UN group has done

    The protection of human rights should be front and centre of any decision to implement AI-based systems regardless of whether they're used as corporate tools such as recruitment or in areas such as law enforcement.

    And unless sufficient safeguards are in place to protect human rights, there should be a moratorium on the sale of AI systems and those that fail to meet international human rights laws should be banned.

    Those are just some of the conclusions from the Geneva-based Human Rights Council (HRC) in a report for the United Nations High Commissioner for Human Rights, Michelle Bachelet.

    Continue reading

Biting the hand that feeds IT © 1998–2021