Offbeat

Science

One man's mistake, missing backups and complete reboot: The tale of Europe's Galileo satellites going dark

Also organizational chaos, secrecy and self-regulation


Key details about the failure of Europe's Galileo satellite system over the summer have started to emerge – and it's not pretty.

While one key official has sought to blame a single individual for the system going dark, insiders warn that organizational chaos, excessive secrecy and some unusual self-regulation is as much to blame.

Combined with those problems, a battle between European organizations over the satellite system, and a delayed independent report into the July cock-up, means things aren't looking good for Europe's answer to America's GPS system. A much needed shake-up may be on its way.

In mid-July, the agency in charge of the network of 26 satellites, the European Global Navigation Satellite Systems Agency (EGSA), warned of a “service degradation” but assured everyone that it would quickly be resolved.

It wasn't resolved however, and six days later the system was not only still down but getting increasingly inaccurate, with satellites reporting that they were in completely different positions in orbit than they were supposed to be – a big problem for a system whose entire purpose is to provide state-of-the-art positional accuracy to within 20 centimeters.

Billions of organizations, individuals, phones, apps and so on from across the globe simply stopped listening to Galileo. It's hard to imagine a bigger mess, aside from the satellites crashing down to Earth.

But despite the outage and widespread criticism over the failure of those behind Galileo to explain what was going on and why, there has been almost no information from the various space agencies and organizations involved in the project.

Inquiry

In September, it was announced that there would be an independent inquiry into what happened – largely as a result of the lack of information. That inquiry's “preliminary recommendations” were due in October – last month. So far, nothing.

Then, earlier this week, the man in overall charge of the system, the EC's deputy director general in charge of space and defense industries Pierre Delsaux, broke the silence at a breakfast meeting on the EU's space policy in Washington DC no less.

In a Q&A session after the presentations, Delsaux was asked about press criticism – including from El Reg – about the lack of communication and transparency and no apparent backup for the system. He blew up, insisting that the problem had been caused by a single individual who made an error and then failed to take the right action to fix it. The error was "unacceptable", he told the audience before declaring "never again!"

Delsaux failed to address the backup question or the lack of communication or transparency. But other EC officials pushed back on that too, pointing out that a presentation had been given at a recent conference – something which led space observers scrambling to the website of the Institute of Navigation Conference and its Miami conference in September.

One of those digging into what is going on has been Bert Hubert, a DNS expert who became intrigued by the Galileo mess this summer and decided to set up an independent resource that would monitor how the system is doing. This week Hubert posted a report into what he's discovered since undertaking that project, including some of the organizational and political problems at the heart of the Galileo.

Ain't got the numbers

Among the most notable details surrounding Galileo are that of its 26 satellites in space, only 21 of them are functional – and there needs to be a minimum of 24 to achieve the accuracy that the system is designed to provide.

More satellites are due to go up next year but those launches look increasingly uncertain, especially with a bun fight going on between the European Space Agency (ESA) and the European Union (EU).

ESA built the Galileo system and is working on the updated version of the system, including new satellite additions. But thanks to European politics, made worse by the UK's Brexit process, the EU now wants to assert more control over the project.

The EU is planning to create a new European Space Agency, called EUSA, which will largely be a renaming of the existing Global Navigation Satellite System agency. Yet another space entity, GSA, will become the EU Agency for the Space Program, and the EC will soon have a new director general position in charge of the “defense industry and space.” In short, there are a lot of political maneuverings and that is causing all kinds of other problems.

Rundown

In the middle of all that comes the complete failure of Europe's flagship satnav system, Galileo, with no one clearly explaining how or why it happened. Here's what we do know based on the report given at the Miami conference in September and additional details dug out by Hubert and others.

So those are the best details we have about what actually happened. But there remains precious little information about how and why it all went so wrong in the first place and why adequate recovery systems weren't in place.

Complexity

It looks increasingly likely that the complex set of organizations that are responsible for operating and developing different parts of the system are a significant part of the problem. It became immediately apparent when things went wrong that clear communication within Galileo does not exist and no doubt there was a significant amount of finger-pointing among the various agencies that only made things worse.

A chart showing just some of Galileo operations & control. Pic: Bert Hubert

On top of that, there is a question over whether one organization – GMV – has an additional degree of responsibility for the whole mess. GMV runs no less than three different parts of the Galileo structure.

But most notably in this case, it runs the part responsible for generating the data that went awry in this case – called ephemerides, the Galileo Orbit Synchronization Processing Facility (OSPF). In addition to generating ephemerides, GMV was also selected as the organization responsible for independently reviewing and monitoring that same data – the Galileo Integrity Processing Facility (IPF).

Galileo, Galileo, Galileo, where to go? Navigation satellite signals flip from degraded to full TITSUP* over span of four days

READ MORE

Did the fact that the same company was monitoring itself in part responsible for the collapse of the Galileo system?

As for public communication, no one in the satellite of organizations around Galileo felt they were authorized to talk about what was going on, leaving it up to EC officials – none of whom knew what was going on either. In other words, a classic clusterfuck of communication.

We still don't know exactly what happened but hopefully the independent inquiry will make its report available soon. It is supposed to be finished by the end of the year.

In the meantime, a dangerous amount of political maneuvering means all the engineers are keeping their heads down. Which is a shame because by all accounts, there is a lot of good work going on, not helped by organizational silos.

In short, Galileo is a classic European venture: a great idea with talented people that has turned into a bureaucratic mess in which no one wants to take the blame for problems caused by unnecessary organizational complexity. ®

Send us news
148 Comments

Facebook and Singapore teams looking for ways to get data centres relaxing in moist tropical climes

Heat and humidity are horrible, and your server can’t quaff a cocktail to cool off

Singapore’s two major universities — Nanyang Technological University (NTU) and the National University of Singapore (NUS) — have announced they are establishing a US$17.2M research program to develop data centre cooling tech that works in tropical environments.

“The new Sustainable Tropical Data Centre Testbed (STDCT) — the first of its kind in the tropics — will serve as an innovation hub for academia and industry to work together to future-proof the region’s data centre industry,” said NUS in their canned statement.

The program is primarily funded by National Research Foundation Singapore (NRF) and Facebook, which has made Singapore its Asian home. Singapore’s Infocomm Media Development Authority (IMDA) is supporting the university research, and additional industry partners include Ascenix Pte Ltd, CoolestDC Pte Ltd, Keppel Data Centres, New Media Express Pte Ltd, and Red Dot Analytics Pte Ltd.

Continue reading

Dem, Repub senators propose tax credits for factories that churn out chips on US soil

It'll be absolutely FABS-ulous, Wyden and Crapo promise

US senators introduced a law bill on Thursday offering semiconductor manufacturers tax credits to encourage them to build more fabrication plants on American soil as well as specialized equipment for chips.

The bipartisan Facilitating American-Built Semiconductors (FABS) Act [PDF] was backed by Senate Finance Committee Chair Ron Wyden (D-OR), and Ranking Member Mike Crapo (R-ID). It’s the latest piece of legislation proposed by Congress aimed at incentivizing the production of semiconductors in the United States as the world grapples with a global shortage of electronics components and materials to build everything from cars to video game consoles.

The FABS Act could help bolster US supply chains, meaning businesses would have to rely less on foreign manufacturers, particularly those across East Asia, and be able to more easily secure the kit they need. Chip production in America has fallen from 37 per cent of global output to 12 per cent over the past two decades as companies manufacture parts abroad because it's cheaper, according to the pair of senators.

Continue reading

Google dishes out homemade SLSA, a recipe to thwart software supply-chain attacks

Try it with phish'n'chips

Google has proposed a framework called SLSA for dealing with supply chain attacks, a security risk exemplified by the recent compromise of the SolarWinds Orion IT monitoring platform.

SLSA – short for Supply chain Levels for Software Artifacts and pronounced "salsa" for those inclined to add convenience vowels – aspires to provide security guidance and programmatic assurance to help defend the software build and deployment process.

"The goal of SLSA is to improve the state of the industry, particularly open source, to defend against the most pressing integrity threats," said Kim Lewandowski, Google product manager, and Mark Lodato, Google software engineer, in a blog post on Wednesday. "With SLSA, consumers can make informed choices about the security posture of the software they consume."

Continue reading

FYI: There's a human-less, AI robot Mayflower ship sailing from the UK to US right now

Follow this Plymouth to Plymouth trip online

A 15-metre-long autonomous human-free ship has begun a 3,000-odd-mile trip from the UK to the US to recreate the historic Mayflower voyage of 1620.

The Mayflower Autonomous Ship (MAS), named after the 17th century vessel that took more than 100 passengers from England to the present day's Massachusetts in America, set sail this week. It was launched from the seaside town Plymouth on Tuesday at 0400 UTC, and if all goes to plan, it’ll reach its American counterpart, the coastal city of Plymouth, MA, in a few weeks.

Built by Promare, a maritime non-profit, with technical support from IBM, the Mayflower has been testing its sailing and navigation skills since September last year to prepare for this trip. It uses radar to detect objects within 2.5 nautical miles, and a suite of cameras connected to an object-recognition system. This machine-learning software has been trained to identify cargo ships, fishing vessels, and floating shipping containers.

Continue reading

TITAN crypto-token does the opposite of zero to $60: Value plummets in hours

And did a code bug throw a spanner in the works for IRON investors?

A cryptocurrency token called TITAN collapsed on Wednesday, going from about $60 apiece to near zero in a matter of hours. The sales frenzy, attributed to a sell-off driven by whales – people who hold large amounts of a cryptocurrency in this context – also destabilized a so-called stablecoin known as IRON.

A stablecoin is pegged to a reserve asset like the dollar, the idea being there's less volatility when investors know they can redeem their fanciful crypto credits for paper depicting dead presidents or some other thing likely to retain value.

IRON, however, proved to be anything but stable because it is only partially collateralized (about 75 per cent) by the US dollar. The remainder of its value came from TITAN tokens, and when the TITAN price collapsed, IRON took a hit too. Among those caught up in the crypto run was tech moneybags Mark Cuban.

Continue reading

Ex-Brave staffer launches GDPR sueball in Germany over tech giants' real-time bidding for ad inventory

Privacy browser's former chief policy officer calls web advertising ecosystem 'the Biggest. Data. Breach. Ever'

Former Brave chief policy officer Johnny Ryan is continuing his crusade against the online advertising industry by filing a lawsuit against Google, Facebook, Amazon, Twitter, and US telco AT&T in Germany.

Ryan's latest campaign organisation, the Irish Council for Civil Liberties (ICCL), said in a statement that online advertising amounts to "the Biggest. Data. Breach. Ever" and accusing internet adland of compiling "secret dossiers" on every single netizen.

"These secret dossiers about you – based on what you think is private – could prompt an algorithm to remove you from the shortlist for your dream job," said Ryan. "A retailer might use the data to single you out for a higher price online. A political group might micro target you with personalised disinformation."

Continue reading

Roger Waters tells Facebook CEO to Zuck off after 'huge' song rights request

Ex-Pink Floyd uber-grouch calls social media mandroid 'one of the most powerful idiots in the world'

Grouchy former Pink Floyd bassist/vocalist Roger Waters launched an expletive-laden attack on human-impersonating Facebook CEObot Mark Zuckerberg after receiving a request from Instagram to use one of his songs in a promotional film.

Speaking at an event in New York to advocate for the release of Julian Assange, Waters held up a printed piece of paper he said he received "on the internet this morning", which included a request to use the Pink Floyd song 'Another Brick In The Wall (Part 2)' in exchange for what Waters described as "a huge, huge amount of money."

Continue reading

Chrome 'Conformance' for JavaScript frameworks says: If you don't follow our rules, your project won't build

Google knows best

Google's Chrome team has introduced projects to assist framework authors with what it considers best practice, starting mainly with the React-based Next.js.

A team of six people (apparently known internally as WebSDK) who work on Chrome introduced what it called project Aurora earlier this week, described as a "collaboration with framework authors." The post talks up the benefits of "strong defaults and opinionated tooling," based on experience with Google applications such as Search and Maps.

Aurora, said Google, will identify weak spots in web frameworks, specifically those that cause "user experience pain," and then fix them in a manner adaptable to other web frameworks as well. All the frameworks mentioned are JavaScript or TypeScript (which compiles to JavaScript), as you would expect from a browser team.

Continue reading

Your spacesuit ran into a problem and needs to restart

ISS solar array installation overran after a good old 'off and on again'

There are two things a spacewalker doesn't want to hear: "Can you turn it off and turn it on again?" and "What's that hissing sound?"

The IT solution of the ancients reached orbit yesterday as one of a pair of astronauts tasked with fitting a new solar array to the International Space Station (ISS) had to make his way back to the airlock in order to restart his spacesuit.

NASA astronaut Shane Kimbrough was not in any great danger during what the US space agency delicately called an "issue" with his spacesuit's display and control module (designed to provide a spacewalker with information on the status of the suit). Controllers also noted a spike in the pressure reading for his sublimator (used to keep things cool) and so sent the astronaut back to the airlock to perform a restart.

Continue reading

Space Force turtle expert uncovers $1.2m Cape Canaveral cocaine haul

30kg stash lost overboard by smugglers enough to get anyone out of their shell

A member of the newly inaugurated US Space Force discovered more than she bargained for as she conducted a survey of turtle nests on the coast around Cape Canaveral last month.

Angy Chambers, a civil engineer and wildlife manager with the 45th Civil Engineer Squadron, was forced to suspend her check on testudinal housing conditions when she noticed that packages containing $1.2m worth of cocaine had washed up on the beach.

Chambers contacted the 45th Security Forces Squadron – another component element of Space Launch Delta 45, the new Space Force unit in charge of Cape Canaveral – to ask them to secure the haul.

Continue reading

Graph DB slinger Neo4j secures $325m round of funding for $2bn valuation

Also touts sharded graph application running on 1,000 servers

Neo4j has secured another $325m in a funding round and said it was ready to demo a distributed graph database with a trillion relationships, sharded across 1,000 servers, returning queries in a matter of milliseconds.

New investors DTCP – formerly Deutsche Telekom Capital Partners – and Lightrock join existing investors One Peak Partners, Creandum, and Greenbridge Partners.

The latest injection of funds – which is said to value Neo4j at $2bn – adds to earlier rounds including $10.6m in 2011, $11m in 2012, $20m in 2015, $36m in 2016, and $80m in 2018.

Continue reading