Concern is gathering over the effects of the backdoor inserted into SolarWinds' network monitoring software on Britain's public sector – as tight-lipped government departments refuse to say whether UK institutions were accessed by Russian spies.
As reported in the small hours of this morning by The Register, it appears the downloads page for SolarWinds' Orion Windows monitoring platform was altered by Kremlin hackers – known as APT29, aka Cozy Bear – so that victims fetched and installed a tampered-with version that included a remote-control backdoor.
This malicious code was detailed by FireEye, which itself said it was earlier hacked by state-level miscreants. Said victims of the Orion job are said to include the Treasury and the Dept of Commerce at the US government. It's not clear at this stage whether FireEye was also hacked via a dodgy Orion install.
Research by The Register has shown that SolarWinds' Orion is used widely across the British public sector, ranging from the Home Office and Ministry of Defence through NHS hospitals and trusts, right down to local city councils.
A job advert for the MoD's Corsham tech bunker lists SolarWinds as one of the tools used by a third-line software support engineer; similarly, a network design engineer job with the MoD's Defence Equipment and Support agency posted in May also listed SolarWinds proficiency as a "nice-to-have" skill.
A list of SolarWinds' UK customers taken from a marketing presentation issued by the company. Click to enlarge.
SolarWinds' products are in regular use in the Royal Navy and Royal Air Force, with the agency also counting GCHQ, the Cabinet Office, and the Ministry of Justice among its customers. Most concerningly, a company brochure [PDF] also stated that the MoD's Defence Equipment and Support agency was a SolarWinds customer. DE&S is the agency that maintains Britain's high-tech fighter jets, submarines, and warships.
Local governments also facing possible storm
Down at local government level, the three London boroughs of Brent, Lewisham, and Southwark all use SolarWinds Orion as part of a joint backend IT venture. Meeting minutes [PDF] from July revealed: "The service has also standardised on the Orion SolarWinds monitoring product, initially for the network infrastructure, but this will be expanded to cover other key components such as server compute and storage."
Other councils around the country also use the product, with the National Cyber Security Centre (NCSC) advising orgs using it to "have these instances installed behind firewalls, disabling internet access for the instances, and limiting the ports and connections to only what are critically necessary".
Yet government departments fobbed off El Reg's questions about the hack, referring us to the NCSC's public statement, which merely said it was "working closely with FireEye and international partners on this incident."
Microsoft has published a detailed technical blog about the SolarWinds compromise, speculating that the Russians may have "compromised internal build or distribution systems of SolarWinds, embedding backdoor code into a legitimate SolarWinds library with the file name SolarWinds.Orion.Core.BusinessLayer.dll". SolarWinds' customers are being urgently advised by the firm to upgrade to Orion Platform version 2020.2.1 HF 1 "as soon as possible to ensure the security of your environment."
And speaking of Microsoft, SolarWinds, in a filing [PDF] to America's securities watchdog, said its Office 365 email and productivity suite account was hacked, which could have led to the quiet tampering of its downloads. SolarWinds also said of its more than 300,000 customers, up to 18,000 of them installed the dodgy Orion update – said to include America's Homeland Security among other US government bodies.
The normally talkative cybersecurity sector has been practically silent about the FireEye hack, which sources suggested to The Register was because smaller firms are scared of being seen to criticise one of the industry's largest players. Meanwhile, the NCSC's refusal to answer any questions about the breach suggests its impact may well be larger than officials want to admit.
Nuke it from orbit
The US government's Cybersecurity and Infrastructure Security Agency (CISA) issued an emergency directive on Sunday evening calling for an immediate IT lockdown by government agencies: specifically, pull the plug on anything running Orion.
"Affected agencies shall immediately disconnect or power down SolarWinds Orion products, versions 2019.4 through 2020.2.1 HF1, from their network," the directive stated.
"Until such time as CISA directs affected entities to rebuild the Windows operating system and reinstall the SolarWinds software package, agencies are prohibited from (re)joining the Windows host OS to the enterprise domain."
In addition, Uncle Sam's IT admins are told to block all incoming and outgoing traffic from machines "where any version of SolarWinds Orion software has been installed," conduct forensic analysis of new user or service accounts, and to analyze network logs to look for suspicious behavior.
CISA also warned that even if servers look clear, administrators should "treat all hosts monitored by the SolarWinds Orion monitoring software as compromised by threat actors and assume that further persistence mechanisms have been deployed." ®