Emergency services call-handling provider: Ransomware forced it to pull servers offline
Advanced's infrastructure still down and out, recovery to take weeks or more
Advanced, the MSP forced to shut down some of its servers last week after identifying an "issue" with its infrastructure hosting products, has confirmed a ransomware attack and says recovery will be in the order of weeks.
The incident was spotted on 4 August and efforts to contain it resulted in server and network connections being taken offline, causing the loss of service on products used by Health & Care customers. Affected hosted products include Adastra, Caresys, Odyssey, Carenotes, Crosscare and Staffplan.
Some 36 customers from the UK's National Health Service (NHS) use services provided by Advanced, including NHS 111, which provides round-the-clock support such as health information. Adastra, for example, is said to work with 85 percent of NHS 111 Services, and call operators were forced to use pen and paper to keep things running.
The turn of events bore all the signs of a serious security strike and in its latest update on 10 August, Advanced confirmed it fell victim to "ransomware."
Third party forensic specialists at Mandiant and Microsoft DART teams are working with Advanced's techies to "ensure our systems are back online securely with enhanced protections."
Advanced said communication is also being maintained with the NHS, the National Cyber Security Centre (NCSC) and UK data watchdog the ICO.
"We want to stress that there is nothing to suggest that our customers are at risk of malware spread and believe that early intervention from our Incident Response Team contained this issue to a small number of servers," the update says.
No further issues have since been detected, the company added.
As for the way forward? Sources told us on 5 August they were informed that services may resume on 9 August but that was seemingly overly optimistic.
Advanced's update says: "We are rebuilding and restoring impacted systems in a separate and secure environment. To help all customers feel confident in reconnecting to our products once service is restored, we have implemented a defined process by which all environments will be systematically checked prior to securely bringing them online.
This process includes:
- Implementing additional blocking rules and further restricting privileged accounts for Advanced staff;
- Scanning all impacted systems and ensuring they are fully patched;
- Resetting credentials;
- Deploying additional endpoint detection and response agents; and
- Conducting 24/7 monitoring.
Following this, Advanced will bring impacted infrastructure back online and reconnect services "as part of a phased return."
"With respect to the NHS, we are working with them and the NCSC to validate the additional steps we have taken, at which point the NHS will begin to bring its services back online.
- Major IT outage forces UK emergency call handlers to use 'pen and paper'
- Schneider Electric to sell Russian ops to local management
- Hospitals to use startup's AI tech to predict A&E traffic
- UK National Crime Agency finds 225 million previously unexposed passwords
"For NHS 111 and other urgent care customers using Adastra and NHS Trusts using eFinancials, we anticipate this phased process to begin within the next few days. For other NHS customers and Care organisations our current view is that it will be necessary to maintain existing contingency plans for at least three to four more weeks. We are working tirelessly to bring this timeline forward, and while we are hopeful to do so, we want our customers to be prepared. We will continue to provide updates as we make progress."
Advanced said it is the "early stages of our investigation into this incident" and has "not yet confirmed the root cause," which it admitted "may take time."
"With respect to potentially impacted data, our investigation is underway, and when we have more information about potential data access or exfiltration, we will update customers as appropriate. Additionally, we will comply with applicable notification obligations," it adds.
It thanked customers for their "continued patience", adding: "We fully understand the challenges this incident has caused for many of our stakeholders."
A security source close to the matter told us there are suggestions the criminals behind the ransomware could have been in Advanced's network for months, and that hundreds of millions of NHS records may have been captured.
We asked Advanced about this, and whether they are negotiating with the extortionists.
In response, the company sent us a statement from Simon Short, chief operating officer:
"We are continuing to make progress in our response to this incident. We are doing this by following a rigorous phased approach, in consultation with our customers and relevant authorities. We thank all our stakeholders for their patience and understanding as our team works around the clock to resume service as safely and securely as possible." ®