Cyber Defence Summit Four researchers from American cybersecurity firm Mandiant have engaged in an eight-month epic battle against hackers behind one of the biggest breaches of this year.
The quartet is not saying who the victim is, nor identifying the attackers. However, it is at the level of, and very-well could be, health insurers Anthem or Premera hit earlier this year.
The breach investigation was so complex and massive that the forensics team tasked with battling the hackers say it is likely the most challenging in the firm's history. That is noteworthy in that the company is among America's most prominent forensics firms.
Investigators Nick Carr and Matthew Dunwoody do not hide their technical admiration for the hacker group they fought; during the live engagement in tit-for-tat hacking and expulsion, the team had to check much of their traditional methods of kicking hackers out of networks at the door.
The offending group, which Mandiant and parent firm FireEye is keeping close to its chest, used remarkably complex and customised malware, backdoors, and tools that rendered much of the forensics men's toolbelt redundant.
“This was the hardest investigation I’ve done,” Carr tells El Reg. “It took eight or nine months and was a very personal engagement.”
More than a 1,000 endpoints were compromised. Some 7,000 MD5 hashes were assigned to the constantly changing malware. Every compromised box had its own unique command and control server that was hosted almost exclusively on hacked third-party sites.
“They were infecting 10 systems a day,” Dunwoody said. “Especially when they were provoked.”
Investigators Matthew Dunwoody (left) and Nick Carr
And they were provoked. The team, which in a very rare exception to the firm's regular jobs included in-house security staff, would mop up one area only to find a backdoor re-emerge behind them. They would shut it off, cutting another of the many tendrils the professional, seven-day-a-week hacking group had into the organisation.
They had to minimise their communications too, so that their opponents could not eavesdrop. But such was the totality of compromise that they did so with the understanding that their enemy knew what they were doing.
Dunwoody calls the hacker's execution of operational security “amazing”. Their handle on stealth was “extreme”. Their backdoors were “one of the coolest things I've ever seen”.
The shape-shifting hacking outfit used Windows Management Instrumentation, custom tools, Kererbos, and Powershell. It didn't need to steal credentials because it could assign its own.
They did “weird one-off things” too, which were an oasis in a desert devoid of tools, tactics, and procedures – the essential tools that forensics researchers need to identify the hallmarks of a hacking outfit.
“We spent time analysing systems with unknown activity, and limited analysis on systems with attacker activity consistent with known tools, tactics, and procedures,” Carr said.
In any regular breach, Mandiant would normally deploy a dozen controllers to sweep through a network in about five days. This job required 40 sweeping every part of every system concurrently. “It had some issues for stability,” Carr said. “It's not something I would recommend doing.”
Much of what the team of four did would not be recommended by Mandiant manuals. Their MIR toolset was rendered a digital doorstop. Malware would be plucked out of the environment and sent to FireEye's FLARE reverse-engineering team for deep analysis.
One of the big wins came with a simple upgrade. The team upgraded to PowerShell version 4, which sported a more robust logging feature. That appeared to catch the enemy off guard and pierced their anonymity enough that they began to leave logs.
The hacked organisation underwent a massive security upgrade so that returning hackers or new belligerents would be slowed by defences and trip through all manner of alarms.
The gang of four now have an edge on the enemy. Carr said they have encountered the mystery hacking team in more breached customer environments and were able to boot them out quickly.
He reckons at least half a dozen organisations are being hosed right now.
“When you have nothing ... you've got to turn their weaknesses into your strengths,” Dunwoody said. “You must match or exceed the attacker's pace, development, visibility, advanced techniques, and intensity.” ®
Darren Pauli travelled to Washington DC as a guest of FireEye.