On-Prem

Networks

That Salesforce outage: Global DNS downfall started by one engineer trying a quick fix

'We feel bad about what happened'


The sound of rumbling rubber could be heard today as Salesforce threw an engineer responsible for a change that knocked it offline under a passing bus.

"We're not blaming one employee," said Chief Availability Officer Darryn Dieken after spending the first half hour of a Wednesday briefing on the outage doing pretty much that very thing.

To recap, on May 11 around 2100 UTC, a configuration change was applied to Salesforce's Domain Name System (DNS) servers that resulted in folks unable to access the software-as-a-service titan's products. For about five hours, clients could not reliably log in, and things got so bad that even the status page was unavailable.

Salesforce has been updating its public root cause analysis ever since, and Dieken said during his briefing to customers that a few more tweaks would be needed before the fix was completed.

It was during that call that the full extent of the screw-up was revealed and the engineer concerned launched buswards.

While Dieken boasted of the automation in place within Salesforce towers, some processes remain manual. One of these is related to DNS (yes, it is always DNS.) A lone engineer was tasked with making a configuration change to connect up a new Salesforce Hyperforce environment in Australia.

A DNS change is not an uncommon occurrence, and the engineer also had a four-year-old script to do the job. However, while Salesforce usually "staggers" changes to reduce the blast radius of blunders, the manual nature of this change meant it was up to the engineer to roll it out slowly.

This, alas, did not happen. The engineer instead decided erroneously, according to Dieken, to shortcut the normal procedures by using a so-called Emergency Break-Fix (EBF) process. The EBF is normally used when something really bad is happening, or an emergency patch is quickly and widely needed.

Going down the EBF route meant fewer approvals and a shortened process that wasn't gradual. Hey, this was a well-used script, the engineer had worked for Salesforce for years and these changes were pretty common. What could possibly go wrong?

In classic Who, Me? fashion, rather a lot.

We don't understand

"For whatever reason that we don't understand, the employee decided to do a global deployment," Dieken went on. The usual staggered approach was therefore bypassed. And a DNS change meant those servers would need restarting.

That in itself would not be a total catastrophe. Maybe a short outage, perhaps. But not the disaster that unfolded.

However, it transpired that lurking within that tried-and-trusted script was a bug. Under load, a timeout could happen that would stop other things from running. And sure enough, as the update was being rolled out across all of Salesforce's data centers, a timeout occurred. This in turn meant that certain tasks were not carried out when the servers were restarted. And that, in turn, meant that those servers did not return to operation correctly. That left customers unable to access Salesforce's products.

And then things got even worse. The Salesforce team has tools to deal with sad servers, and use what Dieken called "our emergency break glass process" to perform rollbacks and restarts.

"In this case," he went on, "we found a circular dependency where the tool that we use to get into production had a dependency on the DNS servers being active."

It is always DNS.

We found a circular dependency where the tool that we use to get into production had a dependency on the DNS servers being active

Of course, staff did eventually get into the servers to fix them up but, as customers felt all too keenly, it took quite some time to undo the mess.

As for forthcoming actions, safeguards are to be put in place to stop manual global deployments like this in future, and the process will be automated. Dieken also acknowledged that the experience had shown up a gap in Salesforce's test coverage – the script needed to be better tested, essentially. Finally, that dependency of the recovery tools on DNS needed to be dealt with.

Customers bemused that they had to get official word of the outage from social media or this very organ, rather than the status page, were doubtless more bemused at the revelation that the reason for the Salesforce status site falling over was due to auto-scale not being turned on for that web property. (During the downtime, Salesforce had to use its documentation site to explain to clients what was going wrong.)

"We over-provisioned enough capacity to make sure that we could handle large spikes," explained Dieken, "but we never foresaw that we'd have this type of load."

Not to worry, though, auto-scale is now on, so should things go south again at least the status site is unlikely to be embarrassingly absent.

And the engineer who sidestepped Salesforce's carefully crafted policies and took down the platform? "We have taken action with that particular employee," said Dieken. ®

Send us news
68 Comments

Petition instructs Jeff Bezos to buy, eat world's most famous painting

Booze-fuelled Change.org campaign implores Amazon founder to 'GOBBLE DA LISA!'

Ultra-billionaire Amazon founder Jeff Bezos has already been the subject of a petition asking him not to return to Earth after he blasts off in his New Shepard rocket on July 20, but even if he is allowed back, Bezos is now facing an even more difficult prospect.

The aerodynamically-pated arch-villain archetype and his vast fortune are increasingly becoming subjects of fascination for the denizens of campaign website Change.org, with multiple petitions currently running, mostly trying to persuade him to divert some of his almost-limitless resources toward good causes.

However, some users are suggesting more novel and entertaining uses for his immense wealth. Change.org user Kane Powell has chosen to use the platform to attempt to persuade Bezos to buy and eat the Mona Lisa, the supposedly priceless Leonardo da Vinci masterpiece housed in the Louvre in Paris.

Continue reading

Microsoft: Try to break our first preview of 64-bit Visual Studio – go on, we dare you

Plus: Updates to .NET 6, ASP.NET Core, and .NET MAUI

Microsoft has unveiled a slew of developer tools, including a preview of the 64-bit Visual Studio 2022, ahead of that developer event set for 24 June.

Preview 1 of Visual Studio 2022 comes direct from the department of never-say-never following version after version of the toolset remaining staunchly 32-bit, even as the hardware world changed around it.

The move to 64-bit was announced earlier this year and is an ambitious one considering the ecosystem and sheer size of the Visual Studio codebase.

Continue reading

Racist malware blocks The Pirate Bay by tampering with victims' Windows hosts file

Hello, 2002 called with one of the oldest low-tech tricks in the book

Malware laced with racial epithets tries to block Windows-based victims from visiting file-sharing sites associated with copyright infringement, according to new Sophos research.

The malicious software amounts to a "goofy process to block people from going to the Pirate Bay," according to Sophos researcher Andrew Brandt, who stumbled across the malware after a colleague mentioned it in passing.

Rather than opening a backdoor for a ransomware gang to exploit or dropping a malicious payload, however, this malware merely sinkholes a bunch of Pirate Bay domain names by adding them to the Windows hosts file and pointing them at 127.0.0.1 – meaning they'll be inaccessible from the victim's machine.

Continue reading

UK gets glowing salute from Bezos-backed General Fusion: Nuclear energy company to build plant in Oxfordshire

Biz will develop Magnetized Target Fusion technology at the site

General Fusion – the Canadian-based atomic outfit backed by Jeff Bezos and a battalion of other major investors – is to build a test facility in Oxfordshire to showcase its power-generating technology.

Following a COVID-friendly handshake, the UK Atomic Energy Authority (UKAEA) has given General Fusion the green light to proceed with its Fusion Demonstration Plant (FDP) at UKAEA's Centre for Fusion Energy Campus in Culham.

The campus – a Royal Navy airbase until it was handed to the UKAEA in 1960 – is home to a cluster of fusion development technologies.

Continue reading

UK financial watchdog dithers over £680k refund from Google (in ad credits, mind you) for running anti-fraud ads

MPs give FCA a telling-off for wasting taxpayer money

The UK's financial regulator is refusing to say whether it will accept an offer by Google to pay back more than £600,000 spent on online ads warning people about the dangers of money scams.

News that Google made the offer came to light earlier this week during oral evidence [PDF] to the Treasury Committee hearing on economic crime. Among those giving evidence was Mark Steward, director of enforcement and market insight at the Financial Conduct Authority (FCA).

He was quizzed by Rushinara Ali, Labour MP for Bethnal Green and Bow, who wanted to know about the £600,000 the FCA is paying Google to run ads warning about online financial scams.

Continue reading

CREST president Ian Glover to retire after 13 years – but where's the transparency, bossman?

UK infosec accreditation body still won't publish exam cheatsheet scandal report nor be interviewed by El Reg

Ian Glover, president of infosec accreditation body CREST, is stepping down from his post, he told the organisation's annual general meeting yesterday.

Sources whispered of Glover's departure to The Register ahead of a mass mailout today to members of the organisation, which oversees some industry-recognised penetration testing exams and certifications in the UK.

"My retirement is something I have been planning for some time and, while I leave with a heavy heart, I am confident CREST will continue to move forward in the hands of an excellent team," said the man himself in a canned statement emailed round CREST member organisations, following his 13 years at the helm.

Continue reading

Playmobil crosses the final frontier with enormous, metre-long Enterprise playset

$500, 136-piece, tribble-laden Star Trek tribute is immense, but clearly illogical

Playmobil is set to boldly go where no three-inch man has gone before with the release of a metre-long replica of the NCC-1701 USS Enterprise from the original Star Trek series.

The enormous model of the Federation Constitution-class vessel will come with standard-scale figures representing the main original series characters – Captain Kirk, Mr Spock, Dr McCoy, Chief Engineer Scott, Lieutenant Uhura, Lieutenant Sulu and Ensign Chekov – and features a removable panel on the disc section revealing "a full 1966-style bridge play environment" to allow children of all ages to recreate their favourite first-contact scenes.

Continue reading

Open standard but not open access: Schematron author complains about ISO paywall

'This is shooting Schematron in the heart ... its heart is individual open source developers'

The original inventor of a popular XML standard, Rick Jelliffe, who created Schematron, has protested that his open source work is now behind a paywall at standards body ISO.

Schematron is a language for validating XML, designed for processing XML documents and reporting on errors. Version 1.0 was developed in 1999, since when it has been enhanced and standardised, with the latest version being ISO/IEC 19757-3:2020.

This replaced the 2016 version: ISO/IEC 19757-3:2016.

Continue reading

Vissles V84: Mechanical keyboard hits all the right buttons for Mac power users

Ideal for Apple fans who appreciate little boxes made of clicky-clacky

Review Mechanical keyboard manufacturers have typically swerved Mac users. It's not personal, it's just business.

The Mac has a fraction of the traditional PC market share, and a significant proportion of mechanical keyboards are intended for competitive gamers, rather than those who type for work (be they developers or writers, or in the case of your correspondent, both).

The Vissles V84 is therefore a bit of an oddity. This compact keyboard (84 keys) ships with a Mac layout by default, although it comes bundled with standard Windows keycaps, as well as the ability to switch into a standard PC layout by pressing down a key combination.

Continue reading

Wanted: Brexit grand fromage. £120k a year. Perks? Hmmmm…

Successful applicant must 'change existing thinking', which is odd as it's all been going so well up to now

No 10 Downing Street - the home of the UK Prime Minister - is looking to hire a big cheese at the Brexit Opportunities Unit to bring a fresh new oomph and zing to Whitehall.

The 17-page job spec (downloadable here) – with perks including hybrid working, childcare benefits, a generous pension, and a loan to buy a bike – is brief.

In a nutshell, it’s this: “The Director, Brexit Opportunities Unit is a high-profile role. It needs someone who can change existing thinking, working across all government departments, developing partnerships with senior stakeholders, including the Prime Minister, to ensure ministerial priorities are met.”

Continue reading

Poltergeist attack could leave autonomous vehicles blind to obstacles – or haunt them with new ones

First 'AMpLe' concept proves worryingly simple to implement with success

Researchers at the Ubiquitous System Security Lab of Zhejiang University and the University of Michigan's Security and Privacy Research Group say they've found a way to blind autonomous vehicles to obstacles using simple audio signals.

"Autonomous vehicles increasingly exploit computer-vision based object detection systems to perceive environments and make critical driving decisions," they explained in the abstract to a newly released paper. "To increase the quality of images, image stabilisers with inertial sensors are added to alleviate image blurring caused by camera jitter.

"However, such a trend opens a new attack surface. This paper identifies a system-level vulnerability resulting from the combination of the emerging image stabiliser hardware susceptible to acoustic manipulation and the object detection algorithms subject to adversarial examples."

Continue reading