Twitter datacenter melted down in Labor Day heat

Bitbarn suffered 'total shutdown' after 113F heatwave

Earlier this month extreme heat downed a Twitter datacenter in California over the Labor Day weekend, leaving the website and app working on bare-bones infrastructure.

According to a memo obtained by CNN, Twitter lost access to its Sacramento (SMF) datacenter "due to extreme weather" on Monday, September 5, 2022. The memo, attributed to VP of engineering Carrie Fernandez, says, "The unprecedented event resulted in the total shutdown of physical equipment in SMF."

The temperature in downtown Sacramento, California, that day – Labor Day in the US – reached a record high of 113°F or 45°C, and local temperature records were broken again within days.

While Twitter has backup datacenters in Atlanta and Portland that can handle the load of traffic that the social network handles each second, the loss of the Sacramento hub put it in a potentially perilous position.

"If we lose one of those remaining datacenters, we may not be able to serve traffic to all Twitter's users," Fernandez warned.

A 2020 study from The Uptime Institute, "The gathering storm: Climate change and datacenter resiliency," warns that extreme temperatures represent a growing threat to datacenters.

"Increasing summer temperatures have put a strain on the cooling of many datacenters, and this is likely to worsen in the years ahead," the report says.

Beyond hindering the ability to operate, high temperatures can make datacenters cost more than anticipated.

"An outdoor air temperature increase of as little as four degrees Fahrenheit (two degrees Celsius) could make free air and evaporative systems ineffective on their own, and uneconomical in some situations," the report says.

About 40 percent of the energy consumed by datacenters goes toward cooling IT equipment, according to a 2015 study, "Data Center Energy and Cost Saving Evaluation."

The memo was sent on Friday, September 9th, and it disallows all production and app changes while service is restored except for what's required "to address service continuity or other urgent operational needs."

Not the worst of Twitter's problems

A whistleblower complaint filed against Twitter last month by its former head of security Peiter "Mudge" Zatko casts doubt on Twitter's data center resilience, citing the company's "Cascading datacenter problems."

"In or around the spring of 2021, Twitter's primary datacenter began to experience problems from a runaway engineering process, requiring the company to move operations to other systems outside of this datacenter," Zatko's complaint says.

"But, the other systems could not handle these rapid changes and also began experiencing problems. Engineers flagged the catastrophic possibility that all the datacenters might go offline simultaneously."

Twitter, currently in the midst of a legal battle to force Elon Musk to honor his commitment to buy the company, brushed off its data center problems.

"There have been no disruptions impacting the ability for people to access and use Twitter at this time," a Twitter spokesperson told The Register in an email. "Our teams remain equipped with the tools and resources they need to ship updates and will continue working to provide a seamless Twitter experience."

We asked Twitter whether having the tools to ship updates means its engineers are actually doing so, given that Fernandez's memo suspended non-essential product updates until operations have been normalized. We'll update this story if we hear back. ®

Similar topics


Other stories you might like

Biting the hand that feeds IT © 1998–2022