Salesforce fell over so hard today, it took out its own server status page

It’s not DNS. There is no way it’s DNS. It was DNS


Updated Salesforce is digging itself out of a multi-hour outage right now that it has blamed on a DNS issue.

At one point today, the IT breakdown was so severe that its status page was pretty much inaccessible for netizens, and staff resorted to posting updates on their help and training sub-site.

"Salesforce is experiencing a major disruption due to what we believe is a DNS issue causing our service to be inaccessible," CTO Parker Harris said in a statement. "We recognize the significant impact on our customers and are actively working on resolution.

"We believe we have isolated the issue and are in the process of bringing affected services back online. Customers may continue to experience issues as we work through remediation. This remains a top priority for Salesforce."

The downtime started shortly before 2200 UTC (1500 PT), knocked pretty much the entire software-as-a-service giant offline, and right now, everything is not yet back to normal. Though products have been restored, we're told, customers can't login if they are using multi-factor authentication.

"At 2146 UTC on May 11, 2021, the Salesforce technology team became aware of an issue impacting multiple Salesforce services," the CRM goliath noted on its status page.

"Customers will experience issues while navigating the Core application, Marketing Cloud, Commerce Cloud, and Experience Cloud (formerly known as Communities).

"The issue is also impacting the Salesforce Trust site as the status.salesforce.com page is only intermittently accessible."

About half an hour ago, the biz added:

The technology team continues to manually restore service. We currently don’t have an estimated time on when full recovery is expected. The team is confident that this incident was the result of an internal DNS change.

The Government Cloud environment is now out of impact. Marketing Cloud, Commerce Cloud, and Heroku services have been restored, however, customers using multifactor authentication will still be unable to log in.

Just as we were going to press, Parker tweeted: "Services are significantly restored though we are still not out of impact." ®

Updated to add on May 12

The outage is now over, and Salesforce has given some details on the cause of the IT breakdown:

The technology team has resolved the issue and called an all-clear as of 02:20 UTC on May 12, 2021.

The team determined the root cause was related to the implementation of an emergency fix that triggered a software issue and caused a DNS network incident. To resolve the issue, the team manually restored DNS service on a data center by data center basis until normal service levels were restored.

Similar topics


Other stories you might like

  • Tesla driver charged with vehicular manslaughter after deadly Autopilot crash

    Prosecution seems to be first of its kind in America

    A Tesla driver has seemingly become the first person in the US to be charged with vehicular manslaughter for a deadly crash in which the vehicle's Autopilot mode was engaged.

    According to the cops, the driver exited a highway in his Tesla Model S, ran a red light, and smashed into a Honda Civic at an intersection in Gardena, Los Angeles County, in late 2019. A man and woman in the second car were killed. The Tesla driver and a passenger survived and were taken to hospital.

    Prosecutors in California charged Kevin George Aziz Riad, 27, in October last year though details of the case are only just emerging, according to AP on Tuesday. Riad, a limousine service driver, is facing two counts of vehicular manslaughter, and is free on bail after pleading not guilty.

    Continue reading
  • AMD returns to smartphone graphics with new Samsung chip for your pocket computer

    We're back in black

    AMD's GPU technology is returning to mobile handsets with Samsung's Exynos 2200 system-on-chip, which was announced on Tuesday.

    The Exynos 2200 processor, fabricated using a 4nm process, has Armv9 CPU cores and the oddly named Xclipse GPU, which is an adaptation of AMD's RDNA 2 mainstream GPU architecture.

    AMD was in the handheld GPU market until 2009, when it sold the Imageon GPU and handheld business for $65m to Qualcomm, which turned the tech into the Adreno GPU for its Snapdragon family. AMD's Imageon processors were used in devices from Motorola, Panasonic, Palm and others making Windows Mobile handsets.

    Continue reading
  • Big shock: Guy who fled political violence and became rich in tech now struggles to care about political violence

    'I recognize that I come across as lacking empathy,' billionaire VC admits

    Billionaire tech investor and ex-Facebook senior executive Chamath Palihapitiya was publicly blasted after he said nobody really cares about the reported human rights abuse of Uyghur Muslims in China.

    The blunt comments were made during the latest episode of All-In, a podcast in which Palihapitiya chats to investors and entrepreneurs Jason Calacanis, David Sacks, and David Friedberg about technology.

    The group were debating the Biden administration’s response to what's said to be China's crackdown of Uyghur Muslims when Palihapitiya interrupted and said: “Nobody cares about what’s happening to the Uyghurs, okay? ... I’m telling you a very hard ugly truth, okay? Of all the things that I care about … yes, it is below my line.”

    Continue reading

Biting the hand that feeds IT © 1998–2022