Oracle NetSuite datacenter plunges offline for a day, customers warned of data loss
Anything could happen in a half hour
Updated Despite what Oracle supremo Larry Ellison might have you believe, Big Red's clouds do in fact go down.
Yesterday some Oracle engineers' Valentine's Day celebrations were perhaps put on hold after the NetSuite enterprise resource planning (ERP) suite experienced an outage at its datacenter in Boston, USA. The downtime, which began around 1215 ET Tuesday, stretched on before services were restored around 1146 today.
The Register spoke with one New York-based military equipment manufacturer via email, who told us the outage took out everything but the phones, with customer service, finance, shipping/receiving, and shop floor all grinding to a halt.
According to complaints shared via Reddit, the outage effectively paralyzed businesses relying on the platform's Boston-based servers. We've seen downtime reports from organizations across America: Tennessee, Wisconsin, Chicago, Texas, North Carolina, Minnesota, Vermont, Nevada, Washington, Louisiana, Delaware, Georgia, and Arizona, at least.
We do note that NetSuite has multiple regions in the United States and some of the world, which weren't directly affected by this particular outage. On February 13, US Ashburn 2 went down for about an hour. Oracle right now claims Netsuite had a 99.97 percent availability over the past 12 months.
While NetSuite, which Oracle acquired in 2016 for $9.3 billion, is now back up and running in that Boston datacenter, the tech goliath notes customers may experience errors on their first attempt logging back in. Notably missing from the status report were any warnings about lost data.
However, Oracle reportedly told one Redditor that approximately 30 minutes of data was lost during the recovery process, as the "restoration point was about 30 minutes prior to the outage."
- IBM cheapens Oracle by delivering promised power-up for some POWER servers
- Refrigeration failure at Hong Kong datacenter takes out Alibaba Cloud and others
- Lessons to be learned from Google and Oracle's datacenter heatstroke
- Oracle clouds never go down, says Oracle's Larry Ellison
The IT giant is said to be working with customers to identify transactions that were created in the 30-minute window.
According to Oracle's NetSuite status page, the cause of the outage was the result of an unspecified event in the Boston area that "forced the local team to disconnect power until the issue is resolved."
It's not clear exactly what the event that necessitated cutting power to the servers was. We've reached out to Oracle for comment and it declined to go on the record.
What we do know is the problem at the Boston datacenter was bad enough that the database supplier began restoring accounts to a disaster recovery site around 2226 ET Tuesday.
The outage flies in the face of Ellison's rather hyperbolic claims made during the company's Q2 earnings call in December. "Our cloud is very secure and extremely reliable," he told financial analysts. "It doesn't go down."
But even at the time, the billionaire's comments overlooked an outage just months earlier. Amid Britain's heatwave this summer, both Google and Oracle cloud datacenters were taken offline by cooling failures after temperatures creeped about 40.3˚C (104.5˚F). ®
Updated to add
Cyxtera, the colocation provider housing Oracle's NetSuite Boston servers, has handed us the following statement, blaming what sounds like a fire in electrical equipment at that datacenter site:
At approximately noon ET yesterday, smoke was reported inside one of our two data centers in Waltham, Massachusetts coming from electrical equipment in the power room. Responding fire department officials enforced a shutdown of power at the site and evacuated the building to ensure the safety of all onsite. (Our second Waltham site was unaffected.)
Cyxtera's engineering and technical personnel have implemented a solution to restore power services for customer environments. The data center is stable, physical access to the site has been restored, and customers have re-entered the facility per normal access procedures. We continue to investigate the cause of the incident and will provide ongoing updates to our customers. We regret the interruption of service and thank our customers for their continued patience.
Thus, it now appears NetSuite's Boston outage was caused by the power being cut to servers after firefighters showed up to the smoking datacenter facility.