Cisco Webex meltdown caused by script that nuked its host VMs

Comms software flinger confesses to ultimate snafu, trigger still under investigation

Cisco has confessed that the cause of the mega Webex outage last week – which it is still trying to clean up – was an automated script "which deleted the virtual machines hosting the service".

A chunk of Webex was KO'd last week, including Teams, Calling, Meetings, Control Hub, Hybrid Services and more, as we reported at the time.

While Webex itself was cured relatively quickly, Webex Teams, the biz chat platform formerly known as Cisco Spark, took far longer than anyone had expected, as the plethora of updates to Cisco's Webex Teams status page show.

The most recent update at the time of writing mentioned that "ongoing data restoration tasks did complete on time, and all data restoration activities have completed" but "some users continue to experience issues accessing and sending messages to specific spaces" with fixes still being cooked up and deployed.

Following some nudging by The Register, Cisco eventually admitted the preliminary cause of the incident:

The service interruption was the result of an automated script running on our Webex Teams platform which deleted the virtual machines hosting the service. This was a process issue, not a technical issue. We continue to investigate the causes of the script being run, however we are confident that this is an isolated incident and processes are in place to prevent any recurrence.

We understand that a full investigation report will be made available once Cisco has figured out how the offending script was triggered.

Were you affected by the outage? Did your productivity crash (or indeed soar) in the absence of Webex or Webex Teams? Leave us a comment and let us know. ®

Biting the hand that feeds IT © 1998–2021