IBM insiders say CEO Arvind Krishna downplayed impact of email troubles, asked for a week to sort things out

Behind the curtain: Server crashes, thumbs down on Slack, staff compared to shoemaker's kids, and more

IBM CEO Arvind Krishna on Wednesday addressed the company's ongoing email woes in his monthly video message to employees.

Krishna, we're told, said the email disruption only lost the company one deal worth about $10,000 and he said the situation would be fully fixed in a week.

The chief exec's comments appear to address The Register's report last week that IBM's partial email outage might have an impact on company revenue. If the figure Krishna cited is correct, the email disruption's sales impact is immaterial in terms of the Big Blue's overall finances. The possibility of brand damage, however, remains.

Our first source within IBM described how Krishna likened IBM employees to "shoemaker's kids," a reference to the proverb that shoemaker's children go barefoot and a suggestion that IBM is too busy tending to its customers to provide reliable email to its employees.

"I took it as 'we are too busy doing other things,'" said the individual who told The Register about the video.

Krishna, we're told, spent several minutes on the subject, claiming that IBM sent 4.2 billion emails a week and that IBM employees have been experiencing only about 30 minutes per day of downtime. Our source told us that's "just not true."

On Tuesday, this individual said thousands of people have been affected and described 27,000 people in the company's Slack help channel posting requests for assistance.

"[Krishna] went down a list of things that should have been done better," a second IBM insider told The Register. "So I expect some people will get canned."

A third IBM insider suggested it was more like 100,000 people having trouble accessing their email as of Tuesday and likened the situation to an outage affecting an entire country. This individual characterized the situation as a major embarrassment for a company that handles IT systems for a living.

But nailing down a specific number of people affected is difficult because, as we understand, email will be fine for some employees on a given day and then die the next day, so not everyone is affected at the same time or the same way.

While a previous source described an incident-free migration, we also heard an account of meetings set up yesterday afternoon that subsequently disappeared and emails with invitations that are visible but won't load.

"My help desk person doesn't even have email," our first source said. "But with Slack we make do."

This individual said Krishna in his video blamed the outage on a lack of resources, specifically people and money allocated to the email migration from HCL to IBM servers. The message was that more capacity, testing, and planning were needed.

This critical situation, he said, didn't come from a single mistake – which would have been fixable right away – but from multiple cumulative decisions made over time due to a lack of resources.

When in doubt, spend

The Register was told that IBM ordered 15 new server clusters on Tuesday to support its email system. It's been suggested to us that this represents an attempt to spread out the server load.

"They keep saying things like they moved 1,400 users from one cluster to another but then people on a 'non-impacted' cluster are still complaining," our first source said.

Our third source described this as diluting the problem without fixing the broken system architecture.

This individual said the situation could be attributed to one or two IBM managers making bad decisions: taking a solid, stable environment and plunging it into chaos requires a manager making bad calls.

The recent email issues have had no material impact on our quarter

The core issue, this person said, is a terribly designed solution. When servers crash, a support request is opened with HCL, they send a patch, it gets applied, and the server crashes again.

Initially, this person said, the CIO had been posting regular updates on Slack. At first, employees appreciated the effort and responded with thumbs-up emojis. Lately, it's been thumbs-down and angry face emojis.

IBM, we're told, moved its email system in-house because HCL, which bought Notes in 2018, decided to shutter its email hosting service, SmartCloud Notes and IBM had no other option but to spin up its own servers. There were various reasons for closing SmartCloud Notes, our third source said, but one had to do with an IBM VP who made it extremely expensive, maintenance intensive, and free of customers.

The failure of the email migration, this person said, is the result of incompetent people.

"The recent email issues have had no material impact on our quarter," an IBM spokesperson told The Register. "Almost all of our mail systems are up and running and we expect any remaining issues to be resolved in the near future." ®