How do you solve the problem that is Twitter?
Technically and leadership-wise what site needs is stability
Opinion It's a toss-up between Elon Musk's management misadventures and Twitter's technical troubles as to which will cause the most damage.
Twitter is in trouble. I mean, who blunders his way into a fight with Apple only to later claim it was all a misunderstanding? But, as idiotic as that is combined with alienating advertisers, Elon Musk's wreaking havoc with Twitter's technical staff may end up causing more damage in the long run.
True, Twitter is still up and running, but some people have already seen smaller failures. For example, Twitter's SMS-based Two-Factor Authentication (2FA) has already failed for some users.
As a reminder, the Titanic didn't sink immediately after hitting the iceberg. It took its own sweet time. And, many passengers still thought all would be well until their feet were awash in the frigid North Atlantic waters.
Don't believe me? Perhaps you'll believe Twitter site reliability engineer (SRE) Ben Krueger, who told MIT Technology Review, "The larger catastrophic failures are a little more titillating, but the biggest risk is the smaller things starting to degrade." Exactly.
Another SRE, MosquitoCapital, says that something as mundane as a random hard drive filling up can "cause cascading failures across systems, even well-engineered fault-tolerant ones with active maintenance. Where's the box? What's filling it up? Who will figure that out?"
Good questions, and with Musk firing half the staff and another thousand plus Twitter employees quitting, it's possible not many at the Big Tweet knows the answer.
And, although you may not know it, that example's more apt than you might think. It's estimated that half of Twitter's infrastructure is devoted to data storage and management. If a leak pops up in Twitter's data lake, there could be a flood of bad things downstream for users quickly.
Then, there's networking. For now, Twitter still chiefly relies on its own datacenters rather than a hyper-cloud. So what happens if ye olde backhoe rips out a key length of fiber? I don't know, and we can only hope someone left at Twitter does.
Or, say, a Distributed Denial of Service (DDoS) attack comes along and kicks Twitter off the net? After all, it has happened to Twitter before. It can happen again. Where are the net admins? Many of them have new, better jobs without a boss who requires them to be hardcore.
Let's look at another example from MosquitoCapital. What if there's a really, really bad code push, instead of the lame failure of paid Blue verification marks? This time around, instead of just a flood of fake users, the new code locks up Twitter's Identity and Access Management (IAM) service for some users. Suddenly, some people, say old blue verified users, can't log in to Twitter. Now what?
You roll back the change, of course. That even a short-handed tech staff can probably pull off. But, what, oh my friends, what happens if they can't pull the change out of the code without freezing IAM up? Or, worse still, the site just goes down. Period. End of statement.
It would require staffers with deep system knowledge, and we'd better hope at least one or two of those are still around.
As Peter Clowes, a former Twitter senior project manager, explained, he quit because he saw no upside to being "the last one standing in multiple large slack rooms and JIRA boards." Clowes continued, "If I stayed, I would have been on-call constantly with little support for an indeterminate amount of time on several additional complex systems I had no experience in."
Musk still has the programmers who stayed until 1:30 in the morning but many of Twitter's top developers have left.
- Twitter set for more layoffs as Musk mulls next move
- Elon Musk issues ultimatum to Twitter staff: Go hardcore or go home
- Investor tells Google: Cut costs now and stop paying staff so much
- Email domain for NPM lib with 6m downloads a week grabbed by expert to make a point
As Will Norris, former Twitter open-source lead, told The Register, "Pretty much all of the key people that were working on open source at Twitter have left. All of the engineers that I worked with on open source are gone." Oh, and in case, you didn't know, Twitter relies on open-source software for pretty much everything.
Many people still believe Twitter will be fine. I can't see it. Between Musk's management missteps, Twitter's ad cash flow receding quickly, and simply not having enough in-house technical expertise to keep things running smoothly, sooner or later, Twitter will stumble badly. My only question is, "How bad will its fall be?" Stick around. We'll soon see. ®