Iowa has already won the worst IT rollout award of 2020: Rap for crap caucus app chaps in vote zap flap

Untested tech, no training, last-minute rollout, buggy code – sound familiar?

It’s all so painfully familiar: with a crunch date of February 3, the Democratic Party in Iowa decided to charge ahead with an IT rollout that comprised an entirely new software system spread out across thousands of sites to record the result of the Democratic caucus for its presidential nominee.

It was, inevitably, a complete failure. The results from the Iowa caucus were supposed to come in nearly 24 hours ago. Instead, it has become a rolling news cycle of tech catastrophe.

We’re not even going to bother to dig into lessons learned because they are the same ones that every sysadmin since the dawn of time has dealt with – and spends their entire career warning the suits about, to greater and lesser degrees of success.

Let’s start with the app. It was produced by a bunch of IT hotshots who have advised previous Democratic campaigns, including those used by Obama and Hillary Clinton, and set up a for-profit company called, for some reason, “Shadow.” The biz won the contract to create an app that would do a simple task: allow people on the ground to type in the result of headcounts in town halls and gymnasiums across Iowa and send them to a central processing point.

All people had to do was download the app, punch in a security code – only one per precinct, of which there are 1,765 – tap in the results and hit send. What could go wrong?

Lots, as it turns out. For one, the app was only launched several weeks ago so nobody that had to use it was familiar with it. That was fine for the occasional 20-something that acted as a caucus chair and who is perfectly comfortable with downloading and trying out new smartphone apps; navigating a new user-interface is just part of life.

But the majority of caucus managers are retired, some in their 70s and 80s, many of whom have done the same job every four years. They needed a smartphone app like they needed a hole in the head.

Sticking PINs in your eyes

But it was fine: Democratic headquarters sent instructions via email about how to use the new software.

So all they had to do was find a smartphone, find the download instructions, download it, figure out how to log in, navigate a new interface, and figure out how to input the right numbers in the right boxes. And read the email instructions. Which include the fact that the test PIN they had been sent to try it out wouldn’t work on the day of voting. For security reasons, they’d get a new one. No one was given any training. And no one was on hand to set it up or answer questions.

One chair from Poweshiek County, John Grennan, told the New York Times that seven of the 10 people running precincts in his area never downloaded the app. But those that did had trouble with the changing security PIN and couldn’t get into it. “A lot of people were like, ‘I’m getting a virus on my phone’ and just quit it,” he told the paper.

Amazingly this last-minute rollout may have been intentional rather than the result of poor project management. Democratic Party heads appeared to believe that providing no information about the app and providing it a week or so before the actual vote was a good security measure. That sound you hear? That’s every IT manager across the world banging their heads on their desk.

The app had only been tested in controlled environments; there was no state-wide testing. No one visited the more rural parts of Iowa where data speeds made the entire download, authentication, data input and sending process an exercise in frustration. There was, seemingly, no independent security review either.

It was also claimed the app's development team was relatively new and inexperienced.

Entirely inevitably, a lot of local managers decided to reject the whole thing. One precinct chair, Deb Copeland, told the Des Moines Register that she “didn’t even bother” to download it. They had phoned in their results for decades and would do the same again in 2020. Others tried. And immediately hit problems: they “couldn’t get the app to download” and “couldn’t get it to work.”

FAQ-you

The brilliant minds at Shadow – who were paid $60,000 to produce the app and have picked up another $60,000 for a different version for Nevada – probably figured people would access its FAQ and Help pages. Had they actually existed, which they didn’t.

But they wouldn’t have anyway. As Deb Copeland noted: “We had an informal Facebook group and people were saying they couldn’t get it to do anything and had so many questions.” So no training, inadequate support, a rushed rollout and a complete inability to think through the obvious resistance to new technology from those required to use the app. Anything else? Oh yes, plenty.

The Iowan Democratic Party – which contracted with Shadow and, somewhat amazingly, defended the app rollout two weeks ago to skeptical journalists – was keen to point out that the app had not been hacked or compromised. Great! Except reports coming in suggest that it simply didn’t work properly.

The data input appears to have worked correctly (although some managers complained that they couldn’t get it to do the maths) but it appears as though it was only partially outputting the results, causing massive tabulation errors. Oh, and the app crashed for lots of people, requiring them to shut it down and log back in again.

It actually gets worse than that: for the first time, the caucuses were reporting three sets of numbers rather than one. That’s thanks to the extremely tight results in 2016 when Hillary Clinton very narrowly beat out Bernie Sanders, Sanders’ team successfully argued that there needed to be results recorded throughout the somewhat convoluted multi-stage voting process that Iowa uses, rather than just the end result.

So all the volunteers were not only required to do their job differently to before – providing three sets of figures instead of one – but they were also expected to do so through an entirely new system that hadn’t been properly bug-tested.

Very hot hotline

And so with the whole system falling over, everyone responsible for reporting their results did the obvious thing and called the election hotline. And guess what? It was overwhelmed by everyone having problems, leaving them on hold for hours.

Another caucus manager said it took her four hours just to get someone on the phone to take the results after she abandoned the app thanks to repeat crashes. Another manager said he managed to successfully send the results but didn’t get a confirmation and so was unsure whether it had really been sent; a fear that only grew as more and more people reported problems.

We could write pages and pages of reports about how differently people experienced this almighty IT cock-up but what’s the point? If you’re reading The Reg you already know what the problem is and the details quickly become irrelevant.

Here’s what’s happened: the suits hired a company because they were swayed by their CVs and sales talk and didn’t run it past anyone that knew what they were doing. Then the suits didn’t listen to all the people telling them it was a bad idea and they should delay rollout. And they didn’t allow sufficient time for testing and training.

In light of the debacle Shadow has apologized for the code cock-up, albeit rather grudgingly. It maintains that the app was "sound and accurate," but says problems occurred in data transmission.

Meanwhile, Nevada has cancelled its contract with Shadow and is presumably asking for its money back. ®