Iowa has already won the worst IT rollout award of 2020: Rap for crap caucus app chaps in vote zap flap

Untested tech, no training, last-minute rollout, buggy code – sound familiar?


It’s all so painfully familiar: with a crunch date of February 3, the Democratic Party in Iowa decided to charge ahead with an IT rollout that comprised an entirely new software system spread out across thousands of sites to record the result of the Democratic caucus for its presidential nominee.

It was, inevitably, a complete failure. The results from the Iowa caucus were supposed to come in nearly 24 hours ago. Instead, it has become a rolling news cycle of tech catastrophe.

We’re not even going to bother to dig into lessons learned because they are the same ones that every sysadmin since the dawn of time has dealt with – and spends their entire career warning the suits about, to greater and lesser degrees of success.

Let’s start with the app. It was produced by a bunch of IT hotshots who have advised previous Democratic campaigns, including those used by Obama and Hillary Clinton, and set up a for-profit company called, for some reason, “Shadow.” The biz won the contract to create an app that would do a simple task: allow people on the ground to type in the result of headcounts in town halls and gymnasiums across Iowa and send them to a central processing point.

All people had to do was download the app, punch in a security code – only one per precinct, of which there are 1,765 – tap in the results and hit send. What could go wrong?

Lots, as it turns out. For one, the app was only launched several weeks ago so nobody that had to use it was familiar with it. That was fine for the occasional 20-something that acted as a caucus chair and who is perfectly comfortable with downloading and trying out new smartphone apps; navigating a new user-interface is just part of life.

But the majority of caucus managers are retired, some in their 70s and 80s, many of whom have done the same job every four years. They needed a smartphone app like they needed a hole in the head.

Sticking PINs in your eyes

But it was fine: Democratic headquarters sent instructions via email about how to use the new software.

So all they had to do was find a smartphone, find the download instructions, download it, figure out how to log in, navigate a new interface, and figure out how to input the right numbers in the right boxes. And read the email instructions. Which include the fact that the test PIN they had been sent to try it out wouldn’t work on the day of voting. For security reasons, they’d get a new one. No one was given any training. And no one was on hand to set it up or answer questions.

One chair from Poweshiek County, John Grennan, told the New York Times that seven of the 10 people running precincts in his area never downloaded the app. But those that did had trouble with the changing security PIN and couldn’t get into it. “A lot of people were like, ‘I’m getting a virus on my phone’ and just quit it,” he told the paper.

Amazingly this last-minute rollout may have been intentional rather than the result of poor project management. Democratic Party heads appeared to believe that providing no information about the app and providing it a week or so before the actual vote was a good security measure. That sound you hear? That’s every IT manager across the world banging their heads on their desk.

The app had only been tested in controlled environments; there was no state-wide testing. No one visited the more rural parts of Iowa where data speeds made the entire download, authentication, data input and sending process an exercise in frustration. There was, seemingly, no independent security review either.

It was also claimed the app's development team was relatively new and inexperienced.

Entirely inevitably, a lot of local managers decided to reject the whole thing. One precinct chair, Deb Copeland, told the Des Moines Register that she “didn’t even bother” to download it. They had phoned in their results for decades and would do the same again in 2020. Others tried. And immediately hit problems: they “couldn’t get the app to download” and “couldn’t get it to work.”

FAQ-you

The brilliant minds at Shadow – who were paid $60,000 to produce the app and have picked up another $60,000 for a different version for Nevada – probably figured people would access its FAQ and Help pages. Had they actually existed, which they didn’t.

But they wouldn’t have anyway. As Deb Copeland noted: “We had an informal Facebook group and people were saying they couldn’t get it to do anything and had so many questions.” So no training, inadequate support, a rushed rollout and a complete inability to think through the obvious resistance to new technology from those required to use the app. Anything else? Oh yes, plenty.

The Iowan Democratic Party – which contracted with Shadow and, somewhat amazingly, defended the app rollout two weeks ago to skeptical journalists – was keen to point out that the app had not been hacked or compromised. Great! Except reports coming in suggest that it simply didn’t work properly.

The data input appears to have worked correctly (although some managers complained that they couldn’t get it to do the maths) but it appears as though it was only partially outputting the results, causing massive tabulation errors. Oh, and the app crashed for lots of people, requiring them to shut it down and log back in again.

It actually gets worse than that: for the first time, the caucuses were reporting three sets of numbers rather than one. That’s thanks to the extremely tight results in 2016 when Hillary Clinton very narrowly beat out Bernie Sanders, Sanders’ team successfully argued that there needed to be results recorded throughout the somewhat convoluted multi-stage voting process that Iowa uses, rather than just the end result.

So all the volunteers were not only required to do their job differently to before – providing three sets of figures instead of one – but they were also expected to do so through an entirely new system that hadn’t been properly bug-tested.

Very hot hotline

And so with the whole system falling over, everyone responsible for reporting their results did the obvious thing and called the election hotline. And guess what? It was overwhelmed by everyone having problems, leaving them on hold for hours.

bruce

You can easily secure America's e-voting systems tomorrow. Use paper – Bruce Schneier

READ MORE

Another caucus manager said it took her four hours just to get someone on the phone to take the results after she abandoned the app thanks to repeat crashes. Another manager said he managed to successfully send the results but didn’t get a confirmation and so was unsure whether it had really been sent; a fear that only grew as more and more people reported problems.

We could write pages and pages of reports about how differently people experienced this almighty IT cock-up but what’s the point? If you’re reading The Reg you already know what the problem is and the details quickly become irrelevant.

Here’s what’s happened: the suits hired a company because they were swayed by their CVs and sales talk and didn’t run it past anyone that knew what they were doing. Then the suits didn’t listen to all the people telling them it was a bad idea and they should delay rollout. And they didn’t allow sufficient time for testing and training.

In light of the debacle Shadow has apologized for the code cock-up, albeit rather grudgingly. It maintains that the app was "sound and accurate," but says problems occurred in data transmission.

Meanwhile, Nevada has cancelled its contract with Shadow and is presumably asking for its money back. ®

Similar topics

Broader topics


Other stories you might like

  • Assange can go to UK Supreme Court (again) to fend off US extradition bid

    Top Brit judges may consider whether an American prison is just too much

    Julian Assange has won a technical victory in his ongoing battle against extradition from the UK to the United States, buying him a few more months in the relative safety of Her Majesty's Prison Belmarsh.

    Today at London's High Court, the Lord Chief Justice of England and Wales Lord Burnett approved a question on a technical point of law, having refused Assange immediate permission to appeal to the UK Supreme Court. The WikiLeaker's lawyers had asked for formal permission to pose this legal conundrum about Assange's likely treatment in US prisons to the Supreme Court:

    Continue reading
  • They see us Cinnamon Rolling, they're rating: GeckoLinux incorporates kernel 5.16 with familiar installation experience

    A nice, clean community distro that works well

    Most distros haven't got to 5.15 yet, but openSUSE's downstream project GeckoLinux boasts 5.16 of the Linux kernel and the latest Cinnamon desktop environment.

    Some of the big-name distros have lots of downstream projects. Debian has been around for decades so has umpteen, including Ubuntu, which has dozens of its own, including Linux Mint, which is arguably more popular a desktop than its parent. Some have only a few, such as Fedora. As far as we know, openSUSE has just the one – GeckoLinux.

    The SUSE-sponsored community distro has two main editions, the stable Leap, which has a slow-moving release cycle synched with the commercial SUSE Linux Enterprise; and Tumbleweed, its rolling-release distro, which gets substantial updates pretty much every day. GeckoLinux does its own editions of both: its remix of Leap is called "GeckoLinux Static", and its remix of Tumbleweed is called "GeckoLinux Rolling".

    Continue reading
  • Running Windows 10? Microsoft is preparing to fire up the update engines

    Winter Windows Is Coming

    It's coming. Microsoft is preparing to start shoveling the latest version of Windows 10 down the throats of refuseniks still clinging to older incarnations.

    The Windows Update team gave the heads-up through its Twitter orifice last week. Windows 10 2004 was already on its last gasp, have had support terminated in December. 20H2, on the other hand, should be good to go until May this year.

    Continue reading
  • Throw away your Ethernet cables* because MediaTek says Wi-Fi 7 will replace them

    *Don't do this

    MediaTek claims to have given the world's first live demo of Wi-Fi 7, and said that the upcoming wireless technology will be able to challenge wired Ethernet for high-bandwidth applications, once available.

    The fabless Taiwanese chip firm said it is currently showcasing two Wi-Fi 7 demos to key customers and industry collaborators, in order to demonstrate the technology's super-fast speeds and low latency transmission.

    Based on the IEEE 802.11be standard, the draft version of which was published last year, Wi-Fi 7 is expected to provide speeds several times faster than Wi-Fi 6 kit, offering connections of at least 30Gbps and possibly up to 40Gbps.

    Continue reading
  • Windows box won't boot? SystemRescue 9 may help

    An ISO image you can burn or drop onto a USB key

    The latest version of an old friend of the jobbing support bod has delivered a new kernel to help with fixing Microsoft's finest.

    It used to be called the System Rescue CD, but who uses CDs any more? Enter SystemRescue, an ISO image that you can burn, or just drop onto your Ventoy USB key, and which may help you to fix a borked Windows box. Or a borked Linux box, come to that.

    SystemRescue 9 includes Linux kernel 5.15 and a minimal Xfce 4.16 desktop (which isn't loaded by default). There is a modest selection of GUI tools: Firefox, VNC and RDP clients and servers, and various connectivity tools – SSH, FTP, IRC. There's also some security-related stuff such as Yubikey setup, KeePass, token management, and so on. The main course is a bunch of the usual Linux tools for partitioning, formatting, copying, and imaging disks. You can check SMART status, mount LVM volumes, rsync files, and other handy stuff.

    Continue reading

Biting the hand that feeds IT © 1998–2022