Remember the Uber self-driving car that killed a woman crossing the street? The AI had no clue about jaywalkers

The last thing we need right now is a rise in ignorant cyber-Judge Dredds

The self-driving Uber car that hit and killed a woman walking her bike across a street wasn’t designed to detect “jaywalking pedestrians.”

That's according to an official dossier published by the US National Safety Transportation Board (NTSB) on Tuesday.

The March 2018 accident was the first recorded death by a fully autonomous vehicle. On-board video footage showed the victim, 49-year-old Elaine Herzberg, pushing her bike at night across a road in Tempe, Arizona, moments before she was struck by the AI-powered SUV at 39 MPH.

Now, an investigation by the NTSB into the crash has pinpointed a likely major contributing factor: the code couldn't recognize her as a pedestrian, because she was not at an obvious designated crossing. Rather than correctly anticipating her movements as a person moving across the road, it ended up running right into her.

“The system design did not include a consideration for jaywalking pedestrians,” the watchdog stated [PDF] in its write-up. “Instead, the system had initially classified her as an 'other' object which are not assigned goals.”

The computer-vision systems in self-driving cars are trained to identify things, such as other vehicles, trees, sign posts, bicycles, and so on, and make decisions on what to do next using that information. It appears Uber’s software wasn’t able to identify Herzberg since there was no classification label for a person not using a proper crossing point, and it wasn't able to make the right decisions.

Countdown to impact

Some 5.6 seconds before hitting her, the car's radar detected Herzberg, and at 5.2 seconds, she was picked out by the Lidar. However, the machine-learning system more or less ignored her, figuring her to be a non-moving object not in the vehicle's way.

As the robo-vehicle drew nearer, it categorized her variously as a vehicle, a bike, or some other thing that was not, or was only partially, in its way.

Just 1.2 seconds before hitting her, it identified her not only as a bicycle but also clearly in the path of its travel, by which point it was far too late to change course.

Crucially, the software was "unable to correctly predict the path" of Herzberg, the report noted. Perhaps if it had correctly identified her early on as a person crossing the street, it could have dramatically slowed down or otherwise tried to avoid her. But it did not anticipate the collision because, for the most part, it considered her to be some unknown object not in the way.

Also, don't forget: the SUV's emergency braking system was deliberately disabled because when it was switched on, the vehicle would act erratically, according to Uber. The software biz previously said “the vehicle operator is relied on to intervene and take action," in an emergency.

The self-driving car was fully autonomous at the time at the accident, though it had a human driver at the wheel. An internal camera caught the Uber worker looking down and away from the road moments before the accident, unaware of Herzberg’s presence before it was too late.

Below is a timetable, produced by the NTSB, detailing the car's decision-making and speed in the seconds before the accident:

The report comes just two weeks before the board is due to hold a public meeting to “determine the probable cause of a crash involving a pedestrian and an Uber test vehicle,” on 19 November.

The safety board also released more than 40 documents totaling at least 430 pages with various bits of supporting evidence and comprehensive accounts into things like the vehicle’s properties and the internal safety culture at Uber.

The massive data dump also revealed that the ride-hailing biz's self-driving cars were involved in 37 smashes between September 2016 and March 2018, prior to the deadly accident. In these 37 incidents, all of the robo-vehicles were driving in autonomous mode, and in 33 of these cases, other vehicles crashed into the self-driving cars.


On Tuesday evening, a spokesperson for Uber told us: “We regret the March 2018 crash involving one of our self-driving vehicles that took Elaine Herzberg’s life. In the wake of this tragedy, the team at Uber Advanced Technologies Group has adopted critical program improvements to further prioritize safety.

"We deeply value the thoroughness of the NTSB’s investigation into the crash and look forward to reviewing their recommendations once issued after the NTSB’s board meeting later this month.”

El Reg asked Uber how it planned to train its autonomous vehicles to detect jaywalkers in future, and the biz told us: "We conduct hundreds of tests in simulation and on our test track to help us improve the overall safety of our system. These tests include taking our software and having it go through scenarios that involve complex situations like jaywalking, hard braking, and unprotected turns." ®

Broader topics

Other stories you might like

  • SEC probes Musk for not properly disclosing Twitter stake
    Meanwhile, social network's board rejects resignation of one its directors

    America's financial watchdog is investigating whether Elon Musk adequately disclosed his purchase of Twitter shares last month, just as his bid to take over the social media company hangs in the balance. 

    A letter [PDF] from the SEC addressed to the tech billionaire said he "[did] not appear" to have filed the proper form detailing his 9.2 percent stake in Twitter "required 10 days from the date of acquisition," and asked him to provide more information. Musk's shares made him one of Twitter's largest shareholders. The letter is dated April 4, and was shared this week by the regulator.

    Musk quickly moved to try and buy the whole company outright in a deal initially worth over $44 billion. Musk sold a chunk of his shares in Tesla worth $8.4 billion and bagged another $7.14 billion from investors to help finance the $21 billion he promised to put forward for the deal. The remaining $25.5 billion bill was secured via debt financing by Morgan Stanley, Bank of America, Barclays, and others. But the takeover is not going smoothly.

    Continue reading
  • Cloud security unicorn cuts 20% of staff after raising $1.3b
    Time to play blame bingo: Markets? Profits? Too much growth? Russia? Space aliens?

    Cloud security company Lacework has laid off 20 percent of its employees, just months after two record-breaking funding rounds pushed its valuation to $8.3 billion.

    A spokesperson wouldn't confirm the total number of employees affected, though told The Register that the "widely speculated number on Twitter is a significant overestimate."

    The company, as of March, counted more than 1,000 employees, which would push the jobs lost above 200. And the widely reported number on Twitter is about 300 employees. The biz, based in Silicon Valley, was founded in 2015.

    Continue reading
  • Talos names eight deadly sins in widely used industrial software
    Entire swaths of gear relies on vulnerability-laden Open Automation Software (OAS)

    A researcher at Cisco's Talos threat intelligence team found eight vulnerabilities in the Open Automation Software (OAS) platform that, if exploited, could enable a bad actor to access a device and run code on a targeted system.

    The OAS platform is widely used by a range of industrial enterprises, essentially facilitating the transfer of data within an IT environment between hardware and software and playing a central role in organizations' industrial Internet of Things (IIoT) efforts. It touches a range of devices, including PLCs and OPCs and IoT devices, as well as custom applications and APIs, databases and edge systems.

    Companies like Volvo, General Dynamics, JBT Aerotech and wind-turbine maker AES are among the users of the OAS platform.

    Continue reading

Biting the hand that feeds IT © 1998–2022