Dr Hannah Fry: We need to be wary of algorithms behind closed doors

UCL researcher on the tragedy of the age of data


Interview Sure, algorithms are insanely useful, but we need to watch we don't become complacent and unable to question them, University College London's Dr Hannah Fry warned in an interview with The Register.

Dr Fry is a lecturer in the mathematics of cities at the Centre for Advanced Spatial Analysis at UCL, where her research "revolves around the study of complex social and economic systems at various scales, from the individual to the urban, regional and the global, and particularly those with a spatial element."

While not engaged in research, however, Dr Fry is quickly becoming one of the UK's favourite mathematicians, known for her work on BBC 4's The Joy of Data, as well as her popular TED talk, 'The Mathematics of Love', which applied statistical and data-scientific models to dating, sex and marriage.

Youtube Video

Chatting to The Register ahead of DataFest2017, the inaugural week-long data science festival in Edinburgh, Dr Fry said she thought the event was going to be "a lot of fun".

"It's perfectly positioned time-wise. It's something people really need to address, and having so many excellent people together in a room at once; it's going to be a great few days."

"Data science as a field has exploded over the past five years," because there's "much more access to data now" said Dr Fry, noting that with "sensors, IoT, with us living more of our lives online" there's now "very little that is untouched by data".

We "realised a few years ago how much data there was," Dr Fry said. "I think the whole thing is very exciting. We have these wonderful opportunities to stand back and rethink how we design our societies, our businesses, almost everything we encounter on a daily basis."

That said, it's still necessary for people to be "paying attention to how biases you have in data can end up feeding through to the analyses you're doing".

Algorithms behind closed doors

Last week, a paper by Julia Powles, an academic at the University of Cambridge – though soon departing for Cornell University in New York – and Hal Hodson, a journalist, described a deal between Google DeepMind and the Royal Free London NHS trust to use patient data without explicit consent as "inexcusable" and potentially in breach of data protection laws.

Dr Fry hadn't read the paper, but believed it was "a conversation that needs to be addressed" especially when it came to ownership of data, access to data, and most importantly, "transparency in terms of the algorithms".

Proprietary software is built with an incentive that might not align with the interests of individual people, who are just data points within it, said Dr Fry. This can be a casual issue or a serious problem, she added, because these algorithms can be used in various situations, from encouraging consumers to purchase particular products, through to establishing whether individuals get loans or decent insurance rates, and have even been used in the US criminal justice system too.

"Algorithms that sit behind closed doors, we need to open those up a bit," said Dr Fry. The issue is that without access to seeing how they function, "you can't argue against them. If their assumptions and biases aren't made open to scrutiny then you're putting a system in the hands of a few programmers who have no accountability for the decisions that they're making."

"In some situations, this doesn't matter," Dr Fry acknowledged. "Netflix is not fundamentally important to the structure of society; but then, some algorithms about predicting reoffending rates for individuals in US are used in sentencing, and the analysis of the data has very serious consequences there.

"An example I use in my talk is of a young man who was convicted of the statutory rape of a young girl – it was a consensual act, but still a statutory crime – and his data was put into this recidivism algorithm and that was used in his sentencing. Because he was so young and it was a sex crime, it judged him to have a higher rate of offending and so he got a custodial sentence. But if he had been 36 instead of 19, he would have received a more lenient sentence, though by any reasonable metric," one might expect a 36-year-old to receive a more punitive sentence.

Collaboration and interest

Dr Fry said the stuff she tends to do "thinks about things from the perspective of the individual in society, rather than as a customer. When designing algorithms as a business owner, your incentive is your profit, something for your business, it's not an incentive to maximise something for the individual. If the two things align then that's great, but generally you're taking care of your business."

The issue is where these two things diverge, when algorithms protect the business rather than individuals, she added. "Classic examples are insurance rates, or banks giving loans, where people from particular backgrounds are very unfairly disadvantaged because of the data category that they're in. You could argue that unfairness extends out to other types of commercial software – there was LinkedIn showing higher paid job advertisements more often to men than women," which was based on dodgy analysis too.

Inevitably there are biases in data because you can't capture the completeness of the real world. Not matter how rich your data sources are, you can't capture the vast richness of reality, and as a result anything you leave out will bias how the world looks through your data. And that's fine, but we have to be aware that that's happening.

"And anytime a programmer makes a decision about how to deal with data, how to average it or clean it, you're imparting more of your own bias on it. Even professionals making their data as impartial as possible, they are expecting the representation of reality that it gives them.

"Sometimes these assumptions and biases can be really hidden, and that can be dangerous," she added, "but at the same time, though, it's not as if live biases don't exist in systems without algorithms and data," noting studies showing that judges have passed harsher sentences just before lunch, or when local football teams have recently lost a game.

It could be sweet

There's a possibility – as with the work of startup Numerai, as covered by Wired – to use algorithms within a social system that is "much more open source and collaborative," said Dr Fry. "That's one way to guard against these biases and unintended consequences that can end up having a damaging effect.

"I work in an interdisciplinary department. When you're looking into the data of social systems or how society's structures, the silos that were created a couple of hundred years ago don't apply. It has to be a collaborative effort.

"Imagine life without any algorithms at all, you wouldn't be able to do anything. This is already completely encompassing. We have a habit of over-trusting what mathematics or computer scientists tell us to do, without questioning it, too much faith in the magical power of analysis.

"I would like people to know more that there are limitations. Algorithms and data should support the human decision, not replace it." ®

Similar topics


Other stories you might like

  • Prisons transcribe private phone calls with inmates using speech-to-text AI

    Plus: A drug designed by machine learning algorithms to treat liver disease reaches human clinical trials and more

    In brief Prisons around the US are installing AI speech-to-text models to automatically transcribe conversations with inmates during their phone calls.

    A series of contracts and emails from eight different states revealed how Verus, an AI application developed by LEO Technologies and based on a speech-to-text system offered by Amazon, was used to eavesdrop on prisoners’ phone calls.

    In a sales pitch, LEO’s CEO James Sexton told officials working for a jail in Cook County, Illinois, that one of its customers in Calhoun County, Alabama, uses the software to protect prisons from getting sued, according to an investigation by the Thomson Reuters Foundation.

    Continue reading
  • Battlefield 2042: Please don't be the death knell of the franchise, please don't be the death knell of the franchise

    Another terrible launch, but DICE is already working on improvements

    The RPG Greetings, traveller, and welcome back to The Register Plays Games, our monthly gaming column. Since the last edition on New World, we hit level cap and the "endgame". Around this time, item duping exploits became rife and every attempt Amazon Games made to fix it just broke something else. The post-level 60 "watermark" system for gear drops is also infuriating and tedious, but not something we were able to address in the column. So bear these things in mind if you were ever tempted. On that note, it's time to look at another newly released shit show – Battlefield 2042.

    I wanted to love Battlefield 2042, I really did. After the bum note of the first-person shooter (FPS) franchise's return to Second World War theatres with Battlefield V (2018), I stupidly assumed the next entry from EA-owned Swedish developer DICE would be a return to form. I was wrong.

    The multiplayer military FPS market is dominated by two forces: Activision's Call of Duty (COD) series and EA's Battlefield. Fans of each franchise are loyal to the point of zealotry with little crossover between player bases. Here's where I stand: COD jumped the shark with Modern Warfare 2 in 2009. It's flip-flopped from WW2 to present-day combat and back again, tried sci-fi, and even the Battle Royale trend with the free-to-play Call of Duty: Warzone (2020), which has been thoroughly ruined by hackers and developer inaction.

    Continue reading
  • American diplomats' iPhones reportedly compromised by NSO Group intrusion software

    Reuters claims nine State Department employees outside the US had their devices hacked

    The Apple iPhones of at least nine US State Department officials were compromised by an unidentified entity using NSO Group's Pegasus spyware, according to a report published Friday by Reuters.

    NSO Group in an email to The Register said it has blocked an unnamed customers' access to its system upon receiving an inquiry about the incident but has yet to confirm whether its software was involved.

    "Once the inquiry was received, and before any investigation under our compliance policy, we have decided to immediately terminate relevant customers’ access to the system, due to the severity of the allegations," an NSO spokesperson told The Register in an email. "To this point, we haven’t received any information nor the phone numbers, nor any indication that NSO’s tools were used in this case."

    Continue reading

Biting the hand that feeds IT © 1998–2021