This article is more than 1 year old

It's your human hubris holding back AI acceptance

Boffins find Dunning-Kruger effect makes us think we know better

Human psychology may prevent people from realizing the benefits of artificial intelligence, according to a trio of boffins based in the Netherlands.

But with training, we can learn to overcome our biases and trust our automated advisors.

In a preprint paper titled "Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems," Gaole He, Lucie Kuiper, and Ujwal Gadiraju, from Delft University of Technology, examine whether the Dunning-Kruger effect hinders people from relying on recommendations from AI systems.

The Dunning-Kruger effect (DKE) dates back to research from 1999 by psychologists David Dunning and Justin Kruger, "Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments."

Dunning and Kruger posit that incompetent people lack the capacity to recognize their incompetence and thus tend to overestimate their abilities.

Assuming DKE exists – something not everyone agrees on – the Delft researchers suggest this cognitive condition means AI guidance may be lost on us. That's not ideal since AI systems presently tend to be pitched as assistive systems that augment human decision-making rather than autonomous systems that operate without oversight. Robo help doesn't mean much if we don't accept it.

"This a particularly important metacognitive bias to understand in the context of human-AI decision making, since one can intuitively understand how inflated self-assessments and illusory superiority over an AI system can result in overly relying on oneself or exhibiting under-reliance on AI advice," state He, Kuiper, and Gadiraju in their paper, which has been conditionally accepted to CHI 2023. "This can cloud human behavior in their interaction with AI systems."

To test this, the researchers asked 249 people to answer a series of multiple choice questions to test their reasoning. The respondents were asked to answer questions first by themselves and then with the help of an AI assistant.

The questions, available in the research project GitHub repository, consisted of a series of questions like this:

Physician: In comparing our country with two other countries of roughly the same population size, I found that even though we face the same dietary, bacterial, and stress-related causes of ulcers as they do, prescriptions for ulcer medicines in all socioeconomic strata are much rarer here than in those two countries. It's clear that we suffer significantly fewer ulcers, per capita, than they do.

The study participants were then asked, Which one of the following, if true, most strengthens the physician's argument?

  1. The two countries that were compared with the physician's country had approximately the same ulcer rates as each other.
  2. The physician's country has a much better system for reporting the number of prescriptions of a given type that are obtained each year than is present in either of the other two countries.
  3. A person in the physician's country who is suffering from ulcers is just as likely to obtain a prescription for the ailment as is a person suffering from ulcers in one of the other two countries.
  4. Several other countries not covered in the physician's comparisons have more prescriptions for ulcer medication than does the physician's country.

After respondents answered, they were presented with the same questions as well as an AI system's recommended answer (D for the question above), and were given the opportunity to change their initial answer. This approach, the researchers say, has been validated by past research [PDF].

Based on the answers they received, the three computer scientists conclude "that DKE can have a negative impact on user reliance on the AI system…"

But the good news, if that's the right term, is that DKE is not destiny. Our mistrust of AI can be trained away.

"To mitigate such cognitive bias, we introduced a tutorial intervention including performance feedback on tasks, alongside manually crafted explanations to contrast the correct answer with the users’ mistakes," the researchers explain. "Experimental results indicate that such an intervention is highly effective in calibrating self-assessment (significant improvement), and has some positive effect on mitigating under-reliance and promoting appropriate reliance (non-significant results)."

Yet if tutorials helped those exhibiting overconfidence (DKE), corrective re-education had the opposite effect on those who initially underestimated their capabilities: It made them either overconfident or possibly algorithm averse – a known consequence [PDF] of seeing machines make mistakes.

In all, the researchers conclude that more work needs to be done to understand how human trust of AI systems can be shaped.

We'd do well to recall the words of HAL, from 2001: A Space Odyssey.

I know I've made some very poor decisions recently, but I can give you my complete assurance that my work will be back to normal. I've still got the greatest enthusiasm and confidence in the mission. And I want to help you.


More about


Send us news

Other stories you might like