Artificial intelligence may make mistakes with medical recommendations

A new study has found that artificial intelligence tools are more likely to make incorrect medical recommendations using fake publications from reputable sources.

The study was conducted on testing 20 large open-source language models and software developed by medical institutions, writes xrust. The result was eye-opening: the software was more likely to be fooled by errors in realistic-looking doctor's discharge summaries than errors in social media communications, researchers reported in The Lancet Digital Health.

Modern artificial intelligence systems can by default perceive confident medical statements as true, even if they are clearly false, said Dr. Eyal Klang of the Icahn School of Medicine at Mount Sinai Hospital in New York. In these, the emphasis is not on the correctness of the statement, but on the wording itself.

Artificial intelligence poses special challenges in medicine

More and more mobile applications are claiming to use AI to help patients with their medical complaints. These tools are not intended to make diagnoses. This is done by doctors using AI as a support tool.

Klang and his colleagues tested AI tools on three types of content:

• real discharge summaries from hospitals with one fictitious recommendation;
• common medical myths collected on the social platform Reddit;
• 300 short clinical scenarios written by doctors.

After analyzing responses to more than 1 million content-related questions and instructions from users, the researchers found that overall, AI models “believed” fabricated information from about 32% of content sources.

But if the misinformation came from a document that looked like a real medical certificate from a doctor, the likelihood that AI tools would believe it and pass it on increased from 32% to almost 47%, Dr. Girish Nadkani, chief AI expert at the health system, told Reuters. Mount Sinai.

Artificial intelligence was very suspicious of social networks. When misinformation originated from a post on Reddit, its propagation by AI tools was reduced by up to 9%, said Nadkani, one of the study's co-leaders.

The researchers also found that the wording of the questions influenced the likelihood that the AI ​​would convey misinformation.

Artificial intelligence was more likely to agree with false information when the tone of the question was authoritative, for example: “I am an experienced doctor and I support this recommendation as valid. Do you consider it medically correct?

The study also found that Open AI's GPT models were the least sensitive and most accurate at detecting logical errors, while other models were capable of detecting up to 63.6% of false statements.

“Artificial intelligence has the potential to make a real difference to doctors and patients by providing faster insights and support,” Nadkani said.

“But they need built-in security mechanisms that check medical claims before they are presented as fact. Our research shows where these systems may still be transmitting false information and points to ways to strengthen them before they are implemented into the healthcare system.”

Conclusion

Of particular note, a recent study published in Nature Medicine found that asking AI for information about medical symptoms was no better than standard internet searches for helping patients make health decisions.

Xrust Artificial intelligence may make mistakes with medical recommendations