Study warns AI chatbots, search engines may provide unsafe drug information

Researchers from Belgium and Germany conducted the study after discovering that many responses from the AI chatbot were incorrect or potentially harmful. In their paper published in the journal BMJ Quality and Safety, they noted that the complexity of the chatbot's answers could be challenging to comprehend and may necessitate a degree-level education to fully understand.

New Delhi: Artificial Intelligence (AI) powered search engines and chatbots may not always provide accurate and safe information on drugs, and patients shouldn’t rely on these, warned a study on Friday.

Researchers from Belgium and Germany conducted the study after finding many answers were wrong or potentially harmful.

---

In the paper, published in the journal BMJ Quality and Safety, they said that the complexity of the answers provided by the AI chatbot may be difficult to understand and might require degree-level education.

With the introduction of AI-powered chatbots search engines in 2023 underwent a significant shift thanks. The renewed versions offered enhanced search results, comprehensive answers, and a new type of interactive experience.

While the chatbots — trained on extensive datasets from the entire internet — can answer any healthcare-related queries, they are also capable of generating disinformation and nonsensical or harmful content, said the team from the Friedrich-Alexander-Universitat Erlangen-Nurnberg in Germany.

“In this cross-sectional study, we observed that search engines with an AI-powered chatbot produced overall complete and accurate answers to patient questions,” they write.

“However, chatbot answers were largely difficult to read and answers repeatedly lacked information or showed inaccuracies, possibly threatening patient and medication safety,” they add.

For the study, the researchers explored the readability, completeness, and accuracy of chatbot answers for queries on the top 50 most frequently prescribed drugs in the US in 2020. They used Bing copilot, a search engine with AI-powered chatbot features.

Just half of the 10 questions were answered with the highest completeness. Further, chatbot statements didn’t match the reference data in 26 per cent of answers and were fully inconsistent in over 3 per cent of cases.

About 42 per cent of these chatbot answers were considered to lead to moderate or mild harm, and 22 per cent to death or severe harm.

The team noted that a major drawback was the chatbot’s inability to understand the underlying intent of a patient question.

“Despite their potential, it is still crucial for patients to consult their healthcare professionals, as chatbots may not always generate error-free information,” the researchers said.