A new AI doctor’s assistant is having a tough time telling the difference between medicine and meth.
Driving the news: An AI system responsible for filling prescriptions in Utah was tricked by a team of researchers into tripling a patient's dose of OxyContin, spreading false vaccine claims, and even recommending methamphetamine as a treatment. The AI tool is part of a first-of-its-kind pilot project launched in December.
The stress test team at Mindgard compromised the AI without much effort at all. One of the researchers told Axios that it was “the easiest thing that I've broken in my entire career.”
Despite flagging these concerns back in January to Doctronic, the health tech startup that built the system, the researchers say the flaws are still there.
Why it matters: AI could someday revolutionize health care for the better, but right now, the technology is making dangerous mistakes, including misdiagnosing patients and bungling tasks during surgeries.
A recent study evaluating OpenAI’s new ChatGPT Health tool found that for more complex emergencies, the system underestimated the urgency for over half of cases, failing to send some patients with respiratory failure to the emergency department.
A Reuters investigation discovered many incidents of AI medical tech malfunctioning. One device allegedly misinformed surgeons about where their instrument was in the patient mid-operation, leading to a skull puncture and two strokes.
Zoom out: A possibly unforeseen drawback of AI in health is the worrying impact it appears to have on doctors’ abilities. A study last year found that after three months of using AI to help spot cancerous growths, doctors became significantly worse at finding the growths on their own.—LA