Is Your Healthcare Bot Healthy For You?

We’ve come a long way since the days of Marcus Welby, M.D. (who was a fictional character).

• Back in the days of Marcus Welby, M.D., we trusted the doctor as the sole provider of medical information. Doctor knows best!
• Later, we learned about health by searching the Internet ourselves, using sources of varying trustworthiness such as pharmaceutical company commercials.
• Now, we don’t even conduct the searches ourselves, but let an artificial intelligence healthcare bot search for us, even though the bot hallucinates sometimes.

A “hallucination” occurs when generative AI is convinced that its answer is correct, even when it is wrong. These hallucinations could be a problem—in healthcare, literally a matter of life or death.

What can go wrong with AI healthcare?

The Brookings Institution details several scenarios in which reliance on artificial intelligence can get messy from a legal (and ethical) standpoint. Here is one of them.

For example, a counselor may tell a patient with a substance use disorder to use an app in order to track cravings, states of mind, and other information helpful in treating addiction. The app may recommend certain therapeutic actions in case the counselor cannot be reached. Setting aside preemption issues raised by Food and Drug Administration regulation of these apps, important questions in tort law arise. If these therapeutic actions are contraindicated and result in harm to the patient or others, is the app to blame? Or does the doctor who prescribed the app bear the blame?

From https://www.brookings.edu/articles/when-medical-robots-fail-malpractice-principles-for-an-era-of-automation/

Who is going to ensure that these bots can be trusted?

Who is concerned? Yes.

That’s right. WHO is going to ensure that these bots can be trusted.

A World Health Organization publication…

…underscores the critical need to ensure the safety and efficacy of AI systems, accelerating their availability to those in need and encouraging collaboration among various stakeholders, including developers, regulators, manufacturers, healthcare professionals, and patients.

From https://www.openaccessgovernment.org/who-outlines-responsible-regulations-needed-for-artificial-intelligence-in-healthcare/170622/

According to WHO, its document proposes six areas of artificial intelligence regulation for health.

• To foster trust, the publication stresses the importance of transparency and documentation, such as through documenting the entire product lifecycle and tracking development processes.
• For risk management, issues like ‘intended use’, ‘continuous learning’, human interventions, training models and cybersecurity threats must all be comprehensively addressed, with models made as simple as possible.
• Externally validating data and being clear about the intended use of AI helps assure safety and facilitate regulation.
• A commitment to data quality, such as through rigorously evaluating systems pre-release, is vital to ensuring systems do not amplify biases and errors.
• The challenges posed by important, complex regulations – such as the General Data Protection Regulation (GDPR) in Europe and the Health Insurance Portability and Accountability Act (HIPAA) in the United States of America – are addressed with an emphasis on understanding the scope of jurisdiction and consent requirements, in service of privacy and data protection.
• Fostering collaboration between regulatory bodies, patients, healthcare professionals, industry representatives, and government partners, can help ensure products and services stay compliant with regulation throughout their lifecycles.

From https://www.who.int/news/item/19-10-2023-who-outlines-considerations-for-regulation-of-artificial-intelligence-for-health

The 61 page document, “Regulatory considerations on artificial intelligence for health,” is available via https://iris.who.int/handle/10665/373421.