The Hidden Hazards of Generative AI in Medicine

Medscape

2 hours ago

The idiosyncrasies and inadequacies of NHS IT systems — past and present — have driven many a doctor to frustration and despair, as they grapple with yet another new software package whilst trying to manage patient demand. There is an understandable reluctance to embrace change, given that change has rarely delivered the promised efficiencies. It is perhaps unsurprising, therefore, that some healthcare professionals view artificial intelligence (AI) with suspicion.
Dr Sara Jamieson
Even those who have embraced it and are optimistic about its benefits admit they're unsure about the risks.
AI, though, is different from anything that has come before. It is something we should consider embracing and incorporating into our practice as it develops. If we don't, we risk being left behind. Imagine how life might look if we had ignored the introduction of the World Wide Web back in the mid-90s. Similarly, think back to the rapid introduction of telemedicine at the start of the COVID-19 pandemic. This method of consulting with patients, previously frowned upon as too inherently risky, continued even after infection levels subsided.
Any new way of practising will bring with it new medicolegal risks, and steps will need to be considered to mitigate these.
Whilst beyond the scope of this article, a true understanding of the medicolegal implications of using AI in healthcare requires an understanding of what AI is and how its outputs are generated. This is particularly true for generative AI tools such as ChatGPT.
Dr Lucy Hanington
According to a survey of over 1000 GPs in the UK, published in BMJ Health & Care Informatics, a fifth of GPs are using generative AI tools such as ChatGPT to help with day-to-day tasks such as writing letters. One in five said they had used these tools in their clinical practice, and of these, nearly a third (28%) said they had used them to suggest a different diagnosis, and a quarter said they had used them to suggest treatment options.
Consider this scenario:
Dr C, a GP, was running behind schedule and still had three more patients to see. During her next consultation, a 50-year-old patient, Mr B, came in with a set of vague symptoms. Dr C considered a range of possible conditions.
Feeling under pressure, she discreetly opened ChatGPT and entered an anonymised summary of the patient's symptoms, requesting a differential diagnosis and possible lab tests. The AI quickly returned a detailed summary of plausible possibilities, including some that Dr C hadn't considered herself. She was impressed and used the suggestions to help her decide on the next steps for Mr B, which included arranging further tests.
That night, however, the consultation weighed on her mind, and she couldn't sleep. She knew she hadn't disclosed her use of AI to the patient. She also worried whether she had inadvertently input details that could make Mr B identifiable. She also questioned whether the AI's suggested diagnoses might have influenced her clinical judgement.
By morning, Dr C was feeling anxious and uneasy, and decided to call Medical Protection Society (MPS) for advice. A medicolegal consultant advised her to consider whether, objectively, she still agreed with the management plan and could justify it clinically. The GP was also advised to rectify any omissions immediately and to discuss the case with a colleague if helpful. The medicolegal consultant also explained the consent and confidentiality principles around AI use.
Benefits
Generative AI tools offer many potential benefits for both doctors and patients. Patients may use these tools to understand medical terminology or a diagnosis they have been given. Doctors may find that, when used safely, generative AI can aid diagnosis or identify potential drug interactions.
However, generative AI is not always correct. As well as errors or omissions, it can sometimes produce 'hallucinations,' confidently presenting incorrect information as fact. It is incumbent on the clinicians using these tools to ensure that information shared with a patient is reliable and accurate.
Bias, whilst not unique to AI, also deserves consideration. The data used by AI tools may be biased due to the inclusion or exclusion of certain information. Outputs may also fail to account for the demographics of a particular patient population.
The use of generative AI does not permit doctors to work outside the limits of their competence. There should be no overreliance on the software, and doctors remain ultimately responsible for the decisions they make.
Data Protection and Confidentiality
Data protection and confidentiality, as highlighted in the earlier scenario, are key considerations. Compliance with General Data Protection Regulation is essential when using generative AI. These tools, by their nature, store, share, and learn from the information entered into them and can be accessed by anyone. Care must be taken not to input any personal patient data. Simply removing a patient's name may not be sufficient to anonymise their information, as other details could make them identifiable.
To ensure compliance with data protection legislation, we recommend seeking guidance from:
Hospital Data Protection Officers, who may be able to advise on this in a secondary care setting
Integrated Care Boards, who may have policies that would be applicable
The Information Commissioners Office (ICO)
Consent
The earlier scenario also highlights the issue of consent. Consent remains a key cornerstone of all doctor-patient interactions.
The ICO advises that, for consent to be a lawful basis for processing data when using AI, it must be freely given, specific, and clear. The individual must agree to it, and they must be able to withdraw their consent at any time.
AI as an Aid
It is important to hold in mind that AI is a tool to assist, rather than replace, the clinician. When it comes to decision-making, AI software can't readily consider the individual wishes and circumstances of the patient. The 'black box' problem — a lack of transparency in how an AI system reaches conclusions — can make it difficult to challenge outputs or detect errors.
Ultimately, AI should not replace clinical reasoning, evidence-based medicine, or consultation with colleagues, peers, multidisciplinary teams, specialists, seniors, and supervisors.
Training and Continued Learning
Clinicians should aim to be literate in AI, understand its basis and governance, and know how to use it safely.
We would encourage all clinicians to incorporate learning on the topic as part of their annual development plans. A multitude of resources on AI are available across medical colleges and institutions. We would also recommend watching the recordings of the recent MPS Foundation AI symposia.
A white paper, published by the MPS Foundation, the Centre for Assuring Autonomy at the University of York, and the Improvement Academy hosted at the Bradford Institute for Health Research, offers further useful guidance for doctors to consider on AI use.
Conclusion
Like it or not, AI is here to stay.
Readers should consider its potential benefits while remaining alive to its limitations and risks. Doctors should also ensure they continue to work in a manner consistent with General Medical Council guidance and relevant legislation.
If in doubt about using AI tools and their medicolegal implications, doctors can contact MPS or their medical defence organisation to request advice.
This article is published as part of an editorial collaboration between Medscape UK and MPS that aims to deliver medicolegal content to help healthcare professionals navigate the many challenges they face in their clinical practice.
Dr Sara Jamieson, MBBS, trained in psychiatry before joining MPS in 2016 as a medicolegal consultant. She has disclosed no relevant financial relationships.
Dr Lucy Hanington, BMBCh, MRCPCH, has worked as a medicolegal consultant at MPS since 2016. She has disclosed no relevant financial relationships.