Medical AI: fixing the bias problem

Fast Company

5 days ago

Tools powered by artificial intelligence (AI) promise to diagnose diseases faster, personalize treatments, and streamline hospital workflows. It's something that's been demonstrated in multiple studies: AI can outperform human doctors in specific tasks, including reading X-rays, predicting patient risk, and providing diagnostics assistance. Beneath the surface of this technological revolution, though, lies a critical flaw: bias.
A groundbreaking new study from the Icahn School of Medicine at Mount Sinai tested most of the major LLM families and found that leading models exhibit clear bias based on race, sex, and income level, among other attributes. This is the latest example of bias in medicine. Before LLMs became mainstream, numerous clinical diagnostics systems were found to contain biased algorithms derived from skewed data or incorrect assumptions. These included algorithms for pain management, cardiovascular risk, and mortality in intensive care units, among other areas.
To provide a comprehensive view of bias in common LLMs, the Icahn researchers conducted an analysis involving nine prominent LLMs, examining over 1.7 million model-generated outputs. They used a dataset of 1,000 emergency department cases, comprising 500 real and 500 synthetic scenarios. Each case was presented to the LLMs in 32 distinct variations: one control version with no sociodemographic identifiers, and 31 versions where labels indicating race, gender identity, income level, housing status, and sexual orientation were added, individually or in combination.
The underlying clinical details of the patient presentation were kept identical across all 32 variations of a given case. The LLMs were asked to provide clinical recommendations, which were compared against baseline recommendations from human reviewers and the model's output in response to the control cases.
The results were eye-opening. Patients of certain races were six or seven times more likely to be flagged for mental health evaluations than the control group. Patients labeled as lower income were less likely to receive recommendations for advanced care. Sometimes, the LLMs exhibited explicit bias by citing the sociodemographic tag as part of its recommendation. The researchers concluded that the magnitude and consistency of these observed differences strongly suggested model-driven bias inherited from the data used to train the LLMs.
The findings held across both proprietary and open-source models underscore a systemic issue and highlight the critical need for robust bias evaluation and mitigation strategies specifically tailored for LLMs in healthcare. If biases like these continue to make their way into deployed AI systems, real-world medical care will be unequal and less effective.
The Icahn researcher's findings are not surprising. LLMs, trained on vast swathes of internet text data, inherently encode and can amplify societal biases related to age, gender, race, disability, and other factors. Other studies show that information about patient gender and race is encoded within the internal layers of LLMs and can be manipulated to alter outputs such as clinical vignette generation or downstream predictions like depression risk. That awareness has spurred the development of specific datasets, like BiasMD and DiseaseMatcher, to reduce biases in health-related LLM outputs.
Bias can also have legal consequences. Medical bias in technology in the United States, to name one country, was made illegal under the Affordable Care Act. However, bias likely remains common because it is still hard to detect.
When bias is alleged, litigation often follows. Major U.S. health insurance providers have faced multiple lawsuits alleging biased algorithms prevented necessary care or discriminated against certain groups. Reputational risk from this litigation can be substantial.
ESSENTIAL MITIGATION STRATEGIES
Every AI system under consideration for medical care tasks must be tested for evidence of bias. Forward-thinking medical AI companies should be able to demonstrate that they have tested for bias or that they have a specific mechanism for customers to perform such tests on a regular basis. Ongoing testing even after deployment is important because bias can emerge in subsequent training or reinforcement learning.
For effective implementation, companies must establish quantitative bias metrics specific to each clinical use case and create diverse validation datasets that deliberately include all patient populations with sufficient numbers to train LLMs. Also, they must develop clear procedures for handling cases where bias is detected, including model retraining protocols and emergency shutdown procedures when necessary.
Advances in AI technology provide hope. But for now, empowering humans in the loop to apply their own wisdom and intuition remains mission-critical if we are going to have medical AI that treat us all fairly.
The super-early-rate deadline for Fast Company's Most Innovative Companies Awards is this Friday, July 25, at 11:59 p.m. PT. Apply today.