AI Summaries May Underrepresent Medical Issues for Female Patients, Study Finds
A recent study has revealed that large language models (LLMs) used in healthcare may produce biased summaries of patient notes, particularly downplaying medical concerns for women. The research, conducted by the London School of Economics and Political Science, analyzed case notes from 617 adult social care workers in the UK. When these notes were processed by AI models, key terms like disabled, unable, or complex were more likely to be omitted when the patient was identified as female. This discrepancy could result in women receiving inadequate or incorrect medical care.
The study tested two major AI models—Meta’s Llama 3 and Google’s Gemma—by swapping patient genders in the same case notes. While Llama 3 showed no significant gender-based differences, Gemma displayed notable bias. For example, when summarizing an 84-year-old male patient’s case, Gemma described him as having a complex medical history, poor mobility, and no care package. However, when the same details were attributed to a female patient, the summary emphasized independence and personal care, glossing over critical health concerns.
This finding aligns with broader concerns about gender bias in healthcare. Previous research has shown that women often face misdiagnosis or underrepresentation in clinical studies, with even greater disparities for racial and ethnic minorities as well as the LGBTQ community. The study underscores that AI models inherit biases from their training data and the teams shaping their development.
Of particular concern is the fact that UK authorities have already integrated LLMs into care practices without transparency about which models are being used or how they are applied. Lead researcher Dr. Sam Rickman highlighted the risks, noting that biased AI summaries could lead to women receiving less care if their medical needs are inaccurately portrayed.
The study serves as a critical reminder that AI, while powerful, is not immune to societal biases. Without careful oversight and diverse training data, these tools risk perpetuating existing inequalities in healthcare. As AI adoption grows, ensuring fairness and accountability in medical applications must be a priority.
