The Role Of Quality Engineering In Safeguarding Medical Decisions

Forbes

24-07-2025

Gopinath Kathiresan is a veteran QE leader combining AI, cybersecurity and automation to build smarter, secure software.
AI is transforming healthcare, from interpreting scans to powering early warning systems that guide diagnosis and treatment. In fact, according to a 2024 McKinsey survey, around 70% of healthcare organizations are now using or planning to use generative AI.
But AI is not without risk. Earlier in my career, I worked at GE Healthcare and Bio-Rad Laboratories, where I saw how small software issues could carry clinical consequences. One misfire in device communication or a delayed result in a diagnostic tool could mean a missed treatment window.
That's why, in healthcare, quality doesn't just mean good code. It means protecting patients in real-world, high-stakes moments.
Why Healthcare AI Needs A New Kind Of Oversight
AI in healthcare is advancing fast. Models now help detect tumors, predict readmissions, flag deteriorating patients and assist with triage decisions. A 2023 study in Lancet Digital Health even found that AI-based diagnostic tools, such as those for skin cancer detection, can sometimes match or outperform specialists, especially novice specialists.
But performance in a controlled environment doesn't always translate to the chaos of real-world hospitals.
We can't assume an algorithm is 'good enough' just because it clears a data scientist's benchmark. We need to engineer for safety from the start. That means moving beyond static accuracy numbers and thinking deeply about how AI performs under pressure, across patient types and in unpredictable situations.
The Limitations Of Traditional QA In Medical AI
Earlier in my career as a quality engineer, I worked on regulated diagnostic software and medical device platforms. I contributed to test automation for verification workflows, validated device communication and ensured clinical accuracy standards were met.
That hands-on experience shaped how I think about risk, reliability and patient safety in software and the importance of quality engineering (QE) in healthcare.
Healthcare organizations have long practiced software testing. But testing AI is a different beast. AI models behave probabilistically. They're trained on datasets that might not reflect the populations they serve. Their outputs evolve as conditions shift.
A tool that performs with 95% accuracy in one hospital might drop dramatically in another, due to differences in demographics, calibrations or data quality. In fact, research has shown that rural hospitals or hospitals with fewer resources, for example, may buy AI tools off-the-shelf, meaning the data the AI is trained on looks different than their hospital's population, which can impact performance.
This is where QE must step in, not just as a checkpoint, but as a design partner focused on:
• Bias detection across patient populations.
• Edge-case testing for rare or high-risk conditions.
• Explainability validation to ensure clinicians can interpret results.
• Monitoring for model drift after deployment.
Quality engineers aren't just testing systems; they're safeguarding patients.
From Bugs To Bias: What A Modern QE Pipeline Looks Like
The FDA has acknowledged this new frontier, and its updates to its software as a medical device (SaMD) guidelines show that the agency is acknowledging both AI's potential and risk. While its guidance isn't finalized, they are focusing on several aspects of how AI can impact medical devices, including ongoing monitoring and performance validation for AI and machine learning tools.
But real-world safety demands more than checkboxes. It requires QE practices that validate AI performance not only at launch but also throughout its lifecycle. So what does 'clinical-grade' QE look like in practice? Emerging best practices include:
• Scenario-Based Test Automation: QE teams should consider how these tools will work in real-world scenarios. For example, researchers at the SIMNOVA Simulation Center developed an AI workflow to create and test realistic healthcare simulation scenarios.
• Synthetic Data Injection: Synthetic data can help to round out data samples during testing. For example, Stanford's RoentGen project improved classifier accuracy using synthetic chest X-rays.
• Demographic Audit Layers: Mainly because of the data it's trained on, AI can perform differently based on factors like a patient's ethnicity or gender, emphasizing the importance of fairness audits
• Collaborative Validation: Subject matter experts, such as clinicians and other healthcare stakeholders, should be included in the AI design process to ensure that issues that impact the real-world use of these tools are considered.
This shift isn't just a technical evolution; it's foundational to trust in healthcare AI.
AI Won't Replace Testers—It'll Make Their Role Even More Critical
There's a common fear that AI might make testers obsolete. In healthcare, the opposite is true. AI may help automate test generation and monitor systems post-deployment, but human quality engineers bring what AI can't: context, curiosity and caution. Humans should ask:
• 'Would this still work in an emergency room?'
• 'Is the model unintentionally learning the wrong patterns?'
• 'Could this mislead a busy nurse under pressure?'
That kind of judgment is forged from hands-on experience. No AI system, no matter how advanced, can replace the human instinct to ask, 'What could go wrong here?' If you're in healthcare tech, this is a call to action:
• Testers: Learn AI model behavior and where quality gaps emerge.
• Developers: Collaborate with QE teams early during integration.
• Leaders: Treat quality as foundational, not a bolt-on.
What matters most isn't what AI can do in theory, but how it behaves when seconds count, when a nurse or physician is making a call that could save a life.
The Role Quality Must Play In Saving Lives
Software doesn't just support healthcare anymore; it is healthcare. And as AI becomes more central to decisions, QE must evolve from assurance to active defense.
I've spent over 15 years working in software quality, including time in regulated healthcare. I've seen how fragile trust becomes when tools fail, and how powerful it is when teams get it right. Now more than ever, we need QE professionals to be the silent guardians of patient safety, testing not just for bugs, but for bias, breakdowns and blind spots.
Because when AI enters the hospital, the question isn't just 'Does it work?' It's about whether it will protect the patient when it truly counts.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?