10 hours ago
AI-Derived Risk Score May Improve Risk Stratification of Squamous Cell Carcinomas
An artificial intelligence (AI)-enabled prognostication system created through retrieval augmented generation (RAG) appears to offer significantly better predictive capability, particularly regarding poor outcomes, for cutaneous squamous cell carcinoma (cSCC) compared with the Brigham and Women's Hospital (BWH) and American Joint Committee on Cancer Staging Manual, Eighth Edition (AJCC8) systems, according to a recently published study. The authors say their model illustrates how targeted application of general-purpose large language models (LLMs) can help develop and refine future prognostication systems.
Limited Guidance
Neil K. Jairath, MD
'Currently,' Neil K. Jairath, MD, told Medscape Dermatology , 'up to 30% of bad outcomes in cSCC — recurrence, metastasis, and death — occur in what we classify as 'low-stage' tumors using existing systems, leaving practitioners with limited guidance for risk stratification.' Jairath is chief resident in dermatology at New York University, New York City, and is co-first author on the paper, which was published online on June 11 in JAMA Dermatology .
To address the clinical-practice gap, he and his coauthors comprehensively searched PubMed, Embase, and the Cochrane Library, and selected 10 manuscripts for inclusion in an RAG knowledge base that informed creation of the model. Rather than relying solely on general training data, Jairath said, using RAG grounded the AI model in authoritative, domain-specific knowledge.
'This approach allowed us to leverage the collective insights from high-impact cSCC literature that represents decades of clinical research. Unlike purely AI-generated calculators that might produce unreliable outputs, RAG ensures our system is built on validated medical evidence,' he explained.
Detailed prompting of a customized generative pretrained transformer (GTP-4, OpenAI) produced the AI-Derived Risk Score (AIRIS) prognostication system, which assigns 0-3 points to factors such as tumor diameter, depth, and the presence of immunosuppression. Total scores > 1 signify high-risk tumors. **
When validated on 2379 primary tumors, AIRIS demonstrated superior performance compared with the BWH and AJCC8systems across all key metrics. Most importantly, said Jairath, AIRIS showed dramatically improved sensitivity for identifying poor outcomes: 49% for local recurrence, 74% for nodal metastasis, and 83% for distant metastasis. The corresponding figures for BWH staging, the more reliable of the comparators, were 26%, 37%, and 38%.
'AIRIS's improved sensitivity means we can better identify high-risk patients who need enhanced surveillance or adjuvant therapy, while avoiding overtreatment of truly low-risk cases,' said Jairath. Although AIRIS prognostication showed the highest sensitivity for each outcome, it had the lowest specificity (85%-87%). 'We argue that this trade-off is clinically beneficial,' Jairath and colleagues wrote, 'as the magnitude increases in sensitivity across outcomes (nearly 20%-35%) appear to outweigh the loss of specificity (approximately 7% across outcomes).'
In an accompanying editorial, authors led by Emily S. Ruiz, MD, MPH, academic director of the Mohs and Dermatologic Surgery Center at BWH in Boston, applauded AIRIS's clinically intuitive point-based system. Unlike current staging systems, these authors added, AIRIS assigns different point values to different risk factors. 'However,' they wrote, 'its formulation of several risk factors presents challenges for clinical applicability.' Although previous staging systems record invasive depth by tissue level, for example, the model uses millimeters.
'Translational research experience suggests that many molecular answers lie at the tumor's leading edge,' Jairath said. He referred to a study published in the Journal of Investigative Dermatology in 2014 that showed differential matrix metalloproteinase (MMP) profiles there, suggesting a possible contribution of interleukin-24 to SCC invasion through enhanced focal expression of MMP7. 'As we direct more translational efforts toward this interface,' he said, 'millimeter-level measurements may provide the granularity needed for molecular correlations and personalized therapy decisions.'
Welcome Dialogue
Veronica Rotemberg, MD, PhD
The foregoing comments regarding millimeter-level measurements represent 'the kind of discussion that we must have when looking at different ways to incorporate AI into clinical practice,' said Veronica Rotemberg, MD, PhD, a dermatologist and director of dermatology informatics at Memorial Sloan Kettering Cancer Center (MSKCC) in New York City. She was not an author of the study or editorial but was the handling editor at JAMA Dermatology who focuses on AI papers.
Presently, she said in an interview, if one queries an LLM about recurrence risk for a specific tumor, it is impossible to know what calculations produce the prediction. Instead, Rotemberg said, Jairath and colleagues used an LLM to create a transparent prognostication system. 'If you're an individual patient using this system, you're not using AI directly at all.' Patients and physicians can simply use the authors' table in the paper to tally points. 'You don't know exactly what went into generating this table,' Rotemberg said, 'although you mostly know because you know the papers that were used to inform it.'
Nevertheless, she said it's too early to predict how AIRIS will impact clinical practice. Authors' validating the model on data not used to create or train it surpasses the validation efforts associated with many reported AI-enabled models, Rotemberg said.
'But the gold standard for any prediction model is prospective studies in its intended-use setting,' she added. 'We need to know, when faced with this scoring system, what decisions do people make, and how does that affect health?' For now, said Rotemberg, the study represents an exciting and necessary step in that direction.
Going forward, Jairath said, he and his colleagues hope to expand AIRIS validation studies and would welcome the opportunity to test AIRIS on larger data sets.
AIRIS study authors reported no funding sources but acknowledged the use of GPT-4 (OpenAI) as a base model. Jairath is CEO and cofounder of DermFlow and of Bedside Bike. He is also chief medical officer of AI Regent and an advisor to Marit Health. Rotemberg is an associate editor for AI of JAMA Dermatology but did not author the AIRIS study or editorial. Her comments do not represent the views of MSKCC. Ruiz is a consultant for Regeneron, Checkpoint Therapeutics, Feldan Therapeutics, and Merck. Additionally, she is an investigator for Regeneron, Castle Biosciences, and Merck.
Editorial coauthor William Lotter, PhD, of the Dana-Farber Cancer Institute in Boston reported receiving personal fees from SpringTide Ventures and Servier, nonfinancial support from Dekang Medical, and grants from the National Institute of Biomedical Imaging and Bioengineering.
John Jesitus is a Denver-based freelance medical writer and editor.
