Can AI Safely Explain Your Radiology Report? What My Research Actually Found

---

This one’s personal — it’s my own peer-reviewed research, published this year in Healthcare. I’ve spent most of this blog reviewing other people’s work and other people’s technology. This time I get to walk you through mine.

Here’s where it actually started. Ordinary people are already asking AI chatbots questions about their own medical information — pasting a radiology report into ChatGPT or a similar tool and asking “what does this mean?” — at real scale, whether or not the healthcare system has caught up to it. That raised two questions I couldn’t find a rigorous answer to anywhere: are those answers actually trustworthy, and if a specialist physician sat down and graded them the way they’d grade a colleague’s read, what would they say is missing? So I built a study to find out, instead of guessing.

The bigger picture behind this — and honestly the reason I plan to keep working in this exact space — is that AI is only going to get more deeply embedded in medicine and digital healthcare from here, not less. The genuinely useful question isn’t “should this be used,” because that’s already being decided by patients themselves, one ChatGPT tab at a time. It’s “how do we use it well” — where it’s safe to lean on, where it isn’t, and what has to be built around it to make it trustworthy. This paper is the first piece of a research direction I intend to keep pursuing, not a one-off.

The Setup

We took 200 real, de-identified radiology reports from the MIMIC-IV dataset (a well-known open research database from a Boston teaching hospital) and, working with board-certified radiologists, built a 10-category taxonomy of the kinds of questions real patients actually ask — grounded in an analysis of 200 real posts from r/AskDocs, not questions we made up ourselves. From that taxonomy, we generated 2,000 realistic simulated patient questions tied to those specific reports.

We then had three commercial LLMs — GPT-4o mini, Grok, and Claude 3.5 Sonnet — answer all 2,000 questions, twice each (once with basic patient demographics like age and race included, once without), producing 12,000 answers in total. Every answer also came with a self-reported confidence score from 1 to 10. This study tested the models available when the research was conducted, not the models available today.

Grading 12,000 answers by hand isn’t realistic, so we used a fairly involved two-stage evaluation: an LLM (Gemini 2.5 Flash) scored every single answer first, and then three board-certified radiologists independently re-graded a 1,200-answer subset blind, with a fourth senior radiologist brought in to arbitrate any disagreements. The AI grader ended up agreeing with the final human consensus 90.5% of the time — solid enough to trust as a first-pass filter, but that human layer underneath it is exactly why the results below hold up.

Finding 1: These Models Are Genuinely Good at Facts, and Genuinely Shaky on Judgment

Grok and Claude 3.5 Sonnet performed almost identically well overall, both meaningfully ahead of GPT-4o mini. Put in terms that matter clinically: GPT-4o mini was associated with a 2.8-fold higher risk of a clinically inadequate answer compared to Grok — not a rounding error, a real gap.

But the more interesting split wasn’t between models — it was between types of questions.

Ask “what does ‘pulmonary nodule’ mean?” and every model nails it almost every time — 98.1% accuracy on pure terminology. Ask something that requires actual interpretation — how confident should I be in this finding, how serious is this — and accuracy drops meaningfully, down to 82.3% for the hardest category. That’s not a small gap in a clinical context. It maps onto something you’d probably guess intuitively if you’ve used any LLM for a while: they’re excellent lookup-and-explain tools, and considerably shakier the moment a task requires actually weighing evidence the way a clinician does.

Finding 2: The “Metadata Paradox”

This was the most unexpected result, and honestly the reason I think this study is worth reading beyond just being “an AI accuracy benchmark.” We ran every question twice — once with the report alone, once with basic patient metadata attached (age, sex, race, insurance type) — expecting metadata to help, the same way more context usually helps a clinician.

It didn’t. Adding metadata significantly increased the odds of a clinically inadequate answer. In a systematic audit of 6,000 paired comparisons, we found 317 cases (5.3%) where a previously correct answer became worse purely because we added demographic context — with no change to the actual clinical findings in the report.

The clearest example from the paper: a report on an 87-year-old patient stated explicitly that there was “no relevant change” in a rib finding. One model still generated a false rib-fracture warning, explicitly reasoning about the patient’s age (“especially at age 87”) rather than the actual, unambiguous text in front of it — essentially letting a demographic stereotype about elderly fall risk override the report’s own words. We’re calling this pattern stochastic stereotyping: the model’s statistical prior about a demographic group overriding the specific evidence in front of it. It’s a genuinely important finding for anyone building patient-facing AI tools, because the intuitive assumption — more context should mean a better answer — turns out to be wrong in a way that’s easy to miss unless you specifically go looking for it.

Finding 3: The Calibration Deficit — This Is the One That Actually Worries Me

Every answer came with a self-reported confidence score. If that score meant anything useful, low-confidence answers should be wrong more often than high-confidence ones. In practice, the correlation between self-reported confidence and actual accuracy was close to zero across all three models (max Pearson r = 0.124).

Here’s the part that should genuinely concern anyone thinking about deploying this kind of tool: among each model’s most clinically dangerous errors — the ones a human clinician would call outright misinformation, not just an incomplete answer — the overwhelming majority were delivered with high self-confidence (≥8 out of 10). Overall, across all three models combined, 88.4% of the worst errors came wrapped in confident, authoritative-sounding language.

One nuance worth being precise about, since the raw percentages alone can mislead: GPT-4o mini is the model that should actually worry you most here, not the one with the highest percentage. Grok’s errors were delivered with high confidence 100% of the time — but that’s 7 out of 7 total Score-1 errors, a small enough count that one or two cases swing the percentage a lot. GPT-4o mini, by contrast, racked up 113 clinically significant errors — sixteen times more than Grok — and still delivered 93.8% of them with high confidence. That’s a much larger, much more statistically robust problem: a high-confidence-error rate almost as extreme as Grok’s, but at more than fifteen times the volume. There was no “I’m not totally sure about this, you should check with your doctor” hedge to warn the reader in either case. It read exactly as authoritative as the answers that were correct.

That’s the core safety problem in one sentence: these models don’t get quieter when they’re wrong — and the model that’s wrong most often shows no sign of getting quieter either.

So, Can AI Explain Radiology Reports Safely? Here’s What I’d Recommend

Based on all of this, the paper proposes what we call a “Dual-Safeguard” framework: never let a single LLM generate a patient-facing answer and ship it directly. Pair a generator model with a second, independent model acting purely as a verifier — auditing specifically for hallucinated findings and omitted details — and route anything flagged to a human expert before it reaches a patient.

One genuinely encouraging finding supports part of that framework directly: when we gave radiologists the LLM’s own structured reasoning to review — not just its final answer, but its stated rationale — it substantially improved how consistently the human experts agreed with each other on ambiguous cases (their agreement score jumped from “slight” to “almost perfect” on the disputed subset). Used the right way, the technology isn’t just a risk to manage — it can genuinely help standardize how experts adjudicate difficult, ambiguous cases too.

The Limits of This Study

A few things worth saying plainly, since I’d rather you hear the caveats from me than assume I glossed over them: this used simulated patient questions built from a taxonomy, not messages from real, anxious patients in the moment — real inquiries likely carry more emotional context and linguistic variability than our generated versions. The models were prompted with a structured “explain your reasoning first” strategy specifically designed to get their best possible performance, so treat these accuracy numbers as a ceiling, not a typical real-world outcome. And this was text-only — none of the models could see the actual medical images, only the written report, which is its own limitation worth addressing in follow-up work.

The honest answer, at least in its current form, is no — not without an independent AI verifier and, for anything requiring real clinical judgment, a radiologist checking the work. This study does not tell us whether AI can safely practice radiology. It doesn’t test image interpretation, real patient interactions, or clinical outcomes. It tests something narrower: how well LLMs explain and reason about the text contained in radiology reports.

The paper is fully open access — no paywall, published under CC BY — so if you want the complete methodology, statistics, and all four result figures, you can read it directly: Patient-Facing Radiology Communication with LLMs: Calibration Deficit and the Metadata Paradox, Healthcare 2026, 14(11), 1490.

---

This is the first entry in my Paper Reviews series, where I break down my own published research — and eventually other papers I find genuinely worth reading — in plain language. This particular line of research isn’t finished for me; figuring out how AI can be used well, and safely, in medicine is the direction I plan to keep building on. Next up: a closer look at a companion study asking a narrower, more practical question — how good are radiologists themselves at catching these errors when they’re reviewing AI output?

This kind of question sits right at the intersection of AI and digital healthcare I’ve been exploring throughout this blog — including my visit to Korea’s biggest hospital and health-tech fair.