Frontier, open-weight or medical models?

Every few weeks, a new AI model is announced, and someone asks us whether it can read X-rays now. The honest answer is that nobody knows yet, including us. That is why Raghotham Sripadraj of Hypervelocity and I started testing it properly. This post explains what we are comparing and how we decided to measure it. The experiments themselves will follow in separate posts.

Three groups of models

AI models have evolved rapidly over the last two years. For our radiology experiments, we're comparing three groups:

Each group performs differently, but we don't yet know which is best suited for radiology.

A quick note on what separates them, for readers who do not follow this closely. Frontier closed models are the large general-purpose models you reach through a company's service; you send the image and get an answer back, but you cannot see or change the model itself. Open-weight models publish their weights, so a team can run them on its own servers and adapt them. Healthcare-focused models are built or tuned specifically on medical data. Each choice comes with different trade-offs in cost, control and how much medical knowledge is built in.

The question we are trying to answer

Most AI solutions in medical imaging today are built around pathology-specific models. But as general-purpose models improve, we still don't know how well they can interpret X-rays. Paired with the right agentic workflow, can they outperform specialised medical models?

Scaling has repeatedly expanded what general-purpose models can do. In some domains, they now match or outperform systems built for a specific task. We want to find out whether the same holds true for radiology.

A pathology-specific model is built to answer one question, such as whether there is a fracture on this wrist film. It can be very good at that one question. A general-purpose model can, in principle, look at the whole study and write the whole report. Whether it does that reliably enough to be useful to a radiologist is exactly what is unproven. (When I came back from RSNA 2025, the tools drawing the most interest were the ones that fitted into the radiologist's day, not the ones promising to replace the radiologist. That is the bar we hold these models to as well.)

This is exactly what Raghotham Sripadraj (Hypervelocity) and I (Naveen K, SOMATIQ) have been exploring, from both a research perspective and through real-world production deployments.

Raghotham has spent more than a decade building AI systems across multiple domains, and my last four years have focused on AI products for radiology. Together, we're evaluating how the three groups perform in practical radiology workflows.

To answer this, we are building a comprehensive set of evals with practising radiologists to compare these models across real X-ray cases. We don't just want benchmark scores. We want to know which cases each model handles well, which it gets wrong, and whether an agentic workflow improves its performance.

Why we started with the eval, not the leaderboard

We picked wrist X-rays to start. Can these models reliably catch fractures, dislocations and soft tissue issues, and write it up as a usable clinical report?

Here's the problem we kept running into: LLMs sound confident even when they're wrong. A model can write a clean, textbook-style report and still mix up a distal radius fracture for a scaphoid fracture. Asking "was it right?" isn't enough. In radiology, not all errors are equal - missing a dislocation is far worse than a minor phrasing issue. Good formatting can hide a dangerous miss.

So before ranking any models, we built an eval framework clinicians can actually rely on:

In practice, that means each model report is compared with the radiologist's final report through a fixed bank of small yes-or-no clinical questions. Did it find the fracture? Did it put it in the right bone? Did it invent one that was not there? Each answer carries a reason and a pointer to the evidence, so a radiologist can disagree with a specific verdict instead of arguing with a single number. Hard caps stop a polished report from scoring well when it misses something major. For agentic workflows, there is a separate trace audit that follows each final claim back through the steps that produced it.

What this means for a centre

If you are being sold an AI product built on a general-purpose model, the model name on the slide tells you very little. Ask how it was evaluated, against what reference, and how it handles a missed finding versus a false one. Ask what happens when the vendor upgrades the model, because a new model version can behave differently in ways a single accuracy number will not show. And ask whether a radiologist still reads the study, because until the evidence says otherwise, they should. Dr. Vishwanath made a related point in AI is with the radiologist: the useful question is how the tool sits beside the person signing the report.

Also ask where the result ends up. A model's finding is only useful if it reaches the viewer and the report in a form the radiologist can check, which is why we care about standard outputs such as DICOM SR for AI findings as much as we care about the model itself.

What comes next

The results will follow in separate posts: first a detailed write-up of how we score a report, then the experiments themselves. We'll share the eval design and what we learn, including what doesn't work, as we go.

So where does this leave a centre that is curious about general-purpose AI? For now, the honest position is that the question is open, and the way to close it is careful measurement on real cases with radiologists involved, not a leaderboard. That is the work we have started, and we will report it plainly, including the parts that do not go well.

FAQ

Can AI models read X-rays?

Some general-purpose and medical models can produce X-ray reports, but how reliable they are is still being tested. We are testing that on wrist X-rays now and will publish the results as they come in.

What is the difference between frontier, open-weight and healthcare-focused models?

Frontier closed models (such as GPT, Claude and Gemini) are general-purpose and used through the provider's service. Open-weight models (such as Kimi, Qwen, DeepSeek and Llama) publish their weights so teams can run and adapt them. Healthcare-focused models (such as MedGemma) are built for medical use.

How do you evaluate an AI radiology report?

We compare it with the radiologist's final report using binary clinical questions, an evidence trail for every verdict, heavier penalties for safety-critical misses and hard caps for major errors. Agentic systems also get a trace audit.

Who is running this benchmark?

Naveen K of SOMATIQ and Raghotham Sripadraj of Hypervelocity, with practising radiologists helping build the evals.