Beyond the algorithm

This is Part 1 of 2. It argues that the model is not the hardest part of medical AI. Part 2 looks at what a model could learn from the corrections radiologists make before they sign a report.

Most conversations about medical AI start with the model: which architecture, how many parameters, what score on which benchmark. This essay starts somewhere else, with what the model is asked to learn and what it never gets to see. I published it on LinkedIn in September 2026 and it appears here as written, with section headings and a reference list added for readers of this site, and a short note at the end on what it means for a centre evaluating AI.

More data cannot solve a clinical problem that was never properly defined.

The old equation

For much of the last decade, medical AI ran on a simple equation:

Data × Compute = Model = Clinical Solution.

Give the machine enough images, enough GPUs and enough labels, and clinical medicine would eventually emerge from the other side. That belief built the first generation of imaging AI, and it was an honest belief for its time.

The 2026 version, stated fairly

The 2026 version is more sophisticated, and it deserves to be stated fairly. It says: scale absorbs context. Feed a frontier multimodal model the entire record, the priors, the notes, the labs, and it will reconstruct the judgement the old narrow models missed. The radiologist's cognition is in the data somewhere. We simply have not given the model all of it yet.

This is the strongest argument against everything I am about to say. It also fails, for one precise reason.

A model can only absorb context that exists somewhere in computable form. Some of the most important context in radiology does not.

The label is not an answer key

Start with the ground truth itself. Most labels are mined from reports. Reports hedge, because hedging is often the correct clinical act. And radiologists disagree with one another at rates the literature has documented for the better part of a century; Garland was reporting roughly 30 per cent disagreement among experienced radiologists reading chest films for tuberculosis in the 1940s [1], and observer variation has never gone away.

So the label is not an answer key. It is one radiologist's judgement, sampled once, stripped of the uncertainty that produced it. We then measure the model against it to two decimal places.

Context changes meaning

Then there is the context that changes meaning. Tuberculosis in an endemic setting can mimic almost anything, including malignancy. The same shadow on the same film carries a different differential in Bengaluru than in Boston, and a different one again depending on what the last scan showed and what the referrer said on the phone.

When the intended clinical signal never fully enters the data, a model does not fail politely; it finds a substitute.

The model was not being stupid. It was solving exactly the problem the data specified, which was not the problem anyone intended.

We became remarkably precise at solving the wrong problem.

Judgement is the product

Radiological expertise is not just recognising abnormalities; it is deciding which findings matter, how they relate, what uncertainty is acceptable and how the conclusion should influence management.

That judgement is the product, and much of it lives outside the pixel.

Here I should be honest about both sides. Engineering often underestimated this depth, while radiologists were rarely embedded deeply enough in the process to correct for it. We stayed out for rational reasons: no protected time, tedious annotation work, genuine medicolegal caution, and development processes that gave us little say in design and brought us in only at the end, validating a model whose problem statement was already frozen.

Both professions behaved sensibly inside a badly designed process.

The real equation

So the real equation looks closer to this:

Data × Compute × Domain Understanding × Validation × Context × Trust × Safety × Workflow Integration × Feedback = Clinical Impact.

It is multiplication. Any essential factor approaching zero can collapse the entire product. A model that dazzles on a curated dataset and fails at the next site is not a clinical solution. A model clinicians do not trust will not be used. A model that adds friction is technically excellent and clinically irrelevant. A system that never learns from deployment drifts away from the world it was built for.

What scale can and cannot absorb

Which brings the argument back to the real limit of the foundation-model case. Scale can absorb extraordinary amounts of context, but only the context available to it.

Some clinical judgement will always resist digitisation. But much of what matters already exists and remains fragmented across prior imaging, reports, history, laboratory data, pathology, workflow events and radiologist corrections.

That is the more tractable problem: reducing how much clinically relevant information the model never gets to see.

Part 2 gets specific: what a model could learn from every correction and edit a radiologist makes before signing, and why the platform around the model may matter as much as the model itself.

Beyond the Algorithm, Part 1 of 2.

What this means for a centre evaluating AI

The equation above doubles as a checklist. When someone offers your centre an AI product, a few of its factors can be checked directly, and they are worth checking before accuracy figures:

1. Validation. Was the model tested on studies like yours, from scanners and patients like yours, or only on a curated dataset from elsewhere?

2. Context. What does the model see besides the pixels? If it cannot see the priors, the history or the previous report, it is working with less than your radiologist has.

3. Workflow integration. Does the result appear inside the worklist, viewer and report your radiologists already use, or in another window? I have argued before, in Radiology did not fail AI, that radiologists need better tools rather than more interfaces.

4. Feedback. When a radiologist corrects or overrules the model, does that correction go anywhere, or is it lost?

Each of these is a factor that can approach zero, and when one does, the accuracy figure on the brochure stops mattering.

References

1. Garland LH. On the scientific evaluation of diagnostic procedures. Radiology. 1949;52(3):309–328. doi:10.1148/52.3.309
2. Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLOS Medicine. 2018;15(11):e1002683. doi:10.1371/journal.pmed.1002683

Related reading

• Radiology did not fail AI — why interpretation is more than pattern recognition.
• AI is with the radiologist — why strong imaging platforms have to come first.
• How we evaluate AI for wrist fractures — binary checks, weighted clinical dimensions and hard caps for safety-critical misses.
• Frontier, open-weight or medical models? — the three groups of models we are comparing.

FAQ

Why isn't more data enough to build clinical AI?

Because a model can only absorb context that exists somewhere in computable form, and some of the most important context in radiology does not. Labels mined from reports carry hedging and inter-observer disagreement, and much of the judgement that shapes a diagnosis lives outside the pixel.

What is shortcut learning in medical imaging AI?

When the intended clinical signal never fully enters the data, a model finds a substitute. Zech and colleagues showed in 2018 that pneumonia models learned to identify which hospital an X-ray came from and used that as a proxy for disease prevalence.

Why is clinical impact a multiplication, not a sum?

Because any essential factor approaching zero collapses the product. Data, compute, domain understanding, validation, context, trust, safety, workflow integration and feedback all have to be present. A model that clinicians do not trust, or that adds friction, will not deliver clinical impact however accurate it is.

What can be done about missing clinical context?

Much of what matters already exists but is fragmented across prior imaging, reports, history, laboratory data, pathology, workflow events and radiologist corrections. Reducing how much of that the model never gets to see is the more tractable problem.