When AI agents read wrist X-rays together

Several AI agents, one wrist X-ray: does dividing the reading make the answer better?

This is the fourth post in our wrist X-ray series, following How we evaluate AI for wrist fractures, Same model, different prompt: Gemini 3.5 Flash on wrist X-rays and Did newer Gemini models get better at reading wrist X-rays?. Those single-model results are the baseline for the agent workflow tested here.

Some of the AI reports described a fracture. But the agent's predicted labels said there were none. Let's find where the answers split.

Let's say we have an AI that takes a wrist X-ray image as input. This AI system reading the wrist X-ray can deliver two kinds of answers. One is a written report for a radiologist or clinician. The other is a structured label that downstream software systems can use, e.g. fracture present or fracture absent.

If the AI is doing both, those two answers should agree.

But in some of our tests, an AI workflow wrote about a fracture in the report while marking the same examination as having no fracture in the structured label. We found it baffling! The doctor reading the report and a system reading the label would have received different answers.

While one might think that adding more AI agents can make the reporting process safer, the disagreement is what we found in our experiments.

A written report will describe the location of a fracture and could have uncertainty. A fracture label reduces that report to a yes or no, one that decision software can use.

We started with a single model and that gave us a useful baseline

In our previous post, we asked three generations of Gemini to read the same wrist studies. We tried a simple prompt and a more detailed reading protocol. The protocol helped one generation of the model; the simple prompt did better with the next. The best approach changed with each generation of the model.

On the shared adult studies, the simple approach scored 85.5%, 85.5%, and 90.3% across the generations. The detailed approach scored 80.6%, 90.3%, and 85.5%.

The prompt that got the prediction right changed with every release of the model. This also means that the one-model results are just a baseline. We cannot hold that "prompt" to be the ideal one.

We wanted to test this: could several focused readers do better than one model reading the whole examination?

On the surface, this idea made sense.

We gave each AI agent a narrower part of the examination

When reporting, a radiologist checks the radius, ulna, wrist bones, and their alignment. We gave AI agents narrower jobs along similar lines, then asked the workflow to check and combine their findings. When it has more attention on each part, it should, in theory, catch more fractures.

An "agent" here is a model assigned a particular job within a larger workflow. Focused agents examine the end of the radius near the wrist, other wrist bones, and how the bones line up. Their observations have to be combined before the workflow can answer.

The findings then went into a shared report that the agents collaborated on. A verifier reviewed them, after which the workflow produced a structured summary and a written report. While dividing the work could give each area more attention, it also created more handoffs: a finding had to move from a focused reader through verification and into both final outputs.

We evaluated the complete workflow. We needed to evaluate all the handoffs because the final result alone cannot identify which step caused an error.

Prediction evaluation

The three approaches were:

For the comparison across model generations, each approach used the same 70 wrist studies. Sixty-two were adult examinations, and the chart below reports results for those adults. The existing radiology reports served as the reference: 35 described a fracture and 27 did not.

Key terms

• Fracture accuracy measures whether each final structured label agreed with the reference.
• Sensitivity measures how many of the 35 studies with a fracture in the reference received a positive label.

The larger Gemini 3.6 and 3.7 runs used 100 studies each. They show whether the result appeared beyond the shared comparison, though individual cases still need checking.

More readers produced a worse final answer

The single-model approaches correctly classified about 81% to 90% of the adult studies, depending on the model and prompt. The agent workflow scored 43.5% in every generation. The workflow marked every adult study as having no fracture.

Every one of them!

It correctly classified the 27 studies whose reference reports showed no fracture and missed the 35 whose reports did. That produced 43.5% accuracy and 0% fracture sensitivity in this comparison.

Each bar uses the same 62 adult studies. The earliest agent run mixed model versions, and changes between run dates may also have affected the comparison. Differences cannot be attributed solely to model generation.

We saw the same pattern, with every label negative, in the larger Gemini 3.6 and 3.7 runs. Each included 89 adult studies, with 49 whose reference reports described fractures. The structured labels missed all 49 in each run.

The baffling and shocking aspect was that some written reports still mentioned a fracture.

The workflow disagreed with itself

Focused agents feed a verifier, but the final label and written report can disagree.

The negative labels do not prove that every agent failed to see every fracture. A report mentioning a fracture establishes only that the written answer contained a fracture prediction. It does not tell us how and where it got the prediction from.

For a clinician, the report and the label must describe the same examination consistently before the workflow can be trusted in a clinical setting.

The inconsistency also changes how we evaluate the AI agents. A score based on the written report may give credit for a useful description. A score based on the structured label will treat the same examination as fracture-negative.

Increasing to nine model requests produced a worse final label

The outcome evaluation and the trace audit answered different questions. We compared final labels and reports with the report-based reference. We inspected execution traces to see what each agent recorded and what the verifier accepted or rejected. The evidence ledger was the shared document the agents collaborated on.

We scored the final result with the same BinEval-style rubric as the single-model runs. The rubric assesses report content and other dimensions; the fracture accuracy above uses the final structured label.

• A report can receive credit for useful text while its fracture label is wrong.
• The traces show which intermediate findings, if any, support each output.

On the full Gemini 3.7 run, the agent workflow used roughly 45,000 tokens and nine model requests per study. The simple prompt used about 5,058 tokens and one request. Tokens measure model input and output. The agent workflow used nearly nine times as many tokens and requests, but even then the final label failed on every adult fracture described in the reference reports.

The traces were used to distinguish among several unproven possibilities:

• A prompt made an agent too cautious when reading the image.
• The verifier discarded a valid finding.
• Label construction turned a positive finding into a negative field.
• The report writer introduced a fracture claim after the label was set.

Learning which one is important because each of them requires a different fix.

Next steps: we need to find the step where the answers split

We should start with cases where the AI report mentions a fracture but the label says no. For each case, we can compare the reference report and follow the agents' observations through verification, structured summary, label, and final report. The trace should show whether a fracture claim was recorded early, rejected later, lost during label construction, or introduced only in the prose. We should also trace report-positive cases where the AI report and label are both negative.

Then we can change the suspect step while holding the model and other settings fixed. If a prompt change improves the observation, prompting was part of the problem. If changing label construction resolves the disagreement while improving agreement with the reference, that would support a fault in label construction. We need the case-level evidence before choosing either fix.

While agreement between the report and label is necessary, both outputs could still be wrong. We need to check the repaired answer against the reference. Then we can test what delegation and verification actually add, using the single-model results as a baseline. Radiologists should also review a sample of the rubric's judgements; a repeatable score does not mean every judgement is clinically correct.

Our initial agent workflow failed this test. We still need to learn whether delegation or verification improves the reading once the output failure is repaired.

Adding agents creates more steps between inspecting an image and delivering an answer. Each step needs to leave a record we can check, and the final report and label need to agree on what the system found.

We have done these. In our next post, we will share the results.

For a centre thinking about AI, the lesson travels beyond wrist X-rays. Whenever an AI system produces both a report and a machine-readable result, someone has to check that the two agree before either reaches a clinician or a worklist. That is a large part of why we argue for AI results in standard, checkable forms such as DICOM SR, and why Dr. Vishwanath wrote in Beyond the algorithm that medical AI needs a clear specification of what it must deliver, not only a model. It is also how we approach AI inside SOMATIQ: the radiologist sees the evidence and signs the report.

Related reading

• How we evaluate AI for wrist fractures — the rubric and the separate trace audit used here.
• Did newer Gemini models get better at reading wrist X-rays? — the single-model baseline this workflow was compared with.
• Same model, different prompt: Gemini 3.5 Flash on wrist X-rays — where the agentic idea was first proposed.
• Beyond the algorithm — why medical AI needs a specification, not just a model.

FAQ

How did the agent workflow do compared with a single model?

Worse. The single-model approaches correctly classified about 81% to 90% of the 62 adult studies, depending on the model and prompt. The agent workflow scored 43.5% in every generation because it marked every adult study as having no fracture, giving 0% fracture sensitivity. The larger Gemini 3.6 and 3.7 runs showed the same pattern, missing all 49 report-positive adult studies in each run.

How can the report mention a fracture while the label says none?

The workflow produced two outputs, a written report and a structured label, and some written reports still mentioned a fracture while the label was negative. We are using the execution traces to work out whether an agent was too cautious, the verifier discarded a valid finding, label construction turned a positive finding into a negative field, or the report writer introduced a fracture claim after the label was set.

Did the extra agents cost more?

Yes. On the full Gemini 3.7 run, the agent workflow used roughly 45,000 tokens and nine model requests per study, against about 5,058 tokens and one request for the simple prompt, and the final label still failed on every adult fracture described in the reference reports.