Did newer Gemini models get better?

Same wrist X-rays, different model generations: does the newer model make fewer mistakes?

This is the third post in our wrist X-ray series; it builds on How we evaluate AI for wrist fractures and Same model, different prompt: Gemini 3.5 Flash on wrist X-rays. The reasons for comparing general-purpose models at all are in Frontier, open-weight or medical models: which can read an X-ray?. The agent experiment will follow in the next post.

The same wrist X-rays, three Gemini generations and two ways of asking the question.

In our previous post, we tested whether giving Gemini a structured wrist-reading protocol changed its interpretation of the same X-rays.

It did. The protocol found two additional fractures documented in the reference reports. It also called three additional fractures in studies where the reference did not describe one.

That left us with something more useful than a slightly higher score. We could see how the prompt changed the mistakes a radiologist would have to deal with.

So we took the experiment further.

Would a newer model make fewer of those mistakes? And would the same reading protocol still help?

We compared three Gemini generations using a simple prompt and a structured reading protocol.

The newer model gave us a promising result with the simplest prompt. With the longer protocol, the pattern went in the other direction.

First, we needed to compare the same examinations

There is a detail worth explaining before looking at the numbers.

The previous post covered 71 adult studies. This post uses the 70 complete cases shared across our broader three-generation benchmark, of which 62 were adult examinations. We retained that matched group for the one-shot comparisons here. The remaining eight were paediatric cases, outside our adult diagnostic evaluation.

We also compared Gemini 3.6 and 3.7 on a larger group of 100 shared studies, containing 89 adult examinations.

The broad rubric scores below cover the full comparison groups, including assessment of how out-of-scope cases were handled. Fracture accuracy, missed fractures and false calls refer only to the adult studies. These are different denominators, and the results should not be read as a direct continuation of the 71-study table in our previous post.

We used the same BinEval-style rubric described earlier. The reference remained the final expert radiology report.

What did we change?

For each generation, this post compares two approaches:

A newer model helped, but the prompt mattered

On the 62 shared adult studies, the one-shot results looked like this:

One-shot results on the 62 shared adult studies: missed fractures, false fracture calls and adult fracture accuracy for each model and prompt.

There were 35 report-positive and 27 report-negative adult studies in this group. A "false fracture call" here means a positive call on a report-negative examination.

With Gemini 3.6, the structured protocol produced the better balance. Compared with the earlier structured run, it missed one fewer fracture and made five fewer false calls.

With Gemini 3.7, the simple prompt performed better. It missed one fracture, while the structured prompt missed four.

This is something I would want an AI team to check whenever it changes the underlying model. The longer prompt helped one generation and hurt the next. A reading protocol that worked well in one experiment did not automatically remain the best way to ask a newer model to examine the images.

The average score barely moved

The larger comparison makes this particularly clear.

With the naive prompt, Gemini 3.6 scored 56.31 and Gemini 3.7 scored 56.32 out of 100. Looking only at those numbers, there was almost nothing to discuss.

But among the 89 adult studies, the errors changed:

Naive prompt: Gemini 3.6 against Gemini 3.7.

Gemini 3.7 corrected three adult fracture labels that 3.6 had wrong. It also got one wrong that 3.6 had correct. The net result was two fewer errors, with about 11% fewer tokens.

That is a modest observed improvement. With this sample, I would be careful about treating it as a reliable advantage on other examinations.

The structured prompt moved in the other direction:

Structured protocol: Gemini 3.6 against Gemini 3.7. In both tables, fracture results cover 89 adult studies. Token counts are the recorded input plus output tokens, averaged across all 100 studies and rounded to whole tokens.

With the structured protocol, Gemini 3.7 also used fewer tokens than 3.6, by about 11%. But missed fractures rose from six to nine and false calls rose from eight to nine.

Within each generation, the systematic protocol used about 6% more tokens than the naive prompt. That additional token use accompanied better fracture accuracy in 3.6 and worse accuracy in 3.7.

The model version alone did not tell us which result we would get. We had to test the model and prompt together.

Why the clinical score does not tell the whole story

Our rubric checks more than fracture presence. It also considers location, characterisation, alignment and consistency. Those parts of a report can change in different directions.

For example, on the smaller shared group, the 3.6 structured prompt made six fewer adult fracture-label errors than the 3.5-era structured prompt. Yet its broad score fell from 60.6 to 57.1.

A lower score sat alongside better fracture detection performance.

The score and error counts measure different things. Broad scores cover 70 studies in the first comparison and 100 in the other two; fracture errors cover the 62 and 89 adult studies respectively. "Elaborate" in the charts refers to our structured protocol.

That does not make the remaining report details unimportant. It means we need to inspect what contributed to the score before deciding whether an output is more useful.

For the clinical side of this collaboration, the question is which errors a reader would need to correct. For the engineering side, it is which part of the system changed those errors. We need both views to decide what to test next.

Which configuration would I take forward?

For the next supervised evaluation, I would start with Gemini 3.7 using the naive prompt.

Gemini 3.6 with the structured protocol is a close alternative. Both reached 84.3% adult fracture accuracy in the larger comparison. The 3.7 naive run missed five fractures and made nine false calls. The 3.6 structured run missed six and made eight.

Two candidates for further supervised evaluation. Error counts cover 89 adult studies; average token use covers all 100 studies.

My preference here gives more weight to the one fewer observed miss, accepting one additional false call. It is a choice about which trade-off to investigate further. We have not shown that it produces better care or makes a radiologist more effective.

Our review rules still flagged 98 of the 100 Gemini 3.7 naive outputs for expert review. We also have not measured reading time, correction burden or whether showing these suggestions helps readers make better decisions.

Those are questions for the next evaluation. Supervision alone does not answer them.

What should we take from this?

The structured protocol helped Gemini 3.6 on fracture accuracy. With Gemini 3.7, the simple prompt gave us the better result. That is why I would repeat the clinical evaluation whenever the model changes, even if the prompt stays exactly the same.

Next: can a team of agents do better?

These one-shot results give us a reference for the agentic experiment we mentioned earlier in the series. In the next post, we look at what happened when focused agents examined different parts of the wrist and combined their findings into a final report.

The practical upshot for anyone running AI in a radiology department is that a model upgrade is a change to a clinical tool, not a routine software update. The prompt, the model and the way results reach the radiologist have to be validated together, and the radiologist still signs the report. It is one reason we keep AI output reviewable inside SOMATIQ's reporting workflow and keep pushing for results in structured formats a report can read.

Related reading

• How we evaluate AI for wrist fractures — the rubric and hard caps behind these numbers.
• Same model, different prompt: Gemini 3.5 Flash on wrist X-rays — the 71-study experiment this post extends.
• Frontier, open-weight or medical models: which can read an X-ray? — why we are benchmarking general-purpose models.

FAQ

Did newer Gemini models read wrist X-rays better?

Partly, and it depended on the prompt. With the naive prompt, Gemini 3.6 scored 56.31 and Gemini 3.7 scored 56.32 out of 100, but 3.7 made two fewer adult fracture-label errors with about 11% fewer tokens. With the structured protocol, 3.7 did worse than 3.6: missed fractures rose from six to nine and false calls from eight to nine.

Which configuration would we take forward?

Gemini 3.7 with the naive prompt, with Gemini 3.6 and the structured protocol as a close alternative. Both reached 84.3% adult fracture accuracy on the 89 adult studies; the 3.7 naive run missed five fractures and made nine false calls; the 3.6 structured run missed six and made eight. Our review rules still flagged 98 of the 100 Gemini 3.7 naive outputs for expert review.

Why retest when only the model changes?

Because the structured protocol helped Gemini 3.6 on fracture accuracy, while with Gemini 3.7 the simple prompt gave the better result. The model version alone did not tell us which result we would get, so the model and prompt have to be tested together.