How we evaluate AI for wrist fractures

This is the first post in our series on testing AI on adult wrist X-rays; it sets out the measuring scale the later posts use. Why we are running the series at all, and which groups of models we are comparing, is explained in Frontier, open-weight or medical models: which can read an X-ray?. The first experiment is Same model, different prompt: Gemini 3.5 Flash on wrist X-rays; later posts will look at newer Gemini models and at AI agents reading wrist X-rays together.

A practical framework for testing whether a model's radiology report is clinically useful.

By the co-founders of Hypervelocity and SOMATIQ

Executive summary

Evaluating wrist-fracture AI takes more than checking whether it says "fracture." Our BinEval-style [1] rubric asks small, auditable clinical questions and weighs errors by their effect on care. Hard caps prevent major misses from being hidden by polished reporting, while a separate trace audit examines how agentic systems reached their conclusions. The rubric distinguishes a missed fracture from an unsupported overcall. Wrong localisation is treated as a major clinical error because it can misdirect downstream care. Every verdict carries a reason label and evidence reference so a radiologist can challenge the score.

From the original post: each part of the wrist maps to a small check, and the checks add up to a reviewable score.

What are we evaluating?

We are building an evaluation for AI systems that interpret adult wrist radiographs. The first target is fracture detection, but a useful report has to do more than return a positive or negative label. It should identify the correct bone and location. When the images support it, the report should describe displacement, comminution, articular extension, angulation, or other features that may affect management. It should inspect carpal alignment and the distal radioulnar joint. It should also state meaningful limitations when views are missing, a cast obscures anatomy, image quality is poor, or an occult injury remains possible.

These expectations follow the way wrist radiographs are read clinically. Check the study and views, move through the anatomy methodically and then assess alignment. Finally, communicate the important findings and limitations in a usable impression. The American College of Radiology guidance mentions that a report should use appropriate terminology, address the clinical question, identify limitations that affect diagnostic confidence and recommend follow-up when appropriate.

Why wrist fractures are hard to score

The easiest way to grade a model report is to ask another model: "How good is this?", and that produces a number. But this number is hard to trust.

There are various ways a report can sound very good but be clinically wrong. A fluent report can miss a distal radius fracture, or a report can detect a fracture but put it in the scaphoid instead of the radius. It can correctly describe the bone and get the articular extension wrong, or it can invent a precise fracture on a normal study. Each report may sound credible, yet each failure creates a different downstream problem.

This matters a lot in this domain because errors do not have equal clinical consequences. Wrong localisation can misdirect immobilisation, surgical planning, specialist referral, or follow-up. Missing a perilunate injury is more serious than omitting a minor soft-tissue observation. A false-positive fracture can lead to avoidable immobilisation, additional imaging, referral, and loss of trust in the system.

An average score can hide all of this. A model should not recover from a missed fracture by earning points for polished prose and complete formatting.

We therefore adapted the central idea from BinEval: ask small questions instead of making one broad judgement.

The evaluation should reveal exactly what failed and preserve enough evidence for an expert to challenge the result.

How we built a more useful evaluation

Our rubric: binary questions, weighted clinical dimensions, hard caps, and a separate trace audit.

For every case, we compare each model report with the reference report. The comparison begins with a fixed bank of binary questions. Examples include:

1. Did the report identify a reference-positive fracture?

2. On a reference-negative study, did it avoid calling a definite fracture?

3. Is the primary finding assigned to the correct bone and clinically useful location?

4. Are management-relevant features described without invention?

5. Is dislocation or major instability handled correctly?

6. Does the recommendation match the finding and level of uncertainty?

7. Do the narrative and structured labels agree?

Each answer is recorded as yes, no, unknown, or not_applicable. It also receives a controlled reason label and a short explanation, with references to the exact evidence used. A failed localisation question might carry wrong_bone_or_joint. A negative study with an unsupported fracture call might carry false_positive_fracture_overcall. This makes the score reviewable. A radiologist can disagree with a particular question or threshold without having to reverse-engineer the entire judgement.

The rubric is purposely designed to be non-linear. A report that misses the central injury should not finish with an apparently safe score because its less important sections were correct.

The questions roll up into ten clinical dimensions.

The remaining weight covers the distal radius, ulna and DRUJ, the carpal bones, technical framing and so on.

The weights express clinical priority. They are not estimates of patient harm. They force the benchmark to care more about errors that can change what happens next.

Weights alone are not enough, so we apply hard caps after the weighted score is calculated. Missing the primary fracture can cap a report at 55. Missing a dislocation or major instability pattern can cap it at 50. A wrong primary location can cap it at 60. Unsafe reassurance, management-changing characterisation errors, unsupported urgency, or a major conflict between the narrative and structured output can also limit the final score.

Looking closer at errors and evidence

Overcalls are handled explicitly. A definite fracture call on a negative reference study is penalised in fracture detection and localisation, with a hard cap when the claim could change management. A minor unsupported comment about swelling receives a smaller penalty. If a soft-tissue sign is used to justify a false fracture or instability diagnosis, the penalty follows the consequential claim. Specificity is part of safety, not an optional companion metric.

Agentic systems receive a second and separate trace audit. We try to follow each important final claim back through the final summary, verifier decision and evidence ledger. We ask whether the system examined the expected anatomy and whether the verifier resolved disagreements. We also check whether the final report matches the structured labels.

A strong trace does not rescue a wrong report. A correct report does not make a contradictory trace acceptable. Clinical performance and process quality answer different questions, so we keep their scores separate.

Where we go from here

With the BinEval style, the current judge is deterministic. It uses the reference report, normalised labels, model output and available harness artefacts to answer a frozen question bank. The result is reproducible and leaves a clear audit trail.

We also need to measure agreement between expert reviewers and the judge. A rubric becomes useful through repeated review and retesting. The aim is not to create an impressive score, but to show where a system is dependable, where it creates work, where it is ambiguous and where it is unsafe.

Only after fixing that measuring scale does it make sense to ask how a particular model performs. In the next post, we apply this framework to Gemini 3.5 Flash on adult wrist radiographs and examine what changed when the same model received a systematic wrist-reading protocol.

For a radiologist or a centre owner reading this, the practical point is simple: before trusting any number attached to an AI report, ask what was counted, what was capped and who can challenge the verdict. That is the same standard Dr. Vishwanath set out in AI is with the radiologist, and it is the standard we hold our own work to.

Reference

1. Cho, S., Chawla, K., Cai, P., Liu, Z., Zhu, C., Zhang, S.-X., and Sahu, S. (2026). Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement.

Related reading

• Frontier, open-weight or medical models: which can read an X-ray? — why we started this benchmark and what we are comparing.
• Gemini 3.5 Flash on wrist X-rays — this rubric applied to one model with two prompts.

FAQ

Why not just ask another AI model to grade the report?

That produces a single number that is hard to trust. A fluent report can miss a distal radius fracture, put the fracture in the wrong bone or invent one on a normal study, and an average score can hide all of this. We ask small binary questions instead, each with a reason label and evidence reference a radiologist can challenge.

How are the clinical dimensions weighted?

The questions roll up into ten clinical dimensions. Fracture detection and localisation each receive 20 per cent, alignment, dislocation and instability receive 15 per cent, and fracture characterisation receives 12 per cent. Hard caps then apply: missing the primary fracture can cap a report at 55, missing a dislocation or major instability pattern at 50, and a wrong primary location at 60.

What is the reference standard?

The final report issued by a human expert radiologist, from which we derive structured labels for fracture, dislocation, alignment abnormality and exclusions such as paediatric cases. It is a practical report-based benchmark, not independent adjudication using follow-up imaging, operative findings, clinical outcomes and reviewer consensus.