Freezing the evidence

A language model scores a picture-naming test from a recording. It returns 11 out of 15. The clinician's reference says 13. Where did the two points go?

There are at least two answers and they call for different work. The model may have heard the participant correctly and judged the response wrongly — counted a semantic paraphasia as correct, or missed a delayed self-correction that a clinician would have accepted. Or it may have judged impeccably, on a transcript that never contained the word the participant actually said.

Those two failures live in different systems. A benchmark that cannot tell them apart cannot report anything useful about either.

Why the obvious design fails

The obvious pipeline runs end to end: take the audio, transcribe it, hand the transcript to a scoring model, compare the result against the clinical reference. Repeat for each model you want to compare, and for each speech recogniser.

The design is factorial — ASR system crossed with scoring model crossed with input condition — and it only works if the transcript is a fixed factor. If each run transcribes afresh, you are no longer comparing scoring models on the same evidence. You are comparing (transcript, scorer) pairs, and any difference between them is unattributable to either half.

Nor is re-transcription free of drift in practice. Hosted endpoints change underneath you. Local decoding is not always bit-identical between runs. Provider retries return different output than the attempt they replaced. Over the weeks an experiment actually takes, "the same audio through the same system" is not a stable object unless you deliberately make it one.

Two stages, one artifact

So the pipeline splits. Stage 1 acquires transcripts and stops. For each combination of task, participant and ASR system it writes exactly one artifact, and freezes it. Stage 2 reads that artifact — the same bytes — for every scoring model, every prompt condition, and every rerun thereafter.

The transcript is now a fixed factor. Seven scoring models crossed with two recognisers gives fourteen cells, and every cell in a column shares its evidence exactly. Differences down a column are scorer effects. Differences across a row are ASR effects. Because the participants are the same in every cell, the comparisons can be paired.

What "frozen" has to mean

An artifact that has been tidied is not frozen; it is a different transcript. Holding the line turns out to require several rules that all feel, individually, like leaving the work half-done.

• Provider-native output is preserved as returned. Whisper's artifact keeps its transcript and timestamped segments with speaker: null. No diarizer is bolted on afterwards to make it resemble Deepgram's. Deepgram's keeps its native utterances, timings and diarization clusters, including a null speaker when the provider returns one. The two artifacts are not normalised into a common shape, because the difference in shape is part of what is being measured.
• Empty segments are retained. A whitespace-only span is evidence. Silence where a response should be is a finding, not a defect to be swept out.
• Diarization clusters stay hypotheses. Nothing maps cluster 0 to "examiner" and cluster 1 to "participant". That mapping is precisely the inference under test; performing it in advance would hand the model the answer to half the problem.
• Nothing is corrected from the answer key. Not a word, not an item boundary, not a speaker label. The moment a transcript is repaired using the target, the benchmark is measuring the repair.

Where evidence is genuinely missing or ambiguous — an unclear item boundary, an absent timestamp — the required output is clinical uncertainty, not reconstruction. A system that guesses confidently through a gap is worse than one that reports the gap, and the benchmark should be able to say so.

What the surrounding code may do

The same discipline applies on the other side of the model. Local code is restricted to work that ought to be reproducible: validating that an output matches a declared contract, binding item numbers to fixed protocol metadata, translating declared decisions into protocol scores, aggregation, hashing the evidence, computing comparison metrics.

It does not repair, reinterpret or replace a clinical answer. If a model returns something invalid, that is a result.

This is harder to hold to than it sounds. Every malformed response is an invitation to write one small rescue heuristic, and each one is individually defensible — the intent was obvious, the parse was nearly right. Collectively they amount to the harness scoring the assessment and the model receiving the credit.

Every rescue heuristic moves a little of the decision from the system under test into the system doing the testing.

Failures stay visible

Invalid, missing and unresolved outputs remain in the coverage and validity metrics rather than being dropped from the denominator. Quietly discarding them is the most reliable way to publish a number better than the system that produced it.

Agreement is therefore always reported next to coverage. A model that answers 60% of items with high agreement and abandons the rest has not outperformed one that answers all of them slightly less well. It has done a different thing, and the pair of numbers is what says so.

The reference is read last

Ground-truth workbooks open during evaluation and never during inference. Participant identifiers, file paths, hashes, model settings and reference scores live in experiment metadata, structurally separate from anything a prompt can reach. What the model receives is genuine scoring evidence and nothing else.

Keeping those two stores apart is mundane engineering, and it is the only reason any resulting number means what it appears to mean.

What this measures, and what it does not

One consequence is worth stating plainly. This design compares the downstream utility of a complete speech recognition system — word recognition, segmentation, timing and whatever diarization it offers, all at once — rather than isolating a word-error-rate effect.

That is deliberate. A scoring model never consumes a WER figure. It consumes an artifact carrying all of that system's properties simultaneously, and it succeeds or fails on the whole of it. Reporting WER alongside would describe a component that nobody uses in isolation.

The cost is a claim you give up. You cannot say that a lower word error rate caused an improvement in scoring agreement. You can say that one complete system produced evidence that scored closer to the clinician's judgement — which is the claim a deployment decision would rest on in any case.

A note on ambition

None of this is sophisticated. It is bookkeeping: one artifact per participant per system, written once, never edited, read many times. It took far longer to get right than the prompting did.

But the entire comparison rests on it. A benchmark whose evidence shifts between conditions is not measuring the thing its column headings claim to measure, and no amount of careful statistics downstream will recover what was lost upstream.