Scoring an interpreter: why word-matching fails, and what we built instead

Engineering

Two excellent interpretations of the same sentence can share almost no vocabulary. So what, exactly, is the machine supposed to compare?

We built a platform that scores people practising for the NAATI CCL — an Australian interpreting test that migrants sit because it is worth points toward a visa. Real stakes: people fail it, pay again, and wait.

The obvious version of this product is easy and useless. The useful version required us to be honest about what a language model can and cannot judge. That distinction is the interesting part, and it generalises well beyond interpreting.

Why you cannot just compare the words

The naive build: store a reference translation, compare the candidate’s answer, score the similarity. Ship it.

This fails immediately, because interpreting is not translation. A skilled interpreter listens to a chunk of speech, discards the surface form entirely, and reconstructs the meaning in the target language. Two excellent interpretations of the same sentence can share almost no vocabulary.

So a similarity score does not merely measure the wrong thing. It actively teaches the wrong skill. You would have built a machine that trains people to fail.

The rubric is the product

The actual examiner is not comparing to a reference. They are working down a deduction rubric — marks come off for specific, named categories of failure:

• Omission — meaning present in the source, absent in the interpretation
• Distortion — meaning present but altered
• Addition — meaning that was never in the source at all
• and each weighted by severity: dropping a politeness marker is not dropping a dosage

Which reframes the entire engineering problem. We were never building a similarity engine. We were building a system that has to answer: what units of meaning were in the source, which of them survived, and how badly does each casualty matter?

That is a fundamentally different piece of software, and you only discover it by reading the marking criteria properly rather than assuming you already understand the domain.

Where the model helps, and where it must not be trusted

Language models are genuinely good at semantic equivalence — at telling you that two differently-worded sentences carry the same meaning. That is precisely the judgment the naive approach needed and could not make. So the model does the part it is good at.

But the model is not the authority on severity. Severity is a domain question with real consequences — in the health module, a mangled dosage is not the same class of error as a clumsy greeting, and no general-purpose model reliably knows the difference in the way NAATI’s examiners do.

Collapse them — just ask the model “score this out of 45” — and you get a number that is fluent, confident, and unaccountable. Candidates would trust it, because it looks exactly like a real mark. That is worse than not scoring at all.

The thing worth taking away

The lesson generalises, and it is the same lesson in almost every serious AI build we do:

• The model is superb at the fuzzy perceptual judgment — did this mean the same thing, is this the same entity, does this document say what the other one says.
• The model is unreliable at the consequential judgment — how much does this matter, what should happen now, who is accountable.
• Good systems put a human-defined rule between the two. Bad systems let the model do both and call it intelligence.

Most of the AI products that will quietly fail over the next few years will fail exactly here. Not because the model was bad, but because someone let it grade its own significance.

The engineering is not the prompt. The engineering is knowing where to put the wall.