cohere: stop degenerate repetition in the greedy decoder

[alexxxcoelho]

cohere: stop degenerate repetition in the greedy decoder

The bug

src/arch/cohere/model.cpp selects each token with a plain argmax over the
logits (three sites: the prompt pass at ~L1063 and both step loops). There is no
repetition penalty, no no_repeat_ngram, and no coverage term. The only exits
are eos_id and the token budget.

That leaves nothing to break a self-reinforcing state: when the most likely
continuation of a phrase is that same phrase, the decoder emits it until the
budget runs out. The result is a transcript that repeats one sentence dozens of
times, returned with TRANSCRIBE_OK and no diagnostic beyond the
output truncated at N tokens warning — which fires because of the loop, not
before it.

Evidence that this is the search, not the weights

Reproduced on 60 problem recordings from a French dictation corpus, decoded
twice with the same audio and the same parameters:

	Q8_0	Q4_K_M
files with an 8-gram repeated ≥5×	1	1
files where Q4 loops more than Q8	—	0
files where Q4 loops less than Q8	—	0

On the worst case the two quantisations loop token for token: 6 repetitions,
43 words, identical output. Quantisation is not the cause.

A related observation: every output truncated at 512 tokens warning in that
corpus landed on a looping file. The cap is a symptom, not the disease.

The fix

A tail-repetition guard in the decode loop, on both the static-graph (GPU) and
dynamic-graph (CPU) paths.

After each token is appended, looping_tail_block() reports the length of a
block that repeats at the end of the generated sequence. A block counts only if
it recurs at least kMinLoopRepeat (3) times and spans at least
kMinLoopTokens (12) tokens in total — so a 1-token block needs 12 repeats
while a 4-token block needs 3. That floor is what keeps an emphatic
"no no no" from being read as a loop.

When a loop is found, trim_looping_tail() drops every copy of the block but
the first, the decoder logs a warning, and decoding stops as if EOS had been
reached. Text preceding the loop is kept — it is usually correct; only the
runaway tail is discarded.

Only the tail is examined, so a deliberate repetition earlier in an utterance
survives untouched.

Results

On the same 60 problem files:

	before	after
files with an 8-gram repeated ≥5×	1	0
maximum repetition observed	6	2
files whose text changed at all	—	4 of 60

56 of 60 files are byte-identical, so the guard is not rewriting healthy output.

Independently verified on a second corpus of 7,724 speech units (28.9 h,
French meeting recordings): 36 loops stopped at the source, and a mechanical
scan of the outputs found no residual loop.

Notes

• No new dependencies; <algorithm> is added for std::max.
• The guard is unconditional. If you would rather gate it, the natural place is
  a field on transcribe_run_params, and I am happy to rework it that way.
• The same greedy-argmax pattern appears in the canary and canary_qwen
  decoders, which carry the same output truncated at %d tokens warning. I have
  not reproduced a loop there, so I have left them alone rather than change code
  I could not test.

[cjpais]

Thanks I'll take a look and likely pull in, I think the reference doesn't do this but this should be helpful in the general case so we should

[cjpais]

The other note that I have mostly for myself, or for you if you want to pick it up, is that likely other models will benefit from this, and it's possible that this abstraction should be more general, and that also might apply for Whisper as well. I'm just curious what you think about this. The main thing is like a bunch of autoregressive models can do degenerate repetition via greedy decoding. So, offering other kinds of decoders is potentially useful. This is a bigger task and is basically why I need to take some time to review this because fundamentally we need to not just apply things for a single model if they can apply to multiple model families

[cjpais]

#166 pulled