Whisper large-v3 kept repeating itself in Danish. Switching off 11 attention heads fixed most of it.

On long Danish recordings, Whisper large-v3 sometimes gets stuck. It writes a sentence, then writes it again, and again. In one five-minute test recording it wrote the same sentence 43 times, and 16 other sentences never made it into the transcript.

I wanted to know if we could fix that inside the model itself. Not by training it, but by finding the part that does the repeating and switching it off.

It worked. The short version:

• Can a weight edit remove the repeats? Yes, mostly. On 30 Danish recordings the edit never saw, the words written by repeats dropped from 1,380 to 85.
• Is it luck? No. Two random edits of the same size did nothing like it.
• Did it pass the strict pass mark we set in advance? Just missed: 76% fewer repeats, where we asked for 80%.
• Is Whisper better at Danish now? On long recordings, yes, because the repeats are gone. On short ones, accuracy stayed the same.

Whisper repeats itself because it copies its own earlier text

Whisper hears audio in 30-second windows. To keep a long transcript consistent, it gets the last couple of hundred words it wrote as a hint for the next window.

That hint is where it goes wrong. Here is one of the 30 recordings, where the speaker says the sentence once:

Original large-v3: "…Der er ikke rapporteret nogen jordskældsaktivitet i Hekla-området inden for de sidste 48 timer, oplyser ICI. Det er også en af de fleste telefonnummer… Der er ikke rapporteret nogen jordskældsaktivitet i Hekla-området inden for de sidste 48 timer, oplyser ICI. Der er ikke rapporteret nogen jordskældsaktivitet i Hekla-området…"

With the edit: "…Der er ikke rapporteret nogen jordskældsaktivitet i Hekla-området inden for de sidste 48 timer, oplyser Islands Meteorologiske Institut. Problemet er sne og klister…"

To check the cause, we rebuilt the hint at the exact moment a repeat started. Taking the earlier copies of the sentence out of the hint stopped every repeat we tested. So the model is not mishearing. It is copying.

This is also why you have not seen it in Codictate. Codictate cuts your recording into pieces of 30 seconds or less and transcribes each one on its own, so there is no earlier text to copy. On that path the original large-v3 had no repeats in any of our tests.

I let Claude edit the weights directly, with no training

The idea came from a paper published a week earlier, Sparks of Weight-Space Programming from MIT. There, an AI agent read a small language model's internals and wrote changes straight into its weights to stop it from going in circles. No training data, no fine-tuning.

So I gave Claude (Anthropic's Opus 5.5) the same role on Whisper, with a few rules written down before any work started:

• No training. Every change is a formula applied to the original weights, which anyone can rerun.
• Two sets of recordings. Claude searched for the change on 10 long Danish recordings. 30 others stayed sealed until the final test.
• Random edits for comparison. Any change gets compared with random changes of the same size, to rule out luck.
• A pass mark, set in advance. Repeats down at least 80%, no more speech lost than before, and Danish accuracy no more than 0.3 points worse.

The recordings are joined from FLEURS, people reading Danish sentences aloud. They run from 90 seconds to 5 minutes.

11 attention heads do the copying

Whisper's text half has 32 layers. Each layer has 20 small parts called attention heads, which decide what earlier text to look at. Claude built 220 pairs of test cases, one with an earlier copy of a sentence in the hint and one without, and traced which heads light up for the copy.

zoom_in

Two groups stood out. Finder heads notice that the words just written appeared before. Writer heads copy out whatever came after them last time. Switching off those 11 heads took the repeat rate in the test cases from 68% to 0%. Switching off 11 random heads in the same layers left it at 68 to 74%.

That surprised me a bit. The research I found before starting (Glazer et al. 2025) blamed the part of Whisper that listens to the audio. Here it was the part that reads its own text.

On 30 recordings it had never seen, long repeats went from 21 to 1

This is the sealed test: 30 recordings, 90 minutes of Danish in total. It counts extra words written by repeats on whisper.cpp's two long-recording modes. Lower is better.

zoom_in

The random edits land on both sides of the original: change anything, and a few recordings flip. The edit lands far below all three.

Repeats of 10 words or more went from 21 to 1. That last one is not a copy. The model made up a phrase ("hvor de har været i en krig") and repeated it. The copying is gone. Making things up is a different problem, and we did not fix that.

It just missed the strict pass mark we set in advance

Before the final test we wrote down five things the edit had to do. Two of them use "Codictate's way" of transcribing: cut the recording into pieces of 30 seconds or less, and transcribe each piece on its own. The 30 recordings hold 460 sentences in total.

The two misses are small:

• Repeats. 5 of the 6 left are tiny slips, three or four words written twice, like "til at booke til at booke". The original and both random edits make those too.
• Left-out sentences. One more out of 460. The random edits left out 15 and 14, so one sentence is normal wobble. And in the long-recording modes, where repeats happen, the edit left out far fewer: 26 sentences instead of 59 with timestamps.

So the edit does the job. It lands just under the line we drew before the test, and that line stays where it is.

Danish got better only where repeats were the problem

My secret hope going in was that a model that stops repeating would also just be better at Danish. That turned out to be only half true.

zoom_in

Where repeats happen, the edit helps a lot. With timestamps, about one word in every nine goes from wrong to right. Where Codictate already cuts the audio into slices, nothing changes, because there was nothing to remove. On all 930 FLEURS Danish test recordings the difference is 0.11 points, which is noise. Cutting into slices is also still the most accurate way to run large-v3 here: 14.2% against 19.7% for the edited model on the whole recording.

That matches what the MIT paper found in its follow-up: an edit like this can remove something in the way, but it cannot teach the model something it does not know.

A few other checks, outside the pass mark:

• English. Codictate has three English test recordings, about 7 minutes in total. With timestamps, the original wrote 160 made-up words in a loop. The edit wrote none, and the word error rate went from 17.6% to 1.7%.
• Smaller download. Compressed to q5_0, about a third of the size, repeats still dropped 79%.
• Hugging Face transformers. On 10 of the recordings with OpenAI's recommended settings, repeats went from 21 to 3.

What this does not tell you

The final test is 30 recordings and one run, all Danish read speech. Whole-recording transcription is chaotic, and the random edits show how much any single recording can swing. English got a handful of recordings, not a real test. Most numbers come from whisper.cpp (CrispASR v0.8.41) on one Apple Silicon Mac; the transformers check was smaller and had no random edits.

We changed the pass mark once. After the search, we dropped a mode that feeds silence straight into the model, because the edit made it worse there on the 10 search recordings. On the 30 sealed recordings the edit made that mode better too, so dropping it was probably not needed. It is written up in the repo. And the edit does nothing about Whisper making up text in silence. You still want voice detection in front of it.

FAQ

Why does Whisper repeat the same sentence? On long audio it gets its own earlier text as a hint. A few attention heads copy from that hint instead of following the audio.

Does Codictate use the edited model? No. Codictate cuts recordings into slices of 30 seconds or less, and on that path large-v3 did not repeat in any of our tests.

Does it make Whisper better at Danish? On long recordings transcribed in one go, yes: the word error rate went from 31.0% to 19.7%, because the repeats are gone. On short recordings it did not change.

Can I use it? Yes, it is MIT licensed like Whisper. It is a research model, so test it on your own audio first.

Try it

The edited model is on Hugging Face as whisper-large-v3-copy-heads-off. There is a version for transformers and two for whisper.cpp, plus a short script that rebuilds it from OpenAI's original. Everything else is in github.com/EmilLykke/whisper-loop-edit: the code, every number, and the full log of what we tried and what failed.

For how large-v3 compares with the other models in Codictate, see the benchmarks page and the speech model benchmarks.