Skip to content
Bernhard Götzendorfer
Debugging & Solutions

When the Agent Replaced 13 Minutes of Speech with "Thank You"

A Whisper default cost me 13 minutes of transcript on my first real end-to-end run – replaced by a hallucination loop. The diagnosis, the fix, and four transferable lessons.

Dark sepia board with an amber ink sketch: an audio waveform tipping rightward into a repeating endless loop labeled "Thank you"

TL;DR

I am building a local, agentic video cutter: Claude in the terminal as the director, local ASR and ffmpeg as the tools, an EDL JSON as the contract between decision and render. On the first real run against a 58-minute live-coding clip, a quiet Whisper default surfaced: condition_on_previous_text=True sent the model into a hallucination loop and replaced roughly 13 minutes of actual speech with a repeated "Thank you." I would never have caught this in the demo material. This article is the honest write-up: the attempt, the wall, the diagnosis, the four-layer fix, and the lessons I am taking forward.

What this is about

Since late 2024 I have worked with AI coding agents every day, and part of that is building my own tools instead of only consuming other people's. One of those tools is an agentic video cutter: I throw in an unedited talking-head or screencast video, and out comes an optimised cut. Filler words gone, long pauses trimmed, retakes removed. It is orchestrated not by a rigid pipeline but by Claude in the terminal acting as the director. The heavy lifting is done by local tools: a speech-to-text model for the transcript, ffmpeg for the frame-accurate cut.

The architecture behind it is deliberately decoupled. Between the decision ("cut here") and the render ("ffmpeg makes the cut") sits an EDL JSON as the contract. The transcript is the basis of every decision. If the transcript is wrong, everything after it is wrong, no matter how cleanly the rest runs. That is exactly where the floor gave out under me.

Up to that point I had only worked with short test clips: one or two minutes, clean audio, clear speech. Everything worked. The cut landed, the captions were right. A textbook happy-path demo. The first real stress test was a 58-minute recording of a live-coding session in German.

The attempt

The standard flow is a chain of steps, each with a clear output:

  1. Transcribe: audio in, time-stamped transcript JSON out.
  2. Clean the transcript: remove obvious artefacts.
  3. Cut decision: derive the EDL JSON from transcript and silence detection.
  4. Review the pause map, approve.
  5. Render and export captions.

I sent the 58-minute clip into step 1 and kept working on the side. The local ASR model took a few minutes, returned a complete JSON, no error, no crash, clean exit code. At first glance: success. That is precisely the trap. A clean exit code only tells you the process ran to completion, not that the result is correct.

The wall

I only noticed it while reviewing the pause map. The cut was absurd in one place: a huge contiguous block marked for trimming, right in the middle of a section where I had demonstrably been talking the whole time. I went back into the transcript JSON and looked.

There, starting around minute 30, across several hundred segments, the same line appeared again and again:

{ "start": 1812.4, "end": 1814.9, "text": " Thank you." }
{ "start": 1814.9, "end": 1817.2, "text": " Thank you." }
{ "start": 1817.2, "end": 1819.8, "text": " Thank you." }

Roughly 13 minutes of real speech, during which I had been explaining code live, had vanished from the transcript entirely and been replaced by a repeated "Thank you" loop. The audio was flawless, I could play it and clearly hear my own voice. The transcript, however, claimed I had said nothing for 13 minutes but a politeness phrase.

A clean exit code only tells you the process ran to completion. It tells you nothing about whether the result is correct.

The insidious part: had the cut algorithm simply waved this block through, the error might never have surfaced. Thirteen minutes of content would have silently disappeared from the finished video. The absurd cut was pure luck, a visible symptom of a silent data corruption further up the chain.

The diagnosis

The pattern was familiar. Anyone who works with Whisper-based models long enough knows the hallucination loop: on silence, music, background noise, or simply a passage the model cannot decode cleanly, it falls into an endless repetition. In German, "Vielen Dank" is a classic candidate; in English it is often "Thank you" or "Thanks for watching," because such phrases are massively over-represented in the training data (subtitles from videos). The model does not guess "nothing," it guesses the most probable stock phrase and clings to it.

The actual root cause is a default parameter that is well-intentioned and, for many cases, even sensible:

# Whisper default -- the actual trigger
result = model.transcribe(
    audio,
    condition_on_previous_text=True,  # default
)

condition_on_previous_text=True means the model receives its own previously generated text as context for each new segment. Normally this makes the transcription more coherent, keeping proper names and terminology consistent. But it also ties the model to its own mistakes. Once it has produced "Thank you," that exact text becomes the context for the next segment. And the most probable next text after "Thank you" is "Thank you" again. A feedback loop that reinforces itself until the model can no longer find its way out.

Three things I had to rule out to be sure this was the cause:

  • Broken audio? No. Played it, clear speech, no noise, no silence at that point.
  • Wrong model or language? No. The first 30 minutes were transcribed cleanly, same run, same model.
  • A bug in my cleaning step? No. The loop was already in the raw ASR output, before my own code ever touched it.

That made it clear: the damage was created directly in the transcription step, triggered by the context default. There was probably a short passage of quieter or less distinct speech at that point in the audio that provoked the first misfire. The coupling did the rest.

The fix, in four layers

The obvious answer is to set condition_on_previous_text to False. I did that, but on its own it is not enough, and it comes at a price. The solution I adopted comes at its core from an earlier project of mine, a macOS push-to-talk tool with local speech recognition. There I had already built a multi-layer defence against exactly this problem. Four layers, because no single one of them covers the whole failure space.

Layer 1: Cut the coupling. The default moves to False. That stops a single misfire from propagating through its own context. Each segment is decoded independently.

# Fix layer 1 -- cut the coupling to its own mistakes
result = model.transcribe(
    audio,
    condition_on_previous_text=False,
)

Layer 2: Detect and remove the loop after the fact. cond=False prevents propagation, but not every individual misfire. So after transcription, a deterministic cleaner runs over the JSON, recognising repeated identical segments as a loop and collapsing them. No model, pure logic: if the same short phrase appears N times in a row with gapless timestamps, that is not speech, it is an artefact.

# Fix layer 2 -- collapse the loop deterministically (simplified)
def collapse_loops(segments, min_repeats=4):
    cleaned, run = [], []

    def flush(run):
        # collapse a detected loop to one occurrence, otherwise keep it all
        return run[:1] if len(run) >= min_repeats else run

    for seg in segments:
        text = seg["text"].strip()
        if run and text == run[-1]["text"].strip():
            run.append(seg)
            continue
        cleaned.extend(flush(run))
        run = [seg]
    cleaned.extend(flush(run))  # do not forget the final run
    return cleaned

Layer 3: Do not decode silence in the first place. An optional voice-activity gate detects up front where there is actually speech and sends only those passages into the model. Silence then cannot become a hallucination at all. This is the cleanest prevention, but I keep it opt-in, because a too-aggressive gate can discard genuine quiet speech.

Layer 4: Make confidence visible. The model returns per-word probabilities. A hallucination loop often has a characteristic confidence pattern. Carrying these values forward instead of throwing them away gives you a later handle to flag suspicious passages automatically rather than believe them.

The order matters: no single layer is the solution. Layer 1 stops propagation, layer 2 clears the remnants, layer 3 prevents the trigger, layer 4 makes the problem measurable. Together they are robust; on their own they all have gaps.

An honest tradeoff remains: cond=False makes the transcript less smooth in places, and it tends to drop genuinely spoken filler words like "uhm" from the text. For a cutter that wants to remove filler anyway, that sounds convenient, but it means I now have to base filler detection on the audio rather than on the transcript. One solved problem has opened a new one. That is almost always how it goes.

What I take from this

Four lessons I consider transferable, well beyond this one bug.

1. Hallucination only shows up in real material

My demo clip was two minutes of clean audio. It would never have shown me this error, because it did not contain the conditions that trigger it: no longer indistinct passages, no point where the model starts to guess. Happy-path demos test whether the mechanics run, not whether the result holds up against real, messy input. The first real 58-minute run was not a nice extra test, it was the actual test.

2. A clean exit code is no proof of correctness

The transcription step ran without error and returned well-formed JSON. At the process level, everything was green. The content was largely garbage all the same. For generative components, "did not crash" is not enough as a success criterion. You need a check on the content: plausibility checks, confidence thresholds, or a human who actually looks before approving.

3. Quiet defaults are the most dangerous

condition_on_previous_text=True is not a bug, it is a deliberate, documented default that does the right thing in many cases. But it is invisible until you know about it, and its failure is silent. The most expensive mistakes rarely come from something obviously broken. They come from a reasonable default that is the wrong choice in your own, slightly different situation.

4. Defence belongs at the contract, not at the end

Because the transcript is the contract for everything downstream, the safeguard had to sit right there, not in the finished video. Catching the loop at the EDL JSON would have been too late; the damage would already have been baked in. If you work with pipelines of several steps, put the check where the data is created, not where it becomes noticeable. I describe the same stance in harness design for AI coding agents: do not trust the output blindly, give the tool the means to check itself.

Conclusion

In order, what made up this failure:

  1. The trigger: a well-intentioned Whisper default, condition_on_previous_text=True, that ties the model to its own mistakes.
  2. The damage: roughly 13 minutes of real speech, replaced by a "Thank you" loop, silently and without any error in the process.
  3. The visibility: noticed only by chance, an absurd cut as a symptom of data corruption further up.
  4. The fix: four layers, from the cut coupling through the deterministic loop cleaner and the opt-in voice gate to the carried-forward confidence.
  5. The price: less smoothing, filler detection moves from the text to the audio. One solved problem opens the next.

Local speech recognition is no black art, but it is not magic either. It is a tool with clear weaknesses, and anyone who uses it seriously has to know those weaknesses and guard against them. If you have seen the same loop before, in whatever language, or built a different defence: write to me, I collect patterns like this. If you are curious about the methodology behind check loops like these, the open resource library at agenticbuilders.at/ressourcen has the material on it. And if you are working on a build of your own and want a sparring partner: I deliberately work with only one or two clients at a time, here is how to reach me.

By the way: that the cutter uses a local model as a tool rather than a cloud service has the same background as my failed attempt to run a coding model locally. Local means control, but it also means you carry every weakness of the model yourself.

← Back to blog