Making Whisper Work in Real Time at the Edge
A live edge-AI transcription case study
Author: Muhammad Shess (10xEngineers)
Date: September 17, 2026
1. Introduction
Running Whisper, an automatic speech recognition (ASR) model on a constrained edge device is achievable. The trouble started once the input stopped behaving like a static recording.
Keeping more audio gave Whisper useful context, but the transcript arrived later and the buffer kept growing; cutting sooner helped latency but damaged words near the cut. The model never changed, the recording strategy built around did.
The application also had a compute problem: its initial RTF[1] left too little margin for continuous speech. Work across model behavior, preprocessing, scheduling, and hardware closed that budget. The latest 100-sample accelerator benchmark reports 0.50× RTF, meaning transcription takes about half the duration of the input audio and leaves substantial headroom for continuous speech.
A queue only helps if the client finishes faster than audio arrives, which sets up the real question: how much live speech should the application hold before handing it to Whisper?
2. The Core Problem
In plain terms, the system must know when to stop collecting one phrase and start transcribing it: a boundary-detection problem. People don’t speak in neat, separated chunks: a pause may fall mid-sentence, while another speaker runs on for several seconds without a clean break. Where the system cuts the audio affects both latency and transcription accuracy.
3. Baseline
The baseline voice activity detector (VAD) told speech from silence: once speech began, a sustained pause closed the phrase and triggered the model. A 15-second maximum phrase length was meant to cap latency, but it was only checked when a non-speech frame arrived at the same branch as the silence rule. It wasn’t a hard ceiling: continuous speech could run past fifteen seconds without closing the phrase.
That exposed the real issue. VAD can detect that someone is speaking; it cannot tell whether they’ve finished a thought. A pause was doing double duty; evidence of a boundary, and permission to start inference.
The accelerator-backed path is the application mode that runs the Whisper model on a dedicated AI hardware accelerator instead of the CPU. In this path, each detected phrase passes directly from the VAD callback to a single serial Whisper call, with no queue between phrase detection and inference (Fig. 1).
Fig 1: CURRENT IMPLEMENTATION — Direct phrase-to-inference handoff; not the queued target design.
This showed that the application could send audio to the model and receive text. However, audio processing still depended on the model being available. The application could not finish one phrase and prepare for the next while the model was transcribing.
Performance improvements made the model fast enough for real-time use. In a new test, it processed 4.464 seconds of audio in 2.654 seconds, giving a real-time factor (RTF) of 0.595×. This is faster than real time, but it does not prove that a synchronous callback can safely record new audio while the model is processing the previous audio.
4. Proposed Design
Fig 2: Proposed Design
The proposed queue separates two concerns; capture and transcription while the edge accelerator remains a single inference worker. While one completed phrase waits or is being transcribed, the microphone keeps collecting the next, so capture, queueing, inference, and output can overlap even though Whisper still processes one phrase at a time.
The queue depends on that speed margin: it doesn’t make the worker faster or change raw RTF, but it lets a sub-one RTF translate into overlap. At end-to-end RTF of 1.0 or higher, the queue would grow instead of draining. So if speech ever arrives faster than the worker can handle it, queue depth exposes the delay instead of burying it inside a blocking callback.
5. Testing the Wrapper
Before touching the streaming design, the wrapper itself had to be ruled out as the source of any accuracy difference, through three checks:
- A direct-call control, to see whether the wrapper changed the model path.
- Fresh repeat runs, to see whether a small output difference was stable.
- A phrase-split test, to see what happens when recordings are cut into live-style segments.
Word error rate (WER)[2] compares a transcript against a reference, lower is better. Transcript similarity compares two generated transcripts against each other. Both are percentages, but they answer different questions.
For the control, the direct call and the wrapper used the same decoded waveform, preprocessing, and generation route on the same edge-accelerator path; source-path parity[3]. That rules out an extra wrapper-side transformation, though the benchmark didn’t hash the model input tensors, so exact tensor equality remains untested.
Result: 99.18% transcript similarity across 100 speech samples. WER came out at 10.47% through the wrapper and 11.07% through the direct path, a difference well inside the three-to-five-transcript-per-hundred variation seen across repeat runs, so it wasn’t treated as a real improvement.
That left the wrapper off the hook as the explanation for the live-transcription gap. The application already runs sub-1.0 RTF on these workloads, meaning inference can outpace incoming audio. Hardware provides the margin, but preprocessing, scheduling, model execution, and output handling decide whether the full client keeps up. The proposed queue exploits that margin through overlap; it doesn’t change raw RTF, and it doesn’t touch the WER question above.
6. The Live-Streaming Gap
The harder test split speech into phrases to simulate live streaming.
Result: 16.87% WER, the key figure for the remaining streaming problem. Unlike the control tests, this larger loss appeared once continuous speech was cut into separate phrases: when a word crosses a boundary, Whisper loses surrounding context and can misrecognize it.
Two limitations shaped this number: the normal neural speech detector[4] was unavailable, so an RMS fallback[5] (loudness-based) stood in for it, and there was no audio overlap or shared context between neighboring phrases. So 16.87% WER measures the cost of the boundary strategy under test, not the expected performance of the finished design.
7. Takeaway
The project surfaced a real difference between running an AI model and deploying a real-time AI system. Whisper could run at the edge; the harder challenge was coordinating what surrounds it — continuous capture, memory, phrase boundaries, inference, and output.
Separating the model-path question from the streaming-boundary problem is what testing was for: a successful model call proves the AI can run, but a reliable deployment is what makes it usable.
8. Method Note
Both evaluations used the first 100 samples of the Common Voice English validation set. Each sample got its own WER score before scores were averaged, a per-sample (macro-average)[6] method, so one long recording can’t dominate the result.
WER compares a transcript against the reference text (lower is better); transcript similarity compares two generated transcripts against each other. Because they answer different questions, the two percentages shouldn’t be compared directly.
Glossary
[1] Real Time Factor (RTF): RTF measures how quickly the inference client processes audio, it’s the ratio between inference time and audio duration.
[2] Word error rate (WER): a lower-is-better measure of words that differ from the reference transcript.
[3] Source-path parity: the direct path and wrapper share the same decoded waveform and preprocessing/generation route; this is not proof that the model saw byte-for-byte identical data.
[4] Neural speech detector: a learned component that decides whether the incoming audio contains speech or silence.
[5] RMS fallback: a basic loudness-based rule used here to identify silence when the normal speech detector was unavailable.
[6] Per-sample average (macro-average): score every sample separately, then average those scores so one long recording does not dominate the result.