Reducing Latency

A voice agent feels quick or slow long before the language model is involved. This page follows one turn from the last syllable to the first sound of the answer, says what each step costs, and points at the settings worth touching.

The budget of one turn

The numbers below were measured on a MacBook Pro M2, in Chromium, over the French half of the Smart Turn test set. Your own will differ, the order of the steps will not.

The first line is the one that surprises people. Before any model has read anything, most of a second has already gone by waiting to be sure the speaker stopped, and that wait is a setting rather than a law.

The silence is the setting

A voice detector cannot know a sentence is over, so it waits. SileroVAD counts redemptionFrames of silence. Eight windows is about a quarter of a second of audio, and measures longer in practice, because the counter only starts once the model stops hearing speech and a voice fades before it disappears.

That number used to be twenty, a compromise with nothing behind it but fear of cutting people off. Turn detection removes the fear, so the default came down. Without a detector, put it back:

import { Micdrop, SileroVAD } from '@micdrop/web'

// No turn detection, so the wait has to cover the hesitations itself
await Micdrop.start({
  url: 'wss://example.com/call',
  vad: new SileroVAD({ redemptionFrames: 20, minSpeechFrames: 8 }),
})

Measured over 120 turns, the defaults close a turn in 760 ms instead of the 1060 ms the old ones took, for four points of accuracy that turn detection is there to absorb.

The two kinds of mistake do not cost the same. A turn held when it should have been answered only waits, and gets its answer when turnMaxWait runs out. A turn answered when it should have been held cuts the speaker off. So read the last column as how often the call feels right rather than as a score.

Every setting, and whether to touch it

Measured one at a time from the same baseline, on the French half of the test set.

The settings that only shave tens of milliseconds each look free on their own and stop being free together: the defaults pay four points for three hundred milliseconds against the old ones, where redemptionFrames alone pays about two.

Everything measured here comes from clean recordings. A noisy room moves the two volume settings first, so tune those against your own audio rather than against this table.

Choosing the detectors

Each voice detector has its own idea of when silence begins, and combining them means waiting for the slowest.

VolumeVAD is quick because it only watches a level, and it takes a door slam for a word. SileroVAD tells a voice from a noise and pays for it in time. Running both is the safe default, and lowering Silero’s redemption is what makes the pair quick again.

Dropping the volume detector and running Silero alone at four frames closes turns in about 360 ms, and gives up far too much: it cuts sentences in the middle, which asks the turn detection model 60% more often and gives it that many more chances to be wrong. Accuracy fell to 75.8% in that setting.

Where turn detection runs

The model itself is cheap next to the silence it saves, provided it runs somewhere sensible.

@micdrop/smart-turn/web picks WebGPU whenever the browser offers it. On a browser without it, weigh the fallback against what your users run, and consider the server side detector instead, which costs a round trip but leaves the phone alone.

Building the features costs 0.5 ms per 100 ms of audio while the speaker talks, so it never lands on the critical path at all.

The rest of the chain

Once the turn closes, the audio is already on the server, since chunks are streamed every 100 ms while the user speaks rather than sent in one go at the end. What remains is transcription, the agent and the voice.

Those three are where a local setup spends its second, and the local models page measures each of them.

The structural one is transcription. A batch model such as Whisper only starts once the turn closes, so its time lands entirely on the critical path, around 440 ms for a three second sentence. A streaming speech to text such as Gladia transcribes while the person is still talking, so the transcript is all but ready when the turn ends and that time mostly disappears. It is the largest single saving available to a local setup, ahead of any model swap.

Three habits then matter more than which model you pick:

Keep the models loaded between calls, and run a warm-up at startup so the first inference happens while the call is being set up rather than while a user waits.

Speak the first sentence as soon as it exists rather than the whole answer once it is finished. The SentenceTTS base class does this, so the voice starts while the agent is still writing.

Count the automatic prompts. Each of autoEndCall, autoSemanticTurn and autoIgnoreUserNoise can cost the agent a tool call, and a second pass through the model costs more than the transcription and the voice put together. autoSemanticTurn in particular has no reason to be on when turn detection runs in the browser.

What is not worth your time

The turn detection model is about 2% of the budget of a turn, and the features it reads are free. Tuning its threshold changes who gets interrupted, not how fast the call feels.

The audio chunks are not a bottleneck either. They leave every 100 ms during speech, so by the time the turn closes the server is holding everything but the tail.

Spend your attention on the silence, on keeping the models warm, and on speaking before the answer is complete.