Semantic Turn Detection

A user who pauses in the middle of a sentence has not finished speaking. Voice activity detection cannot tell the difference, so the server answers a half question. Semantic turn detection waits for the thought to land.

The server offers two ways of doing it, and Micdrop also offers a third that runs in the browser.

Reading the sound of the sentence

turnDetector hands the audio of each stretch of speech to a small model, which answers in a few tens of milliseconds without spending a token.

import { MicdropServer } from '@micdrop/server'
import { SmartTurn } from '@micdrop/smart-turn'
import { setSmartTurnOptions } from '@micdrop/smart-turn/node'

// Keep the checkpoint next to the code so the server never reaches out at boot
setSmartTurnOptions({ model: './smart-turn-v3.2-cpu.onnx' })

new MicdropServer(socket, {
  agent,
  stt,
  tts,
  turnDetector: new SmartTurn(),
  turnMaxWait: 4000,
})

When the sentence sounds unfinished the server sends SkipAnswer and waits for the rest, which arrives as a second user message. The agent then answers with both in front of it.

A model can hear an unfinished sentence where there is none, so the wait has a deadline. turnMaxWait says how long it lasts, four seconds by default, and the agent answers on its own once it runs out. The clock stops as soon as the speaker starts again, so a long answer full of hesitations never runs out of time.

What it costs

The model reads the last eight seconds of the stretch that just ended and answers in about 13 ms on a laptop processor, 40 ms on a single server core. It works on every call, whatever the browser, and the audio never leaves your infrastructure.

The delay it adds is the one that made the client the better place for it. By the time the server hears the pause, the client has already stopped its turn, so lowering the silence the browser waits for is out of reach here.

Asking the agent

autoSemanticTurn puts the question to the language model instead, as a tool it can call on the transcript.

const agent = new OpenaiAgent({
  apiKey: process.env.OPENAI_API_KEY || '',
  systemPrompt: 'You are a helpful assistant',
  autoSemanticTurn: true,
  // or provide your own wording:
  // autoSemanticTurn: 'Last user message is an incomplete sentence',
})

Reading the transcript catches what the sound alone cannot, a sentence that is grammatically complete yet obviously leads somewhere. It costs a full round trip to the language model and its tokens on every pause, so it sits well in a call where the agent is already thinking between turns, and poorly in one tuned for latency.

What each of them changes

Take a user saying “I want to…” and pausing.

Without any of this, the agent answers the fragment and talks over the rest of the sentence.

With detection in the client, nothing is sent, the turn stays open, and “book a flight” joins the same message. The agent sees one question and answers it once.

With detection on the server, the fragment reaches the agent and the answer is held back. “book a flight” arrives as a second message, and the agent answers with both.

Combining them

The three can be layered. The client decides when to close its turn, the server decides whether the audio deserves an answer, and the agent has the last word on the transcript. Start with the client alone, and add the others when you meet a case it misses.