๐ŸŽคMicdrop

Voice Activity Detection in the Micdrop Client

Voice activity detection (VAD) classifies short audio frames as speech or non-speech. Micdrop runs VAD in its browser client, which holds the microphone, so it can decide when to send audio to the server without a network round trip. VAD detects speech and silence. Turn detection decides whether a pause ends the speakerโ€™s thought.

Micdrop offers a volume detector and Silero. Each trades latency against false positives, and Silero, WebRTC VAD and a volume threshold react differently to background noise.

Micdrop keeps recent audio in reserve while the detector confirms speech, so the first syllable reaches the server with the rest of the sentence.

When interruptions are enabled, the microphone records speech over the assistantโ€™s reply, with echo cancellation keeping the assistantโ€™s voice out of the recording. When interruptions are disabled, Micdrop does not send microphone audio captured while the assistant speaks or for 300 ms afterward. The client disables interruptions automatically when the browser reports that echo cancellation is off.

Echo and Interruptions

A browser can report echo cancellation on while the microphone still hears the assistant, as iOS does with an audio context. The assistant then sounds like someone speaking over it, and cuts itself off. Micdrop handles this on its own, with nothing to configure.

Very few people speak over a voice that has only just started, so speech confirmed during the first 2 seconds of the first answer of a call is treated as echo. The assistant keeps talking, the server hears nothing of it, and the call moves to the next stage. Micdrop.echoStage tells which stage it is on.

echoStageWhat the call does
'normal'Plays as usual. Every call starts here.
'fallback'Plays by the other route of the platform, an audio element in a browser. The same watch runs for the next 2 seconds of sound, and if the answer ends first, it carries over to the next one. When the call passes it, it stays here.
'disabled'Disables interruptions until the call ends, as disableInterruption does. Reached on a second detection, or on the first one when the platform has no other route or the browser refuses it.

When the first watch ends without hearing anything, the call stays on 'normal' and interruptions work as usual from then on. Nothing is remembered from one call to the next.

This has three limits:

  • When someone really does speak in those first 2 seconds, they are taken for an echo and those words are lost.
  • When echo cancellation needs a moment to adapt, the call can leave the normal route without needing to.
  • When an echo only appears later in the call, once the volume goes up for instance, it goes unnoticed.

Pass disableInterruption: true to skip all of this and never let the user interrupt. It also applies when the browser reports that echo cancellation is off.

Supported VAD Types

Micdrop supports the following VADs by name:

  • 'volume': Volume-based VAD (default)
  • 'silero': AI-based VAD using Silero

You can also pass instances of these VADs, or combine them in an array. See below for details.

Note: Only 'volume' and 'silero' are supported as string names. Custom VADs must be passed as instances.

Quick Start

Install the browser client:

Terminal window
npm install @micdrop/web

Configure VAD when starting a call:

import { Micdrop } from '@micdrop/web'
// Use volume-based detection (default)
await Micdrop.start({
url: 'ws://localhost:8081',
vad: 'volume',
})
// Use AI-based detection for better accuracy
await Micdrop.start({
url: 'ws://localhost:8081',
vad: 'silero',
})
// Combine multiple VADs for best results
await Micdrop.start({
url: 'ws://localhost:8081',
vad: ['volume', 'silero'],
})

Or when starting the microphone (before starting the call):

Micdrop.startMic({ vad: 'volume' })

Volume VAD: Speech detection based on volume

By default, MicdropClient uses VolumeVAD for speech detection. You can use it explicitly when starting Micdrop:

Micdrop.start({ vad: 'volume' })

or when starting the microphone (before starting the call):

Micdrop.startMic({ vad: 'volume' })

It is inspired by hark and triggers speech detection events based on volume changes.

The level VolumeVAD follows is the loudest frequency rather than the overall energy: a voice concentrates its power in a few bands while a fan spreads it over all of them, which separates the two by about 25 dB instead of the 7 dB a plain average would give.

You can also pass an instance of VolumeVAD to MicdropClient:

const vad = new VolumeVAD({
history: 5, // Number of frames to consider for volume calculation
threshold: -55, // Threshold in decibels for speech detection
})
Micdrop.start({ vad })
  • Default options: { history: 5, threshold: -55 }
  • Persistence: Options are saved to localStorage in a browser, and wherever setMicdropStorage() points on a phone.

When to use Volume VAD:

  • โœ… Low latency requirements
  • โœ… Quiet environments
  • โœ… Clear speech patterns
  • โŒ Noisy environments
  • โŒ Soft-spoken users

Silero VAD: Human speech detection with AI

To use SileroVAD for speech detection:

Micdrop.start({ vad: 'silero' })

It runs the Silero model on ONNX Runtime Web in a browser and on the native runtime on a phone. The model gives each window of sound a probability of speech. Micdrop decides from those probabilities when speech starts and stops, with the same algorithm as ricky0123/vad, the open source project that runs Silero in the browser. The model is about two megabytes, fetched once, on the first call.

SileroVAD is more accurate than VolumeVAD and works better with low voice, since it hears the difference between a voice and a noise rather than measuring how loud the room is.

The ONNX runtime weighs more than the rest of Micdrop put together, so it only enters your bundle if you ask for it:

import '@micdrop/web/silero'

In an app built with Vite, add the Micdrop Vite plugin. It ships the model and the ONNX runtime with your app. Without it, they are fetched from jsDelivr and unpkg when a call starts, and voice detection fails wherever those CDNs are unreachable or blocked.

With another bundler, serve the model and the files of the ONNX runtime (ort-wasm-simd-threaded.jsep.mjs and ort-wasm-simd-threaded.jsep.wasm, from onnxruntime-web/dist) from your own domain:

import { setSileroOptions } from '@micdrop/web/silero'
setSileroOptions({
model: '/assets/silero_vad_v5.onnx',
wasmPath: '/assets/onnxruntime/',
})

wasmPath: false leaves the files to the ONNX runtime, which looks for them next to itself, for a bundler that emits them on its own.

On a phone the model runs on the native runtime, which asks for one more package and a patch. See Voice activity detection on React Native.

You can also pass an instance of SileroVAD to MicdropClient:

const vad = new SileroVAD({
positiveSpeechThreshold: 0.18, // Threshold for positive speech detection
negativeSpeechThreshold: 0.11, // Threshold for negative speech detection
minSpeechFrames: 4, // Windows of speech before a turn is confirmed
redemptionFrames: 8, // Windows of silence before a turn is closed
})
Micdrop.start({ vad })
  • Default options: { positiveSpeechThreshold: 0.18, negativeSpeechThreshold: 0.11, minSpeechFrames: 4, redemptionFrames: 8 }
  • Persistence: Options are saved to localStorage in a browser, and wherever setMicdropStorage() points on a phone.

redemptionFrames is eight windows, about a quarter of a second, so a turn closes quickly and leans on turn detection to keep the floor when a speaker only paused. Running Silero without a turn detector, raise it to twenty, or expect the agent to answer during hesitations. See Reducing Latency for what each window costs and what it buys.

When to use Silero VAD:

  • โœ… Noisy environments
  • โœ… Soft-spoken users
  • โœ… Multiple speakers
  • โœ… Background music/TV
  • โŒ Extremely low latency needs (adds ~50ms processing)

Multiple VAD: Combine multiple VADs

Combining multiple VADs is useful to get more accurate speech detection:

  • Volume to ignore low voice
  • Silero to detect human speech

A turn opens once every VAD has heard the speech and at least one of them is sure of it. A short word like โ€œNoโ€ is over before the volume VAD confirms it, and still gets through as long as it was loud enough to be noticed.

You can combine multiple VADs by passing an array of VAD names:

Micdrop.start({ vad: ['volume', 'silero'] })

Or with instances:

const vad = [new VolumeVAD(), new SileroVAD()]
Micdrop.start({ vad })

Or mix names and instances:

await Micdrop.start({
vad: ['volume', new SileroVAD({ positiveSpeechThreshold: 0.15 })],
})

How it works:

  • StartSpeaking is emitted when any VAD detects possible speech.
  • ConfirmSpeaking is emitted only when all VADs confirm speech.
  • StopSpeaking is emitted when all VADs detect silence.
  • CancelSpeaking is emitted if all VADs agree speech was a false positive.

This approach reduces false positives while maintaining quick response times.

VAD Events

VADs emit the following events:

  • StartSpeaking: Possible speech detected (not yet confirmed)
  • ConfirmSpeaking: Speech confirmed
  • CancelSpeaking: Speech start was a false positive (noise, etc.)
  • StopSpeaking: Speech ended
  • ChangeStatus: Status changed (Silence, MaybeSpeaking, Speaking)

Monitor VAD activity in your application:

Micdrop.vad.on('StartSpeaking', () => {
console.log('๐ŸŽค Possible speech detected...')
showListeningIndicator()
})
Micdrop.vad.on('ConfirmSpeaking', () => {
console.log('โœ… Speech confirmed - recording')
highlightMicrophoneButton()
})
Micdrop.vad.on('StopSpeaking', () => {
console.log('๐Ÿ”‡ Speech ended')
resetMicrophoneButton()
})
Micdrop.vad.on('CancelSpeaking', () => {
console.log('โŒ False positive - not speech')
hideListeningIndicator()
})
Micdrop.vad.on('ChangeStatus', (status) => {
console.log('VAD status:', status) // 'Silence', 'MaybeSpeaking', 'Speaking'
})

Silero on Another Runtime

@micdrop/web and @micdrop/react-native load the Silero model on their ONNX runtime when @micdrop/web/silero or @micdrop/react-native/silero is imported. Another runtime plugs in with setSileroModelLoader(), given a function that loads a SileroModel: it scores windows of 512 samples at 16 kHz, forgets the previous call on reset(), and frees its runtime on release().

LazySession holds the inference session of such a model. Callers asking while it loads share the same load, a release during the load disposes of the session as soon as it lands, use() runs inference that a release waits for, and a failed load is tried again on the next call. createDownloadCache fetches each model address once for as long as the app runs, and forgets a failed download.

import {
createDownloadCache,
LazySession,
SileroModel,
setSileroModelLoader,
} from '@micdrop/client'
const download = createDownloadCache(async (url) => {
const response = await fetch(url)
if (!response.ok) throw new Error(`Download failed: ${response.status}`)
return response.arrayBuffer()
})
class MySileroModel implements SileroModel {
private session = new LazySession('Silero', async () =>
createMySession(await download(MODEL_URL))
)
process(frame: Float32Array) {
return this.session.use((session) => session.score(frame))
}
reset() {
// Clears the state the model carries from one window to the next
}
release() {
return this.session.release()
}
}
// A new model for each call, released when the call stops
setSileroModelLoader(async () => new MySileroModel())

Custom VAD

The VAD class is exported, so another method can be plugged in. It listens to the microphone and reports four moments: speech may have started, it is confirmed, it was only noise, it has ended.

import { MicSource, VAD } from '@micdrop/web'
class MyVAD extends VAD {
private mic?: MicSource
get isStarted() {
return !!this.mic
}
get isPaused() {
return false
}
async start(mic: MicSource) {
this.mic = mic
mic.on('Frames', this.onFrames)
}
async stop() {
this.mic?.off('Frames', this.onFrames)
this.mic = undefined
}
async pause() {}
async resume() {}
private onFrames = (frames: Float32Array, sampleRate: number) => {
// Decide, then report
// this.emit('StartSpeaking')
// this.emit('ConfirmSpeaking')
// this.emit('CancelSpeaking')
// this.emit('StopSpeaking')
}
}
Micdrop.start({ vad: new MyVAD() })

Frames carries mono samples between -1 and 1, Volume carries the level in dBFS. The class lives in @micdrop/client, so the same detector runs in a browser and on a phone, where it is imported from @micdrop/react-native.

See VolumeVAD as a complete example.

VAD Delay

Every VAD declares in delay the longest time it may take to notice that someone started speaking, in milliseconds. Micdrop keeps that much audio in reserve and sends it with the turn, so the first word arrives whole. Silero asks for about 320 ms, the volume VAD for 500 ms, and a combination of VADs for the largest of its members. A custom VAD that needs several samples before it reacts should raise it (the default is 100 ms).

Tuning VAD Performance

Volume VAD Tuning

Adjust sensitivity based on environment:

// Quiet environment - more sensitive
const quietVad = new VolumeVAD({
threshold: -65, // Lower threshold for quiet voices
history: 3, // Faster response
})
// Noisy environment - less sensitive
const noisyVad = new VolumeVAD({
threshold: -45, // Higher threshold to ignore noise
history: 8, // More frames for stability
})

Silero VAD Tuning

Fine-tune AI detection:

// More sensitive - catches quiet speech
const sensitiveVad = new SileroVAD({
positiveSpeechThreshold: 0.15, // Lower threshold
minSpeechFrames: 6, // Faster confirmation
})
// More conservative - reduces false positives
const conservativeVad = new SileroVAD({
positiveSpeechThreshold: 0.22, // Higher threshold
minSpeechFrames: 8, // More confirmation needed
redemptionFrames: 20, // Longer silence confirmation
})

Dynamic VAD Configuration

You can update VAD settings in real-time without restarting:

// Update Volume VAD settings
const volumeVad = Micdrop.vad as VolumeVAD
volumeVad.setOptions({ threshold: -45 })
// Update Silero VAD settings
const sileroVad = Micdrop.vad as SileroVAD
sileroVad.setOptions({ positiveSpeechThreshold: 0.15 })
// Reset to default options
volumeVad.resetOptions()
sileroVad.resetOptions()

Persistent Settings

Both VolumeVAD and SileroVAD settings are automatically saved to localStorage and restored when loading with their names ('volume' or 'silero') and not instances.

React VAD Settings UI

The demo client gives each VAD its own React settings component, ready to copy into your app: VolumeVADSettings for the volume thresholds and SileroVADSettings for the Silero model.