Voice Activity Detection (VAD)

Audio is only sent to the server while someone is speaking, and what decides that is the VAD. It is the same code on a phone as in a browser, in @micdrop/client, so the options, the events, the way several detectors combine and how to write one of your own are all in Voice Activity Detection. This page covers what a phone does differently.

Volume detection, the default

VolumeVAD needs nothing beyond the package and is what a call uses when you say nothing:

import { Micdrop, VolumeVAD } from '@micdrop/react-native'

await Micdrop.start({
  url: 'wss://example.com/call',
  vad: new VolumeVAD({ threshold: -55, history: 5 }),
})

A browser saves these options to localStorage and finds them again on the next visit. A phone remembers nothing until the app says where to write, with a store that answers synchronously since a VAD is built with its saved options in hand:

import { setMicdropStorage } from '@micdrop/react-native'
import { MMKV } from 'react-native-mmkv'

const mmkv = new MMKV()

setMicdropStorage({
  getItem: (key) => mmkv.getString(key) ?? null,
  setItem: (key, value) => mmkv.set(key, value),
  removeItem: (key) => mmkv.delete(key),
})

The chosen microphone and output are kept there too.

Silero

SileroVAD runs the same model as in a browser, on the native ONNX runtime. It hears the difference between a voice and a noise, where the volume detection only hears how loud the room is, which is worth it in a car or in a street.

npx expo install onnxruntime-react-native

The native runtime adds a good chunk to the app, so it is only linked in when you import it:

import '@micdrop/react-native/silero'

await Micdrop.start({ url: 'wss://example.com/call', vad: 'silero' })

The model is about two megabytes, fetched once on the first call and kept for as long as the app runs. To ship it with the app instead:

import { setSileroOptions } from '@micdrop/react-native/silero'

setSileroOptions({ model: modelPathOnTheDevice })

model takes an address to fetch, a file already on the device, or the bytes themselves. Everything else about tuning the model is the same on both platforms, see Silero VAD.

Knowing when the turn is over

Voice detection says whether someone is speaking. Deciding that the turn is over then means waiting for a fixed amount of silence, which cuts people off mid-hesitation or makes every answer late.

Turn detection reads the sound of the sentence instead, and runs on the same native ONNX runtime as Silero.