Voice Activity Detection (VAD)

Audio is only sent to the server while someone is speaking. What decides that is the VAD, and on React Native it follows the level of the microphone.

For the concepts behind it, and how volume detection compares with a model, read Voice Activity Detection in the Browser.

The default

VolumeVAD measures the microphone about ten times a second and needs a few loud samples in a row before opening a turn, so a door slam or a keyboard click does not start one. The turn closes once the room has been quiet for about half a second.

The level it follows is the loudest frequency rather than the overall energy: a voice concentrates its power in a few bands while a fan spreads it over all of them, which separates the two by about 25 dB instead of the 7 dB a plain average would give.

import { Micdrop, VolumeVAD } from '@micdrop/react-native'

await Micdrop.start({
  url: 'wss://example.com/call',
  vad: new VolumeVAD({
    threshold: -55, // level above which audio counts as speech
    history: 5, // samples kept to make up its mind
  }),
})

Lower the threshold in a quiet room to catch a soft voice, raise it in a noisy one. Passing 'volume' instead of an instance uses the defaults.

Micdrop keeps the last moments of audio in reserve, so the syllable spoken before the VAD reacted is sent along with the rest of the sentence.

Silero

SileroVAD runs the same model as in a browser, on the native ONNX runtime. It hears the difference between a voice and a noise, where the volume detection only hears how loud the room is, which is worth it in a car or in a street.

npx expo install onnxruntime-react-native

The native runtime adds a good chunk to the app, so it is only linked in when you import it:

import '@micdrop/react-native/silero'

await Micdrop.start({ url: 'wss://example.com/call', vad: 'silero' })

The model is about two megabytes, fetched once on the first call and kept for as long as the app runs. To ship it with the app instead:

import { setSileroOptions } from '@micdrop/react-native/silero'

setSileroOptions({ model: modelPathOnTheDevice })

The state machine that turns the model’s answers into turns is the very same code the browser runs, in @micdrop/client, so the two platforms behave alike once the model is loaded.

Your own detection

The VAD class is exported, so another method can be plugged in. It listens to the microphone and reports four moments: speech may have started, it is confirmed, it was only noise, it has ended.

import { MicSource, VAD } from '@micdrop/react-native'

class MyVAD extends VAD {
  private mic?: MicSource

  get isStarted() {
    return !!this.mic
  }

  get isPaused() {
    return false
  }

  async start(mic: MicSource) {
    this.mic = mic
    mic.on('Frames', this.onFrames)
  }

  async stop() {
    this.mic?.off('Frames', this.onFrames)
    this.mic = undefined
  }

  async pause() {}
  async resume() {}

  private onFrames = (frames: Float32Array, sampleRate: number) => {
    // Decide, then report
    // this.emit('StartSpeaking')
    // this.emit('ConfirmSpeaking')
    // this.emit('CancelSpeaking')
    // this.emit('StopSpeaking')
  }
}

Frames carries mono samples between -1 and 1, Volume carries the level in dBFS. Several detectors can be combined, a turn then opens when any of them hears something and closes when all of them agree it is over:

await Micdrop.start({
  url: 'wss://example.com/call',
  vad: ['volume', new MyVAD()],
})