Whisper

Local speech to text for @micdrop/server, running Whisper in your Node process through Transformers.js and ONNX Runtime.

Nothing leaves the machine, and running it takes no API key and no server to start next to your own. The weights are downloaded on first use and cached, then shared by every call in the process.

Installation

npm install @micdrop/whisper

Usage

import { WhisperSTT } from '@micdrop/whisper'
import { MicdropServer } from '@micdrop/server'

const stt = new WhisperSTT({
  model: 'base',
  language: 'en',
})

// Use with MicdropServer
new MicdropServer(socket, {
  stt,
  // ... other options
})

Options

Models

The shorthands point at the community ONNX exports. Any other repository holding an ONNX export of Whisper works too.

Latencies were measured on a MacBook Pro M2 with 24 GB of memory, on CPU. Most of each latency is fixed. Whisper reads thirty seconds of audio whatever the length of the utterance, so a one second sentence costs nearly as much as a ten second one. Only the decoding scales with what is being said.

Picking a model for another language

A checkpoint fine-tuned on one language works better than a much heavier generic one. Over five French sentences carrying proper nouns, numbers and homophones, on the same machine:

french costs what small costs and reads French better than turbo, which weighs twice as much and answers three times slower. Part of the gap is that the generic checkpoints write "17h30" where the French one writes "dix-sept heures trente". A voice agent wants the spelled out form, since the answer will be read out loud.

The other languages have no shorthand yet. Search the Hugging Face hub for a whisper-small fine-tuned on your language and exported to ONNX, then pass its repository id as model for the same benefit.

How the audio is read

Micdrop hands over one utterance at a time, already cut by the client’s voice activity detection, so the model reads the whole utterance in one pass when the stream ends rather than as it arrives. The result comes back a few hundred milliseconds after the user stops speaking.

Hallucinations on silence

Whisper describes what it hears even when nobody has spoken, so a breath or a door closing comes back as a sound tag such as [BLANK_AUDIO], a music note, or one of the subtitle credits its training data is full of. The voice activity detection fires on those sounds too, so they show up regularly.

A transcript carrying no speech at all comes back as an empty string, so the server skips the answer. A transcript mixing a tag and real words keeps its words. Set filterHallucinations to false to receive the raw text.

Devices

cpu is the default and, on a Mac, the fastest option, since Transformers.js exposes no GPU device on macOS and the CoreML provider reached directly is three to five times slower on the Whisper encoder. On Windows dml uses DirectML, and on Linux with an NVIDIA card cuda is worth trying. The local models guide lists the measurements.

Memory and startup

The weights are loaded once per configuration and shared by every call, so a second concurrent call costs nothing more. Loading takes from one to a few seconds depending on the model, and the first inference pays for the graph warm-up, which is why warmup runs one pass on silence while the call is being set up.

The first run of a given model downloads it, so that run takes much longer than the ones after it.

Documentation

Read the guide on running Micdrop with local models to see how Whisper fits with a local LLM and a local voice.

License

Whisper is released by OpenAI under the MIT license.