Audio sampling rate in a voice pipeline, from the microphone to the speech model and back

The audio sampling rate is the number of times per second a recording measures the sound wave, given in hertz: 16,000 measurements a second is 16 kHz. A voice agent handles three rates at once. The browser microphone records at 44.1 or 48 kHz, most speech to text models read 16 kHz (OpenAI’s Realtime API wants 24 kHz), and voice engines generate at 16, 22.05 or 24 kHz. Each boundary between two of them needs a conversion. A rate declared wrong at one boundary is the usual reason an agent mishears its user or speaks like a slowed tape. This article lists the rate each stage uses, what a conversion costs, why sending more than the model reads brings nothing, and which format to stream to a speech model.

What the audio sampling rate measures

A microphone produces a continuous wave. To store it, the sound card measures its height at regular intervals. Each measurement is a sample. The sampling rate says how many samples are taken per second.

The rate sets the highest frequency the recording can hold, which is half the rate. Engineers call that limit the Nyquist frequency. A 16 kHz recording carries sound up to 8 kHz, and a 48 kHz one up to 24 kHz, beyond what an adult hears. The human voice carries most of what makes words intelligible between 300 Hz and 3,400 Hz, the band the telephone network was designed for. Consonants such as “s” and “f” put energy higher, up to about 8 kHz, which is why speech recognition settled on 16 kHz.

The browser microphone records at the rate of the device

getUserMedia accepts a sampleRate constraint, but Chrome ignores it. A Chromium change from 2017 removed the sample rate, sample size and channel count constraints from the device selection code, because that code only looked at what the hardware reports. You ask for 16 kHz and receive the rate of the device, usually 48 kHz, sometimes 44.1 kHz.

The Web Audio API adds a second rate. An AudioContext runs at the preferred rate of the output device unless you pass one. Chrome accepts new AudioContext({ sampleRate: 16000 }) and converts the microphone on the fly. Firefox refuses: connecting a microphone stream to a context at another rate throws a NotSupportedError, which its own test suite checks. Bluetooth headsets add a third case. In call mode they switch to the hands-free profile, which carries the microphone at 16 kHz, or 8 kHz on older devices.

So the code that captures the microphone has to read the rate it actually gets, every time. An AudioWorklet sees it as a global named sampleRate and can send it along with the samples:

class PcmProcessor extends AudioWorkletProcessor {
  process(inputs) {
    const channel = inputs[0]?.[0]
    if (channel) {
      // `sampleRate` is the real rate of the context, 44100 or 48000 in practice
      this.port.postMessage({ frames: new Float32Array(channel), sampleRate })
    }
    return true
  }
}

registerProcessor('pcm-processor', PcmProcessor)

The page then converts those frames to the rate the server expects, once, before they leave the browser. Sending 16 kHz rather than 48 kHz divides the upload by three. A neural voice activity detector in the browser such as Silero needs the conversion anyway, since it reads 8 or 16 kHz only.

The sample rate each speech model and voice expects

Each provider documents its own rate, which rarely matches the next one’s. Micdrop, an open source TypeScript library for real-time voice conversations with AI, integrates most of these engines, so we read their rates straight from its code. AssemblyAI, Deepgram and Twilio appear as well, since you will meet them in other stacks.

Speech to text models read audio at these rates:

Text to speech engines generate audio at these rates:

The hosted voices let you ask for the rate you need, so the cheapest conversion is the one the provider does for you. Local engines generate at their own rate, which differs from one local text to speech engine to another. Kokoro and Qwen3-TTS produce 24 kHz, while Piper changes rate from one voice to the next and writes it in the voice’s JSON file. A voice stack that runs entirely on your machine has to convert each of them on the server.

A sample rate mismatch makes the voice slow and deep, or fast and high

Raw audio carries no label. A stream of 16 bit numbers holds no record of how many of them make a second, so the receiver uses the rate you declare. When you declare the wrong rate, the audio plays at the wrong speed and at the wrong pitch.

Audio recorded at 48 kHz and declared as 16 kHz lasts three times longer and sounds more than an octave lower. A speech model fed with it returns nonsense, or nothing. The reverse turns the voice into a fast, high-pitched squeak. A 24 kHz voice played at 16 kHz is 1.5 times slower and seven semitones lower. Every word stays intelligible, so it is tempting to blame the voice engine.

The 44.1 against 48 kHz confusion is the hardest to spot. Audio recorded at 44.1 kHz and declared as 48 kHz plays 8.8% faster and about one and a half semitones higher. Everyone still understands it, so it can stay unnoticed through a whole demo.

The quickest check is to listen. Dump the raw bytes your server receives to a file and play them at the rate you declared, here with ffmpeg’s player:

ffplay -f s16le -ar 16000 -ch_layout mono received.pcm

If the voice sounds normal, the rate is right. If it sounds too slow or too fast, fix the step that declared the rate.

What a resampling step costs

Converting audio from one rate to another computes new samples between the old ones. The simplest method, linear interpolation, reads two neighbouring samples and takes a weighted average. Going from 16 to 24 kHz that means 24,000 small calculations a second per call, a negligible load for a server. The delay is just as small. A streaming resampler keeps only the last sample of a chunk to compute the next one, which holds back 62.5 microseconds of audio at 16 kHz.

The hard part is keeping track of the position. Audio arrives in chunks, sometimes with a 16 bit sample cut in two by the network, so the resampler has to carry its position between two samples over from one chunk to the next. A resampler that starts from zero on each chunk adds a tiny jump every 100 ms, which you hear as a regular crackle. Micdrop’s server package exports Pcm16Resampler, which keeps that state between chunks, so feeding it a stream chunk by chunk gives the same audio as converting it all at once:

import { Pcm16Resampler } from '@micdrop/server'

// A voice engine that returns 24 kHz PCM, sent to a client that plays 16 kHz
const resampler = new Pcm16Resampler(24000, 16000)

for await (const chunk of engineAudio) {
  socket.send(resampler.process(chunk))
}

// The next sentence is a new stream, start it from a clean state
resampler.reset()

Lowering the rate also costs some quality. Before dropping from 48 to 16 kHz, a careful resampler filters out everything above 8 kHz, because sound above the new limit folds back into the audible band as noise, an effect called aliasing. Linear interpolation skips that filter. Micdrop accepts that loss, since speech carries little energy above 8 kHz and the conversion has to run on every chunk in the browser. For music, or a recording you keep, use a filtering resampler such as libsamplerate, the Speex resampler or ffmpeg’s aresample.

Convert as few times as you can. Each conversion approximates the samples it computes. Going up also adds no information: 16 kHz audio converted to 24 kHz for OpenAI still stops at 8 kHz.

Audio above the model’s rate only adds bytes

Whisper converts whatever you give it to 16 kHz before computing its spectrogram, and most streaming speech models do the same on their side. Sending 48 kHz audio means paying for three times the bytes so the provider can throw two thirds of them away.

The bytes matter on the upload, which is the narrow side of a mobile connection. Raw 16 bit mono PCM weighs 32 KB a second at 16 kHz and 96 KB a second at 48 kHz. An API that takes audio as base64 inside JSON adds a third on top. A fibre line carries that easily. On a weak 4G uplink, a phone takes longer to send it, which delays the moment the model hears the end of the sentence.

The voice going back to the user is the one place where a higher rate can be heard. A 24 kHz voice keeps sound up to 12 kHz, and on good headphones it sounds crisper than the same voice at 16 kHz. Micdrop sends 16 kHz both ways so the client handles a single format. The browser then converts that audio to the rate of the speaker for playback.

PCM audio is the format a voice pipeline carries

PCM, for pulse code modulation, stores each sample as a plain number, one after the other, with no compression. Three settings describe a PCM stream, and both ends have to agree on all three:

• the sampling rate, such as 16 kHz,
• the bit depth, the size of each number,
• the number of channels.

Speech pipelines use 16 bit signed integers, written little endian, which providers call pcm_s16le, linear16 or audio/pcm. Sixteen bits give 65,536 levels and about 96 dB between the quietest and the loudest sound, more than a microphone in a room delivers. The Web Audio API hands out 32 bit floats between -1 and 1, so the browser converts them before sending:

function floatToPcm16(input: Float32Array): Int16Array {
  const output = new Int16Array(input.length)
  for (let i = 0; i < input.length; i++) {
    const sample = Math.max(-1, Math.min(1, input[i]))
    output[i] = sample < 0 ? sample * 0x8000 : sample * 0x7fff
  }
  return output
}

Speech uses one channel. A second channel doubles the bytes and gives a speech model nothing it can use, so record and send mono.

A WAV file is PCM with a 44 byte header in front, which holds the rate, the depth and the channel count. A raw PCM stream drops that header, so the rate travels in the provider’s parameters or in your own protocol. The slow, deep voices and the high squeaks of a rate mismatch come from that missing header.

The best audio format to send to a speech to text API

For a live conversation, send raw 16 bit mono PCM at the rate the model reads: 16 kHz for most providers, 24 kHz for OpenAI. The provider reads it without decoding, every streaming API accepts it, and a 100 ms chunk at 16 kHz weighs 3.2 KB.

For a recorded file, a lossless format at 16 kHz or more keeps everything the model needs. FLAC compresses to about half the size of the same WAV without losing a sample.

Compressed formats trade bytes for quality. Opus keeps a voice clear at a few tens of kilobits per second, which is why WebRTC uses it. Some speech APIs accept it directly. A low bitrate MP3 removes detail the model would have used. Phone audio arrives as 8 kHz μ-law. Converting it to 16 kHz restores nothing above 4 kHz, so pick a model trained for telephone audio rather than resampling it. Sending PCM over a WebSocket or Opus over WebRTC is a separate choice, with its own trade-offs between WebSocket and WebRTC for a voice agent.

How Micdrop handles sample rates

Micdrop picks one format for the whole conversation: 16 bit mono PCM at 16 kHz, from the browser to the server and back. Every integration converts to and from it, so changing provider never changes what the browser sends.

On the browser side, @micdrop/client reads the rate the device actually delivers, converts each 100 ms of speech to 16 bit PCM at 16 kHz and sends it over the WebSocket. It adapts when the rate changes during a call, for instance when a Bluetooth headset connects. For playback, it converts the 16 kHz audio it receives to the rate of the speaker. All of that happens inside the Micdrop browser client, which also manages the microphone and the speaker.

On the server side, each integration of @micdrop/server declares the rate its provider expects. The OpenAI transcription and Realtime integrations convert 16 kHz to 24 kHz on the way in, and OpenAI TTS, Gemini, Kokoro, Piper, Pocket TTS and Qwen3-TTS convert their output to 16 kHz on the way out. ElevenLabs, Cartesia and Gradium are asked for 16 kHz directly.

graph TD
    A[Microphone at 44.1 or 48 kHz] --> B[Browser converts to 16 kHz, 16 bit mono PCM]
    B -->|WebSocket, 32 KB per second| C[Node server]
    C -->|16 kHz as is| D[Whisper, Voxtral, Gemini, Gladia]
    C -->|Converted to 24 kHz| E[OpenAI transcription and Realtime]
    F[Voice at 16, 22.05 or 24 kHz] -->|Converted to 16 kHz| C
    C -->|WebSocket, 16 kHz| G[Browser converts to the speaker rate]

An integration you write yourself emits 16 kHz PCM, with Pcm16Resampler converting the output of a provider that speaks at another rate. You then pass your integration to MicdropServer like a bundled one.

Doing the same work by hand means writing an AudioWorklet, the float to 16 bit conversion and a resampler that keeps its position between chunks, then handling a rate change in the middle of a call and one conversion per provider and per direction. A hosted platform handles it for you and charges by the minute: ElevenLabs Agents charged $0.08 a minute on every plan, with the LLM billed on top, when we read its pricing on September 24, 2026. Micdrop is MIT licensed and runs on your server with your own provider keys. In exchange, you host that server yourself and build the call history and the dashboard a platform would give you.

Frequently asked questions

A voice agent uses three sampling rates: the device’s rate in the browser, 16 kHz for most speech models, and 16 to 24 kHz for the voices. Read the real rate at the microphone, convert once to what the model reads, keep the resampler’s state between chunks, and send 16 bit mono PCM.

Micdrop does that conversion in the browser and in each server integration, so your code only sees 16 kHz audio. You can run a first voice call in about five minutes, then plug in the speech to text and the voice of your choice.