🎤Micdrop

Build a Voice AI Agent with Node.js and TypeScript

A real-time voice AI agent needs a browser client that captures the microphone and detects when the user speaks, and a Node.js server that receives the audio over a WebSocket. On that server, speech to text transcribes what the user says, an LLM writes the answer and text to speech streams it back as audio. Micdrop is an open source TypeScript library for both the browser client and the server. Follow the setup below to get a first call running in five minutes.

How a real-time voice AI agent works

A call is a loop of turns. Each turn goes through six steps:

  1. The browser captures the microphone and runs voice activity detection (VAD) to tell when the user is speaking. Audio leaves the browser only during speech.
  2. A WebSocket carries that audio to your Node.js server. Your provider API keys stay on the server, so the browser never needs a session token.
  3. Speech to text (STT) transcribes the audio as it arrives.
  4. When the user stops speaking, the LLM writes the answer.
  5. Text to speech (TTS) turns the answer into audio while the LLM is still writing it. The server streams that audio to the browser, which plays it.
  6. When the user speaks over the answer, the browser stops playback and the server cancels the LLM and the TTS.

Interruptions need echo cancellation. Without it, the microphone picks up the assistant’s voice. If interruptions stayed on, the assistant would answer itself, so Micdrop turns them off. The browser’s echo cancellation usually removes the assistant’s voice from the microphone signal. With headphones, that voice never reaches the microphone.

A realtime speech to speech model, such as the OpenAI Realtime API or Gemini Live, can replace speech to text, the LLM and text to speech on its own. It answers sooner and hears the tone of the voice. With three separate providers, you choose each one and often pay less per minute. Micdrop lets you switch a call to a realtime model on the server and keep the same client code.

The OpenAI Realtime API and LiveKit connect the browser over WebRTC. Micdrop connects it over a WebSocket to your own backend instead, so you deploy only the server you already run. WebRTC is worth choosing over a WebSocket for group calls, video, phone lines and weak mobile networks.

Micdrop is MIT licensed and runs on your infrastructure with your own provider keys. You build yourself what a hosted platform includes: transcript storage, call history and a dashboard.

Quick Setup

Client (Browser)

Install the browser package and start capturing voice input:

Terminal window
npm install @micdrop/web

If you’re using React, you can also install @micdrop/react package to get a ready-to-use React hooks.

import { Micdrop } from '@micdrop/web'
// Start a voice conversation
Micdrop.start({
url: 'ws://localhost:8081',
})
// Listen for events
Micdrop.on('StateChange', (state) => {
console.log('Conversation:', state.conversation)
console.log('isAssistantSpeaking:', state.isAssistantSpeaking)
})
Micdrop.on('EndCall', () => {
console.log('Call ended by assistant')
})

Client (React Native)

The same call runs on a phone, against the same server:

Terminal window
npm install @micdrop/react-native react-native-audio-api
import { Micdrop, useMicdropState } from '@micdrop/react-native'
function Call() {
const state = useMicdropState()
return (
<Button
title={state.isStarted ? 'Hang up' : 'Start the call'}
onPress={() =>
state.isStarted
? Micdrop.stop()
: Micdrop.start({ url: 'ws://localhost:8081' })
}
/>
)
}

Audio goes through native code there, see React Native for the setup.

Server (Node.js)

Install the server package and set up AI integrations:

Terminal window
npm install @micdrop/server @micdrop/openai @micdrop/gladia @micdrop/elevenlabs

This example uses OpenAI, Gladia and ElevenLabs. You can use other providers with provided or custom integrations.

import { MicdropServer } from '@micdrop/server'
import { OpenaiAgent } from '@micdrop/openai'
import { GladiaSTT } from '@micdrop/gladia'
import { ElevenLabsTTS } from '@micdrop/elevenlabs'
import { WebSocketServer } from 'ws'
const wss = new WebSocketServer({ port: 8081 })
wss.on('connection', (socket) => {
// Handle voice conversation
new MicdropServer(socket, {
firstMessage: 'How can I help you today?',
agent: new OpenaiAgent({
apiKey: process.env.OPENAI_API_KEY,
systemPrompt: 'You are a helpful assistant',
}),
stt: new GladiaSTT({
apiKey: process.env.GLADIA_API_KEY,
}),
tts: new ElevenLabsTTS({
apiKey: process.env.ELEVENLABS_API_KEY,
voiceId: process.env.ELEVENLABS_VOICE_ID,
}),
})
})

Don’t forget to set the API keys in the environment variables.

Examples

To hear Micdrop work, start with the basic example. Every other example app and demo, with the command that runs it, is listed in Examples and demos.

Setup Documentation MCP

To vibe code your Micdrop implementation faster with Cursor, Claude Code or any other agent, install Context7 MCP.

Once the MCP is installed, you can prompt like this:

create a turbo monorepo in current folder with a client and a server in a new /apps folder:
- server: a simple fastify server with Typescript that implements a micdrop server (use context7)
- client: a simple React webapp with micdrop client. show play/stop/pause buttons and conversation, no other UI. make it beautiful and futuristic.

Troubleshooting

Turn on the logs first

Both sides stay quiet by default. On the client, debugLog prints each step of the call: microphone, WebSocket, reconnections.

Micdrop.start({ url: 'ws://localhost:8081', debugLog: true })
Micdrop.on('Error', (error) => {
console.error(error.code, error.message)
})

On the server, every piece takes a Logger. Errors from the providers (a rejected API key, a wrong model name, a quota reached) only show up there, so give one to each piece while you set things up:

import { Logger, MicdropServer } from '@micdrop/server'
const server = new MicdropServer(socket, { agent, stt, tts })
server.logger = new Logger('Server')
agent.logger = new Logger('Agent')
stt.logger = new Logger('STT')
tts.logger = new Logger('TTS')

The error code the client receives tells you where to look next.

The call stops at once with a Mic error

The microphone was refused, and error.message carries the browser’s reason. NotAllowedError means the user or the site settings denied access, NotFoundError means no microphone is plugged in.

Browsers only give the microphone to secure pages, served over HTTPS or from localhost. Opening your dev server from a phone at http://192.168.1.20:3000 fails for this reason, so go through an HTTPS tunnel instead. On React Native, the permission has to be declared in the app, see Installation.

Connection errors, or a client that keeps reconnecting

The client gives the server 5 seconds to answer, then retries every second with no limit (see the reconnect option). The usual causes:

  • The URL misses the route. Behind Fastify or NestJS, the socket lives on a path such as ws://localhost:8081/call.
  • An HTTPS page has to connect with wss://, browsers block ws:// from it. A reverse proxy in front of the server must forward the Upgrade and Connection headers.
  • A phone has its own localhost, so it needs your machine’s address on the network, as explained in Reaching the server.

InternalServer, Unauthorized, BadRequest or NotFound

These come from the server closing the socket. InternalServer means your connection handler threw: handleError prints the error in the server console, most often a missing environment variable. The other three are the MicdropError your own code throws when it checks the parameters and authentication, and the client does not retry them.

The call connects but the assistant stays silent

A provider is failing, and the server logs tell you which one. Follow a turn through them:

  1. With firstMessage set, hearing it proves the speech synthesis and the speakers work.
  2. Received chunk lines show the audio arrives. None while you talk means the client never detected your voice, see the next section.
  3. User transcript shows speech to text works. Its absence points at the STT key or language.
  4. Adding assistant message shows the agent answered. Its absence points at the LLM key, model name or quota.
  5. Send audio chunk shows the answer is being spoken. Its absence points at the TTS key or voice ID.

A provider that gives up after its retries emits Failed, and the call stays open without a word. In production, put a fallback behind each piece: STT, agent and TTS.

The assistant does not hear you, or never stops listening

Voice detection runs in the browser, and the volume detector used by default depends on the room and the microphone. A quiet microphone never crosses the threshold, a noisy room never drops below it. Tune it or switch to Silero, which recognises human speech, as described in VAD. Check also that the right input is selected with device management, and that the call is neither muted nor paused.

The assistant interrupts itself, or you cannot interrupt it

Without echo cancellation, the microphone hears the speakers and the assistant would answer its own voice. Micdrop then turns interruptions off (isInterruptionDisabled in the state), and debugLog prints No echo cancellation, interruptions are disabled. Headphones fix it, and so does a microphone the browser can cancel echo on (some USB and Bluetooth headsets lose it).

Some browsers report echo cancellation on and hear the assistant anyway. A call detects it during the first seconds of the first answer and adapts on its own, which can end with interruptions off for that call. Micdrop.echoStage tells where it stands, see Echo and interruptions.

No sound on Safari or iOS

Browsers keep audio locked until the user interacts with the page. Call Micdrop.start() from a click or tap handler, never on page load.

Answers come too late

Most of the delay is the silence the client waits for before it ends your turn. Latency breaks down a turn and lists the settings worth changing.

Next steps