---
title: "Realtime Models | Micdrop"
description: "Run a call on a speech-to-speech model such as Gemini Live or the OpenAI Realtime API, in place of speech to text, an agent and a voice, with the same client."
url: "https://micdrop.dev/docs/server/realtime"
---

*   [Getting Started](/docs/getting-started)
*   [Client (Browser)](/docs/client)
    
    *   [Installation](/docs/client/installation)
    *   [React Hooks](/docs/client/react-hooks)
    *   [Start/Stop Call](/docs/client/start-stop-call)
    *   [Pause/Resume Call](/docs/client/pause-resume-call)
    *   [Mute/Unmute Call](/docs/client/mute-unmute-call)
    *   [Call State](/docs/client/call-state)
    *   [Display Conversation Messages](/docs/client/display-conversation-messages)
    *   [Handling Tool Calls](/docs/client/handling-tool-calls)
    *   [Device Management](/docs/client/devices-management)
    *   [Voice Activity Detection (VAD)](/docs/client/vad)
    *   [Turn Detection](/docs/client/turn-detection)
    *   [Reducing Latency](/docs/client/latency)
    *   [Error Handling](/docs/client/error-handling)
    *   Utility Classes
        
        *   [Mic](/docs/client/utility-classes/mic)
        *   [MicdropClient](/docs/client/utility-classes/micdrop-client)
        *   [MicRecorder](/docs/client/utility-classes/mic-recorder)
        *   [Speaker](/docs/client/utility-classes/speaker)
        
    
*   [Client (React Native)](/docs/react-native)
    
    *   [Installation](/docs/react-native/installation)
    *   [Hooks and Call State](/docs/react-native/hooks)
    *   [Audio Output and Devices](/docs/react-native/audio-output)
    *   [Voice Activity Detection (VAD)](/docs/react-native/vad)
    *   [Turn Detection](/docs/react-native/turn-detection)
    *   [Using Another Audio Library](/docs/react-native/custom-audio)
    
*   [Server (Node.js)](/docs/server)
    
    *   [Installation](/docs/server/installation)
    *   [With Fastify](/docs/server/with-fastify)
    *   [With NestJS](/docs/server/with-nestjs)
    *   [Auth and Parameters](/docs/server/auth-and-parameters)
    *   [First Message](/docs/server/first-message)
    *   [Dictation and Text-Only Calls](/docs/server/dictation)
    *   [Realtime Models](/docs/server/realtime)
    *   [Partial Messages](/docs/server/partial-messages)
    *   [Save Messages](/docs/server/save-messages)
    *   [Resume a Conversation](/docs/server/resume-conversation)
    *   [Recording Audio](/docs/server/recording-audio)
    *   [Error Handling](/docs/server/error-handling)
    *   [Tools](/docs/server/tools)
    *   [Extract Value from Answer](/docs/server/extract)
    *   [Auto End Call](/docs/server/auto-end-call)
    *   [Semantic Turn Detection](/docs/server/semantic-turn-detection)
    *   [Noise Filtering](/docs/server/noise-filtering)
    *   [Micdrop Protocol](/docs/server/protocol)
    
*   [AI Integrations](/docs/ai-integration)
    
    *   Provided Integrations
        
        *   [AI SDK](/docs/ai-integration/provided-integrations/ai-sdk)
        *   [Cartesia](/docs/ai-integration/provided-integrations/cartesia)
        *   [ElevenLabs](/docs/ai-integration/provided-integrations/elevenlabs)
        *   [Gemini](/docs/ai-integration/provided-integrations/gemini)
        *   [Gladia](/docs/ai-integration/provided-integrations/gladia)
        *   [Gradium](/docs/ai-integration/provided-integrations/gradium)
        *   [Kokoro](/docs/ai-integration/provided-integrations/kokoro)
        *   [Mistral](/docs/ai-integration/provided-integrations/mistral)
        *   [OpenAI](/docs/ai-integration/provided-integrations/openai)
        *   [Piper](/docs/ai-integration/provided-integrations/piper)
        *   [Pocket TTS](/docs/ai-integration/provided-integrations/pocket-tts)
        *   [Qwen3-TTS](/docs/ai-integration/provided-integrations/qwen-tts)
        *   [Whisper](/docs/ai-integration/provided-integrations/whisper)
        
    *   Custom Integrations
        
        *   [Agent (LLM)](/docs/ai-integration/custom-integrations/custom-agent)
        *   [Speech-to-Text (STT)](/docs/ai-integration/custom-integrations/custom-stt)
        *   [Text-to-Speech (TTS)](/docs/ai-integration/custom-integrations/custom-tts)
        
    *   Fallback Strategies
        
        *   [FallbackAgent](/docs/ai-integration/fallback-strategies/agent-fallback)
        *   [FallbackSTT](/docs/ai-integration/fallback-strategies/stt-fallback)
        *   [FallbackTTS](/docs/ai-integration/fallback-strategies/tts-fallback)
        
    *   [Local Models](/docs/ai-integration/local-models)
        
        *   [Local LLM](/docs/ai-integration/local-models/agent)
        *   [Local STT](/docs/ai-integration/local-models/speech-to-text)
        *   [Local TTS](/docs/ai-integration/local-models/text-to-speech)
        *   [Latency and Memory](/docs/ai-integration/local-models/performance)
        *   [Explorations](/docs/ai-integration/local-models/explorations)
            
            *   [MiniCPM5-2B](/docs/ai-integration/local-models/explorations/minicpm)
            *   [Mistral 7B](/docs/ai-integration/local-models/explorations/mistral-7b)
            *   [Voxtral Mini 3B](/docs/ai-integration/local-models/explorations/voxtral-stt)
            *   [Voxtral TTS 4B](/docs/ai-integration/local-models/explorations/voxtral-tts)
            *   [AuK and AuK-Flash](/docs/ai-integration/local-models/explorations/auk)
            
        
    *   [IA Vocale Souveraine 🇫🇷🇪🇺](/docs/ai-integration/sovereign-voice-ai)
    
*   [Migration](/docs/migration)
    
    *   [Upgrade to v3](/docs/migration/v3)
    

[Micdrop](/) › [Documentation](/docs/getting-started) › [Server (Node.js)](/docs/server)

# Realtime Models

A realtime model hears the user and answers with its own voice. One connection replaces the three parts of a regular call, the speech to text, the agent and the text to speech, which cuts the delay before the answer and keeps the tone of the voice in what the model understands.

Micdrop ships two of them, [`GeminiLive`](/docs/ai-integration/provided-integrations/gemini) and [`OpenaiRealtime`](/docs/ai-integration/provided-integrations/openai#openai-realtime).

## Usage

Pass the model as `realtime`, instead of `stt`, `agent` and `tts`:

```
import { GeminiLive } from '@micdrop/gemini'import { MicdropServer } from '@micdrop/server'
new MicdropServer(socket, {  generateFirstMessage: true,  realtime: new GeminiLive({    apiKey: process.env.GEMINI_API_KEY || '',    systemPrompt: 'You are a helpful assistant',    autoEndCall: true,  }),})
```

The client stays the same. It still detects when the user speaks, and the turn it sends goes to the model as audio rather than to a transcription service.

## What keeps working

A realtime model is an [agent](/docs/ai-integration/custom-integrations/custom-agent): it holds the conversation, takes tools and emits the same events. So the rest of the server works as it does with three separate parts.

*   The conversation is written from the transcripts of both voices, in the order they were spoken, and reaches the client as messages.
*   [Tools](/docs/server/tools) are added with `addTool()` and run on your server.
*   [`firstMessage` and `generateFirstMessage`](/docs/server/first-message) open the call. A fixed first message is read out by the model in its own voice.
*   `server.speak(text)` has the model read a text, word for word.
*   [Partial messages](/docs/server/partial-messages) follow the transcript of the answer.
*   An interruption stops the answer, and OpenAI also forgets the part the user did not hear.
*   [Turn detection](/docs/client/turn-detection) and the [server-side turn detector](/docs/server/semantic-turn-detection) decide when the turn goes to the model.
*   [Recording](/docs/server/recording-audio) and [saving messages](/docs/server/save-messages) read the same events.
*   [`autoEndCall`](/docs/server/auto-end-call), `autoSemanticTurn` and `autoIgnoreUserNoise` are available as options.

## What changes

The model speaks on its own, so the options that edit the text before it is spoken belong to text agents only. `extract` and `onBeforeAnswer` are absent from the options of a realtime model, and the voice is one of the provider’s.

The transcript of an answer is written by the provider as the model speaks. It can differ slightly from the words heard, and it lands in the conversation once the answer ends.

Each provider limits the length of a session. `GeminiLive` resumes the session on a new connection when Gemini asks for it, and `OpenaiRealtime` reconnects with the conversation written so far.

## Writing your own

Extend the `Realtime` class of `@micdrop/server`. It keeps what every provider shares: when a turn opens, which answer the output belongs to, dropping the output of an interrupted answer, and the order of the conversation. A provider implements the calls to its API.

Method

Called when

`openTurn()`

The user starts speaking, right before the first audio.

`appendAudio(chunk)`

Audio of the user arrives, PCM 16 bits, 16 kHz, mono.

`closeTurn()`

The turn ends, and the model has to answer it.

`clearTurn()`

The turn is dropped, when the user mutes the call.

`generate()`

The model speaks first, with no turn of the user to answer.

`speakText(text)`

The model reads a text, word for word.

`cancelAnswer()`

The user interrupts the answer.

`updateTools()`

A tool was added or removed.

`injectMessage(...)`

The application adds a message the model did not hear.

The provider reports what the model does with these protected methods:

Method

Call it when

`emitAudio(chunk)`

Audio of the answer arrives, PCM 16 bits, 16 kHz, mono.

`addAnswerTranscript(text)`

A piece of the transcript of the answer arrives.

`endAnswer()`

The model finished its answer.

`addUserTranscript(text)`

A piece of the transcript of the user arrives.

`commitUserTranscript()`

The transcript of the user is complete.

`runTool(toolCall)`

The model calls a tool, to run it and record it.

The sources of [`GeminiLive`](https://github.com/Godefroy/micdrop/blob/main/packages/gemini/src/GeminiLive.ts) and [`OpenaiRealtime`](https://github.com/Godefroy/micdrop/blob/main/packages/openai/src/OpenaiRealtime.ts) show both ways a provider can handle tools and interruptions.

[Previous← Dictation and Text-Only Calls](/docs/server/dictation)[NextPartial Messages →](/docs/server/partial-messages)

On this page

*   [Usage](#usage)
*   [What keeps working](#what-keeps-working)
*   [What changes](#what-changes)
*   [Writing your own](#writing-your-own)