Realtime Models

A realtime model hears the user and answers with its own voice, over a single connection. The answer starts sooner, and the model understands the tone of the voice as well as the words.

Micdrop ships two of them, GeminiLive and OpenaiRealtime. Gemini Live and the OpenAI Realtime API differ in price per minute, session length and reasoning. A pipeline of speech to text, an agent and a voice loses the tone of the voice, but it lets you pick each provider and often costs less per minute.

Usage

Pass the model to the server as realtime:

import { GeminiLive } from '@micdrop/gemini'
import { MicdropServer } from '@micdrop/server'

new MicdropServer(socket, {
  generateFirstMessage: true,
  realtime: new GeminiLive({
    apiKey: process.env.GEMINI_API_KEY || '',
    systemPrompt: 'You are a helpful assistant',
    autoEndCall: true,
  }),
})

OpenaiRealtime takes the same place, with the same options:

import { OpenaiRealtime } from '@micdrop/openai'

new MicdropServer(socket, {
  generateFirstMessage: true,
  realtime: new OpenaiRealtime({
    apiKey: process.env.OPENAI_API_KEY || '',
    systemPrompt: 'You are a helpful assistant',
  }),
})

On the browser side, the client connects to this server like to any other. A React Native app connects the same way, which lets a phone talk to the OpenAI Realtime API without WebRTC.

Options

Every realtime model accepts these options, next to the ones of its provider (API key, model, voice).

How a turn runs

The client listens to the microphone and tells the server when the user starts and stops speaking. The provider’s own voice detection stays off, so the call behaves the same whatever the model.

1. The user starts speaking. The server opens a turn and streams the audio to the model as it arrives.
2. The user stops speaking. The server closes the turn, and the model answers.
3. The voice of the model streams to the client, which plays it.
4. The transcripts of both voices land in the conversation, the question first and the answer after it.

When the user speaks over the answer, the model stops, and the server opens the next turn. OpenaiRealtime also cuts its own memory of the answer where the user most likely stopped hearing it.

Turn detection in the client and the server-side turn detector hold the turn open while the sentence sounds unfinished, so the model answers whole sentences.

Conversation

The model hears audio, and Micdrop writes the conversation from the transcripts the provider sends for each voice. The client receives it as messages, and Message events let you save it.

The transcript of an answer is written as the model speaks. It follows the words heard closely, sometimes with small differences, and joins the conversation when the answer ends. Until then, partial messages show it word by word.

Recording keeps the audio of each message, the voice of the user as well as the voice of the model.

Tools

Tools are added with addTool() and run on your server. The model calls them in the middle of an answer and speaks about their result, and each call is written in the conversation.

import { z } from 'zod'

realtime.addTool({
  name: 'get_weather',
  description: 'Get the current weather of a city',
  inputSchema: z.object({ city: z.string() }),
  execute: async ({ city }) => ({ city, temperature: 21 }),
})

Steering the call

The model can open the call. With generateFirstMessage, it greets the user as its instructions say, and a fixed firstMessage is read out in its voice.

server.speak(text) has the model read a text word for word, for an announcement decided by your application.

realtime.addMessage(role, text) adds a message to the conversation and passes it to the model, which takes it into account from its next answer on. It also rebuilds a previous conversation before the call starts.

Voice and instructions

The model produces its answer directly as audio. Its voice is one of the provider’s, chosen with the voice option, and the system prompt is where you shape what it says and how it says it: length, tone, pace, language.

Session length

Each provider limits the length of a session, and the model reconnects on its own when it ends. GeminiLive resumes the session on a new connection when Gemini asks for it, and OpenaiRealtime opens a new session with the conversation written so far.

Writing your own

Extend the Realtime class of @micdrop/server. It keeps what every provider shares: when a turn opens, which answer the output belongs to, dropping the output of an interrupted answer, and the order of the conversation. A provider implements the calls to its API.

The provider reports what the model does with these protected methods:

The sources of GeminiLive and OpenaiRealtime show both ways a provider can handle tools and interruptions.