---
title: "OpenAI Realtime API and Gemini Live in Node.js | Micdrop"
description: "Run a call on a speech-to-speech model such as Gemini Live or the OpenAI Realtime API, in place of speech to text, an agent and a voice, with the same client."
url: "https://micdrop.dev/docs/realtime"
---

# OpenAI Realtime API and Gemini Live in Node.js

A realtime model hears the user and answers with its own voice, over a single connection. The answer starts sooner, and the model understands the tone of the voice as well as the words.

Micdrop ships two of them, [`GeminiLive`](/docs/realtime/gemini-live) and [`OpenaiRealtime`](/docs/realtime/openai-realtime). Gemini Live and the OpenAI Realtime API [differ in price per minute, session length and reasoning](/blog/gpt-live-vs-gpt-realtime-vs-gemini-live). A pipeline of speech to text, an agent and a voice loses the tone of the voice, but it [lets you pick each provider and often costs less per minute](/blog/openai-realtime-api-vs-pipeline).

## Usage

Pass the model to the server as `realtime`:

```typescript
import { GeminiLive } from '@micdrop/gemini'
import { MicdropServer } from '@micdrop/server'

new MicdropServer(socket, {
  generateFirstMessage: true,
  realtime: new GeminiLive({
    apiKey: process.env.GEMINI_API_KEY || '',
    systemPrompt: 'You are a helpful assistant',
    autoEndCall: true,
  }),
})
```

`OpenaiRealtime` takes the same place, with the same options:

```typescript
import { OpenaiRealtime } from '@micdrop/openai'

new MicdropServer(socket, {
  generateFirstMessage: true,
  realtime: new OpenaiRealtime({
    apiKey: process.env.OPENAI_API_KEY || '',
    systemPrompt: 'You are a helpful assistant',
  }),
})
```

On the browser side, the [client](/docs/client) connects to this server like to any other. A React Native app connects the same way, which lets a phone [talk to the OpenAI Realtime API without WebRTC](/blog/openai-realtime-api-react-native).

## Options

Every realtime model accepts these options, next to the ones of its provider (API key, model, voice).

Option

Type

Default

Description

`systemPrompt`

`string`

Required

Instructions of the model

[`autoEndCall`](/docs/server/auto-end-call)

`boolean | string`

`false`

Ends the call when the user asks for it

`autoSemanticTurn`

`boolean | string`

`false`

Lets the model wait silently for the end of a sentence

[`autoIgnoreUserNoise`](/docs/server/noise-filtering)

`boolean | string`

`false`

Drops a turn that carries no meaning, such as a cough or a “hmm”

## How a turn runs

The client listens to the microphone and tells the server when the user starts and stops speaking. The provider’s own voice detection stays off, so the call behaves the same whatever the model.

1.  The user starts speaking. The server opens a turn and streams the audio to the model as it arrives.
2.  The user stops speaking. The server closes the turn, and the model answers.
3.  The voice of the model streams to the client, which plays it.
4.  The transcripts of both voices land in the conversation, the question first and the answer after it.

When the user speaks over the answer, the model stops, and the server opens the next turn. `OpenaiRealtime` also cuts its own memory of the answer where the user most likely stopped hearing it.

[Turn detection in the client](/docs/client/turn-detection) and the [server-side turn detector](/docs/server/semantic-turn-detection) hold the turn open while the sentence sounds unfinished, so the model answers whole sentences.

## Conversation

The model hears audio, and Micdrop writes the conversation from the transcripts the provider sends for each voice. The client receives it as messages, and [`Message` events](/docs/server/save-messages) let you save it.

The transcript of an answer is written as the model speaks. It follows the words heard closely, sometimes with small differences, and joins the conversation when the answer ends. Until then, [partial messages](/docs/server/partial-messages) show it word by word.

[Recording](/docs/server/recording-audio) keeps the audio of each message, the voice of the user as well as the voice of the model.

## Tools

[Tools](/docs/server/tools) are added with `addTool()` and run on your server. The model calls them in the middle of an answer and speaks about their result, and each call is written in the conversation.

```typescript
import { z } from 'zod'

realtime.addTool({
  name: 'get_weather',
  description: 'Get the current weather of a city',
  inputSchema: z.object({ city: z.string() }),
  execute: async ({ city }) => ({ city, temperature: 21 }),
})
```

## Steering the call

The model can open the call. With `generateFirstMessage`, it greets the user as its instructions say, and a fixed [`firstMessage`](/docs/server/first-message) is read out in its voice.

`server.speak(text)` has the model read a text word for word, for an announcement decided by your application.

`realtime.addMessage(role, text)` adds a message to the conversation and passes it to the model, which takes it into account from its next answer on. It also rebuilds a [previous conversation](/docs/server/resume-conversation) before the call starts.

## Voice and instructions

The model produces its answer directly as audio. Its voice is one of the provider’s, chosen with the `voice` option, and the system prompt is where you shape what it says and how it says it: length, tone, pace, language.

## Session length

Each provider limits the length of a session, and the model reconnects on its own when it ends. `GeminiLive` resumes the session on a new connection when Gemini asks for it, and `OpenaiRealtime` opens a new session with the conversation written so far.

## Writing your own

To run one of the other [speech to speech APIs](/blog/top-speech-to-speech-apis-voice-agents), such as Grok Voice or Amazon Nova Sonic, write a class of your own.

Extend the `Realtime` class of `@micdrop/server`. It keeps what every provider shares: when a turn opens, which answer the output belongs to, dropping the output of an interrupted answer, and the order of the conversation. A provider implements the calls to its API.

Method

Called when

`openTurn()`

The user starts speaking, right before the first audio.

`appendAudio(chunk)`

Audio of the user arrives, PCM 16 bits, 16 kHz, mono.

`closeTurn()`

The turn ends, and the model has to answer it.

`clearTurn()`

The turn is dropped, when the user mutes the call.

`generate()`

The model speaks first, with no turn of the user to answer.

`speakText(text)`

The model reads a text, word for word.

`cancelAnswer()`

The user interrupts the answer.

`updateTools()`

A tool was added or removed.

`injectMessage(...)`

The application adds a message the model did not hear.

The provider reports what the model does with these protected methods:

Method

Call it when

`emitAudio(chunk)`

Audio of the answer arrives, PCM 16 bits, 16 kHz, mono.

`addAnswerTranscript(text)`

A piece of the transcript of the answer arrives.

`endAnswer()`

The model finished its answer.

`addUserTranscript(text)`

A piece of the transcript of the user arrives.

`commitUserTranscript()`

The transcript of the user is complete.

`runTool(toolCall)`

The model calls a tool, to run it and record it.

The sources of [`GeminiLive`](https://github.com/Godefroy/micdrop/blob/main/packages/gemini/src/GeminiLive.ts) and [`OpenaiRealtime`](https://github.com/Godefroy/micdrop/blob/main/packages/openai/src/OpenaiRealtime.ts) show both ways a provider can handle tools and interruptions.
