OpenAI Realtime API or a swappable STT → LLM → TTS pipeline?

You are adding a voice mode to a web application, and the first architecture decision splits in two. Either you open a single connection to a speech-to-speech model and let it hear and answer, or you assemble three services: speech-to-text, a language model, text-to-speech. OpenAI’s Realtime API made speech to speech easy enough that plenty of teams take it without weighing the pipeline. Both are good answers, to different products.

The Realtime API and Micdrop can share a codebase. The Realtime API is a model endpoint, Micdrop is a TypeScript library that orchestrates voice conversations, and it already uses that endpoint to transcribe. The question this article answers is where the audio should go.

What the Realtime API gives you in one connection

You open one session, stream audio in as the microphone captures it, and receive audio frames back for playback. The model hears the user, generates the answer and speaks it, with no orchestration code handing text between services.

You can open that session over three transports. WebRTC suits browser and mobile clients that capture and play audio directly. WebSocket suits a server that already receives raw audio from a media pipeline or a call system. SIP connects the model to a phone number. Function calling and remote MCP servers work inside the session, so the model can query your database mid-sentence. A transcription-only session type also exists, for applications that want the text without a spoken answer.

Prosody stays intact. A hesitation, a laugh, a sarcastic lilt, someone trailing off before finishing a thought: the model receives all of it as sound. A pipeline transcribes first, and “um, I think, no wait, actually yes” reaches the model as a flat string with every acoustic clue removed. For a language tutor correcting pronunciation, or an interview agent reading how confident someone sounds, hearing the audio is the feature itself.

graph TD
%% Mermaid lays disconnected subgraphs out in reverse declaration order, so the
%% pipeline is declared first to render on the right. direction TB keeps each
%% flow vertical, which a subgraph does not inherit from the parent graph.
    subgraph PL[Swappable pipeline]
        direction TB
        A2[Browser audio] --> B2[Speech-to-text]
        B2 --> C2[Language model]
        C2 --> D2[Text-to-speech]
        D2 --> E2[Audio answer]
    end
    subgraph SP[Speech to speech]
        direction TB
        A1[Browser audio] --> B1[Realtime session]
        B1 --> C1[Audio answer]
    end

What one connection costs you

Provider choice collapses to one. Hearing, reasoning and speaking come from the same provider, on the same bill, in the same regions. A price change, a quality regression on a new model snapshot or an incident hits all three functions at once, and you have no second path to route around it.

The voice list is short. The realtime models ship a handful of voices, two of them exclusive to the current generation and the others inherited from earlier snapshots. ElevenLabs publishes a library of more than ten thousand community voices and Cartesia around five hundred, both with cloning. If your product needs a recognisable brand voice, or a specific accent in a language OpenAI’s voices pronounce unnaturally, you need a text-to-speech provider of your own.

Everything is billed as audio. OpenAI’s pricing page lists the full-size realtime models at $32 per million audio input tokens and $64 per million audio output tokens, with a mini tier at $10 and $20, as of August 2026.

Data residency gets harder. OpenAI offers regional processing in Europe for eligible projects, with zero data retention, and its own data controls guide notes that tracing is not currently EU data residency compliant for the realtime endpoint. Beyond the endpoint itself, a single-provider architecture rules out a fully European stack, transcription, model and voice included.

The transcript is a byproduct. In a pipeline the text is the substrate, already there for your logs, your evaluations, your compliance archive and your prompt iteration. Speech to speech gives you audio, and you collect the text from the session events.

When speech to speech wins

Pick the Realtime API when the acoustics carry meaning. Language learning, accent coaching, roleplay and emotional register all need the model to hear how something was said rather than what was said.

Pick it when your latency budget is the product. Each stage of a pipeline adds a wait, and the model cannot start reasoning until the transcription of the turn is final. A single session removes those waits.

Take it when you want the fewest moving parts. A browser connecting over WebRTC with an ephemeral key needs almost no backend, and a phone agent over SIP needs no telephony stack of your own. For a prototype that has to exist by Friday, that is the whole argument.

It also fits a team already committed to OpenAI everywhere else. If your product runs on their models, their embeddings and their infrastructure, the provider diversity a pipeline buys you is a benefit you were never going to use.

When a swappable pipeline wins

One of the three stages usually turns out to have a requirement of its own. A specific voice is the most common. Outside English, a dedicated transcription provider is often more accurate than a general-purpose one.

Cost is the largest difference, and OpenAI publishes every rate it needs. Audio counts as one token per 100 ms it hears and one token per 50 ms it speaks, so a minute of conversation shared evenly between the user and the agent is 300 tokens in and 600 tokens out.

Those figures come from audio at $32 and $64 per million tokens on gpt-realtime, $0.017 a minute on gpt-live-transcribe, $12 per million on gpt-4o-mini-tts, $0.05 and $0.40 per million on gpt-5-nano over four turns of 1,500 tokens, $0.75 an hour on Gladia streaming, $0.50 and $1.50 per million on Mistral Large, and Gradium’s S plan at $43 for 900,000 characters, which its own count of 45,000 characters an hour puts at 375 characters for that half minute.

Speech billed as audio costs several times the same content billed as text, and one rate applies to every turn, however trivial. A pipeline prices each stage on its own, so you buy each one from whoever fits the requirement. On small OpenAI models that lands about three times below the realtime session. The European stack costs more than that and still undercuts the single session, because a premium voice is the expensive stage and Gradium charges for it. Paying more on one stage while staying cheaper overall is the whole point of pricing them separately. Live transcription bills by audio duration and costs more than transcribing a recording afterwards, so a batch price list makes a pipeline look cheaper than it is.

Both architectures re-send the conversation history on every turn, and they pay for it differently. The pipeline pays in text tokens, which the reasoning figure already includes. A realtime session pays in audio tokens, cached at a discount but still added on top, which is what makes long calls diverge. The mini realtime tier also closes part of the difference on its own, at $10 and $20 per million.

Regulation leads to a pipeline too. A European deployment can run Mistral for the model, Gladia for transcription and Gradium for the voice, where the data stays in the EU and every company in the chain is European. Regional processing on a US endpoint moves the servers, and the chain stays American.

Resilience is the argument teams discover last and care about most. When the three services are separate, one of them failing takes down one stage instead of the conversation. Micdrop’s FallbackAgent switches to a second model mid-conversation, from the same history, and FallbackSTT and FallbackTTS do the same for transcription and voice. A single session has nowhere to fall back to.

Driving the Realtime API from TypeScript with Micdrop

Micdrop connects to the Realtime endpoint today, in transcription mode. OpenaiSTT opens a WebSocket to wss://api.openai.com/v1/realtime, streams the browser’s audio into it after resampling to the rate the endpoint expects, and emits transcripts as they arrive:

import { MicdropServer } from '@micdrop/server'
import { OpenaiAgent, OpenaiSTT, OpenaiTTS } from '@micdrop/openai'

new MicdropServer(socket, {
  stt: new OpenaiSTT({ apiKey: process.env.OPENAI_API_KEY, language: 'en' }),
  agent: new OpenaiAgent({
    apiKey: process.env.OPENAI_API_KEY,
    systemPrompt: 'You are a helpful assistant',
  }),
  tts: new OpenaiTTS({ apiKey: process.env.OPENAI_API_KEY, voice: 'alloy' }),
})

That is an all-OpenAI stack, running as a pipeline. You get OpenAI’s realtime transcription quality and their voices, and you keep three separate arguments you can change one at a time. The OpenAI integration page documents every option of the three classes.

Micdrop stops short of running the speech-to-speech model as an agent. Its Agent base class takes conversation history and writes text into a stream, which the text-to-speech stage then speaks, so a model that answers in audio does not fit that interface. For a full speech-to-speech session, use OpenAI’s own SDK and connect the browser over WebRTC. Micdrop covers the pipeline architecture, in TypeScript on both sides of the socket.

Switching providers without rewriting the orchestration

The reason to keep the three stages separate shows up the day one of them disappoints you. Changing a provider is one import and one constructor, and everything around it stays as it was:

import { GladiaSTT } from '@micdrop/gladia'
import { ElevenLabsTTS } from '@micdrop/elevenlabs'

new MicdropServer(socket, {
  stt: new GladiaSTT({ apiKey: process.env.GLADIA_API_KEY }),
  agent, // unchanged
  tts: new ElevenLabsTTS({
    apiKey: process.env.ELEVENLABS_API_KEY,
    voiceId: 'your-voice-id',
  }),
})

Your voice activity detection, your interruption handling, your tools and your UI keep working. The AI integration guide lists the providers that ship with Micdrop, and the abstract classes behind them let you write your own when your provider is not on the list.

Picking an architecture

graph TD
    A[Voice feature to build] --> B{Tone or hesitation<br>changes the answer?}
    B -->|Yes| R[Realtime API]
    B -->|No| C{Specific voice or<br>European stack?}
    C -->|Yes| P[Swappable pipeline]
    C -->|No| D{Failover across<br>providers?}
    D -->|Yes| P
    D -->|No| E{Backend already<br>all OpenAI?}
    E -->|Yes| R
    E -->|No| P

If you are still unsure, build the first version on whichever architecture you can get running this week, and keep your prompt and your tool logic in your own repository rather than in a session configuration. Both architectures read the same prompt and call the same functions, so changing your mind later means swapping the audio path rather than rewriting the product.

The getting started guide gets a browser talking to a Node server in about five minutes. From there, the OpenAI integration page covers the agent, transcription and voice classes, so the pipeline runs on OpenAI’s models with any one of them swappable later.

Frequently asked questions