🎤Micdrop

ElevenLabs TTS in Node.js, with Eleven v4 Turbo

@micdrop/elevenlabs streams ElevenLabs text to speech from a Node.js server, for @micdrop/server. It sends the agent’s answer to ElevenLabs over a WebSocket as the agent writes it, so the audio comes back before the answer is finished. The package works with Eleven v4 Turbo, Eleven v4, Eleven v3, Flash v2.5, Turbo v2.5 and Multilingual v2, and picks the WebSocket each model needs.

ElevenLabs and every other voice engine in Micdrop implement the same interface, so moving to an ElevenLabs alternative only means replacing new ElevenLabsTTS(...) with the other engine’s class.

Installation

Terminal window
npm install @micdrop/elevenlabs

ElevenLabs TTS (Text-to-Speech)

Usage with MicdropServer

import { ElevenLabsTTS } from '@micdrop/elevenlabs'
import { MicdropServer } from '@micdrop/server'
const tts = new ElevenLabsTTS({
apiKey: process.env.ELEVENLABS_API_KEY || '',
voiceId: '21m00Tcm4TlvDq8ikWAM', // ElevenLabs voice ID
modelId: 'eleven_v4_turbo', // Optional: model to use
language: 'en', // Optional: language code
voiceSettings: {
stability: 0.5,
},
})
// Use with MicdropServer
new MicdropServer(socket, {
tts,
// ... other options
})

Usage without MicdropServer

import { ElevenLabsTTS } from '@micdrop/elevenlabs'
import { Readable } from 'stream'
const tts = new ElevenLabsTTS({
apiKey: process.env.ELEVENLABS_API_KEY || '',
voiceId: '21m00Tcm4TlvDq8ikWAM',
})
// Audio is raw PCM, 16 bits, 16 kHz, mono
tts.on('Audio', (chunk) => console.log('Audio:', chunk.length, 'bytes'))
tts.on('Failed', (texts) => console.error('Failed:', texts))
tts.speak(Readable.from(['Hello! ', 'What can I do for you?']))

Events

EventPayloadDescription
AudioBufferA chunk of audio, PCM 16 bits, 16 kHz, mono, ready to be played.
Failedstring[]The text left unspoken when ElevenLabsTTS gives up after its retries.

See the TTS interface for the full contract.

Choosing a model

ElevenLabs streams speech over two WebSockets, each with its own protocol. Eleven v3 and v4 only stream over the Text to Dialogue WebSocket, while Flash v2.5, Turbo v2.5 and Multilingual v2 use the Text to Speech one. ElevenLabsTTS opens the right one for the modelId you pass, so moving from Flash to v4 Turbo only means changing the model ID.

ModelWebSocketLanguagesAudio tagsBest for
eleven_v4_turboText to Dialogue90+PerformedExpressive voice agents, with a median inference latency of about 100 ms
eleven_v4Text to Dialogue90+PerformedNarration and characters, where the quality of the voice comes first
eleven_v3_conversationalText to Dialogue70+PerformedExpressive real-time speech, the generation before v4 Turbo
eleven_v3Text to Dialogue70+PerformedExpressive speech, the generation before v4
eleven_flash_v2_5Text to Speech32Read outThe lowest latency, about 75 ms
eleven_multilingual_v2Text to Speech29Read outLong narration, where the voice stays the most stable
eleven_turbo_v2_5Text to Speech32Read outExisting servers, since ElevenLabs deprecated it in favor of Flash v2.5

The latencies are the ones ElevenLabs gives, network excluded. A Micdrop test measured the first audio in a streamed voice agent, network included: about 90 ms on v4 Turbo and about 850 ms on v3. eleven_turbo_v2_5 stays the default of the package, so an existing server keeps its voice until you pick another model.

Eleven v4 and audio tags

Eleven v3 and v4 perform audio tags, short directions in square brackets. A tag like [whispers], [laughs] or [excited] goes right before the words it changes. Eleven v4 also plays sound effects like [door slams] and follows free directions like [strong French accent]. ElevenLabs lists the tags it tested with Eleven v4.

The LLM writes the tags, so tell it in the system prompt which ones it may use:

import { OpenaiAgent } from '@micdrop/openai'
const agent = new OpenaiAgent({
apiKey: process.env.OPENAI_API_KEY || '',
systemPrompt: `You are a warm and playful assistant. Your answers are spoken out loud.
Your voice performs audio tags: short directions in square brackets, placed right before the words they change. Use one or two per answer, where they fit:
- How you speak: [whispers], [excited], [curious], [sarcastic]
- How you react: [laughs], [sighs], [clears throat]
Use an ellipsis for a pause. The words you say always stay outside the brackets.`,
})

The tags reach the client with the rest of the answer, in the conversation and in the partial messages. Run the storyteller demo to hear what the tags change. Its page shows each tag apart from the words, and lets you switch the voice to Flash v2.5, where the same storyteller speaks without tags.

Only Eleven v3 and v4 perform the tags. Flash v2.5, Turbo v2.5 and Multilingual v2 read them out loud, so leave the tag instructions out of the system prompt when the voice runs on one of those models. When one of those models only serves as a backup voice, keep the tag instructions and strip the audio tags from the text it receives.

How Eleven v3 and v4 stream

The Text to Dialogue WebSocket holds text back until it has about 40 characters and 8 words. ElevenLabsTTS flushes it at the end of every sentence, so a short first sentence such as “Sure!” starts playing without waiting for the next one. The package counts an ellipsis as part of the sentence, since Eleven v4 performs it as a pause.

The Text to Dialogue WebSocket has no message to stop a synthesis halfway. When the user interrupts the assistant, ElevenLabsTTS closes the connection and opens a new one while the user is still speaking, so the interrupted answer stops where the user cut it. ElevenLabs closes a connection after 20 seconds without a message, so the package sends a keep-alive every 15 seconds.

ElevenLabs counts each connection to the Text to Dialogue WebSocket as one dialogue session for as long as it stays open, speaking or silent, so each call holds one session from start to end. ElevenLabs allows 14 sessions at once on its free plan and 105 on Scale, and publishes the session limit of each plan. Past the limit, it rejects new connections with too_many_concurrent_requests. ElevenLabs counts only the time spent generating audio on the Text to Speech WebSocket, so the same plan usually allows more calls at once with Flash.

Interruptions and reconnections

Flash v2.5, Turbo v2.5 and Multilingual v2 have no message to stop a synthesis either. When the user interrupts the assistant, ElevenLabsTTS drops their connection too and opens a new one while the user is still speaking, so no audio of the interrupted answer plays after the cut.

When a connection drops in the middle of an answer, ElevenLabsTTS reconnects and sends the answer again. Flash v2.5, Turbo v2.5 and Multilingual v2 pick it up where its audio stopped, while Eleven v3 and v4 speak it again from its start. ElevenLabs accepts a connection before it checks the key and the voice, so a connection that closes before carrying any audio counts as a failed attempt. After more than maxRetry failed attempts in a row, ElevenLabsTTS emits Failed with the text left unspoken, and the next call to speak() connects again with a full retry budget.

Options

OptionTypeDefaultDescription
apiKeystringRequiredYour ElevenLabs API key
voiceIdstringRequiredElevenLabs voice ID
modelIdElevenLabsModelId'eleven_turbo_v2_5'Model to use for speech synthesis, one of the supported models
languagestringOptionalLanguage code (e.g., ‘en’, ‘fr’)
outputFormatTextToSpeechStreamRequestOutputFormat'pcm_16000'Audio output format
voiceSettingsVoiceSettingsOptionalVoice customization settings. Eleven v3 and v4 only read stability
connectionTimeoutnumber5000Timeout in milliseconds for WebSocket connection
retryDelaynumber1000Delay in milliseconds between reconnection attempts
maxRetrynumber3Failed connection attempts in a row allowed before emitting Failed

Voice Settings

The voiceSettings option takes the VoiceSettings type of the ElevenLabs SDK, with its keys in camel case:

const tts = new ElevenLabsTTS({
apiKey: process.env.ELEVENLABS_API_KEY || '',
voiceId: 'your-voice-id',
modelId: 'eleven_flash_v2_5',
voiceSettings: {
stability: 0.5, // 0 to 1, lower is more expressive, higher is steadier
similarityBoost: 0.75, // 0 to 1, how closely to match the original voice
style: 0.5, // 0 to 1, exaggeration of the style
useSpeakerBoost: true, // Boost the similarity to the speaker
speed: 1, // 1 is the normal pace
},
})

Eleven v3 and v4 only read stability. A lower value gives the voice a broader emotional range, so it responds more to the audio tags.

Supported Languages

The languages depend on the model: more than 90 for Eleven v4 and v4 Turbo, more than 70 for Eleven v3, 32 for Flash v2.5 and Turbo v2.5, and 29 for Multilingual v2. ElevenLabs keeps the list of languages of each model. Every model speaks these languages:

CodeLanguageCodeLanguageCodeLanguage
enEnglishesSpanishfrFrench
deGermanitItalianptPortuguese
plPolishtrTurkishruRussian
nlDutchcsCzecharArabic
zhChinesejaJapanesesvSwedish
koKoreanhiHindifiFinnish

Getting Started

  1. Sign up for an ElevenLabs account and create an API key
  2. Choose a voice from the ElevenLabs voice library or create a custom voice
  3. Install the package and configure with your credentials
import { ElevenLabsTTS } from '@micdrop/elevenlabs'
const tts = new ElevenLabsTTS({
apiKey: process.env.ELEVENLABS_API_KEY || '',
voiceId: 'your-voice-id', // Get this from ElevenLabs dashboard
modelId: 'eleven_v4_turbo', // Choose based on your needs
language: 'en',
})
// Use with MicdropServer
new MicdropServer(socket, {
tts,
// ... other options
})

Creating an ElevenLabs API key

You create API keys on the API keys page of your ElevenLabs settings. Name each key after the server that uses it, so you can revoke one server’s key and leave the other servers running.

Restrict the key to Text to Speech access. @micdrop/elevenlabs only opens the Text to Speech and Text to Dialogue WebSockets, so that one permission covers everything the package does. A leaked key can then only generate speech. Add Voices read access if your own code lists the voices of your account. You can also give each key a monthly credit limit and an IP allowlist, so the key stops once it has spent its credits for the month and works only from your servers’ addresses.

Set the key in the ELEVENLABS_API_KEY environment variable of your server, then pass process.env.ELEVENLABS_API_KEY as apiKey, as the code samples on this page do:

Terminal window
ELEVENLABS_API_KEY=your-elevenlabs-api-key

The key stays on the server, since the browser talks to @micdrop/server and only the server connects to ElevenLabs. ElevenLabs checks the key only after it accepts the connection. With a wrong key or a missing permission, each connection closes before any audio, until ElevenLabsTTS runs out of retries and emits Failed.

Finding Voice IDs

You can find voice IDs in the ElevenLabs dashboard, by browsing the voices of your account, or through the public voice library API.

FAQ

Does ElevenLabs have an API?

Yes. ElevenLabs has a REST API for text to speech, speech to text, voices and its other products, plus WebSockets that stream speech while the text is still arriving. Both take an API key in the xi-api-key header. A voice agent needs the WebSockets, which @micdrop/elevenlabs wraps for Node.js. The free plan includes API access, within its own character quota and concurrency limit.

How do I get an API key for ElevenLabs?

Sign in to ElevenLabs, open the API keys page of your settings and create a key. Restrict it to Text to Speech access, which covers everything @micdrop/elevenlabs does, then set it as ELEVENLABS_API_KEY on your server. A free account gets a key too, with the monthly character quota of the free plan. You can also give the key a credit limit and an IP allowlist.

Which ElevenLabs WebSocket supports Eleven v4?

Eleven v4 and v4 Turbo stream only over the Text to Dialogue WebSocket, wss://api.elevenlabs.io/v1/text-to-dialogue/stream-input, like Eleven v3. The Text to Speech WebSocket refuses them with unsupported_model. ElevenLabsTTS opens the Text to Dialogue one as soon as modelId names a v3 or v4 model, and the Eleven v4 Turbo test details what changes in its protocol.

How do I stream ElevenLabs text to speech in Node.js?

Install @micdrop/elevenlabs, create an ElevenLabsTTS with your API key and a voice ID, and pass it to MicdropServer as tts. The server sends each chunk of the LLM answer to ElevenLabs as it arrives and forwards the audio to the browser. Outside Micdrop, call tts.speak() with a readable stream of text and listen to the Audio event, as in the usage without MicdropServer.

Which ElevenLabs model has the lowest latency?

Flash v2.5, at about 75 ms according to ElevenLabs, network excluded. Eleven v4 Turbo follows at about 100 ms and performs audio tags, which Flash reads out loud. Network included, a Micdrop test measured the first audio of v4 Turbo at about 90 ms in a streamed voice agent. The table of models compares them all.

Does Flash v2.5 read audio tags out loud?

Yes. Flash v2.5, Turbo v2.5 and Multilingual v2 speak a tag like [laughs] as words, and only Eleven v3 and v4 perform it. When Flash serves as a backup voice behind v4 Turbo, keep the tags in the system prompt and strip them from the text Flash receives.