Write a Custom Text to Speech (TTS)
The TTS class is the core abstraction for text-to-speech functionality in Micdrop. It provides a standardized interface for integrating various text-to-speech providers into real-time voice conversations.
Available Implementations
- ElevenLabsTTS from @micdrop/elevenlabs
- CartesiaTTS from @micdrop/cartesia
- MistralTTS from @micdrop/mistral, a
SentenceTTSthat reads one sentence per request - MockTTS for testing
For automatic failover between multiple TTS providers, see FallbackTTS.
Overview
The TTS class is an abstract base class that manages:
- Real-time text stream processing
- Audio stream generation and output
- Cancellation and cleanup mechanisms
- Integration with logging systems
- Audio format conversion and optimization
export abstract class TTS extends EventEmitter<TTSEvents> { public logger?: Logger
// Convert text stream to audio (emits Audio event) abstract speak(textStream: Readable): void
// Cancel current speech generation abstract cancel(): void
// Protected logging method protected log(...message: any[]): void
// Cleanup destroy(): void}Events
The TTS class emits the following events:
Audio
Emitted when an audio chunk is ready to be sent to the client.
tts.on('Audio', (audioChunk: Buffer) => { console.log('Received audio chunk:', audioChunk.length, 'bytes') // Send to client})Failed
Emitted when the TTS service fails after exhausting all retries. This event provides the buffered text chunks that were pending synthesis.
tts.on('Failed', (textChunks: string[]) => { console.error('TTS failed with', textChunks.length, 'pending text chunks') // Handle failure (e.g., notify user, fallback to another TTS)})Abstract Methods
speak(textStream: Readable): void
Converts a streaming text input into audio, emitted as Audio events.
Implementation Requirements:
- Accept a
Readablestream containing text chunks - Emit each audio chunk with
this.emit('Audio', chunk), in PCM 16 bits at 16 kHz - A new
speak()replaces the utterance still being spoken - Handle real-time streaming for optimal latency
- Support cancellation via the
cancel()method - Process text incrementally as it arrives
- Handle text stream errors gracefully
cancel(): void
Cancels the current speech generation process. Should:
- Abort any ongoing API requests
- Stop audio stream generation
- Clean up resources (WebSocket connections, timers, etc.)
- End audio output stream gracefully
Public Methods
destroy()
Cleans up the TTS instance:
- Calls
cancel()to stop ongoing operations - Logs destruction event
- Performs final cleanup
Debug Logging
Enable detailed logging for development:
// Enable debug loggingtts.logger = new Logger('CustomTTS')Custom TTS Implementation
Creating a WebSocket-based Streaming TTS
For real-time streaming text-to-speech services. ReconnectingSocket, described
on the STT page, holds the
connection, so the class only deals with the protocol.
import { ReconnectingSocket, TTS } from '@micdrop/server'import { Readable } from 'stream'
interface CustomStreamingTTSOptions { apiKey: string voiceId: string language?: string}
export class CustomStreamingTTS extends TTS { private socket: ReconnectingSocket // Numbers the utterances, so the audio of a cancelled one is dropped private sessionId = 0 // Text of the utterance being spoken, sent again on a new connection private textSent = ''
constructor(private readonly options: CustomStreamingTTSOptions) { super() this.socket = new ReconnectingSocket({ url: `wss://api.example.com/v1/tts/stream?api_key=${options.apiKey}`, handshake: { start: (socket) => socket.send( JSON.stringify({ type: 'config', voice_id: options.voiceId, language: options.language || 'en', output_format: { encoding: 'pcm_s16le', sample_rate: 16000 }, }) ), }, log: (...message) => this.log(...message), }) this.socket.on('Open', this.onOpen) this.socket.on('Message', (data) => this.handleMessage(JSON.parse(data.toString())) ) this.socket.on('Failed', () => { const text = this.textSent this.cancel() // A FallbackTTS speaks the rest instead this.emit('Failed', text ? [text] : []) }) this.socket.connect() }
speak(textStream: Readable) { const sessionId = ++this.sessionId this.textSent = ''
textStream.on('data', (chunk: Buffer) => { if (sessionId !== this.sessionId) return const text = chunk.toString('utf-8') this.textSent += text // Without a connection, the text goes out on `Open` this.send({ type: 'text', text, session_id: sessionId }) })
textStream.on('end', () => { if (sessionId !== this.sessionId) return this.send({ type: 'finalize', session_id: sessionId }) }) }
cancel() { // The audio still on its way belongs to no utterance any more this.sessionId++ this.textSent = '' }
destroy() { super.destroy() this.socket.destroy() }
// A new connection lost the utterance in progress, which starts over private onOpen = ({ reconnected }: { reconnected: boolean }) => { if (!reconnected || !this.textSent) return this.send({ type: 'text', text: this.textSent, session_id: this.sessionId }) }
private handleMessage(message: any) { if (message.session_id !== this.sessionId) return if (message.type === 'audio') { this.emit('Audio', Buffer.from(message.data, 'base64')) } else if (message.type === 'audio_end') { this.textSent = '' } else if (message.type === 'error') { this.log('TTS error:', message.error) } }
private send(message: object) { this.socket.send(JSON.stringify(message)) }}Using CustomStreamingTTS with MicdropServer
// Create custom TTSconst tts = new CustomStreamingTTS({ apiKey: process.env.CUSTOM_TTS_API_KEY || '', voiceId: process.env.CUSTOM_VOICE_ID || '', language: 'en',})
// Add loggingtts.logger = new Logger('CustomTTS')
// Create server with custom TTSconst server = new MicdropServer(socket, { tts, // ... other options})Creating a Fetch-based TTS Implementation
For services that process complete text before generating audio:
import { TTS } from '@micdrop/server'import { Readable } from 'stream'import { text } from 'stream/consumers'
interface CustomFetchTTSOptions { apiKey: string voiceId: string model?: string language?: string}
export class CustomFetchTTS extends TTS { private currentRequest?: AbortController
constructor(private readonly options: CustomFetchTTSOptions) { super() }
speak(textStream: Readable) { // A new utterance replaces the one being spoken this.cancel() const request = new AbortController() this.currentRequest = request this.generateSpeech(textStream, request) }
private async generateSpeech(textStream: Readable, request: AbortController) { try { // Collect all text first this.log('Collecting text content...') const textContent = await text(textStream) if (!textContent.trim() || request.signal.aborted) return
this.log(`Generating speech for: "${textContent}"`)
// Make API request const response = await fetch('https://api.example.com/v1/tts/generate', { method: 'POST', headers: { Authorization: `Bearer ${this.options.apiKey}`, 'Content-Type': 'application/json', }, body: JSON.stringify({ text: textContent, voice_id: this.options.voiceId, model: this.options.model || 'neural-v1', language: this.options.language || 'en', output_format: 'pcm_s16le', // 16-bit signed little-endian PCM at 16kHz }), signal: request.signal, })
if (!response.ok) { throw new Error( `TTS API error: ${response.status} ${response.statusText}` ) }
if (!response.body) throw new Error('No audio data in response') const reader = response.body.getReader() while (true) { const { done, value } = await reader.read() if (done || request.signal.aborted) break this.emit('Audio', Buffer.from(value)) } } catch (error) { // A cancelled request throws, which is no failure if (!request.signal.aborted) { this.log('TTS generation failed:', error) } } finally { if (this.currentRequest === request) this.currentRequest = undefined } }
cancel(): void { this.currentRequest?.abort() this.currentRequest = undefined }}Using CustomFetchTTS with MicdropServer
// Create and configure custom TTSconst tts = new CustomFetchTTS({ apiKey: process.env.CUSTOM_TTS_API_KEY || '', voiceId: process.env.CUSTOM_VOICE_ID || '', language: 'en',})
// Add loggingtts.logger = new Logger('CustomTTS')
// Create server with custom TTSconst server = new MicdropServer(socket, { tts, // ... other options})Simple Echo TTS Example
For testing and development:
import { TTS } from '@micdrop/server'import { readFileSync } from 'fs'import { Readable } from 'stream'
export class EchoTTS extends TTS { private sampleAudio: Buffer
constructor(sampleAudioPath: string) { super() // Raw PCM 16 bits at 16 kHz, played as the "speech" of any text this.sampleAudio = readFileSync(sampleAudioPath) }
speak(textStream: Readable) { textStream.once('data', () => { this.log('Echoing sample audio for any text input') this.emit('Audio', this.sampleAudio) }) }
cancel(): void { // Nothing to cancel for this simple implementation }}Using EchoTTS
const echoTTS = new EchoTTS('/path/to/sample-audio.pcm')echoTTS.logger = new Logger('EchoTTS')
echoTTS.on('Audio', (chunk) => { console.log(`Audio chunk received: ${chunk.length} bytes`)})
const textStream = new PassThrough()echoTTS.speak(textStream)textStream.end('Hello, world!')Sharing a Local Model Between Calls
Each call creates its own TTS, while a local model takes from a second to a
minute to load and holds its weights in memory. SharedInstances keeps one
instance per configuration for the whole process, so every call after the
first reuses it.
import { SharedInstances } from '@micdrop/server'
const synthesizers = new SharedInstances<Synthesizer>()
function loadSynthesizer(model: string, device: string) { // Calls asking while it loads share the same load return synthesizers.load([model, device], () => Synthesizer.load(model, { device }) )}The key is any JSON value, usually the options the instance is built from. A
failed load is forgotten, so the next call tries again. get(key) returns the
instance loaded or loading without starting a load, has(key) tells whether
there is one, delete(key) forgets one that broke, a crashed worker for
instance, so the next call loads a new one, and clear() forgets them all.
Releasing what a forgotten instance holds is left to the caller.
Audio Format Considerations
Output Format
TTS implementations should output audio in PCM format (specifically pcm_s16le - 16-bit signed little-endian PCM) for optimal compatibility and performance with Micdrop.
The recommended audio specifications are:
- Format: PCM (pcm_s16le)
- Sample Rate: 16000 Hz
- Channels: 1 (mono)
- Bit Depth: 16-bit signed integers
Why PCM?
PCM format is recommended because:
- Low latency: No encoding/decoding overhead
- Universal compatibility: Supported by all audio systems
- Real-time streaming: Optimal for live conversation scenarios
- Simplicity: Direct audio data without compression artifacts
Streaming Considerations
For optimal real-time performance:
- Chunk Size: Balance between latency and efficiency (typically 1-4KB chunks)
- Buffering: Minimize buffering to reduce latency
- Error Handling: Gracefully handle network interruptions
- Cancellation: Support immediate cancellation for natural conversation flow
Error Handling
Robust TTS implementations should handle:
// A broken text stream ends the utterance with what was receivedtextStream.on('error', (error) => { this.log('Text stream error:', error)})
// API errorsif (!response.ok) { throw new Error(`TTS API error: ${response.status}`)}
// A cancelled request throws too, which is no failureif (request.signal.aborted) return
// The service is out of reach for good: hand the unspoken text over, so a// FallbackTTS speaks it insteadthis.emit('Failed', [unspokenText])