🎤Micdrop

Write a Custom Text to Speech (TTS)

The TTS class is the core abstraction for text-to-speech functionality in Micdrop. It provides a standardized interface for integrating various text-to-speech providers into real-time voice conversations.

Available Implementations

For automatic failover between multiple TTS providers, see FallbackTTS.

Overview

The TTS class is an abstract base class that manages:

  • Real-time text stream processing
  • Audio stream generation and output
  • Cancellation and cleanup mechanisms
  • Integration with logging systems
  • Audio format conversion and optimization
export abstract class TTS extends EventEmitter<TTSEvents> {
public logger?: Logger
// Convert text stream to audio (emits Audio event)
abstract speak(textStream: Readable): void
// Cancel current speech generation
abstract cancel(): void
// Protected logging method
protected log(...message: any[]): void
// Cleanup
destroy(): void
}

Events

The TTS class emits the following events:

Audio

Emitted when an audio chunk is ready to be sent to the client.

tts.on('Audio', (audioChunk: Buffer) => {
console.log('Received audio chunk:', audioChunk.length, 'bytes')
// Send to client
})

Failed

Emitted when the TTS service fails after exhausting all retries. This event provides the buffered text chunks that were pending synthesis.

tts.on('Failed', (textChunks: string[]) => {
console.error('TTS failed with', textChunks.length, 'pending text chunks')
// Handle failure (e.g., notify user, fallback to another TTS)
})

Abstract Methods

speak(textStream: Readable): void

Converts a streaming text input into audio, emitted as Audio events.

Implementation Requirements:

  • Accept a Readable stream containing text chunks
  • Emit each audio chunk with this.emit('Audio', chunk), in PCM 16 bits at 16 kHz
  • A new speak() replaces the utterance still being spoken
  • Handle real-time streaming for optimal latency
  • Support cancellation via the cancel() method
  • Process text incrementally as it arrives
  • Handle text stream errors gracefully

cancel(): void

Cancels the current speech generation process. Should:

  • Abort any ongoing API requests
  • Stop audio stream generation
  • Clean up resources (WebSocket connections, timers, etc.)
  • End audio output stream gracefully

Public Methods

destroy()

Cleans up the TTS instance:

  • Calls cancel() to stop ongoing operations
  • Logs destruction event
  • Performs final cleanup

Debug Logging

Enable detailed logging for development:

// Enable debug logging
tts.logger = new Logger('CustomTTS')

Custom TTS Implementation

Creating a WebSocket-based Streaming TTS

For real-time streaming text-to-speech services. ReconnectingSocket, described on the STT page, holds the connection, so the class only deals with the protocol.

import { ReconnectingSocket, TTS } from '@micdrop/server'
import { Readable } from 'stream'
interface CustomStreamingTTSOptions {
apiKey: string
voiceId: string
language?: string
}
export class CustomStreamingTTS extends TTS {
private socket: ReconnectingSocket
// Numbers the utterances, so the audio of a cancelled one is dropped
private sessionId = 0
// Text of the utterance being spoken, sent again on a new connection
private textSent = ''
constructor(private readonly options: CustomStreamingTTSOptions) {
super()
this.socket = new ReconnectingSocket({
url: `wss://api.example.com/v1/tts/stream?api_key=${options.apiKey}`,
handshake: {
start: (socket) =>
socket.send(
JSON.stringify({
type: 'config',
voice_id: options.voiceId,
language: options.language || 'en',
output_format: { encoding: 'pcm_s16le', sample_rate: 16000 },
})
),
},
log: (...message) => this.log(...message),
})
this.socket.on('Open', this.onOpen)
this.socket.on('Message', (data) =>
this.handleMessage(JSON.parse(data.toString()))
)
this.socket.on('Failed', () => {
const text = this.textSent
this.cancel()
// A FallbackTTS speaks the rest instead
this.emit('Failed', text ? [text] : [])
})
this.socket.connect()
}
speak(textStream: Readable) {
const sessionId = ++this.sessionId
this.textSent = ''
textStream.on('data', (chunk: Buffer) => {
if (sessionId !== this.sessionId) return
const text = chunk.toString('utf-8')
this.textSent += text
// Without a connection, the text goes out on `Open`
this.send({ type: 'text', text, session_id: sessionId })
})
textStream.on('end', () => {
if (sessionId !== this.sessionId) return
this.send({ type: 'finalize', session_id: sessionId })
})
}
cancel() {
// The audio still on its way belongs to no utterance any more
this.sessionId++
this.textSent = ''
}
destroy() {
super.destroy()
this.socket.destroy()
}
// A new connection lost the utterance in progress, which starts over
private onOpen = ({ reconnected }: { reconnected: boolean }) => {
if (!reconnected || !this.textSent) return
this.send({ type: 'text', text: this.textSent, session_id: this.sessionId })
}
private handleMessage(message: any) {
if (message.session_id !== this.sessionId) return
if (message.type === 'audio') {
this.emit('Audio', Buffer.from(message.data, 'base64'))
} else if (message.type === 'audio_end') {
this.textSent = ''
} else if (message.type === 'error') {
this.log('TTS error:', message.error)
}
}
private send(message: object) {
this.socket.send(JSON.stringify(message))
}
}

Using CustomStreamingTTS with MicdropServer

// Create custom TTS
const tts = new CustomStreamingTTS({
apiKey: process.env.CUSTOM_TTS_API_KEY || '',
voiceId: process.env.CUSTOM_VOICE_ID || '',
language: 'en',
})
// Add logging
tts.logger = new Logger('CustomTTS')
// Create server with custom TTS
const server = new MicdropServer(socket, {
tts,
// ... other options
})

Creating a Fetch-based TTS Implementation

For services that process complete text before generating audio:

import { TTS } from '@micdrop/server'
import { Readable } from 'stream'
import { text } from 'stream/consumers'
interface CustomFetchTTSOptions {
apiKey: string
voiceId: string
model?: string
language?: string
}
export class CustomFetchTTS extends TTS {
private currentRequest?: AbortController
constructor(private readonly options: CustomFetchTTSOptions) {
super()
}
speak(textStream: Readable) {
// A new utterance replaces the one being spoken
this.cancel()
const request = new AbortController()
this.currentRequest = request
this.generateSpeech(textStream, request)
}
private async generateSpeech(textStream: Readable, request: AbortController) {
try {
// Collect all text first
this.log('Collecting text content...')
const textContent = await text(textStream)
if (!textContent.trim() || request.signal.aborted) return
this.log(`Generating speech for: "${textContent}"`)
// Make API request
const response = await fetch('https://api.example.com/v1/tts/generate', {
method: 'POST',
headers: {
Authorization: `Bearer ${this.options.apiKey}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
text: textContent,
voice_id: this.options.voiceId,
model: this.options.model || 'neural-v1',
language: this.options.language || 'en',
output_format: 'pcm_s16le', // 16-bit signed little-endian PCM at 16kHz
}),
signal: request.signal,
})
if (!response.ok) {
throw new Error(
`TTS API error: ${response.status} ${response.statusText}`
)
}
if (!response.body) throw new Error('No audio data in response')
const reader = response.body.getReader()
while (true) {
const { done, value } = await reader.read()
if (done || request.signal.aborted) break
this.emit('Audio', Buffer.from(value))
}
} catch (error) {
// A cancelled request throws, which is no failure
if (!request.signal.aborted) {
this.log('TTS generation failed:', error)
}
} finally {
if (this.currentRequest === request) this.currentRequest = undefined
}
}
cancel(): void {
this.currentRequest?.abort()
this.currentRequest = undefined
}
}

Using CustomFetchTTS with MicdropServer

// Create and configure custom TTS
const tts = new CustomFetchTTS({
apiKey: process.env.CUSTOM_TTS_API_KEY || '',
voiceId: process.env.CUSTOM_VOICE_ID || '',
language: 'en',
})
// Add logging
tts.logger = new Logger('CustomTTS')
// Create server with custom TTS
const server = new MicdropServer(socket, {
tts,
// ... other options
})

Simple Echo TTS Example

For testing and development:

import { TTS } from '@micdrop/server'
import { readFileSync } from 'fs'
import { Readable } from 'stream'
export class EchoTTS extends TTS {
private sampleAudio: Buffer
constructor(sampleAudioPath: string) {
super()
// Raw PCM 16 bits at 16 kHz, played as the "speech" of any text
this.sampleAudio = readFileSync(sampleAudioPath)
}
speak(textStream: Readable) {
textStream.once('data', () => {
this.log('Echoing sample audio for any text input')
this.emit('Audio', this.sampleAudio)
})
}
cancel(): void {
// Nothing to cancel for this simple implementation
}
}

Using EchoTTS

const echoTTS = new EchoTTS('/path/to/sample-audio.pcm')
echoTTS.logger = new Logger('EchoTTS')
echoTTS.on('Audio', (chunk) => {
console.log(`Audio chunk received: ${chunk.length} bytes`)
})
const textStream = new PassThrough()
echoTTS.speak(textStream)
textStream.end('Hello, world!')

Sharing a Local Model Between Calls

Each call creates its own TTS, while a local model takes from a second to a minute to load and holds its weights in memory. SharedInstances keeps one instance per configuration for the whole process, so every call after the first reuses it.

import { SharedInstances } from '@micdrop/server'
const synthesizers = new SharedInstances<Synthesizer>()
function loadSynthesizer(model: string, device: string) {
// Calls asking while it loads share the same load
return synthesizers.load([model, device], () =>
Synthesizer.load(model, { device })
)
}

The key is any JSON value, usually the options the instance is built from. A failed load is forgotten, so the next call tries again. get(key) returns the instance loaded or loading without starting a load, has(key) tells whether there is one, delete(key) forgets one that broke, a crashed worker for instance, so the next call loads a new one, and clear() forgets them all. Releasing what a forgotten instance holds is left to the caller.

Audio Format Considerations

Output Format

TTS implementations should output audio in PCM format (specifically pcm_s16le - 16-bit signed little-endian PCM) for optimal compatibility and performance with Micdrop.

The recommended audio specifications are:

  • Format: PCM (pcm_s16le)
  • Sample Rate: 16000 Hz
  • Channels: 1 (mono)
  • Bit Depth: 16-bit signed integers

Why PCM?

PCM format is recommended because:

  • Low latency: No encoding/decoding overhead
  • Universal compatibility: Supported by all audio systems
  • Real-time streaming: Optimal for live conversation scenarios
  • Simplicity: Direct audio data without compression artifacts

Streaming Considerations

For optimal real-time performance:

  • Chunk Size: Balance between latency and efficiency (typically 1-4KB chunks)
  • Buffering: Minimize buffering to reduce latency
  • Error Handling: Gracefully handle network interruptions
  • Cancellation: Support immediate cancellation for natural conversation flow

Error Handling

Robust TTS implementations should handle:

// A broken text stream ends the utterance with what was received
textStream.on('error', (error) => {
this.log('Text stream error:', error)
})
// API errors
if (!response.ok) {
throw new Error(`TTS API error: ${response.status}`)
}
// A cancelled request throws too, which is no failure
if (request.signal.aborted) return
// The service is out of reach for good: hand the unspoken text over, so a
// FallbackTTS speaks it instead
this.emit('Failed', [unspokenText])