🎤Micdrop

Write a Custom Speech to Text (STT)

The STT class is the core abstraction for speech-to-text functionality in Micdrop. It provides a standardized interface for integrating various speech-to-text providers into real-time voice conversations.

Available Implementations

For automatic failover between multiple STT providers, see FallbackSTT.

MicdropServer accepts an STT on its own, yours included, for dictation without an agent or a voice, where each sentence reaches the client as a message.

Overview

The STT class is an abstract base class that extends EventEmitter and manages:

  • Real-time audio stream processing
  • Automatic audio format detection
  • Event emission for transcription results
  • Integration with logging systems
  • Resource cleanup and cancellation
export abstract class STT extends EventEmitter<STTEvents> {
public logger?: Logger
// Transcribe audio stream to text (emits Transcript event)
abstract transcribe(audioStream: Readable): void
// Cleanup
destroy(): void
}

Events

The STT class emits the following events:

Transcript

Emitted when a transcription is ready.

stt.on('Transcript', (text: string) => {
console.log('Transcript:', text)
})

Failed

Emitted when the STT service fails after exhausting all retries. This event provides the buffered audio chunks that were pending transcription.

stt.on('Failed', (audioChunks: Buffer[]) => {
console.error('STT failed with', audioChunks.length, 'pending audio chunks')
// Handle failure (e.g., notify user, fallback to another STT)
})

Debug Logging

Enable detailed logging for development:

// Enable debug logging
stt.logger = new Logger('CustomSTT')

Custom STT Implementation

Creating a Real-time STT Implementation

For services that support real-time streaming transcription. The server gives each utterance its own stream and expects exactly one Transcript per stream, in the order they were spoken, even when a new utterance starts before the transcript of the previous one came back.

import { ReconnectingSocket, STT } from '@micdrop/server'
import { Readable } from 'stream'
interface Utterance {
chunks: Buffer[]
ended: boolean
text: string
timeout?: NodeJS.Timeout
}
export class CustomRealtimeSTT extends STT {
private socket: ReconnectingSocket
// Waiting for their transcript, in the order they were spoken
private utterances: Utterance[] = []
constructor(private options: { apiKey: string; language?: string }) {
super()
this.socket = new ReconnectingSocket({
url: `wss://api.example.com/v1/stream?key=${options.apiKey}`,
// The session is ready once the service accepts its configuration
handshake: {
start: (socket) =>
socket.send(
JSON.stringify({ type: 'config', language: options.language })
),
read: (data) => JSON.parse(data.toString()).type === 'ready',
},
keepAlive: { interval: 30000, message: '{"type":"ping"}' },
log: (...message) => this.log(...message),
})
this.socket.on('Open', this.onOpen)
this.socket.on('Message', (data) =>
this.handleMessage(JSON.parse(data.toString()))
)
this.socket.on('Failed', this.onFailed)
this.socket.connect()
}
transcribe(audioStream: Readable) {
const utterance: Utterance = { chunks: [], ended: false, text: '' }
this.utterances.push(utterance)
audioStream.on('data', (chunk: Buffer) => {
utterance.chunks.push(chunk)
// Without a connection, the chunk goes out with the others on `Open`
this.socket.send(chunk)
})
audioStream.on('end', () => {
utterance.ended = true
this.socket.send(JSON.stringify({ type: 'end_stream' }))
// A transcript that never comes still settles the utterance
utterance.timeout = setTimeout(() => this.finish(utterance), 4000)
})
}
destroy() {
super.destroy()
this.utterances.forEach((utterance) => clearTimeout(utterance.timeout))
this.utterances = []
this.socket.destroy()
}
// A new session knows nothing of the previous one: the audio waiting for a
// transcript goes again
private onOpen = ({ reconnected }: { reconnected: boolean }) => {
if (!reconnected) return
for (const utterance of this.utterances) {
utterance.text = ''
utterance.chunks.forEach((chunk) => this.socket.send(chunk))
if (utterance.ended) {
this.socket.send(JSON.stringify({ type: 'end_stream' }))
}
}
}
private handleMessage(message: any) {
// The service answers in order, so its words belong to the oldest one
const utterance = this.utterances[0]
if (!utterance) return
if (message.type === 'transcript' && message.is_final) {
utterance.text += message.text
} else if (message.type === 'stream_ended') {
this.finish(utterance)
}
}
private finish(utterance: Utterance) {
if (!this.utterances.includes(utterance)) return
// Those spoken before go first, with what they have
while (this.utterances.length > 0) {
const first = this.utterances.shift()!
clearTimeout(first.timeout)
this.emit('Transcript', first.text.trim())
if (first === utterance) break
}
}
// Every retry failed: the audio goes back to the caller, a FallbackSTT
// for instance
private onFailed = () => {
const chunks = this.utterances.flatMap((utterance) => utterance.chunks)
this.utterances.forEach((utterance) => clearTimeout(utterance.timeout))
this.utterances = []
this.emit('Failed', chunks)
}
}

Connecting over a WebSocket

ReconnectingSocket holds the connection of a provider and reconnects on its own, so an integration only deals with its protocol. It tries one connection at a time, gives up after maxRetry failed attempts in a row, reconnects after a close from the server whatever its code, and silences a socket it left behind so it never acts on the next one. A connection that closes before it proved usable, with a handshake, a first message or by staying open, counts as a failed attempt.

OptionTypeDefaultDescription
urlstring | () => string | Promise<string>Address, or a function giving it for each attempt
clientOptionsClientOptionsOptions of the ws client, headers included
connectionTimeoutnumber5000How long an attempt may take, handshake included, in ms
retryDelaynumber1000Delay before a new attempt, in ms
maxRetrynumber3Failed attempts in a row before giving up
handshake{ start?, read? }Sets the session up before the connection counts as open
keepAlive{ interval, message }Message sent on a regular basis while open
log(...message) => voidReceives the log lines

status reads idle, connecting, open, reconnecting, failed or destroyed, and isOpen tells whether a message sent now goes out. send() returns false when the message did not go out, and ready() resolves to true once open, or to false if the connection failed or was destroyed meanwhile. restart() drops the connection for a new one right away, for a provider that can only stop a synthesis by closing. After failed, connect() starts over with a full budget. destroy() closes the connection for good, a retry still waiting included, and settles ready() with false.

EventPayloadDescription
Open{ reconnected: boolean }The moment to send again what a dropped connection lost
MessageRawData, booleanA message, once the handshake is done
Closenumber, stringThe connection dropped, a new attempt follows
FailedErrorEvery retry failed
StatusChangeReconnectingSocketStatusThe status changed

Using CustomRealtimeSTT with MicdropServer

// Create custom STT
const stt = new CustomRealtimeSTT({
apiKey: process.env.CUSTOM_STT_API_KEY || '',
language: 'en',
})
// Add logging
stt.logger = new Logger('CustomSTT')
// Create server with custom STT
const server = new MicdropServer(socket, {
stt,
// ... other options
})