AI Integration

Integrate speech-to-text, text-to-speech, and AI agents from multiple providers or build custom implementations.

A classifier can run next to them, to turn what the user says into typed decisions.

Overview

Micdrop provides a modular AI architecture allowing you to:

• Mix and match providers for optimal cost and quality
• Run everything locally, with no API key and no data leaving your machine
• Build custom integrations using abstract base classes

Splitting transcription, the agent and the voice into three services is one of two architectures. The other sends audio straight to a realtime model such as Gemini Live or the OpenAI Realtime API, where one connection comes with a faster answer, the voice of that provider and a single bill.

Provider Categories

Provided Integrations

Ready-to-use implementations for popular AI services:

Speech-to-Text (STT):

• Gladia - Fast, accurate multilingual transcription
• Gemini Transcribe - Streaming transcription, 85+ languages
• OpenAI Whisper - High-quality speech recognition
• Mistral Voxtral - Real-time transcription
• Gradium - Real-time transcription
• Whisper - Local transcription, in your Node process

The hosted ones differ in price, languages and time to the final transcript.

Text-to-Speech (TTS):

• ElevenLabs - High-quality voice synthesis
• Cartesia - Low-latency streaming TTS
• Gemini TTS - 30 voices, directed with plain instructions
• Kokoro - Local voice, in your Node process, English only
• Piper - Local voice, around forty languages
• Pocket TTS - Local voice cloning, in your Node process, English only
• Qwen3-TTS - Local voice, ten languages, on an mlx-audio server

AI Agents (LLM):

• AI SDK - Universal provider using Vercel AI SDK
• OpenAI - GPT models for conversation
• Mistral - Open-source and commercial LLMs
• Gemini - Gemini Flash models for conversation

Realtime models (speech to speech):

• Gemini Live - Hears and speaks in one connection
• OpenAI Realtime - Hears and speaks in one connection

Classifiers:

• Jev (TypeSafe), typed decisions about what the user says, in a few hundred ms

Running Locally

The three parts above all have a local counterpart, so a whole conversation can run on your machine. Read the local models guide to set it up.

Custom Integrations

Build your own integrations using abstract base classes:

• Custom Agent - Create custom AI agents
• Custom STT - Implement speech-to-text services
• Custom TTS - Build text-to-speech providers
• Custom Classifier, classify each turn of the user with any model

Quick Start

Basic Setup

import { MicdropServer } from '@micdrop/server'
import { OpenaiAgent } from '@micdrop/openai'
import { ElevenLabsTTS } from '@micdrop/elevenlabs'
import { GladiaSTT } from '@micdrop/gladia'

new MicdropServer(socket, {
  // AI Agent for conversation
  agent: new OpenaiAgent({
    apiKey: process.env.OPENAI_API_KEY,
    model: 'gpt-4-turbo-preview',
  }),

  // Speech-to-Text
  stt: new GladiaSTT({
    apiKey: process.env.GLADIA_API_KEY,
    settings: { language_config: { languages: ['en'] } },
  }),

  // Text-to-Speech
  tts: new ElevenLabsTTS({
    apiKey: process.env.ELEVENLABS_API_KEY,
    voiceId: 'voice-id-here',
  }),
})

Cost-Optimized Setup

// Use different providers for optimal cost/quality balance
new MicdropServer(socket, {
  agent: new MistralAgent({
    // Cost-effective LLM
    apiKey: process.env.MISTRAL_API_KEY,
    model: 'mistral-large-latest',
  }),

  stt: new GladiaSTT({
    // Fast, affordable STT
    apiKey: process.env.GLADIA_API_KEY,
  }),

  tts: new CartesiaTTS({
    // Low-latency TTS
    apiKey: process.env.CARTESIA_API_KEY,
    voiceId: 'cartesia-voice-id',
  }),
})

Models Comparison

Speech-to-Text Comparison

Text-to-Speech Comparison

Costs are the published list price of the entry paid plan in August 2026, per hour of audio generated or transcribed. ElevenLabs bills its API in dollars at the same price on every plan, $40 per million characters on Flash v2.5 and $80 on Multilingual v2 (list prices read on October 2, 2026). Every provider charges less as the volume grows, and Gladia goes furthest with real-time transcription at $0.25 an hour on a commitment. The prices per character are converted at the 45,000 characters an hour Gradium publishes for its own voices.

AI Agent Comparison