AI Integration

Integrate speech-to-text, text-to-speech, and AI agents from multiple providers or build custom implementations.

Overview

Micdrop provides a modular AI architecture allowing you to:

• Mix and match providers for optimal cost and quality
• Run everything locally, with no API key and no data leaving your machine
• Build custom integrations using abstract base classes

Splitting transcription, the agent and the voice into three services is one of two architectures. The other sends audio straight to a speech-to-speech model such as the OpenAI Realtime API, where one connection comes with a fixed voice and a single provider.

Provider Categories

Provided Integrations

Ready-to-use implementations for popular AI services:

Speech-to-Text (STT):

• Gladia - Fast, accurate multilingual transcription
• OpenAI Whisper - High-quality speech recognition
• Mistral Voxtral - Real-time transcription
• Gradium - Real-time transcription
• Whisper - Local transcription, in your Node process

Text-to-Speech (TTS):

• ElevenLabs - High-quality voice synthesis
• Cartesia - Low-latency streaming TTS
• Kokoro - Local voice, in your Node process, English only
• Piper - Local voice, around forty languages
• Pocket TTS - Local voice cloning, in your Node process, English only

AI Agents (LLM):

• AI SDK - Universal provider using Vercel AI SDK
• OpenAI - GPT models for conversation
• Mistral - Open-source and commercial LLMs

Running Locally

The three parts above all have a local counterpart, so a whole conversation can run on your machine. Read the local models guide to set it up.

Custom Integrations

Build your own integrations using abstract base classes:

• Custom Agent - Create custom AI agents
• Custom STT - Implement speech-to-text services
• Custom TTS - Build text-to-speech providers

Quick Start

Basic Setup

import { MicdropServer } from '@micdrop/server'
import { OpenaiAgent } from '@micdrop/openai'
import { ElevenLabsTTS } from '@micdrop/elevenlabs'
import { GladiaSTT } from '@micdrop/gladia'

new MicdropServer(socket, {
  // AI Agent for conversation
  agent: new OpenaiAgent({
    apiKey: process.env.OPENAI_API_KEY,
    model: 'gpt-4-turbo-preview',
  }),

  // Speech-to-Text
  stt: new GladiaSTT({
    apiKey: process.env.GLADIA_API_KEY,
    settings: { language_config: { languages: ['en'] } },
  }),

  // Text-to-Speech
  tts: new ElevenLabsTTS({
    apiKey: process.env.ELEVENLABS_API_KEY,
    voiceId: 'voice-id-here',
  }),
})

Cost-Optimized Setup

// Use different providers for optimal cost/quality balance
new MicdropServer(socket, {
  agent: new MistralAgent({
    // Cost-effective LLM
    apiKey: process.env.MISTRAL_API_KEY,
    model: 'mistral-large-latest',
  }),

  stt: new GladiaSTT({
    // Fast, affordable STT
    apiKey: process.env.GLADIA_API_KEY,
  }),

  tts: new CartesiaTTS({
    // Low-latency TTS
    apiKey: process.env.CARTESIA_API_KEY,
    voiceId: 'cartesia-voice-id',
  }),
})

Models Comparison

Speech-to-Text Comparison

Text-to-Speech Comparison

Costs are the published list price of the entry paid plan in August 2026, per hour of audio generated or transcribed. Every provider charges less as the volume grows, and Gladia goes furthest with real-time transcription at $0.25 an hour on a commitment. The plans priced per character are converted at the 45,000 characters an hour Gradium publishes for its own voices.

AI Agent Comparison