--- title: "OpenAI | Micdrop" description: "OpenAI implementation for @micdrop/server." url: "https://micdrop.dev/docs/ai-integration/provided-integrations/openai" --- * [Getting Started](/docs/getting-started) * [Client (Browser)](/docs/client) * [Installation](/docs/client/installation) * [React Hooks](/docs/client/react-hooks) * [Start/Stop Call](/docs/client/start-stop-call) * [Pause/Resume Call](/docs/client/pause-resume-call) * [Mute/Unmute Call](/docs/client/mute-unmute-call) * [Call State](/docs/client/call-state) * [Display Conversation Messages](/docs/client/display-conversation-messages) * [Handling Tool Calls](/docs/client/handling-tool-calls) * [Device Management](/docs/client/devices-management) * [Voice Activity Detection (VAD)](/docs/client/vad) * [Error Handling](/docs/client/error-handling) * Utility Classes * [Mic](/docs/client/utility-classes/mic) * [MicdropClient](/docs/client/utility-classes/micdrop-client) * [MicRecorder](/docs/client/utility-classes/mic-recorder) * [Speaker](/docs/client/utility-classes/speaker) * [Server (Node.js)](/docs/server) * [Installation](/docs/server/installation) * [With Fastify](/docs/server/with-fastify) * [With NestJS](/docs/server/with-nestjs) * [Auth and Parameters](/docs/server/auth-and-parameters) * [First Message](/docs/server/first-message) * [Save Messages](/docs/server/save-messages) * [Resume a Conversation](/docs/server/resume-conversation) * [Recording Audio](/docs/server/recording-audio) * [Error Handling](/docs/server/error-handling) * [Tools](/docs/server/tools) * [Extract Value from Answer](/docs/server/extract) * [Auto End Call](/docs/server/auto-end-call) * [Semantic Turn Detection](/docs/server/semantic-turn-detection) * [Noise Filtering](/docs/server/noise-filtering) * [Micdrop Protocol](/docs/server/protocol) * [AI Integrations](/docs/ai-integration) * Provided Integrations * [AI SDK](/docs/ai-integration/provided-integrations/ai-sdk) * [Cartesia](/docs/ai-integration/provided-integrations/cartesia) * [ElevenLabs](/docs/ai-integration/provided-integrations/elevenlabs) * [Gladia](/docs/ai-integration/provided-integrations/gladia) * [Gradium](/docs/ai-integration/provided-integrations/gradium) * [Mistral](/docs/ai-integration/provided-integrations/mistral) * [OpenAI](/docs/ai-integration/provided-integrations/openai) * Custom Integrations * [Agent (LLM)](/docs/ai-integration/custom-integrations/custom-agent) * [Speech-to-Text (STT)](/docs/ai-integration/custom-integrations/custom-stt) * [Text-to-Speech (TTS)](/docs/ai-integration/custom-integrations/custom-tts) * Fallback Strategies * [FallbackAgent](/docs/ai-integration/fallback-strategies/agent-fallback) * [FallbackSTT](/docs/ai-integration/fallback-strategies/stt-fallback) * [FallbackTTS](/docs/ai-integration/fallback-strategies/tts-fallback) * [IA Vocale Souveraine šŸ‡«šŸ‡·šŸ‡ŖšŸ‡ŗ](/docs/ai-integration/sovereign-voice-ai) [Micdrop](/) ā€ŗ [Documentation](/docs/getting-started) # OpenAI OpenAI implementation for [@micdrop/server](/docs/server). This package provides AI agent, speech-to-text and text-to-speech implementations using OpenAIā€™s API. ## Installation Terminal window ``` npm install @micdrop/openai ``` ## OpenAI Agent ### Usage ``` import { OpenaiAgent } from '@micdrop/openai'import { MicdropServer } from '@micdrop/server' const agent = new OpenaiAgent({ apiKey: process.env.OPENAI_API_KEY || '', model: 'gpt-4o', // Default model systemPrompt: 'You are a helpful assistant', // Custom OpenAI Responses API settings (optional) settings: { temperature: 0.7, max_output_tokens: 150, },}) // Use with MicdropServernew MicdropServer(socket, { agent, // ... other options}) ``` ### Options Option Type Default Description `apiKey` `string` Required\* Your OpenAI API key (required if `openai` not provided) `openai` `OpenAI` Optional OpenAI instance (alternative to `apiKey`) `model` `string` `'gpt-4o'` OpenAI model to use `systemPrompt` `string` Required System prompt for the agent `retryDelay` `number` `1000` Delay in milliseconds between retry attempts `maxRetry` `number` `3` Maximum number of retries on API failures `maxSteps` `number` `5` Maximum number of steps (for tool calls) `autoEndCall` `boolean | string` `false` [Auto-detect when user wants to end call](/docs/server/auto-end-call) `autoSemanticTurn` `boolean | string` `false` [Handle incomplete user sentences](/docs/server/semantic-turn-detection) `autoIgnoreUserNoise` `boolean | string` `false` [Filter meaningless user sounds](/docs/server/noise-filtering) `extract` [`ExtractJsonOptions`](/docs/server/extract#json-extraction) | [`ExtractTagOptions`](/docs/server/extract#custom-tag-extraction) `undefined` Extract structured data from responses `onBeforeAnswer` `function` `undefined` Hook called before answer generation - return `true` to skip generation `settings` `object` `{}` Additional OpenAI Responses API parameters The OpenAI Agent supports adding and removing custom tools to extend its capabilities. For detailed information about tool management, see the [Tools documentation](/docs/server/tools). ### Advanced Features The OpenAI Agent supports advanced features for improved conversation handling: * **[Auto End Call](/docs/server/auto-end-call)**: Automatically detect when users want to end the conversation * **[Semantic Turn Detection](/docs/server/semantic-turn-detection)**: Handle incomplete sentences for natural flow * **[User Noise Filtering](/docs/server/noise-filtering)**: Filter out meaningless sounds and filler words * **[Extract Value from Answer](/docs/server/extract)**: Extract structured data from responses * **[Tools](/docs/server/tools)**: Add custom tools to the agent ### Langfuse Integration You can integrate Langfuse for observability by using the `openai` option with a Langfuse-wrapped OpenAI client: ``` import { OpenaiAgent } from '@micdrop/openai'import { Langfuse, observeOpenAI } from 'langfuse'import OpenAI from 'openai' // Initialize Langfuseconst langfuse = new Langfuse({ secretKey: process.env.LANGFUSE_SECRET_KEY, publicKey: process.env.LANGFUSE_PUBLIC_KEY, baseUrl: process.env.LANGFUSE_BASE_URL, // Optional, defaults to https://cloud.langfuse.com}) // Get system prompt from Langfuseconst systemPrompt = await langfuse.getPrompt('voice-assistant-system-prompt') // Create OpenAI client and wrap with Langfuse observabilityconst openai = observeOpenAI( new OpenAI({ apiKey: process.env.OPENAI_API_KEY }), { sessionId: 'session-123', userId: 'user-456', }) // Create agent with Langfuse-wrapped OpenAI clientconst agent = new OpenaiAgent({ openai, model: 'gpt-4o', systemPrompt: systemPrompt.prompt,}) ``` This integration will automatically track all OpenAI API calls, token usage, and conversation flows in your Langfuse dashboard with session and user context. ## OpenAI STT (Speech-to-Text) Real-time speech-to-text implementation using OpenAIā€™s WebSocket-based real-time transcription API. ### Usage ``` import { OpenaiSTT } from '@micdrop/openai'import { MicdropServer } from '@micdrop/server' const stt = new OpenaiSTT({ apiKey: process.env.OPENAI_API_KEY || '', model: 'gpt-4o-transcribe', // Default real-time transcription model language: 'en', // Optional: specify language for better accuracy prompt: 'Transcribe the incoming audio in real time.', // Optional: custom prompt transcriptionTimeout: 4000, // Optional: timeout in ms for transcription}) // Use with MicdropServernew MicdropServer(socket, { stt, // ... other options}) ``` ### Options Option Type Default Description `apiKey` `string` Required Your OpenAI API key `model` `string` `'gpt-4o-transcribe'` Real-time transcription model to use `language` `string` `'en'` Language code for transcription `prompt` `string` `'Transcribe the incoming audio in real time.'` Custom prompt to guide transcription behavior `connectionTimeout` `number` `5000` Timeout in milliseconds for WebSocket connection `transcriptionTimeout` `number` `4000` Timeout in milliseconds to wait for transcription result `retryDelay` `number` `1000` Delay in milliseconds between reconnection attempts `maxRetry` `number` `3` Maximum number of reconnection attempts before failing ## OpenAI TTS (Text-to-Speech) Text-to-speech implementation using OpenAIā€™s speech API. The incoming text is buffered into sentences and each sentence is synthesized as soon as it is complete, so playback can start without waiting for the whole answer. ### Usage ``` import { OpenaiTTS } from '@micdrop/openai'import { MicdropServer } from '@micdrop/server' const tts = new OpenaiTTS({ apiKey: process.env.OPENAI_API_KEY || '', model: 'gpt-4o-mini-tts', // Default model voice: 'alloy', // Default voice // Prosody control, only for gpt-4o-mini-tts (optional) instructions: 'Speak in a calm and friendly tone',}) // Use with MicdropServernew MicdropServer(socket, { tts, // ... other options}) ``` ### Options Option Type Default Description `apiKey` `string` Required\* Your OpenAI API key (required if `openai` not provided) `openai` `OpenAI` Optional OpenAI instance (alternative to `apiKey`) `model` `string` `'gpt-4o-mini-tts'` Speech model to use (`gpt-4o-mini-tts`, `tts-1`, `tts-1-hd`) `voice` `string` `'alloy'` Voice to use (e.g. `alloy`, `ash`, `ballad`, `coral`, `sage`, ā€¦) `instructions` `string` `undefined` Prosody/accent/tone control. Only works with `gpt-4o-mini-tts` `speed` `number` `undefined` Speech speed from `0.25` to `4.0`. Only works with `tts-1`/`tts-1-hd` ā„¹ļø Note **Language**: OpenAIā€™s speech API has no language parameter, the voice follows the language of the input text. To influence the spoken language or accent, use `instructions` (e.g. `'Speak in French'`) with the `gpt-4o-mini-tts` model. [Previousā† Mistral](/docs/ai-integration/provided-integrations/mistral)[NextAgent (LLM) ā†’](/docs/ai-integration/custom-integrations/custom-agent) On this page * [Installation](#installation) * [OpenAI Agent](#openai-agent) * [Usage](#usage) * [Options](#options) * [Advanced Features](#advanced-features) * [Langfuse Integration](#langfuse-integration) * [OpenAI STT (Speech-to-Text)](#openai-stt-speech-to-text) * [Usage](#usage-1) * [Options](#options-1) * [OpenAI TTS (Text-to-Speech)](#openai-tts-text-to-speech) * [Usage](#usage-2) * [Options](#options-2)