Mistral Agent, Voxtral STT and TTS in Node.js
Mistral AI implementation for @micdrop/server.
This package provides an AI agent on Mistral AI’s API, real-time speech to text with Voxtral Mini Transcribe Realtime 3, and text to speech with Voxtral Speech 3. The agent, the transcription and the voice can all run on one Mistral account.
Mistral guarantees inference in the EU only on its European endpoint, api.eu.mistral.ai, billed 10% more. MistralTTS reaches it through its baseUrl option. MistralAgent and MistralSTT call the global endpoint, api.mistral.ai, where Mistral makes no commitment on where requests run. To keep the agent in Europe, use @micdrop/ai-sdk with a Mistral model set to api.eu.mistral.ai, as in this sovereign voice AI setup.
Installation
npm install @micdrop/mistralMistral Agent
Usage with MicdropServer
import { MistralAgent } from '@micdrop/mistral'import { MicdropServer } from '@micdrop/server'
const agent = new MistralAgent({ apiKey: process.env.MISTRAL_API_KEY || '', model: 'mistral-large-latest', systemPrompt: 'You are a helpful assistant',})
// Use with MicdropServernew MicdropServer(socket, { agent, // ... other options})Usage without MicdropServer
import { MistralAgent } from '@micdrop/mistral'
const agent = new MistralAgent({ apiKey: process.env.MISTRAL_API_KEY || '', systemPrompt: 'You are a helpful assistant',})
agent.on('Message', (message) => console.log('Message:', message))
agent.addUserMessage('Hello, what can you do?')
// The answer is a text stream, written as the model generates itagent.answer().on('data', (chunk) => process.stdout.write(chunk))Events
| Event | Payload | Description |
|---|---|---|
Message | MicdropConversationItem | A message, a tool call or a tool result was added to the conversation. |
ToolCall | MicdropToolCall | A tool declared with emitOutput ran, with its parameters and output. |
CancelLastUserMessage | none | The last user message was dropped because it carried no intent. |
SkipAnswer | none | The agent stays silent and waits for the user to finish their sentence. |
EndCall | none | The agent decided that the call is over. |
Failed | none | The agent gave up generating an answer after its retries. |
See the Agent interface for the full contract.
Options
| Option | Type | Default | Description |
|---|---|---|---|
apiKey | string | Required | Your Mistral AI API key |
model | string | 'mistral-large-latest' | Mistral AI model to use |
systemPrompt | string | Required | System prompt for the agent |
maxRetry | number | 3 | Maximum number of retries on API failures |
retryDelay | number | 500 | Delay in milliseconds between retries |
maxSteps | number | 5 | Maximum number of steps (for tool calls) |
autoEndCall | boolean | string | false | Auto-detect when user wants to end call |
autoSemanticTurn | boolean | string | false | Handle incomplete user sentences |
autoIgnoreUserNoise | boolean | string | false | Filter meaningless user sounds |
extract | ExtractJsonOptions | ExtractTagOptions | undefined | Extract structured data from responses |
onBeforeAnswer | function | undefined | Hook called before answer generation, returns true to skip it or a text to answer instead |
settings | object | {} | Additional Mistral AI API parameters |
Cancellation and Retries
When the user interrupts, the request to Mistral is aborted and the unfinished answer stays out of the conversation. A request that fails before writing anything is retried up to maxRetry times, retryDelay apart, unless the answer was cancelled meanwhile. A request cut off after part of the answer reached the stream ends there, and the part already spoken is the answer recorded, so no sentence is ever spoken twice.
Available Models
Mistral AI offers several models you can choose from:
ministral-8b-latest- Fast and efficient 8B parameter model (default)mistral-large-latest- Most capable model for complex tasksmistral-small-latest- Balanced performance and costcodestral-latest- Specialized for code generation
Settings Object
The settings parameter accepts any additional options from the Mistral AI Chat Completions API:
const agent = new MistralAgent({ apiKey: process.env.MISTRAL_API_KEY || '', systemPrompt: 'You are a helpful assistant', settings: { temperature: 0.7, // Controls randomness (0-1) max_tokens: 1000, // Maximum tokens in response top_p: 0.9, // Nucleus sampling parameter random_seed: 42, // For reproducible outputs safe_prompt: false, // Enable/disable safety filtering },})Advanced Features
The Mistral Agent supports advanced features for improved conversation handling:
- Auto End Call: Automatically detect when users want to end the conversation
- Semantic Turn Detection: Handle incomplete sentences for natural flow
- User Noise Filtering: Filter out meaningless sounds and filler words
- Extract Value from Answer: Extract structured data from responses
- Tools: Add custom tools to the agent
Mistral STT (Speech-to-Text)
Real-time transcription with Voxtral Mini Transcribe Realtime 3, in more than 40 languages, with a final transcript in under half a second. It costs $0.008 per minute. MistralSTT connects to Mistral’s global endpoint, so pick Gladia or Gradium when the audio has to stay in the EU.
Voxtral also runs on its own. With stt as the server’s only option, you get a dictation tool that transcribes each sentence and stays quiet.
Usage with MicdropServer
import { MistralSTT } from '@micdrop/mistral'import { MicdropServer } from '@micdrop/server'
const stt = new MistralSTT({ apiKey: process.env.MISTRAL_API_KEY || '',})
// Use with MicdropServernew MicdropServer(socket, { stt, // ... other options})Usage without MicdropServer
import { MistralSTT } from '@micdrop/mistral'import { createReadStream } from 'fs'
const stt = new MistralSTT({ apiKey: process.env.MISTRAL_API_KEY || '',})
stt.on('Transcript', (text) => console.log('Transcript:', text))stt.on('Failed', (chunks) => console.error('Failed:', chunks.length, 'chunks'))
// Audio is raw PCM, 16 bits, 16 kHz, monostt.transcribe(createReadStream('speech.pcm'))Events
| Event | Payload | Description |
|---|---|---|
Transcript | string | Transcription of one utterance. The text is empty when nothing was recognized. |
Failed | Buffer[] | Transcription gave up after its retries, with the audio chunks left pending. |
See the STT interface for the full contract.
Options
| Option | Type | Default | Description |
|---|---|---|---|
apiKey | string | Required | Your Mistral AI API key |
model | string | 'voxtral-mini-transcribe-realtime-3-0' | Realtime transcription model to use |
encoding | MistralAudioEncoding | 'pcm_s16le' | Audio encoding of the incoming stream |
targetStreamingDelayMs | number | Optional | Target streaming delay in milliseconds |
connectionTimeout | number | 5000 | Timeout in milliseconds for WebSocket connection |
transcriptionTimeout | number | 4000 | Timeout in milliseconds to wait for transcription |
retryDelay | number | 1000 | Delay in milliseconds between reconnection attempts |
maxRetry | number | 3 | Maximum number of reconnection attempts before failing |
Each stream given to transcribe() is one utterance and gets exactly one Transcript event, sent when Mistral finalizes the utterance after its flush. With no answer within transcriptionTimeout, the utterance is emitted with the text received so far, and the late answer is discarded so it never lands on the next utterance.
A connection that drops, even a clean close by Mistral, is opened again within a budget of maxRetry attempts in a row, retryDelay apart, and the audio of every utterance still waiting for its transcript is sent again. Once the budget runs out, Failed hands that audio back, and the next utterance tries to connect again.
The Micdrop client streams 16kHz PCM16 mono audio, which matches Voxtral’s default input format, so no resampling is required.
The previous model, voxtral-mini-transcribe-realtime-2602, still works through the model option.
Mistral TTS (Text-to-Speech)
Voxtral Speech 3 reads the answers of your agent in more than 35 languages, with preset voices or a voice cloned from a few seconds of audio. The speech endpoint reads a whole text at once, so MistralTTS cuts the answer into sentences and sends the audio of each one as Mistral streams it. A sentence starts playing in about half a second and is generated several times faster than it is spoken, so the next one is ready before the current one ends.
Usage with MicdropServer
import { MistralTTS } from '@micdrop/mistral'import { MicdropServer } from '@micdrop/server'
const tts = new MistralTTS({ apiKey: process.env.MISTRAL_API_KEY || '', model: 'voxtral-mini-tts-3-0', // Default model voiceId: 'voice_592', // Marie, French})
new MicdropServer(socket, { tts, // ... other options})Usage without MicdropServer
import { MistralTTS } from '@micdrop/mistral'import { Readable } from 'stream'
const tts = new MistralTTS({ apiKey: process.env.MISTRAL_API_KEY || '' })
// Audio is raw PCM, 16 bits, 16 kHz, monotts.on('Audio', (chunk) => console.log('Audio:', chunk.length, 'bytes'))tts.on('Failed', (texts) => console.error('Failed:', texts))
tts.speak(Readable.from(['Hello! ', 'What can I do for you?']))Events
| Event | Payload | Description |
|---|---|---|
Audio | Buffer | A chunk of audio, PCM 16 bits, 16 kHz, mono, ready to be played. |
Failed | string[] | Synthesis gave up after an error, with the text that stayed unspoken. |
See the TTS interface for the full contract.
Options
| Option | Type | Default | Description |
|---|---|---|---|
apiKey | string | Required | Your Mistral AI API key |
model | string | 'voxtral-mini-tts-3-0' | Speech model to use |
voiceId | string | 'voice_3150' | Preset voice, or the id of a voice saved in your Mistral account |
refAudio | string | undefined | A few seconds of speech to clone, base64 encoded, used over voiceId |
settings | Record<string, unknown> | undefined | Extra fields of the request body, such as phoneme_hints |
baseUrl | string | 'https://api.mistral.ai' | API address, such as https://api.eu.mistral.ai to stay in Europe |
Voxtral Speech 3 costs $13 per million characters.
Voices
Each preset voice has a native language and reads the others too. The full list comes from GET /v1/audio/voices on the Mistral API.
| Voice | Name | Language |
|---|---|---|
voice_3150 | James | en-US |
voice_478 | Sheila | en-US |
voice_3149 | Hugo | en-GB |
voice_590 | Harriet | en-GB |
voice_592 | Marie | French |
voice_3141 | Camille | French |
voice_3137 | Manon | French |
voice_589 | Lea | French |
voice_481 | Heidi | German |
voice_483 | Elena | Spanish |
voice_482 | Lorenzo | Italian |
voice_480 | Gabriela | Portuguese |
Mistral also offers voices in Arabic, Catalan, Hindi, Japanese, Korean, Dutch, Polish, Russian, Chinese and more. The Voxtral 2 voices, such as gb_jane_neutral or fr_marie_happy, still work.
The text can carry emotion and pace tags such as [emotion=...], which Voxtral Speech 3 performs rather than reads out. Ask your agent to write them in its system prompt.