--- title: "Latency and Memory | Micdrop" description: "What a local call costs per turn, how much memory it holds, and why transcription and the voice stay on the CPU." url: "https://micdrop.dev/docs/ai-integration/local-models/performance" --- * [Getting Started](/docs/getting-started) * [Client (Browser)](/docs/client) * [Installation](/docs/client/installation) * [React Hooks](/docs/client/react-hooks) * [Start/Stop Call](/docs/client/start-stop-call) * [Pause/Resume Call](/docs/client/pause-resume-call) * [Mute/Unmute Call](/docs/client/mute-unmute-call) * [Call State](/docs/client/call-state) * [Display Conversation Messages](/docs/client/display-conversation-messages) * [Handling Tool Calls](/docs/client/handling-tool-calls) * [Device Management](/docs/client/devices-management) * [Voice Activity Detection (VAD)](/docs/client/vad) * [Error Handling](/docs/client/error-handling) * Utility Classes * [Mic](/docs/client/utility-classes/mic) * [MicdropClient](/docs/client/utility-classes/micdrop-client) * [MicRecorder](/docs/client/utility-classes/mic-recorder) * [Speaker](/docs/client/utility-classes/speaker) * [Server (Node.js)](/docs/server) * [Installation](/docs/server/installation) * [With Fastify](/docs/server/with-fastify) * [With NestJS](/docs/server/with-nestjs) * [Auth and Parameters](/docs/server/auth-and-parameters) * [First Message](/docs/server/first-message) * [Save Messages](/docs/server/save-messages) * [Resume a Conversation](/docs/server/resume-conversation) * [Recording Audio](/docs/server/recording-audio) * [Error Handling](/docs/server/error-handling) * [Tools](/docs/server/tools) * [Extract Value from Answer](/docs/server/extract) * [Auto End Call](/docs/server/auto-end-call) * [Semantic Turn Detection](/docs/server/semantic-turn-detection) * [Noise Filtering](/docs/server/noise-filtering) * [Micdrop Protocol](/docs/server/protocol) * [AI Integrations](/docs/ai-integration) * Provided Integrations * [AI SDK](/docs/ai-integration/provided-integrations/ai-sdk) * [Cartesia](/docs/ai-integration/provided-integrations/cartesia) * [ElevenLabs](/docs/ai-integration/provided-integrations/elevenlabs) * [Gladia](/docs/ai-integration/provided-integrations/gladia) * [Gradium](/docs/ai-integration/provided-integrations/gradium) * [Kokoro](/docs/ai-integration/provided-integrations/kokoro) * [Mistral](/docs/ai-integration/provided-integrations/mistral) * [OpenAI](/docs/ai-integration/provided-integrations/openai) * [Piper](/docs/ai-integration/provided-integrations/piper) * [Pocket TTS](/docs/ai-integration/provided-integrations/pocket-tts) * [Whisper](/docs/ai-integration/provided-integrations/whisper) * Custom Integrations * [Agent (LLM)](/docs/ai-integration/custom-integrations/custom-agent) * [Speech-to-Text (STT)](/docs/ai-integration/custom-integrations/custom-stt) * [Text-to-Speech (TTS)](/docs/ai-integration/custom-integrations/custom-tts) * Fallback Strategies * [FallbackAgent](/docs/ai-integration/fallback-strategies/agent-fallback) * [FallbackSTT](/docs/ai-integration/fallback-strategies/stt-fallback) * [FallbackTTS](/docs/ai-integration/fallback-strategies/tts-fallback) * [Local Models](/docs/ai-integration/local-models) * [Choosing the Models](/docs/ai-integration/local-models/choosing-models) * [Latency and Memory](/docs/ai-integration/local-models/performance) * [Explorations](/docs/ai-integration/local-models/explorations) * [IA Vocale Souveraine 🇫🇷🇪🇺](/docs/ai-integration/sovereign-voice-ai) [Micdrop](/) › [Documentation](/docs/getting-started) # Latency and Memory Latency limits a local call long before memory does. These measurements come from a MacBook Pro M2 with 24 GB of memory, on CPU, with the models already loaded. ## Latency Step Time Whisper `base` on a 3 second sentence ~440 ms Whisper `french` on a 3 second sentence ~1100 ms First token from Qwen3 4B Instruct ~80 ms First token when a tool call is answered 740 to 1200 ms First sentence from Kokoro 900 to 1300 ms First sentence from Piper ~390 ms First sentence from Pocket TTS 250 to 430 ms A local call therefore answers in about one second with Piper or Pocket TTS, and closer to two with Kokoro. When the model answers one of the automatic prompts, the turn runs a second pass, and that pass costs more than the transcription and the voice put together. Micdrop keeps the models loaded between turns by sharing one instance per configuration, and Ollama does the same through its `keep_alive`, so the models load once and stay loaded. Run the warm-up at startup, and the first inference happens while the call is being set up rather than while the user waits for the first sentence. ## Memory Qwen3 4B in `Q4`, Whisper `base` and Kokoro together take around 4 GB including the key value cache, which leaves plenty of room on a 16 GB machine and is comfortable on 24 GB. Pocket TTS in place of Kokoro adds about half a gigabyte, since Pocket TTS holds 600 MB while it generates, against 80 MB for Kokoro. Latency is the binding constraint here, so spend the spare memory on a larger LLM rather than on a larger transcription model. ## Why transcription and the voice stay on the CPU The language model, by far the heaviest part of a call, already runs on the GPU. Ollama uses Metal on a Mac and CUDA on a machine with an NVIDIA card, and `ollama ps` shows `100% GPU` while a call is in progress. The runtime is what keeps transcription and the voice on the CPU: * Transformers.js declares no GPU device on macOS. Windows gets DirectML and Linux on x64 gets CUDA, both reachable through the `device` option of `WhisperSTT` and `KokoroTTS`. * WebGPU, the fast path Transformers.js uses in a browser, does not exist in Node, with or without a flag. * The CoreML provider is compiled into the ONNX Runtime that ships with Node and can be reached directly, so it was measured rather than assumed. That last one deserves its numbers. On the encoder of the `french` checkpoint: Provider Encoder Session load cpu 586 ms 207 ms coreml 1728 ms 14552 ms coreml `MLProgram` 2975 ms 12655 ms CoreML is three to five times slower, and needs another twelve seconds to compile the model when the session opens. The CoreML log shows why: CoreML covers 577 of the 873 nodes and splits the graph into 88 partitions, so the run spends its time copying tensors back and forth across those boundaries instead of computing. So the CPU is the right answer on a Mac today, and quantized weights are the fastest setting measured on it, which is why `q8` is the default. On Linux with an NVIDIA card, try `device: 'cuda'`, and measure a server built around such a card yourself, since these numbers describe a personal machine. `src/tests/onnx-device-bench.ts` in the demo server runs this comparison against any encoder from the Transformers.js cache. Run it to see what a different machine does. Piper stays on the CPU for a different reason. Its command line has a `--cuda` flag, with no Metal equivalent. Piper is fast enough that the missing GPU path rarely matters, since almost all of its cost is loading the voice once. [Previous← Choosing the Models](/docs/ai-integration/local-models/choosing-models)[NextExplorations →](/docs/ai-integration/local-models/explorations) On this page * [Latency](#latency) * [Memory](#memory) * [Why transcription and the voice stay on the CPU](#why-transcription-and-the-voice-stay-on-the-cpu)