Mistral AI: Voxtral TTS (text to speech), Vision & AI Agents

Watch the agent start listening the moment the session opens, with no button to hold and no push-to-talk. A long answer gets cut off mid-sentence when the speaker talks over it, and the debug panel shows the barge-in event that caused it. The same session asks for the time in Ljubljana and for a dice roll, both answered by real function calls, and then the agent recalls something it was told several turns earlier.

The design behind the demo: four states, idle, listening, processing and speaking, with every transition triggered by an event, and barge-in as a direct jump from speaking back to listening. One WebSocket carries microphone audio and control messages, and a dispatcher splits them into two queues so an interrupt signal never waits behind streaming audio. The spec then pins the models, the silence timeout, the tools the agent must expose, and the acceptance criteria you check your own build against.

The code that decides when you have stopped talking: a timestamp refreshed on every transcription event, and a silence timeout of one and a half seconds that only starts counting once speech has been detected. Two async tasks run at the same time, one sending microphone audio to Voxtral and one receiving transcription events back. On the interrupt side you read the browser code that measures audio loudness and fires only after three consecutive loud frames, the backend code that stops speaking and returns to listening, and the echo cancellation setting that keeps the agent from picking up its own voice.

The finished app on one worksheet: the photo goes up through a single upload area, and an explanation comes back and starts reading itself aloud, with each sentence highlighted as it is spoken. A child asks a follow-up out loud, the words appear as a live transcript while they talk, and the answer works the fractions problem from that same page. You also see the three explanation modes applied to one photo, and a short reference clip in Settings that makes the app read in a familiar voice.

How the five Mistral capabilities connect, and why one photo goes to two services: on a real worksheet, OCR returned every printed sentence but only a placeholder where the bar chart sits, while vision described the chart and the hand-drawn number line. You get the rule for your own projects, that text-only pages need OCR alone and anything visual adds vision, along with the parallel call pattern that runs both at once. The spec walkthrough covers the four model names, the reasoning effort values this model accepts and the HTTP 400 you get for the others, the plain text display rules, and the acceptance criteria for your build.

Watch the finished assistant handle real requests: a research question answered with sources, a salary chart built by running code, a watercolor illustration, and a follow-up the conversation remembers. Every handoff between agents shows up in the sidebar as it happens.

How the five-agent system is designed before any code is written. One router reads agent descriptions and delegates; four specialists carry their own tools; and a written spec tells an AI coding tool exactly what to build and how to verify that it works.

The code behind the assistant. Create agents with the Agents API, wire handoffs so the router can pass work to specialists, start conversations, and parse the typed outputs that come back. You read along in server.py, the same file you can run yourself.

Where the agents get their capabilities: web search, a sandboxed Python environment that builds real charts, and FLUX image generation, each wired with a single tool type. Then the output parsing that makes charts and sources actually render, and MCP, the open standard for connecting tools you build yourself.

The pieces that make it a real assistant: conversation state that survives follow-up messages, the always-start routing rule and the append bug behind it, server-sent events that stream progress to the browser, and agent cleanup on shutdown. Ends with the full demo walkthrough, now that you know what every piece is doing.