Building Low-Latency, Interruptible Voice Agents with AWS

Set up an AWS account, create an IAM user for CLI access, generate and download the access keys CSV, and follow the principle of least privilege.

Fork the voice agents repository on GitHub, create a local voice course directory, and add hotel_agent.py to paste and explore the code line by line.

Explain how the Nova Sonic Premier speech-to-speech model replaces traditional speech-to-text plus text-to-speech pipelines with a unified, streaming audio brain that processes input to output audio.

Learn how AWS bidirectional streaming enables low-latency, interruptible voice interactions by keeping a continuous two-way data pipe of audio chunks, text, and tool-use events in real time.

Configure audio settings for low latency voice agents by setting input 16 kHz, output 24 kHz, mono pi audio int16, and 1024-sample chunks to balance latency and CPU usage.

Explain how debug mode prints a line with timestamp, function name, and message. Use two timing functions for sync and async tasks, inspecting the stack to reveal the real caller.

Explore non-blocking asyncio in Python, using the event loop to run concurrent tasks like audio processing and DynamoDB lookups without freezing the voice agent.

Learn how the process tool async function schedules tool calls as tasks on the event loop, using coroutines and executor to run DynamoDB lookups non-blocking for a voice agent.

Learn how the voice agent checks a guest profile using the check guest tool, querying DynamoDB for name, loyalty tier, contact details, and preferences in a non-blocking async flow.

Explore how the check reservation status tool retrieves a guest's upcoming and past stays using a DynamoDB lookup, filters by guest name, and builds messages including balances due.

Updating reservations enables changing the room type and appending special requests via a dynamic DynamoDB update expression, requiring a reservation ID and returning the updated item.

Discover how the bedrock stream manager enables bidirectional streaming with AWS Bedrock using event templates, including start session and content events, and 16kHz mono 16-bit base64 audio.

Explore how audio and text event templates drive Nova Sonic, encoding microphone audio in base64 and routing events to Bedrock with prompt and content names.

Define and invoke a prompt start event to configure Nova Sonic's tool access, with guest, reservation, and update tools, and include input schemas and 24,000 Hz audio settings.

Build the tool result event returned to Nova Sonic after your Python tool completes, focusing on the content json string, the unique content name, and the DynamoDB-derived content.

Initializing the bedrock stream manager configures the model id and region, and sets up input and output queues, response task, and bedrock client scaffolding for low-latency, interruptible voice processing.

Initialize the bedrock real-time client from environment credentials, then open a bidirectional stream to start a session, send system prompts, and launch background tasks to process audio and responses.

Send raw event encodes a json payload to bedrock using the invoke model bidirectional stream input chunk, utf-8 encoded, while checking the stream is active and logging debug information.

Process audio input by initiating a content start event and streaming base64-encoded audio chunks from the microphone queue to Bedrock via Nova Sonic, maintaining an active bidirectional connection.

Push audio chunks into the bedrock stream queue and base64 encode them for Nova Sonic. Manage tool start and tool result events, content end event, and session in bidirectional streaming.

Process incoming bedrock responses by listening to the stream and parsing event payloads. Manage display flags, text output labeling (assistant or user), and non-blocking tool calls.

Handle tool requests in a low-latency voice agent by triggering asynchronous background tasks, tagging results with a uuid, and cleaning up with completion handlers.

Explain how to execute a tool and send results, using await to avoid blocking, and implement a graceful shutdown of the bedrock stream with task cancellation and error handling.