OpenAI Realtime

Name: Strands Agents SDK
Author: Strands Agents

The OpenAI Realtime API is a speech-to-speech interface that enables low-latency, natural voice conversations with AI. Key features include:

Bidirectional Interaction: The user and the model can provide input and output at the same time.
Interruptibility: Allows users to interrupt the AI mid-response, like in human conversations.
Multimodal Streaming: The API supports streaming of text and audio data.
Tool Use and Function Calling: Can use external tools to perform actions and get context while maintaining a real-time connection.
Secure Authentication: Uses tokens for secure client-side authentication.

Installation

OpenAI Realtime is configured as an optional dependency in Strands Agents.

To install it, run:

pip install 'strands-agents[bidi-io,bidi-openai,bidi-pyaudio]'

Or to install all bidirectional streaming providers at once:

pip install 'strands-agents[bidi-all,bidi-pyaudio]'

Usage

After installing the OpenAI Realtime and local audio extras, create a voice agent:

import asyncio

from strands.experimental.bidi.agent import BidiAgent
from strands.experimental.bidi.io import BidiAudioIO
from strands.experimental.bidi.models import OpenAIRealtimeModel
from strands.experimental.tools import stop
from strands.vended_tools import notebook


async def main() -> None:
    model = OpenAIRealtimeModel(
        model_id="gpt-realtime",
        voice="coral",
        api_key="<OPENAI_API_KEY>",
    )
    # stop tool allows user to verbally stop agent execution.
    agent = BidiAgent(model=model, tools=[notebook, stop])

    audio_io = BidiAudioIO()
    await agent.run(inputs=[audio_io.input()], outputs=[audio_io.output()])


if __name__ == "__main__":
    asyncio.run(main())

Configuration

Client Options

Parameter	Description	Example	Options
`api_key`	OpenAI API key used for authentication	`sk-...`	reference
`organization`	Organization associated with the connection. Used for authentication if required.	`myorg`	reference
`project`	Project associated with the connection. Used for authentication if required.	`myproj`	reference
`timeout_s`	OpenAI documents a 60 minute limit on realtime sessions (docs). However, OpenAI does not emit any warnings when approaching the limit. As a workaround, we allow users to configure a timeout (in seconds) on the client side to gracefully handle the connection closure.	`3000`	`[1, 3000]` (in seconds)

Model Config

Parameter	Description	Example	Options
`model_id`	OpenAI Realtime model identifier.	`"gpt-realtime"`	OpenAI models
`voice`	Output voice identifier. Defaults to `"alloy"`.	`"coral"`	Voice options
`params`	OpenAI Realtime session parameters. Audio must remain mono PCM at 24000 Hz.	`{"max_output_tokens": 4096}`	`session.update`
`connection`	Reconnect timing overrides.	`{"auto_reconnect": false}`	reference

Additional Provider Options

Use direct options such as voice for common settings, and pass additional OpenAI Realtime options through params.

from strands.experimental.bidi.models import OpenAIRealtimeModel

model = OpenAIRealtimeModel(
    api_key="<OPENAI_API_KEY>",
    voice="coral",
    params={"audio": {"input": {"turn_detection": {"threshold": 0.3}}}},
)

Nested dictionaries in params merge with the existing configuration, preserving unspecified fields. If a setting overlaps with a default or direct option, params takes precedence.

Calling update_config(params=...) replaces the entire params dictionary. The new values take effect on the next start() or restart().

Connection Restart

The OpenAI Realtime API exposes no server-side resume handle, so when a connection nears its time limit, BidiAgent restarts it and replays the accumulated conversation history into the new connection. The provider then sends a short system instruction so the model continues in the same language without re-introducing itself. The agent emits a BidiConnectionRestartEvent when this happens; treat it as informational rather than an error.

Set timeout_s in the client config to control when the client-side timeout fires ahead of OpenAI’s session limit (see the table above). For the full lifecycle, see Connection Restart.

Troubleshooting

Module Not Found

If you encounter the error ModuleNotFoundError: No module named 'websockets', this means the WebSocket dependency hasn’t been properly installed in your environment. To fix this, run pip install 'strands-agents[bidi-openai]'.

Authentication Errors

Set the OPENAI_API_KEY environment variable or pass the key through api_key.