openai / gpt-realtime-2.1

Overview

GPT Realtime 2.1 is OpenAI’s speech-to-speech model for live conversations. It runs as a stateful session that listens, reasons, and speaks back with natural timing. It can also call tools mid-conversation, so your agent can take actions and report results. It’s useful when you need a single model to handle speech, text, and interruptions.

What you can build

• A phone-style support agent that handles barge-in and noisy callers
• A voice concierge for scheduling, FAQs, and account lookups using tool calls
• A live coaching or tutoring voice assistant that can explain step-by-step
• A hands-free app interface that reads results aloud and confirms actions
• A voice-driven operations bot that runs structured workflows through tools

Inputs

• A selected assistant voice preset (for example Marin, Cedar, Alloy, Ash, Ballad, Coral, Echo, Fable, Nova, Onyx, Sage, Shimmer, Verse)
• System instructions that define the assistant’s behavior and tone, provided as text up to 5,000 characters
• An optional reasoning intensity level that trades speed for deeper problem solving
• An optional speech transcription model selection for turning microphone audio into text during the session
• Your microphone audio encoding, chosen from PCM 16-bit or telephony codecs (G.711 μ-law or A-law)
• Your microphone sample rate selection, typically 24 kHz for full quality or 8 kHz for telephony audio
• The assistant audio output encoding, chosen from PCM 16-bit or G.711 μ-law or A-law
• The assistant audio output sample rate selection, typically 24 kHz or 8 kHz
• Optional turn detection tuning, including a sensitivity value from 0 to 1 and a silence duration from 200 to 2,000 ms

Outputs

• Generated assistant speech as audio bytes in your chosen output encoding (PCM 16-bit or G.711 μ-law or A-law)
• Assistant text content for the same turn, suitable for captions, chat logs, or QA review
• Tool call requests when the assistant decides it needs an external action

Recommended settings

• For highest audio quality: use PCM 16-bit at 24 kHz for both input and output
• For telephony: use G.711 μ-law at 8 kHz end-to-end
• For most voice agents: keep reasoning intensity at a low level and raise it only for complex turns
• For best overall voice quality: start with Marin or Cedar
• For turn detection: start near 0.5 sensitivity and about 500 ms of silence, then tune per environment

Limitations

• This model does not support video.
• Image can be accepted as input, but audio output remains the main output path.
• Structured outputs are not supported, so don’t expect strict JSON-only responses.
• Once the session has produced audio, many realtime setups can’t change the voice for that session.
• Realtime sessions have a maximum duration of 60 minutes.
• If you send audio in chunks (common in streaming setups), each chunk has a fixed maximum size.
• Low-quality input audio can cause missed turns, bad transcription, or wrong intent.
  • Common issues include clipping, background noise, far-field microphones, and overlapping speakers.
• Voice availability can vary by realtime voice set.
  • If a selected voice errors, switch to a documented realtime voice like Marin or Cedar.

Safety & compliance

• Follow OpenAI’s Usage Policies and any applicable laws for your users and region.
• Get clear consent before recording, transcribing, or storing a user’s voice.
• Don’t build impersonation or deceptive voice experiences.
• Disclose when users are speaking with an AI voice agent.
• Use moderation and human review for high-stakes domains like finance, medical, or legal decisions.
• Treat tool outputs as untrusted input, and validate before taking real-world actions.