Realtime Voice

Overview

Realtime voice models enable two-way audio conversations with AI. Unlike standard model runs that process a single input and return a result, realtime sessions maintain a persistent WebSocket connection where you stream microphone audio and receive AI speech in real time.

The flow is:

1. Run the realtime model via POST /Run to get a socketaccesstoken
2. Connect to the WebSocket and send task_info with your token
3. Wait for task_stream_ready: the model is ready to receive audio
4. Stream microphone audio as binary frames
5. Receive AI audio as binary frames and play them
6. End the session with task_session_end

Run Parameters

Each realtime model has its own set of parameters. Use POST /Tool/Detail to discover the exact parameters for a specific model. See Model Parameters for details on parameter types.

Available realtime conversation models include:

• openai/gpt-realtime-mini: GPT Mini Realtime Voice Assistant
• openai/gpt-realtime: GPT Realtime Voice Assistant
• elevenlabs/realtime-conversational-ai: ElevenLabs Conversational AI

Common parameters across realtime models typically include voice selection, system instructions, and audio format settings. Example run for an OpenAI realtime model:

curl -X POST "https://api.wiro.ai/v1/Run/openai/gpt-realtime-mini" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "voice": "marin",
    "system_instructions": "You are a friendly assistant.",
    "input_audio_format": "audio/pcm",
    "output_audio_format": "audio/pcm",
    "input_audio_rate": "24000",
    "output_audio_rate": "24000"
  }'

Connection & Registration

After running the task, connect to the WebSocket and register with task_info:

var ws = new WebSocket("wss://socket.wiro.ai/v1");

ws.onopen = function() {
  ws.send(JSON.stringify({
    type: "task_info",
    tasktoken: "YOUR_SOCKET_ACCESS_TOKEN"
  }));
};

Realtime Events

During a realtime session, you'll receive these WebSocket events:

Audio Format

Both directions (microphone → server, server → client) use the same format:

Binary Frame Format

Every binary WebSocket frame (in both directions) is structured as:

[tasktoken]|[PCM audio data]

The pipe character | (0x7C) separates the token from the raw audio bytes.

Sending Microphone Audio

Capture microphone at 24 kHz using the Web Audio API with an AudioWorklet. Convert Float32 samples to Int16, prepend your task token, and send as a binary frame.

Key steps:

1. Request microphone with getUserMedia (enable echo cancellation and noise suppression)
2. Create an AudioContext at 24,000 Hz sample rate
3. Use an AudioWorklet to buffer and convert samples to Int16
4. Send each chunk as tasktoken|pcm_data binary frame

Receiving AI Audio

AI responses arrive as binary WebSocket frames in the same PCM Int16 24 kHz format. To play them:

1. Check if the message is a Blob (binary) before parsing as JSON
2. Find the pipe | separator and extract audio data after it
3. Convert Int16 → Float32 and create an AudioBuffer
4. Schedule gapless playback using AudioBufferSourceNode

Transcripts

Both user and AI speech are transcribed automatically. Transcripts arrive as task_output messages with a string prefix:

• TRANSCRIPT_USER: for what the user said
• TRANSCRIPT_AI: for what the AI said

// Example task_output message
{
  "type": "task_output",
  "message": "TRANSCRIPT_USER:What's the weather like today?"
}

{
  "type": "task_output",
  "message": "TRANSCRIPT_AI:I'd be happy to help, but I don't have access to real-time weather data."
}

Interrupting AI Speech

While the AI is speaking, you can interrupt it to take over the conversation. Send task_session_interrupt:

{
  "type": "task_session_interrupt",
  "tasktoken": "YOUR_SOCKET_ACCESS_TOKEN"
}

When the server receives this:

• The AI immediately stops generating audio
• A final task_stream_end is sent for the interrupted turn
• The session continues: the user can speak and the AI will respond to the next input

On the client side, stop audio playback immediately when the user triggers an interrupt. This gives instant feedback (AI voice cuts off) while the server processes the signal.

Ending a Session

To gracefully end a realtime session, send task_session_end:

{
  "type": "task_session_end",
  "tasktoken": "YOUR_SOCKET_ACCESS_TOKEN"
}

After sending this, the server will process any remaining audio, send final cost/transcript events, and then emit task_postprocess_end. Wait for task_postprocess_end before closing the WebSocket.