ProtecteddataProtectedinputMeter over the USER's microphone, when the driver attached one.
ProtectedoutputMeter over the AGENT's audio output, when the driver attached one.
ProtectedsessionStatic ReadonlyASSISTANT_Static ReadonlyMAX_Static ReadonlyPLAYBACK_Static ReadonlyPLAYBACK_Static ReadonlyPLAYBACK_Static ReadonlyPREBILL_true while model audio is AUDIBLY playing in the browser. Distinct from
IsBusy — audio plays at realtime while generation finishes early, so the
model can be "idle" while speech is still coming out of the speaker. Hosts must
gate narration on BOTH, or queued utterances come out late and stale.
true while a model response is in flight (generation started and not yet done).
Distinct from IsAudioPlaying: generation runs ahead of playback.
Hosts use this to gate interim narration so it never interrupts a reply.
ProtectedadoptProtectedattachAttaches (replacing + closing any previous) the USER-microphone meter. null detaches.
ProtectedattachAttaches (replacing + closing any previous) the AGENT-output meter. null detaches.
Cancels an active model response and flushes local playback.
ProtectedcloseReleases both meters — every driver calls this from its disconnect/teardown path.
Connects to the OpenAI Live WebRTC endpoint via the application server SDP broker.
Optional_cameraStream: MediaStreamProtectedcreateCreates a hidden <audio> element to sink the model's audio output.
ProtectedcreateCreates the peer connection. Production returns a real RTCPeerConnection.
Tears down the connection, data channel, audio tracks, and audio sink.
ProtectedemitProtectedemitEmits a true barge-in interruption to the registered handler (if any).
ProtectedemitEmits the model/avatar's remote video stream to the registered handler (video drivers only).
ProtectedemitEmits a state change to the registered handler (if any).
ProtectedemitEmits a tool-call request to the registered handler (if any).
ProtectedemitEmits a transcript event to the registered handler (if any).
ProtectedemitEmits a usage update (token deltas for a completed response/turn) to the registered handler (if any).
The session's current audible activity, or null when this driver attached no
meters at all (hosts then keep their turn-state-driven visuals). Sampled by hosts
per animation frame — implementations are cheap, allocation-light reads of an
AnalyserNode; no per-call provider traffic.
Returns the agent's remote audio MediaStream if available.
Registers the (single) error handler. Receives both fatal transport errors (which are
also followed by an 'error' state change) and non-fatal provider error frames.
Registers the (single) interruption handler.
True barge-in only: fires ONLY when USER INPUT CUT OFF ACTIVE MODEL OUTPUT — a model response in flight or audio audibly playing when the user took the floor. A user simply taking their normal turn while the model is idle is NOT an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a response is actually active or audio is playing).
On interruption, drivers must also flush locally-owned playback and report
IsAudioPlaying === false promptly (driver obligation #3). Hosts typically
use this hook to abort in-flight delegated work so a stale result is never narrated
into a conversation that has moved on.
Registers a callback for when the agent's remote audio MediaStream lands.
Registers the (single) remote-VIDEO handler — the model/avatar's video track for a VIDEO session (a talking-head the host renders, e.g. as the agent's tile). Invoked once the provider publishes its video track.
Optional capability: audio-only drivers (the default) never emit — registering is always safe, but hosts must not assume a video track arrives. Video-capable drivers (BaseRealtimeModel.SupportsVideo) call emitRemoteVideo when the track is live.
Invoked with the remote video MediaStream when it becomes available.
Registers the (single) state-change handler. States may be re-emitted (e.g. repeated
'listening' on successive barge-ins); hosts should treat emissions as idempotent.
Registers the (single) tool-call handler. The host executes the tool and calls SendToolResult with the outcome.
Registers the (single) transcript handler. Receives interim deltas (IsFinal: false)
and final turn transcripts (IsFinal: true) for both roles, tagged with their
RealtimeClientTranscript.Kind.
Registers the (single) usage handler.
Emissions carry token deltas for the response/turn that just completed (see
RealtimeClientUsage — deltas preferred; cumulative-only providers must convert
in the driver). Hosts accumulate and relay/persist on their own cadence (e.g. the voice
session service debounces a RelayRealtimeUsage mutation onto the co-agent prompt run).
Optional capability: drivers whose provider exposes no usage telemetry simply never emit — registering a handler is always safe, but hosts must not assume emissions arrive. See RealtimeClientUsage for per-driver availability.
ProtectedpostExchanges local SDP offer for remote answer SDP. Posts to application server broker or direct endpoint.
Requests a brief interim narration utterance during delegated background work.
Injects background context without triggering model speech.
Uses session.thinking.append with content parameter and delegation_id: null.
Injects user text into the live session and requests a model response. SendText implies barge-in and must NOT synthesize a user transcript (obligation #4).
Delivers a completed tool execution result back to the model. OpenAI Live remote reasoning plane requires response.item.create followed by response.create. Client delegation plane uses session.commentary.append with the delegation_id.
Mutes or unmutes the microphone capture tracks.
Browser WebRTC client driver for the OpenAI Live Realtime API (
gpt-live-1).Implements the browser-direct WebRTC topology specified in Section 5 of the OpenAI Live plan:
POST /v1/live/sessions), keeping the project API key and session configuration on the trusted server.'oai-events'data channel beforecreateOffer().iceGatheringState === 'complete').session.starton the data channel.session.input_audio.appendandsession.output_audio.deltaare forbidden on the data channel.audio.formatentirely (SDP negotiates codecs).Registered under
'openai-live'and'OpenAILiveRealtime'.