Member Junction
    Preparing search index...

    Browser WebRTC client driver for the OpenAI Live Realtime API (gpt-live-1).

    Implements the browser-direct WebRTC topology specified in Section 5 of the OpenAI Live plan:

    1. Exchanges SDP offer for answer with an application server SDP broker (POST /v1/live/sessions), keeping the project API key and session configuration on the trusted server.
    2. Creates the 'oai-events' data channel before createOffer().
    3. Waits for non-trickle ICE (iceGatheringState === 'complete').
    4. The HTTP SDP exchange starts the session — never sends session.start on the data channel.
    5. Media rides WebRTC tracks: session.input_audio.append and session.output_audio.delta are forbidden on the data channel.
    6. Omits audio.format entirely (SDP negotiates codecs).
    7. Accounts for OpenAI Live's 15-second WebRTC session pre-bill.

    Registered under 'openai-live' and 'OpenAILiveRealtime'.

    Hierarchy (View Summary)

    Index

    Constructors

    Properties

    Accessors

    Methods

    Constructors

    Properties

    callsFromBeforeReseed: Set<string> = ...

    Tool call IDs that were in flight when a non-native seed reconnect occurred.

    configuredCallDurationSeconds: number

    Duration of a single provider connection in seconds from ClientRealtimeSessionConfig.ConnectionMaxSeconds.

    configuredReconnectBufferSeconds: number = DEFAULT_REALTIME_RECONNECT_BUFFER_SECONDS

    Buffer in seconds before CallDurationSeconds expiry at which Reconnect is triggered.

    configuredSessionMaxSeconds: number

    Credential/session token lifetime in seconds from ClientRealtimeSessionConfig.SessionMaxSeconds (or MaxSessionSeconds).

    configuredTrustNativeResumption: boolean

    Optional metadata override for whether native resumption is trusted on this endpoint.

    conversationMemory: RealtimeConversationMemory = ...

    Rolling conversation memory used to re-seed provider context on reconnect.

    dataChannel: IRealtimeDataChannel = null
    inputAudioMeter: IRealtimeAudioMeter = null

    Meter over the USER's microphone, when the driver attached one.

    outputAudioMeter: IRealtimeAudioMeter = null

    Meter over the AGENT's audio output, when the driver attached one.

    reconnectController: RealtimeReconnectController = ...

    Unified I/O-free reconnect controller shared with server-side BaseRealtimeSession.

    sessionConfig: JSONObject = null
    tokenRefreshCallback: RealtimeClientTokenRefreshCallback = null

    Optional callback installed by RealtimeSessionRuntime to refresh the provider token mid-call.

    tracks: RealtimeTrack[] = []

    Tracks requested, established, or denied on this session.

    ASSISTANT_SAFETY_BACKSTOP_MS: 3000
    MAX_OUTBOUND_QUEUE_SIZE: 100
    PLAYBACK_DRAIN_SILENCE_MS: 200
    PLAYBACK_DRAIN_SILENCE_THRESHOLD: 0.01
    PLAYBACK_DRAIN_TRANSCRIPT_GAP_MS: 300
    PREBILL_DURATION_SECONDS: 15

    Accessors

    • get CallDurationSeconds(): number

      Effective connection ceiling in seconds. Metadata and runtime caps narrow the driver's provider ceiling (ClampSessionSecondsValue(DefaultCallDurationSeconds, RuntimeCallDurationSeconds, configuredCallDurationSeconds)), never widen it.

      Returns number

    • get InboundVideoRate(): number

      The frame rate (frames per second) the live inbound video track was negotiated at — the model's ceiling or the session's request, whichever is lower — or undefined when no inbound video track is live or none declared a rate. Anything pacing a video feed reads THIS, never a constant of its own.

      Returns number

    • get IsAudioPlaying(): boolean

      true while model audio is AUDIBLY playing in the browser. Distinct from IsBusy — audio plays at realtime while generation finishes early, so the model can be "idle" while speech is still coming out of the speaker. Hosts must gate narration on BOTH, or queued utterances come out late and stale.

      Returns boolean

    • get IsReconnectCycleActive(): boolean

      True from the start of a reconnect cycle until it succeeds or fails. Unlike IsReconnecting, it stays true while a planned move waits between attempts with the call live again on its old connection. A driver that would otherwise end the call on a loss of that connection checks this and hands the loss to the running cycle (TriggerReconnect('connection-lost')) instead.

      Returns boolean

    • get MaxInboundVideoStreams(): number

      How many concurrent inbound video streams the model accepts: the ceiling the driver declared when it negotiated tracks, else 1 when an inbound video track is live and 0 when it is not.

      A source arbiter reads this to decide whether several live video sources must be narrowed to one (with a note to the model on each switch) or passed through untouched.

      Returns number

    • get RequiresPostTurnResumptionHandle(): boolean

      Whether this driver's provider emits a fresh resumable handle shortly after each turnComplete (Gemini / Vertex AI Live), so planned moves wait for that post-turn handle.

      Returns boolean

    • get RuntimeCallDurationSeconds(): number

      Optional runtime ceiling override (e.g. Gemini's 120s ceiling when uncompressed video is active). Narrowed against DefaultCallDurationSeconds and configuredCallDurationSeconds via ClampSessionSecondsValue.

      Returns number

    • get StateToRestoreOnOldConnection(): RealtimeClientState

      The state to report again when ResumeOnOldConnection goes back to the old connection, or null when the old connection has reported a state of its own since the move started (it keeps delivering events during the attempt), which is current. 'speaking' while the turn is open, else 'listening': the turn, not the state the move replaced, says whether the agent is answering, since a driver can leave 'speaking' without reporting it (a tool call) and the old connection can start an answer during the attempt. A 'speaking' report opens a turn, and one with no answer behind it would hold the next attempt until the last call. 'connected' is never reported: a host keeps showing the move for it.

      Returns RealtimeClientState

    • get SupportsInboundVideo(): boolean

      Whether the model takes inbound video on this session: the driver declared an inbound video track it supports when it negotiated tracks, and a stream limit above zero. No video track need be live; one is added when a capture starts (AddTrack). false before tracks are negotiated, and for a driver that does not negotiate them.

      Returns boolean

    • get SupportsRelayTransport(): boolean

      Whether this driver can open a relay session (ClientRealtimeSessionConfig.Transport 'relay'): connect to MJAPI's realtime relay at the session's RelayUrl rather than to its provider. false here; a driver that speaks its provider's protocol through the relay opts in by overriding it (the Gemini Enterprise client does).

      Returns boolean

    Methods

    • Adds a track to a running session, such as inbound video when the user starts their camera, resolved against what the driver supported when it negotiated at connect. The new track is 'live', or 'unsupported' with the reason (the model takes no such track, or no more inbound video streams). A live track with the same direction, modality and source is returned as it is. Other tracks keep their ids.

      The base re-negotiates locally, which is all a driver needs when its provider accepts the media without a setup change (Gemini Live takes video frames at any time). A driver whose wire must change overrides this and calls it.

      Parameters

      Returns RealtimeTrack

      The track, or null when the session has not negotiated tracks: before it connects, or with a driver that does not negotiate them (only the Gemini driver does today).

    • Arms (or re-arms) the soft and hard connection-ceiling timers for the current transport socket.

      • Soft timer opens the turn-boundary window before CallDurationSeconds.
      • Hard timer fires at CallDurationSeconds - ReconnectBufferSeconds and forces a reconnect; while the model is answering it waits for the turn to end, up to shortly before CallDurationSeconds. Both measure from the connection deadline instead when it comes before CallDurationSeconds (connectionDeadlineAtMs: the expiry of the credential the connection opened on, on a driver whose connections close with it).

      Returns void

    • Delivers a completed tool execution result back to the model. OpenAI Live remote reasoning plane requires response.item.create followed by response.create. Client delegation plane uses session.commentary.append with the delegation_id.

      Parameters

      • callID: string
      • outputJson: string

      Returns boolean | void

    • Returns the AGENT's remote-audio MediaStream when this driver owns a tappable agent-audio plane (a WebRTC driver's remote track, or a PCM driver's playout output), or null otherwise. Hosts use it to MIX the agent's voice into a browser-side recording alongside the mic; null (drivers with no tappable plane, or before the stream lands) degrades gracefully to mic-only capture. Hosts may still call it as client.GetRemoteMediaStream?.(). Pure read — never mutates.

      Returns MediaStream

    • Whether the user said something the current connection has not answered yet, with no turn open for it. The hard-point move then waits as it does for an open turn, until shortly before the provider limit. false here: a driver whose provider opens the turn only once the answer starts overrides it, and calls reconnectController.NotifyIdleStateChanged() when the hold ends without a turn.

      Returns boolean

    • Registers the (single) interruption handler.

      True barge-in only: fires ONLY when USER INPUT CUT OFF ACTIVE MODEL OUTPUT — a model response in flight or audio audibly playing when the user took the floor. A user simply taking their normal turn while the model is idle is NOT an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a response is actually active or audio is playing).

      On interruption, drivers must also flush locally-owned playback and report IsAudioPlaying === false promptly (driver obligation #3). Hosts typically use this hook to abort in-flight delegated work so a stale result is never narrated into a conversation that has moved on.

      Parameters

      • handler: () => void

      Returns void

    • Registers a handler invoked when the agent's remote-audio stream becomes available — IMMEDIATELY (synchronously) if it has already landed, otherwise when the driver publishes it (a WebRTC track typically lands AFTER Connect resolves). Lets a host attach the agent voice to a recording that began before the stream landed. Handler errors are isolated (logged) so a host bug never disturbs the call. Handlers are session-scoped: clearRemoteMediaStream (called from Disconnect) drops them.

      Parameters

      • handler: (stream: MediaStream) => void

      Returns void

    • Registers the (single) remote-VIDEO handler — the model/avatar's video for a VIDEO session (a talking-head the host renders, e.g. as the agent's tile). Invoked once the provider's video is available.

      The video is a MediaVideoSource: a live stream (a WebRTC track), or a player that must own the <video> element (MSE or WebCodecs playout of encoded avatar video). Hosts show either with AttachVideoSource.

      Optional capability: audio-only drivers (the default) never emit — registering is always safe, but hosts must not assume video arrives. A driver whose model sends video (the server driver declares an outbound video track in its session's Capabilities.SupportedOutboundTracks, or renders an avatar, BaseRealtimeModel.SupportsAvatarOutput) calls emitRemoteVideo when it is live.

      Parameters

      • handler: (video: MediaVideoSource) => void

        Invoked with the remote video when it becomes available.

      Returns void

    • Registers the (single) usage handler.

      Emissions carry token deltas for the response/turn that just completed (see RealtimeClientUsage — deltas preferred; cumulative-only providers must convert in the driver). Hosts accumulate and relay/persist on their own cadence (e.g. the voice session service debounces a RelayRealtimeUsage mutation onto the co-agent prompt run).

      Optional capability: drivers whose provider exposes no usage telemetry simply never emit — registering a handler is always safe, but hosts must not assume emissions arrive. See RealtimeClientUsage for per-driver availability.

      Parameters

      Returns void

    • Registers the single handler invoked when the provider's voice-activity detection saw the user start to speak. Fires once per detected onset and may fire again inside one turn after a pause; it carries no text. When the provider's event names the conversation item this speech belongs to (OpenAI-protocol input_audio_buffer.speech_started.item_id), itemId carries that id so the host can pair the onset with the matching user transcript (RealtimeClientTranscript.ItemId); otherwise itemId is undefined. Unlike OnInterruption, it fires whether or not the model was speaking.

      Parameters

      • handler: (itemId?: string) => void

      Returns void

    • A planned move's attempt failed while the old connection still carries the call: go back to live on it (state, held input) until the next attempt, which waits for the next idle boundary or the hard point, and return true. Return false when the old connection cannot carry the call any more; the cycle then retries on its schedule. IsReconnecting is already false when this runs, so sends go to the old connection. false here: a driver opts in by overriding it. See RealtimeReconnectControllerOptions.ReturnToOldConnection.

      Returns boolean

    • Feeds an executed tool's result back to the model, correlated by callID, and ensures the model SPEAKS the result as soon as possible: immediately when idle, otherwise queued behind the in-flight response (e.g. a progress narration) so the trigger is never dropped by the provider.

      Template method: dispatches to doSendToolResult first (so the wire send happens before soft-window reconnect evaluation and rejected/unknown call IDs can return false), then records the result into conversationMemory via recordToolResult.

      Parameters

      • callID: string

        The CallID from the originating RealtimeClientToolCall.

      • outputJson: string

        The tool's result as a JSON-stringified string.

      Returns void

    • Streams one video frame as base64-encoded image data to the model. Optional capability — drivers that support inbound video tracks implement this.

      Parameters

      • base64Image: string

        Base64-encoded image data.

      • OptionalmimeType: string

        Image MIME type (defaults to 'image/jpeg').

      • OptionalsourceId: string

        Which source this frame came from. A driver whose model takes ONE inbound video stream ignores it (the source arbiter already chose what to send); a driver whose model takes several routes the frame to the track bound to that SourceID.

      Returns boolean

      true if accepted and sent; false if dropped (throttled, unestablished, etc.).