Class BaseRealtimeClientAbstract

true while model audio is AUDIBLY playing in the browser. Distinct from IsBusy — audio plays at realtime while generation finishes early, so the model can be "idle" while speech is still coming out of the speaker. Hosts must gate narration on BOTH, or queued utterances come out late and stale.

true while a model response is in flight (generation started and not yet done). Distinct from IsAudioPlaying: generation runs ahead of playback. Hosts use this to gate interim narration so it never interrupts a reply.

Cancels the model's ACTIVE spoken response and flushes pending playback so a new user turn can take the floor immediately; no-op when nothing is active. MUST NOT affect server-side delegated work — cancelling speech is a floor-control action, not an abort of in-flight tools/agents (hosts abort delegated work from OnInterruption / their own policy, never from this call).

Implementations must leave IsBusy and IsAudioPlaying honest after the cancel (response inactive, local playback flushed).

Opens the provider connection using the server-minted ephemeral credential and applies config.SessionConfig once the control channel is ready — so prompt and tool authority stay server-side even though the browser owns the socket. The payload's SHAPE is a private pact between this driver and the same-keyed server driver that minted it (see driver obligation #8); how it is applied is entirely driver-specific.

Tears down the provider connection and all client-held resources (control channel, transport, mic tracks, audio sink) and emits a final 'closed' state (unless the session already ended in 'error'). Safe to call more than once.

The session's current audible activity, or null when this driver attached no meters at all (hosts then keep their turn-state-driven visuals). Sampled by hosts per animation frame — implementations are cheap, allocation-light reads of an AnalyserNode; no per-call provider traffic.

Returns the AGENT's remote-audio MediaStream when this driver owns a tappable remote-audio plane (e.g. a WebRTC peer-connection driver routes the model's audio track here), or null otherwise. Hosts use it to MIX the agent's voice into a browser-side recording alongside the mic; a null return (the default, and what every non-WebRTC driver gives) degrades gracefully to mic-only capture.

Optional capability: the method itself is optional — call sites must use client.GetRemoteMediaStream?.() ?? null. Drivers that can't expose a remote stream simply don't implement it (or return null).

Registers the (single) error handler. Receives both fatal transport errors (which are also followed by an 'error' state change) and non-fatal provider error frames.

Registers the (single) interruption handler.

True barge-in only: fires ONLY when USER INPUT CUT OFF ACTIVE MODEL OUTPUT — a model response in flight or audio audibly playing when the user took the floor. A user simply taking their normal turn while the model is idle is NOT an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a response is actually active or audio is playing).

On interruption, drivers must also flush locally-owned playback and report IsAudioPlaying === false promptly (driver obligation #3). Hosts typically use this hook to abort in-flight delegated work so a stale result is never narrated into a conversation that has moved on.

Optional capability: registers a handler invoked when the agent's remote-audio stream becomes available — immediately if it has already landed, otherwise when the WebRTC track arrives (typically AFTER Connect resolves). Lets a host attach the agent voice to a recording that began before the track landed. Call as client.OnRemoteMediaStream?.(cb); drivers without a remote stream simply don't implement it.

Registers the (single) remote-VIDEO handler — the model/avatar's video track for a VIDEO session (a talking-head the host renders, e.g. as the agent's tile). Invoked once the provider publishes its video track.

Optional capability: audio-only drivers (the default) never emit — registering is always safe, but hosts must not assume a video track arrives. Video-capable drivers (BaseRealtimeModel.SupportsVideo) call emitRemoteVideo when the track is live.

Registers the (single) state-change handler. States may be re-emitted (e.g. repeated 'listening' on successive barge-ins); hosts should treat emissions as idempotent.

Registers the (single) tool-call handler. The host executes the tool and calls SendToolResult with the outcome.

Registers the (single) transcript handler. Receives interim deltas (IsFinal: false) and final turn transcripts (IsFinal: true) for both roles, tagged with their RealtimeClientTranscript.Kind.

Registers the (single) usage handler.

Emissions carry token deltas for the response/turn that just completed (see RealtimeClientUsage — deltas preferred; cumulative-only providers must convert in the driver). Hosts accumulate and relay/persist on their own cadence (e.g. the voice session service debounces a RelayRealtimeUsage mutation onto the co-agent prompt run).

Optional capability: drivers whose provider exposes no usage telemetry simply never emit — registering a handler is always safe, but hosts must not assume emissions arrive. See RealtimeClientUsage for per-driver availability.

Asks the model for ONE brief interim utterance (e.g. a spoken progress update while delegated work runs). Implementations MUST tag the resulting turn's transcripts with Kind: 'narration' and MUST NOT let this response collide with a pending tool-result reply — the tool result is queued and spoken as soon as the narration finishes.

Collision rule — drivers MUST queue or skip: when a model response is already in flight, the driver must either defer the update to the next turn boundary or skip it outright (skipping is acceptable — narration is disposable by contract; a stale "still working…" line has no value once the real result lands). Hosts SHOULD still gate calls on IsBusy / IsAudioPlaying for timing quality — a well-timed update beats a queued-then-stale one — but the driver is the safety net.

Injects background context into the model's conversation WITHOUT forcing a reply — e.g. delegated-run progress the model can draw on the next time it speaks.

Injects typed text into the live session as a USER turn and asks the model to respond. Implementations must route the reply through the same collision-safe path as tool results so it never collides with an in-flight response. No-op when the control channel is not open.

SendText implies barge-in: an active spoken response is cancelled (via the driver's own CancelActiveResponse path) before the text is injected, so the typed turn takes the floor immediately instead of waiting behind stale speech.

Implementations must NOT synthesize a user-role transcript event for the injected text — the host owns the local echo (driver obligation #4).

Feeds an executed tool's result back to the model, correlated by callID, and ensures the model SPEAKS the result as soon as possible: immediately when idle, otherwise queued behind the in-flight response (e.g. a progress narration) so the trigger is never dropped by the provider.

Mutes / unmutes the microphone (implementations toggle the mic tracks' enabled flag — the transport stays up; the provider just receives silence while muted).