true while a model response is in flight (generation started and not yet done).
Distinct from IsAudioPlaying: generation runs ahead of playback.
Hosts use this to gate interim narration so it never interrupts a reply.
ProtectedattachAttaches (replacing + closing any previous) the USER-microphone meter. null detaches.
ProtectedattachAttaches (replacing + closing any previous) the AGENT-output meter. null detaches.
has no cancel frame — the client OWNS the audio plane, so cancelling means:
flush the local playout queue (speech stops immediately), mark the in-flight response
inactive, and drain queued sends (a queued tool result takes the floor next — delivery
is never dropped by a cancel). Residual server-side generation for the cancelled turn is
simply never played; the platform's own VAD handles SPOKEN barge-in (emitting
interruption). No-op when nothing is active.
ProtectedcloseReleases both meters — every driver calls this from its disconnect/teardown path.
Opens the client-direct conversation: socket to the signed URL, the initiation frame
carrying the server-authored prompt override, then — once the metadata confirms the
session config is applied — the audio plane at the NEGOTIATED sample rates. Reports
'listening' only after all of that (obligation #7).
ProtectedcreateCreation seam for the mic-capture pipeline at the NEGOTIATED input rate. Production
delegates to the shared createPcmMicCapture; unit tests override this with a
no-op fake (and may capture onPcmChunk to simulate mic frames).
ProtectedcreateCreation seam for the playout engine at the NEGOTIATED output rate. Production returns the shared RealtimePcmPlayback.
ProtectedcreateCreation seam for the conversation websocket. Production wraps the platform-global
WebSocket opened against the signed URL; unit tests override this to return an
in-memory fake. Handlers are attached by Connect AFTER this returns, so the
implementation must not require them at construction time.
Tears down the socket, mic capture, mic tracks, and playout engine, resets the response
state machine, and emits a final 'closed' (unless already 'error'). Safe to call
more than once.
ProtectedemitProtectedemitEmits a true barge-in interruption to the registered handler (if any).
ProtectedemitEmits the model/avatar's remote video stream to the registered handler (video drivers only).
ProtectedemitEmits a state change to the registered handler (if any).
ProtectedemitEmits a tool-call request to the registered handler (if any).
ProtectedemitEmits a transcript event to the registered handler (if any).
ProtectedemitEmits a usage update (token deltas for a completed response/turn) to the registered handler (if any).
The session's current audible activity, or null when this driver attached no
meters at all (hosts then keep their turn-state-driven visuals). Sampled by hosts
per animation frame — implementations are cheap, allocation-light reads of an
AnalyserNode; no per-call provider traffic.
OptionalGetReturns the AGENT's remote-audio MediaStream when this driver owns a tappable
remote-audio plane (e.g. a WebRTC peer-connection driver routes the model's audio track
here), or null otherwise. Hosts use it to MIX the agent's voice into a browser-side
recording alongside the mic; a null return (the default, and what every non-WebRTC
driver gives) degrades gracefully to mic-only capture.
Optional capability: the method itself is optional — call sites must use
client.GetRemoteMediaStream?.() ?? null. Drivers that can't expose a remote stream
simply don't implement it (or return null).
Registers the (single) error handler. Receives both fatal transport errors (which are
also followed by an 'error' state change) and non-fatal provider error frames.
Registers the (single) interruption handler.
True barge-in only: fires ONLY when USER INPUT CUT OFF ACTIVE MODEL OUTPUT — a model response in flight or audio audibly playing when the user took the floor. A user simply taking their normal turn while the model is idle is NOT an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a response is actually active or audio is playing).
On interruption, drivers must also flush locally-owned playback and report
IsAudioPlaying === false promptly (driver obligation #3). Hosts typically
use this hook to abort in-flight delegated work so a stale result is never narrated
into a conversation that has moved on.
OptionalOnOptional capability: registers a handler invoked when the agent's remote-audio stream
becomes available — immediately if it has already landed, otherwise when the WebRTC track
arrives (typically AFTER Connect resolves). Lets a host attach the agent voice to a
recording that began before the track landed. Call as client.OnRemoteMediaStream?.(cb);
drivers without a remote stream simply don't implement it.
Registers the (single) remote-VIDEO handler — the model/avatar's video track for a VIDEO session (a talking-head the host renders, e.g. as the agent's tile). Invoked once the provider publishes its video track.
Optional capability: audio-only drivers (the default) never emit — registering is always safe, but hosts must not assume a video track arrives. Video-capable drivers (BaseRealtimeModel.SupportsVideo) call emitRemoteVideo when the track is live.
Invoked with the remote video MediaStream when it becomes available.
Registers the (single) state-change handler. States may be re-emitted (e.g. repeated
'listening' on successive barge-ins); hosts should treat emissions as idempotent.
Registers the (single) tool-call handler. The host executes the tool and calls SendToolResult with the outcome.
Registers the (single) transcript handler. Receives interim deltas (IsFinal: false)
and final turn transcripts (IsFinal: true) for both roles, tagged with their
RealtimeClientTranscript.Kind.
Registers the (single) usage handler.
Emissions carry token deltas for the response/turn that just completed (see
RealtimeClientUsage — deltas preferred; cumulative-only providers must convert
in the driver). Hosts accumulate and relay/persist on their own cadence (e.g. the voice
session service debounces a RelayRealtimeUsage mutation onto the co-agent prompt run).
Optional capability: drivers whose provider exposes no usage telemetry simply never emit — registering a handler is always safe, but hosts must not assume emissions arrive. See RealtimeClientUsage for per-driver availability.
Triggers ONE short spoken update. EMULATED as a user_message carrying the
instructions, with the resulting response stamped Kind: 'narration' at send time
(reset on the response boundary) — see the class-level fidelity caveat. Queued behind
any in-flight response (a user_message sent mid-response would barge in on it), per
the base contract's collision rule.
Injects background context via contextual_update — NATIVE on this provider: the
platform's purpose-built non-interrupting context channel. Sent immediately even while
a response is in flight (the platform guarantees it never triggers or disturbs
generation), so unlike the Gemini driver no queueing applies.
Injects typed text as a user_message (the platform's "respond to this" trigger).
No-op when the session is not open.
SendText implies barge-in (base-contract rule): an active spoken response is cancelled via CancelActiveResponse first — playback flushed, response marked inactive, queued sends drained — so the typed turn takes the floor immediately. If a drained queued send (e.g. a pending tool result) starts a new response, the text queues behind it, preserving the tool-result delivery invariant. Server-side the fresh user turn takes the floor on its own; no explicit cancel frame exists or is needed.
Feeds an executed tool's result back via client_tool_result, correlated by the
platform's tool_call_id. EXACTLY-ONCE: the pending id is consumed when the result is
accepted, and a duplicate (or unknown) callID is dropped with a warning. Sent
immediately when idle — the platform speaks the result as the turn's continuation —
otherwise queued behind the in-flight response (e.g. a progress narration) so the
trigger is never lost.
Mutes / unmutes by toggling the mic tracks' enabled flag: the capture pipeline stays
up and streams SILENCE while muted (the provider's VAD sees a continuous stream and the
un-mute is glitch-free — same policy as the OpenAI and Gemini client drivers).
StaticParseParses a pcm_<rate> audio-format tag from the initiation metadata. Non-PCM formats
(e.g. ulaw_8000 — a telephony-only configuration) are not playable/encodable by the
shared PCM pipeline; the driver warns and falls back to the platform default so the
session degrades loudly rather than throwing.
StaticParseParses a JSON-stringified tool result into a structured value for the result slot,
falling back to the raw string so a free-text result still round-trips.
ElevenLabs Agents implementation of BaseRealtimeClient: a browser-direct conversation websocket authenticated with the server-minted SIGNED URL (the
EphemeralTokenIS thewss://…&token=…URL — no API key ever reaches the browser).Registered with the ClassFactory under the key
'elevenlabs'— theProviderstring the server'sElevenLabsRealtimedriver stamps on itsClientRealtimeSessionConfig.Owns ALL ElevenLabs wire concerns (the behavioral twin of the Gemini client driver — the audio plane is client-owned over a websocket, no WebRTC):
conversation_initiation_client_datacarrying the server-authored prompt override (from theSessionConfigpact) → wait forconversation_initiation_metadata→ negotiate the PCM rates fromuser_input_audio_format/agent_output_audio_format→ build the audio plane →'listening'. The state is gated on the metadata (driver obligation #7): only then has the platform confirmed the session — including the per-session system prompt — is applied.user_audio_chunkframes (note: a bare-key frame, not atyped one — that is the wire contract).audioevents decoded into the shared RealtimePcmPlayback at the negotiated output rate; IsAudioPlaying comes from its playout clock.user_transcript/agent_responseevents (no interim deltas), so every transcript this driver emits hasIsFinal: true.agent_response_correction(post-barge-in truncation) re-finalizes the assistant turn with the text that was ACTUALLY spoken — hosts persisting transcripts should treat a final assistant transcript arriving right after an interruption as the authoritative replacement of the previous one.IsBusyis set on the firstaudio/agent_responseof a turn (and eagerly when this client triggers a response) and cleared onagent_response_complete, oninterruption, and on aclient_tool_callframe (the agent has yielded the floor pending the result — deadlock guard, obligation #2).user_messageand the response kind is stamped'narration'AT SEND TIME (sends are the turn triggers; there is noresponse.created-style confirmation frame to stamp on), then reset on the response boundary. Fidelity caveat: the instruction enters the conversation as a user turn, so the agent may occasionally reference it; hosts should phrase narration instructions accordingly.interruption), and a typed SendText takes the floor server-side as a fresh user turn.