ProtectedinputMeter over the USER's microphone, when the driver attached one.
ProtectedoutputMeter over the AGENT's audio output, when the driver attached one.
ProtectedtracksTracks requested, established, or denied on this session.
ProtectedvideoStatic ReadonlyASSISTANT_All tracks evaluated for this session (including 'unsupported' or 'denied').
Established tracks on this session (tracks in the 'live' state).
The frame rate (frames per second) the live inbound video track was negotiated at — the model's
ceiling or the session's request, whichever is lower — or undefined when no inbound video track
is live or none declared a rate. Anything pacing a video feed reads THIS, never a constant of its own.
directly from the playout engine's playhead clock — this client OWNS the output buffer (no WebRTC playback events exist on Gemini), so "audibly playing" is precisely "scheduled audio extends beyond the audio context's current time". In an avatar session the avatar's video counts too, while some of an answer's media is still ahead of its playhead: its MP4 usually carries the voice. The idle video that follows an answer does not count.
true while a model response is in flight (generation started and not yet done).
Distinct from IsAudioPlaying: generation runs ahead of playback.
Hosts use this to gate interim narration so it never interrupts a reply.
Whether a turn is in progress. A turn opens at its first sign: the user's first transcribed words, a turn this
client sends (typed text, a spoken update), or the model's output (a transcription delta, a thought, a tool call,
an audio part, which in an avatar session must play as the voice, or, on an interactionStatus model, the
IN_PROGRESS status; the video an avatar streams between turns never opens one). It closes at the turn's
turnComplete (the true idle signal on an interactionStatus model) once no tool call is pending, or when the last
pending call's result goes out after that; and at a resume. The session moves to a new connection only between
turns.
How many concurrent inbound video streams the model accepts: the ceiling the driver declared when it
negotiated tracks, else 1 when an inbound video track is live and 0 when it is not.
A source arbiter reads this to decide whether several live video sources must be narrowed to one (with a note to the model on each switch) or passed through untouched.
Returns the latest session resumption handle reported by the server, if any.
Whether the model takes inbound video on this session: the driver declared an inbound video track it supports when it
negotiated tracks, and a stream limit above zero. No video track need be live; one is added when a capture starts
(AddTrack). false before tracks are negotiated, and for a driver that does not negotiate them.
ProtectedSupportsWhether this driver can open a relay session (ClientRealtimeSessionConfig.Transport 'relay'): connect to
MJAPI's realtime relay at the session's RelayUrl rather than to its provider. false here; a driver that speaks
its provider's protocol through the relay opts in by overriding it (the Gemini Enterprise client does).
Returns the count of video frames successfully sent over the established video track.
Returns the cumulative active video duration in seconds across sent video frames. Represents the wall-clock span between first and last sent frames (span-not-sum) for stream telemetry. Provider-reported ImageTokens remains the authoritative financial billing basis.
Adds a track to a running session, such as inbound video when the user starts their camera, resolved against
what the driver supported when it negotiated at connect. The new track is 'live', or 'unsupported' with the
reason (the model takes no such track, or no more inbound video streams). A live track with the same
direction, modality and source is returned as it is. Other tracks keep their ids.
The base re-negotiates locally, which is all a driver needs when its provider accepts the media without a setup change (Gemini Live takes video frames at any time). A driver whose wire must change overrides this and calls it.
The track, or null when the session has not negotiated tracks: before it connects, or with a driver
that does not negotiate them (only the Gemini driver does today).
ProtectedAssertRefuses a session this driver cannot open: a relay session, unless the driver opts in through SupportsRelayTransport. Every driver calls this first in Connect, before it touches the session's media or opens a connection (driver obligation #11). The message names the provider key and never quotes the relay URL, which carries the relay's ticket.
The session the host asked this driver to open.
ProtectedattachAttaches (replacing + closing any previous) the USER-microphone meter. null detaches.
ProtectedattachAttaches (replacing + closing any previous) the AGENT-output meter. null detaches.
has no explicit cancel frame — the client OWNS the audio plane, so cancelling means: flush the local playout queue (GeminiPcmPlayback) so speech stops immediately, mark the in-flight turn inactive, and drain queued sends (a queued tool result or context note takes the floor next — tool-result delivery is never dropped by a cancel). Server-side, the next client content sent naturally interrupts any residual generation per the Live API contract. The interrupted turn's accumulated transcript is kept — the provider's trailing frames finalize what WAS spoken. No-op when nothing is active. An avatar's video stops with the voice.
ProtectedclearDriver hook for Disconnect: drops the stream AND the handlers. Handlers must not survive
into a later Connect on a reused instance, or a stale host callback would receive the
next session's stream.
ProtectedcloseReleases both meters — every driver calls this from its disconnect/teardown path.
Opens the client-direct Gemini Live session: creates the playout engine, connects with
the ephemeral token + the server-built SessionConfig ({ model, config } — the same
values the server LOCKED into the token, so tampering is ignored by the API), negotiates
tracks, then wires the mic-capture worklet and optional video capture.
Reports 'connected' once Google confirms the session's setup (setupComplete): input
sent before that is dropped, so nothing goes out on the connection, and the microphone
does not start, until then. Reports 'listening' once audio is flowing.
When the server granted an avatar (the minted avatar block) and the host shows it (the
outbound video track is live), the avatar's video goes to the host once the connection is
open. When the host doesn't show it, the session connects audio only.
OptionalcameraStream: MediaStreamProtectedconnectCreation seam for the Gemini Live session. Production constructs a GoogleGenAI client
with the ephemeral token as the API key on the v1alpha API version (the only
version that accepts auth_tokens), then opens live.connect with the server-built
model + config. Rejects if the socket errors or closes before Gemini acknowledges the setup.
Unit tests override this to return an in-memory fake.
ProtectedconnectOpens a Live connection with connect, which passes the given callbacks to live.connect, and rejects if the
socket errors or closes before Gemini acknowledges the setup. live.connect (@google/genai 2.11+) resolves only on
the server's setupComplete frame and never rejects when the socket errors or closes first (a rejected token, model
or setup field), so without this the connect waits forever. A promise settles once, so callbacks after a successful
connect only reach the handlers. Every connectLiveSession override should open through this.
ProtectedcreateCreation seam for the mic-capture pipeline. Production delegates to the shared
createPcmMicCapture (a 16 kHz AudioContext, inline-Blob capture worklet, and a
zero-gain tail; each worklet block is PCM16-encoded and handed to onPcmChunk as base64).
Unit tests override this with a no-op fake (and may capture onPcmChunk to simulate mic
frames).
ProtectedcreateCreation seam for the playout engine. Production returns GeminiPcmPlayback.
ProtectedCreateCreation seam for the avatar's video player, called at connect when the server granted an avatar and the host shows it. Production returns VideoPlayout (Media Source Extensions); unit tests return a fake.
The type the grant names, whether the video carries the voice, and the hook that routes the element's audio into the PCM playback's Web Audio graph.
Tears down the session, mic capture, mic tracks, camera capture, and playout engine,
resets the response state machine, and emits a final 'closed' (unless already 'error').
Safe to call more than once.
ProtectedemitProtectedemitEmits a true barge-in interruption to the registered handler (if any).
ProtectedemitEmits the model/avatar's remote video to the registered handler (video drivers only). A driver with a plain
MediaStream passes it as it is; it is handed on as a 'stream' source.
ProtectedemitEmits a state change to the registered handler (if any).
ProtectedemitEmits a tool-call request to the registered handler (if any).
ProtectedemitEmits a track state change to the registered handler (if any).
ProtectedemitEmits a transcript event to the registered handler (if any).
ProtectedemitEmits a usage update (token deltas for a completed response/turn) to the registered handler (if any).
ProtectedemitEmits a user-speech-onset to the registered handler (if any).
The session's current audible activity, or null when this driver attached no
meters at all (hosts then keep their turn-state-driven visuals). Sampled by hosts
per animation frame — implementations are cheap, allocation-light reads of an
AnalyserNode; no per-call provider traffic.
Returns the AGENT's remote-audio MediaStream when this driver owns a tappable
agent-audio plane (a WebRTC driver's remote track, or a PCM driver's playout output),
or null otherwise. Hosts use it to MIX the agent's voice into a browser-side recording
alongside the mic; null (drivers with no tappable plane, or before the stream lands)
degrades gracefully to mic-only capture. Hosts may still call it as client.GetRemoteMediaStream?.().
Pure read — never mutates.
Returns whether a track of the requested modality and direction is established and 'live'.
ProtectednegotiateNegotiates requested tracks against model capability. Audio (inbound + outbound) is the baseline floor of every realtime session; any additional requested tracks (such as channel video) are unioned onto this baseline floor.
OptionalmaxInboundVideoStreams: numberRegisters the (single) error handler. Receives both fatal transport errors (which are
also followed by an 'error' state change) and non-fatal provider error frames.
Registers the (single) interruption handler.
True barge-in only: fires ONLY when USER INPUT CUT OFF ACTIVE MODEL OUTPUT — a model response in flight or audio audibly playing when the user took the floor. A user simply taking their normal turn while the model is idle is NOT an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a response is actually active or audio is playing).
On interruption, drivers must also flush locally-owned playback and report
IsAudioPlaying === false promptly (driver obligation #3). Hosts typically
use this hook to abort in-flight delegated work so a stale result is never narrated
into a conversation that has moved on.
Registers a handler invoked when the agent's remote-audio stream becomes available —
IMMEDIATELY (synchronously) if it has already landed, otherwise when the driver publishes
it (a WebRTC track typically lands AFTER Connect resolves). Lets a host attach the
agent voice to a recording that began before the stream landed. Handler errors are
isolated (logged) so a host bug never disturbs the call. Handlers are session-scoped:
clearRemoteMediaStream (called from Disconnect) drops them.
Registers the (single) remote-VIDEO handler — the model/avatar's video for a VIDEO session (a talking-head the host renders, e.g. as the agent's tile). Invoked once the provider's video is available.
The video is a MediaVideoSource: a live stream (a WebRTC track), or a player that must own the
<video> element (MSE or WebCodecs playout of encoded avatar video). Hosts show either with
AttachVideoSource.
Optional capability: audio-only drivers (the default) never emit — registering is always safe,
but hosts must not assume video arrives. A driver whose model sends video (the server driver declares an
outbound video track in its session's Capabilities.SupportedOutboundTracks, or renders an avatar,
BaseRealtimeModel.SupportsAvatarOutput) calls emitRemoteVideo when it is live.
Invoked with the remote video when it becomes available.
Registers the (single) state-change handler. States may be re-emitted (e.g. repeated
'listening' on successive barge-ins); hosts should treat emissions as idempotent.
Registers the (single) tool-call handler. The host executes the tool and calls SendToolResult with the outcome.
Registers the track-state-change handler — invoked when tracks transition in lifecycle
(e.g. established to 'live', 'denied', or 'unsupported').
Registers the (single) transcript handler. Receives interim deltas (IsFinal: false)
and final turn transcripts (IsFinal: true) for both roles, tagged with their
RealtimeClientTranscript.Kind.
Registers the (single) usage handler.
Emissions carry token deltas for the response/turn that just completed (see
RealtimeClientUsage — deltas preferred; cumulative-only providers must convert
in the driver). Hosts accumulate and relay/persist on their own cadence (e.g. the voice
session service debounces a RelayRealtimeUsage mutation onto the co-agent prompt run).
Optional capability: drivers whose provider exposes no usage telemetry simply never emit — registering a handler is always safe, but hosts must not assume emissions arrive. See RealtimeClientUsage for per-driver availability.
Registers the single handler invoked when the provider's voice-activity detection saw the user start to speak. Fires once per detected onset and may fire again inside one turn after a pause; it carries no text. Unlike OnInterruption, it fires whether or not the model was speaking.
ProtectedpublishDriver hook: records the agent's remote-audio stream and fans it out to every registered
handler. Passing null (a driver with no tappable plane) just records null and
notifies nobody.
Ends a track on a running session, such as inbound video when the user stops their camera: the track is
reported 'ended' and leaves AllTracks. Audio, the session's floor, cannot be removed.
Whether a track was ended.
Rebinds the PCM capture and the input meter to the stream's current track (obligation #10); the socket stays open.
Triggers ONE short spoken update. Gemini has no per-response instructions (OpenAI's
response.create.instructions), so this is EMULATED: the instructions ride as a
realtime-text user turn (sendRealtimeInput({ text }) — the path that triggers
generation on native-audio models, where mid-call sendClientContent is inert), and
the resulting turn is stamped Kind: 'narration' at send time (reset on
turnComplete) — mirroring the OpenAI driver's narration semantics. Queued behind any
in-flight turn so it can never interrupt a pending reply.
ProtectedresumeOpens a replacement connection that resumes the session from handle, switches to it once
Google confirms its setup, and closes the old one. Until then the old connection stays in
use, so what it still sends (the rest of a turn the deadline cut, its usage) is handled.
Mic capture and playout carry over untouched. Rejects when the connection can't be opened or
its setup fails, so RealtimeSessionResumption can retry.
The resume reuses the session's ephemeral token, which Google accepts until the token's
expireTime (the server driver mints 30 minutes). After that every attempt fails and the
session ends with a fatal error.
The resumption handle Google issued.
Marks the attempt abandoned after a timeout or a consumer close; a connection that opens after that is closed instead of used.
Injects background context as a user turn with turnComplete: false — appended to the
conversation WITHOUT starting generation, so the model draws on it the next time it
speaks. Gemini Live turns have no system role, so the user role carries it (the host owns
prefixing policy). Queued while a turn is in flight because ANY client content interrupts
in-flight generation on Gemini (a divergence from the OpenAI driver, which can inject
items mid-response safely).
SIDE EFFECT: turnComplete: false leaves the client content turn OPEN — the server
holds generation until a turnComplete: true commit arrives. openClientTurn
tracks this so SendToolResult can commit the turn and unblock the spoken reply.
Injects typed text as a realtime-text user turn (sendRealtimeInput({ text }) —
Gemini's "respond now" trigger on every Live model generation, including native-audio
models that ignore mid-call sendClientContent). No-op when the session is not open.
SendText implies barge-in (base-contract rule): an active spoken response is cancelled via CancelActiveResponse first — playback flushed, turn marked inactive, queued sends drained — so the typed turn takes the floor immediately. If a drained queued send (e.g. a pending tool result) starts a new turn, the text queues behind it, preserving the tool-result delivery invariant. On the wire, sending the user turn itself interrupts any residual server-side generation (Gemini Live's any-client-content-interrupts contract), so no explicit cancel frame exists or is needed.
Feeds an executed tool's result back via sendToolResponse, supplying the function name
cached from the originating tool call (Gemini requires it; the contract only carries the
callID). Sent immediately when idle — Gemini speaks the result as its next turn —
otherwise queued until the in-flight turn (e.g. a progress narration) completes.
If context notes left a client content turn OPEN (turnComplete: false), the tool
response is followed by an empty-turn commit (sendClientContent({ turnComplete: true }))
— without it the server keeps waiting for more client input and NEVER starts the spoken
reply (observed live as the model staying silent after a delegated agent's result).
Streams one base64 image frame over the established inbound video track.
Enforces a minimum inter-frame spacing as a backstop with deliberate jitter headroom for the
upstream pacers (the source arbiter, ChannelInboundVideoBridge, frameCapture, channel-level
gates such as OnScreencastFrame). The spacing is derived from the rate the track NEGOTIATED
(MinVideoFrameSpacingMs of BaseRealtimeClient.InboundVideoRate), so it follows the
model's profile rather than a constant: at the 1 fps every Live model accepts today it is the
750 ms gate this driver has always applied, and a faster model needs no change here. Upstream
generators are the primary enforcers of the nominal rate; this gate absorbs event-loop and async
dispatch jitter without dropping intended frames, while preventing an unpaced caller from bursting.
Gemini accepts ONE inbound video stream, so sourceId is ignored: choosing which source feeds that
stream is the arbiter's job, and by the time a frame arrives here the choice is made.
If inbound video is not established, returns false without error or frame sends (fallback).
A frame that comes while no connection is ready (before Google confirms the setup, or while
the session moves to a new connection) is dropped too: the source sends the next one at the track's rate.
Optional_sourceId: stringtrue if the frame was dispatched to the session; false if dropped (throttled,
track unestablished, or no connection ready).
Mutes / unmutes by toggling the mic tracks' enabled flag: the capture pipeline stays
up and streams SILENCE while muted (chosen over gating the worklet send so the provider's
VAD sees a continuous stream and the un-mute is glitch-free — same policy as the OpenAI
client driver).
Google Gemini implementation of BaseRealtimeClient: a browser-direct Gemini Live websocket session authenticated with the server-minted ephemeral auth token (
v1alphaauth_tokensmechanism — the token is passed as the SDKapiKey).Registered with the ClassFactory under the key
'gemini'— theProviderstring the server'sGeminiRealtimedriver stamps on itsClientRealtimeSessionConfig— so hosts resolve it without referencing this class directly.Owns ALL Gemini wire concerns (the behavioral twin of OpenAIRealtimeClient, adapted to Gemini's client-owned audio plane — there is no WebRTC here):
AudioWorklet(createMicCapture seam) and streamed withsendRealtimeInput.GeminiAvatarOutput). A host that shows no avatar gets an audio-only session.inputTranscription→ User deltas/finals,outputTranscription→ Assistant deltas/finals (accumulated like the OpenAI driver),toolCall.functionCalls→ OnToolCall (callID→name cached for SendToolResult),interrupted→ playback flush +'listening',turnComplete→ busy cleared + queued sends flushed,usageMetadata→ OnUsage (per-turn prompt/response token deltas with their modality split — see handleUsageMetadata). An avatar's generated video seconds go to OnUsage in updates of their own (OutputTokenDetails.VideoSeconds) at each turn boundary.response.createdframe, soIsBusyis set EAGERLY when this client triggers a response (text / narration / tool result) and on the first model output of a turn (audio part or output-transcription delta); cleared onturnComplete, and on atoolCallframe (the model has yielded the floor pending the tool result — so a slowturnCompletecan never deadlock the queued result). In an avatar session, video sets it only while the model's answer is under way: Vertex AI streams the avatar's video between answers too, and that idle video plays without marking the agent speaking or busy.sendClientContentinterrupts in-flight Gemini generation (per the Live API contract), so text / narration / context-note / tool-result sends issued while a turn is in flight are queued and flushed in order onturnComplete(the flush stops at the first send that starts a new response).goAway, and the session resumes on a new one with Google's handle (RealtimeSessionResumption). A planned move waits until no turn is in progress (IsTurnOpen). A connection is put to use only once Google confirms its setup (setupComplete): input that reaches it earlier is dropped. During a move, sends queue and the latest 2 s of mic audio are held; both go out on the new connection once it is ready.turnComplete: falseclient content, which tells the Live API MORE INPUT IS COMING — the server holds ALL generation (including the normally-automatic continuation after a tool response) until aturnComplete: truecommit. SendToolResult therefore followssendToolResponsewith an empty-turn commit whenever a note left the turn open, so the model speaks the result immediately (the behavioral equivalent of the OpenAI driver's explicitresponse.create).sendRealtimeInput({ text })— the Live API's documented in-conversation text path. Native-audio Live models treatsendClientContentas initial-history seeding only: a mid-callturnComplete: trueclient turn appends to history WITHOUT starting generation (the model stays silent until the user's next spoken turn), while realtime text triggers an immediate response on every model generation. Context notes stay onsendClientContent(turnComplete: false) — the silent history-append is exactly the contract they want.'narration'at send time (sends ARE the turn triggers on Gemini, unlike OpenAI whereresponse.createdconfirms) and reset to'normal'when the turn completes.