ProtectedcallsTool call IDs that were in flight when a non-native seed reconnect occurred.
ProtectedconfiguredDuration of a single provider connection in seconds from ClientRealtimeSessionConfig.ConnectionMaxSeconds.
ProtectedconfiguredBuffer in seconds before CallDurationSeconds expiry at which Reconnect is triggered.
ProtectedconfiguredCredential/session token lifetime in seconds from ClientRealtimeSessionConfig.SessionMaxSeconds (or MaxSessionSeconds).
ProtectedconfiguredOptional metadata override for whether native resumption is trusted on this endpoint.
Protected ReadonlyconversationRolling conversation memory used to re-seed provider context on reconnect.
ProtectedinputMeter over the USER's microphone, when the driver attached one.
ProtectedoutputMeter over the AGENT's audio output, when the driver attached one.
Protected ReadonlyreconnectUnified I/O-free reconnect controller shared with server-side BaseRealtimeSession.
ProtectedtokenOptional callback installed by RealtimeSessionRuntime to refresh the provider token mid-call.
ProtectedtracksTracks requested, established, or denied on this session.
All tracks evaluated for this session (including 'unsupported' or 'denied').
Effective connection ceiling in seconds. Metadata and runtime caps narrow the driver's
provider ceiling (ClampSessionSecondsValue(DefaultCallDurationSeconds, RuntimeCallDurationSeconds, configuredCallDurationSeconds)),
never widen it.
Accessor for the client's rolling conversation memory ledger.
ProtectedDefaultDriver fallback connection ceiling in seconds when model metadata omits connectionMaxSeconds.
ProtectedDefaultEstablished tracks on this session (tracks in the 'live' state).
The frame rate (frames per second) the live inbound video track was negotiated at — the model's
ceiling or the session's request, whichever is lower — or undefined when no inbound video track
is live or none declared a rate. Anything pacing a video feed reads THIS, never a constant of its own.
true while a model response is in flight (generation started and not yet done).
Distinct from IsAudioPlaying: generation runs ahead of playback.
Hosts use this to gate interim narration so it never interrupts a reply.
ProtectedIsTrue from the start of a reconnect cycle until it succeeds or fails. Unlike IsReconnecting, it stays true
while a planned move waits between attempts with the call live again on its old connection. A driver that would
otherwise end the call on a loss of that connection checks this and hands the loss to the running cycle
(TriggerReconnect('connection-lost')) instead.
ProtectedIsTrue while a mid-call reconnect is actively in flight.
How many concurrent inbound video streams the model accepts: the ceiling the driver declared when it
negotiated tracks, else 1 when an inbound video track is live and 0 when it is not.
A source arbiter reads this to decide whether several live video sources must be narrowed to one (with a note to the model on each switch) or passed through untouched.
Seconds before CallDurationSeconds when the reconnect buffer window opens.
ProtectedRequiresWhether this driver requires a freshly minted single-use credential on each reconnect.
ProtectedRequiresWhether this driver's provider emits a fresh resumable handle shortly after each turnComplete
(Gemini / Vertex AI Live), so planned moves wait for that post-turn handle.
Latest recorded provider resumption handle or conversation ID.
ProtectedRuntimeOptional runtime ceiling override (e.g. Gemini's 120s ceiling when uncompressed video is active).
Narrowed against DefaultCallDurationSeconds and configuredCallDurationSeconds via ClampSessionSecondsValue.
ProtectedStateThe state to report again when ResumeOnOldConnection goes back to the old connection, or null when the
old connection has reported a state of its own since the move started (it keeps delivering events during the
attempt), which is current. 'speaking' while the turn is open, else 'listening': the turn, not the state the
move replaced, says whether the agent is answering, since a driver can leave 'speaking' without reporting it
(a tool call) and the old connection can start an answer during the attempt. A 'speaking' report opens a turn,
and one with no answer behind it would hold the next attempt until the last call. 'connected' is never
reported: a host keeps showing the move for it.
Whether the model takes inbound video on this session: the driver declared an inbound video track it supports when it
negotiated tracks, and a stream limit above zero. No video track need be live; one is added when a capture starts
(AddTrack). false before tracks are negotiated, and for a driver that does not negotiate them.
ProtectedSupportsWhether this concrete client subclass implements mid-call Reconnect.
ProtectedSupportsWhether this driver can open a relay session (ClientRealtimeSessionConfig.Transport 'relay'): connect to
MJAPI's realtime relay at the session's RelayUrl rather than to its provider. false here; a driver that speaks
its provider's protocol through the relay opts in by overriding it (the Gemini Enterprise client does).
Whether native provider resumption is trusted on the first reconnect attempt.
Adds a track to a running session, such as inbound video when the user starts their camera, resolved against
what the driver supported when it negotiated at connect. The new track is 'live', or 'unsupported' with the
reason (the model takes no such track, or no more inbound video streams). A live track with the same
direction, modality and source is returned as it is. Other tracks keep their ids.
The base re-negotiates locally, which is all a driver needs when its provider accepts the media without a setup change (Gemini Live takes video frames at any time). A driver whose wire must change overrides this and calls it.
The track, or null when the session has not negotiated tracks: before it connects, or with a driver
that does not negotiate them (only the Gemini driver does today).
Reads catalog-stamped timing and resumption settings from the typed fields of config, falling
back to config.SessionConfig (connectionMaxSeconds, sessionMaxSeconds, reconnectBufferSeconds,
trustNativeResumption), records the initial credential snapshot, and schedules proactive
credential/relay refresh.
Arms (or re-arms) the soft and hard connection-ceiling timers for the current transport socket.
CallDurationSeconds.CallDurationSeconds - ReconnectBufferSeconds and forces a reconnect; while the model is
answering it waits for the turn to end, up to shortly before CallDurationSeconds.
Both measure from the connection deadline instead when it comes before CallDurationSeconds
(connectionDeadlineAtMs: the expiry of the credential the connection opened on, on a driver whose
connections close with it).ProtectedAssertRefuses a session this driver cannot open: a relay session, unless the driver opts in through SupportsRelayTransport. Every driver calls this first in Connect, before it touches the session's media or opens a connection (driver obligation #11). The message names the provider key and never quotes the relay URL, which carries the relay's ticket.
The session the host asked this driver to open.
ProtectedattachAttaches (replacing + closing any previous) the USER-microphone meter. null detaches.
ProtectedattachAttaches (replacing + closing any previous) the AGENT-output meter. null detaches.
has no cancel frame — the client OWNS the audio plane, so cancelling means:
flush the local playout queue (speech stops immediately), mark the in-flight reply
inactive, SUPPRESS residual reply.audio chunks of the cancelled reply until the next
reply boundary, and drain queued sends (a queued narration takes the floor next —
delivery is never dropped by a cancel). No-op when nothing is active.
ProtectedclearDriver hook for Disconnect: drops the stream AND the handlers. Handlers must not survive
into a later Connect on a reused instance, or a stale host callback would receive the
next session's stream.
ProtectedcloseReleases both meters — every driver calls this from its disconnect/teardown path.
Opens the client-direct session: socket to the token-authenticated endpoint, the
server-authored session.update as the first frame, then — once session.ready
confirms the config is applied — the audio plane at the provider's fixed 24 kHz
format. Reports 'listening' only after all of that (obligation #7).
ProtectedConnectionWhether the provider closes a connection when the credential it opened on expires, even after the client holds a
newer one. false (the default) for a credential that only gates opening the connection, as OpenAI's
client_secret does. A driver returning true gets a planned move onto the refreshed credential before the old
one expires (see connectionDeadlineAtMs), instead of a drop at the expiry.
ProtectedcreateCreation seam for the mic-capture pipeline at the provider's fixed 24 kHz rate.
Production delegates to the shared createPcmMicCapture; unit tests override
this with a no-op fake (and may capture onPcmChunk to simulate mic frames).
ProtectedcreateCreation seam for the playout engine at the provider's fixed 24 kHz rate. Production returns the shared RealtimePcmPlayback.
ProtectedcreateCreation seam for the agent websocket. Production wraps the platform-global
WebSocket opened against the token-authenticated URL; unit tests override this to
return an in-memory fake. Handlers are attached by Connect AFTER this returns,
so the implementation must not require them at construction time.
ProtectedCredentialHow many connections credential may open, or undefined for no limit (the default). A driver whose provider
counts uses on a credential overrides it, so the client re-mints before it would open a connection the
credential has no use left for, instead of spending a failed attempt finding out.
Tears down the socket, mic capture, mic tracks, and playout engine, resets the response
state machine, and emits a final 'closed' (unless already 'error'). Sends
session.end first — without it the provider holds the session (billable) for a
30-second resume window. Safe to call more than once.
ProtectedDisposeCancels all reconnect and duration timers when the session is disconnected.
ProtecteddoFeeds an executed tool's result back via tool.result, correlated by the provider's
call_id. EXACTLY-ONCE: the pending id is consumed when the result is accepted, and a
duplicate (or unknown) callID is dropped with a warning. Sent immediately when idle —
the provider speaks the result as the turn's continuation (no explicit generation
trigger exists or is needed) — otherwise queued behind the in-flight reply (e.g. a
progress narration) so the trigger is never lost and the narration's kind tagging is
not clobbered. The wire result slot expects a JSON-STRING, which is exactly the
contract's outputJson shape — it passes through verbatim.
ProtectedemitProtectedemitEmits a true barge-in interruption to the registered handler (if any).
ProtectedemitEmits the model/avatar's remote video to the registered handler (video drivers only). A driver with a plain
MediaStream passes it as it is; it is handed on as a 'stream' source.
ProtectedemitEmits a state change to the registered handler (if any) and flushes any deferred turn-boundary reconnect.
ProtectedemitEmits a tool-call request to the registered handler (if any) and records it in memory.
ProtectedemitEmits a track state change to the registered handler (if any).
ProtectedemitEmits a transcript event to the registered handler (if any) and records final normal turns in memory.
ProtectedemitEmits a usage update (token deltas for a completed response/turn) to the registered handler (if any).
ProtectedemitEmits a user-speech-onset to the registered handler (if any).
OptionalitemId: stringThe session's current audible activity, or null when this driver attached no
meters at all (hosts then keep their turn-state-driven visuals). Sampled by hosts
per animation frame — implementations are cheap, allocation-light reads of an
AnalyserNode; no per-call provider traffic.
Returns the AGENT's remote-audio MediaStream when this driver owns a tappable
agent-audio plane (a WebRTC driver's remote track, or a PCM driver's playout output),
or null otherwise. Hosts use it to MIX the agent's voice into a browser-side recording
alongside the mic; null (drivers with no tappable plane, or before the stream lands)
degrades gracefully to mic-only capture. Hosts may still call it as client.GetRemoteMediaStream?.().
Pure read — never mutates.
ProtectedholdsWhether the user said something the current connection has not answered yet, with no turn open for it. The
hard-point move then waits as it does for an open turn, until shortly before the provider limit. false here: a
driver whose provider opens the turn only once the answer starts overrides it, and calls
reconnectController.NotifyIdleStateChanged() when the hold ends without a turn.
ProtectedisReturns true when the current credential is expired or within marginSeconds of expiry.
ProtectedisReturns whether the host environment reports a live network interface (navigator.onLine !== false).
Overridable by subclasses and tests so offline retry attempts do not invalidate a resumable provider handle.
ProtectedisWhether a tool result's payload requests SILENT scheduling (meaning no post-tool model reply is expected).
Subclasses that gate scheduling on model capability (e.g. GeminiRealtimeClient.supportsScheduling) can override.
Returns whether a track of the requested modality and direction is established and 'live'.
ProtectedisProtectedisWhether the mic meter hears the user right now (always false when the driver attached no input meter).
ProtectednegotiateNegotiates requested tracks against model capability. Audio (inbound + outbound) is the baseline floor of every realtime session; any additional requested tracks (such as channel video) are unioned onto this baseline floor.
OptionalmaxInboundVideoStreams: numberSchedules a turn-boundary reconnect (e.g. on a provider GoAway frame).
Runs immediately if the session is idle at a turn boundary, or defers until the active turn ends.
OptionaltimeLeftMs: numberRegisters the (single) error handler. Receives both fatal transport errors (which are
also followed by an 'error' state change) and non-fatal provider error frames.
Registers the (single) interruption handler.
True barge-in only: fires ONLY when USER INPUT CUT OFF ACTIVE MODEL OUTPUT — a model response in flight or audio audibly playing when the user took the floor. A user simply taking their normal turn while the model is idle is NOT an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a response is actually active or audio is playing).
On interruption, drivers must also flush locally-owned playback and report
IsAudioPlaying === false promptly (driver obligation #3). Hosts typically
use this hook to abort in-flight delegated work so a stale result is never narrated
into a conversation that has moved on.
ProtectedonProtectedonProtectedonRegisters a handler invoked when the agent's remote-audio stream becomes available —
IMMEDIATELY (synchronously) if it has already landed, otherwise when the driver publishes
it (a WebRTC track typically lands AFTER Connect resolves). Lets a host attach the
agent voice to a recording that began before the stream landed. Handler errors are
isolated (logged) so a host bug never disturbs the call. Handlers are session-scoped:
clearRemoteMediaStream (called from Disconnect) drops them.
Registers the (single) remote-VIDEO handler — the model/avatar's video for a VIDEO session (a talking-head the host renders, e.g. as the agent's tile). Invoked once the provider's video is available.
The video is a MediaVideoSource: a live stream (a WebRTC track), or a player that must own the
<video> element (MSE or WebCodecs playout of encoded avatar video). Hosts show either with
AttachVideoSource.
Optional capability: audio-only drivers (the default) never emit — registering is always safe,
but hosts must not assume video arrives. A driver whose model sends video (the server driver declares an
outbound video track in its session's Capabilities.SupportedOutboundTracks, or renders an avatar,
BaseRealtimeModel.SupportsAvatarOutput) calls emitRemoteVideo when it is live.
Invoked with the remote video when it becomes available.
Registers the (single) state-change handler. States may be re-emitted (e.g. repeated
'listening' on successive barge-ins); hosts should treat emissions as idempotent.
Registers the (single) tool-call handler. The host executes the tool and calls SendToolResult with the outcome.
Registers the track-state-change handler — invoked when tracks transition in lifecycle
(e.g. established to 'live', 'denied', or 'unsupported').
Registers the (single) transcript handler. Receives interim deltas (IsFinal: false)
and final turn transcripts (IsFinal: true) for both roles, tagged with their
RealtimeClientTranscript.Kind.
Registers the (single) usage handler.
Emissions carry token deltas for the response/turn that just completed (see
RealtimeClientUsage — deltas preferred; cumulative-only providers must convert
in the driver). Hosts accumulate and relay/persist on their own cadence (e.g. the voice
session service debounces a RelayRealtimeUsage mutation onto the co-agent prompt run).
Optional capability: drivers whose provider exposes no usage telemetry simply never emit — registering a handler is always safe, but hosts must not assume emissions arrive. See RealtimeClientUsage for per-driver availability.
Registers the single handler invoked when the provider's voice-activity detection saw
the user start to speak. Fires once per detected onset and may fire again inside one
turn after a pause; it carries no text. When the provider's event names the conversation
item this speech belongs to (OpenAI-protocol input_audio_buffer.speech_started.item_id),
itemId carries that id so the host can pair the onset with the matching user
transcript (RealtimeClientTranscript.ItemId); otherwise itemId is undefined. Unlike
OnInterruption, it fires whether or not the model was speaking.
ProtectedpublishDriver hook: records the agent's remote-audio stream and fans it out to every registered
handler. Passing null (a driver with no tappable plane) just records null and
notifies nobody.
ProtectedReconnectRe-establishes the AssemblyAI Voice Agent session on a fresh credential/socket while
keeping the local mic capture and playback pipeline intact. When UseNativeResumption
is true and a ResumptionHandle (session_id) is available, attempts session.resume;
otherwise starts a fresh session and seeds prior conversation memory into system_prompt
via RenderRealtimeMemorySeedAsContextNote.
Records a provider resumption handle or conversation ID, along with whether it is resumable now.
ProtectedrecordRecords a completed tool result into conversationMemory and clears its in-flight guard. Invoked automatically by the SendToolResult template method after doSendToolResult succeeds.
OptionaltoolName: stringProtectedrefreshMints a fresh credential via the registered host callback, updates currentCredential, and re-arms the refresh timer.
Optionaloptions: RealtimeClientTokenRefreshOptionsEnds a track on a running session, such as inbound video when the user stops their camera: the track is
reported 'ended' and leaves AllTracks. Audio, the session's floor, cannot be removed.
Whether a track was ended.
Rebinds the PCM capture and the input meter to the stream's current track (obligation #10); the socket stays open.
A spoken TURN: the same reply trigger as the update, with the reply kept normal (MJ#4591).
Triggers ONE short spoken update — NATIVE via reply.create with per-response
instructions. The resulting reply is stamped Kind: 'narration' at send time (reset
on the reply boundary). Queued behind any in-flight reply per the base contract's
collision rule (an overlapping reply.create would collide with active generation).
ProtectedresolveResolves the tool name recorded when callID was emitted, or fallbackName ?? callID.
OptionalfallbackName: stringProtectedResumeA planned move's attempt failed while the old connection still carries the call: go back to live on it (state,
held input) until the next attempt, which waits for the next idle boundary or the hard point, and return true.
Return false when the old connection cannot carry the call any more; the cycle then retries on its schedule.
IsReconnecting is already false when this runs, so sends go to the old connection. false here: a driver
opts in by overriding it. See RealtimeReconnectControllerOptions.ReturnToOldConnection.
Injects background context — EMULATED via the mutable system_prompt: the note is
appended under a "Background updates" heading and the FULL prompt is re-sent via
session.update. A config write never triggers or disturbs generation, so it is sent
immediately even while a reply is in flight (no queueing applies).
Optionaloptions: RealtimeContextNoteOptionsInjects typed text — EMULATED via reply.create (the protocol accepts no typed user
input), with instructions framing the text as the user's turn. No-op when the session
is not open. See the class-level fidelity caveat.
SendText implies barge-in (base-contract rule): an active spoken reply is cancelled via CancelActiveResponse first — playback flushed, reply marked inactive, queued sends drained — so the typed turn takes the floor immediately. If a drained queued send starts a new reply, the text queues behind it, preserving delivery order.
Feeds an executed tool's result back to the model, correlated by callID, and ensures
the model SPEAKS the result as soon as possible: immediately when idle, otherwise
queued behind the in-flight response (e.g. a progress narration) so the trigger is
never dropped by the provider.
Template method: dispatches to doSendToolResult first (so the wire send happens
before soft-window reconnect evaluation and rejected/unknown call IDs can return false),
then records the result into conversationMemory via recordToolResult.
The CallID from the originating RealtimeClientToolCall.
The tool's result as a JSON-stringified string.
OptionalSendStreams one video frame as base64-encoded image data to the model. Optional capability — drivers that support inbound video tracks implement this.
Base64-encoded image data.
OptionalmimeType: stringImage MIME type (defaults to 'image/jpeg').
OptionalsourceId: stringWhich source this frame came from. A driver whose model takes ONE inbound video
stream ignores it (the source arbiter already chose what to send); a driver whose model takes
several routes the frame to the track bound to that SourceID.
true if accepted and sent; false if dropped (throttled, unestablished, etc.).
Mutes / unmutes by toggling the mic tracks' enabled flag: the capture pipeline stays
up and streams SILENCE while muted (the provider's VAD sees a continuous stream and the
un-mute is glitch-free — same policy as the other client drivers).
Registers the host callback used to mint a fresh credential before expiry or reconnect.
Initiates an in-place reconnect that retries through its recovery window (REALTIME_RESUMPTION_RECOVERY_WINDOW_MS,
counted from the old connection's end for a planned move), with the rapid-reconnect circuit breaker and Attempt-2
fallback from native handle to history seeding.
AssemblyAI Voice Agent implementation of BaseRealtimeClient: a browser-direct agent websocket authenticated with the server-minted ONE-TIME client token (the
EphemeralToken— no API key ever reaches the browser).Registered with the ClassFactory under the key
'assemblyai'— theProviderstring the server'sAssemblyAIRealtimedriver stamps on itsClientRealtimeSessionConfig.Owns ALL AssemblyAI wire concerns (the behavioral sibling of the ElevenLabs client driver — the audio plane is client-owned over a websocket, no WebRTC):
wss://…/v1/ws?token=…→ send the server-authoredsession.update(from theSessionConfigpact: system prompt, tools, voice, turn detection) as the FIRST frame → wait forsession.ready→ build the audio plane at the provider's FIXED 24 kHz PCM16 format →'listening'. The state is gated onsession.ready(driver obligation #7): only then has the provider confirmed the session config is applied (audio sent earlier would be dropped anyway).input.audioframes.reply.audiochunks decoded into the shared RealtimePcmPlayback at 24 kHz; IsAudioPlaying comes from its playout clock.transcript.user.deltafragments (IsFinal: falsedeltas) finalized bytranscript.user; agent turns arrive as ONE finaltranscript.agent(after a barge-in it carriesinterrupted: trueand the text already reflects the truncated turn — no ElevenLabs-style correction event follows).IsBusyis set onreply.started/ firstreply.audio(and eagerly when this client triggers a reply) and cleared onreply.doneand on atool.callframe (the agent has yielded the floor pending the result — deadlock guard, obligation #2).reply.createwith the instructions (a real per-response-instructions channel — no user-turn emulation). The response kind is stamped'narration'at send time (sends are the turn triggers) and reset on the reply boundary; the kind tagging assumestranscript.agentprecedes itsreply.done, which is the provider's emission order.reply.create: the protocol accepts no typed user input, so the typed text rides as response instructions framing it as the user's turn. Fidelity caveat: the text enters as instructions rather than a conversation user turn, so the agent may occasionally reference "the message you sent" — hosts should treat typed input on this provider as best-effort.system_prompt: notes are appended under a "Background updates" heading and the full prompt is re-sent viasession.update— a config write that never triggers or disturbs generation, so it is sent immediately even mid-reply.input.speech.startedwhile a reply is active (or audio audibly playing) is the snappy flush point (per the provider's own guidance — ~300 ms faster than waiting forreply.done);reply.donewithstatus: 'interrupted'is the authoritative verdict and is the fallback flush when the speech-start gate missed. A speech start while idle is a user simply taking their turn — NOT an interruption (base contract).reply.audioframes of the cancelled reply until the next reply boundary — so late-arriving chunks of a cancelled turn are never played or allowed to re-assert'speaking'.