AbstractProtectedactiveThe kind of the response currently in flight. Event ordering (confirmed against the live
OpenAI API): response.created → transcript deltas → *_audio_transcript.done →
response.done. The transcript-done frame therefore arrives while the kind is still set,
letting onAssistantDone classify the turn; response.done then resets it.
ProtectedcallsTool call IDs that were in flight when a non-native seed reconnect occurred.
ProtectedconfiguredDuration of a single provider connection in seconds from ClientRealtimeSessionConfig.ConnectionMaxSeconds.
ProtectedconfiguredBuffer in seconds before CallDurationSeconds expiry at which Reconnect is triggered.
ProtectedconfiguredCredential/session token lifetime in seconds from ClientRealtimeSessionConfig.SessionMaxSeconds (or MaxSessionSeconds).
ProtectedconfiguredOptional metadata override for whether native resumption is trusted on this endpoint.
ProtectedconfirmedTrue while a response the provider has CONFIRMED (response.created seen, response.done not
yet) is in flight. Distinct from responseActive, which is set EAGERLY before a local
response.create is even sent: if the provider REJECTS that create (an error frame with no
response.created), responseActive would otherwise stay stuck true forever. This flag lets
onErrorFrame tell "the eager flag is a phantom for a rejected create" (no confirmed
response) from "a real response is genuinely active" (e.g. a concurrent VAD turn), so it clears
the phantom without disturbing a live turn. Set on response.created, cleared on response.done.
Protected ReadonlyconversationRolling conversation memory used to re-seed provider context on reconnect.
ProtectedcurrentThe client's own view of the session state — mirrors what emitStateChange last
reported, EXCEPT after a tool call is emitted: the host typically shows its own busy
state then, so the client silently leaves 'speaking' (no emission) to preserve the
host's indicator until the result reply starts (see onToolCallFrame).
ProtectedinputMeter over the USER's microphone, when the driver attached one.
ProtectedmicThe mic stream owned by the current connection (used by the shared SetMuted).
ProtectedoutputMeter over the AGENT's audio output, when the driver attached one.
ProtectedpendingAccumulates the in-flight assistant transcript across delta frames.
ProtectedpendingCount of locally-initiated response.creates whose response.created echo has not arrived
yet. While non-zero, a response.done belongs to an EARLIER (typically just-cancelled)
response — it must not clear the busy flag, flush the queued trigger, or flip the state for
the response we just started (usage is still emitted). Consumed by response.created.
ProtectedpendingSet by RequestSpokenUpdate just before it sends its response.create, and
CONSUMED by the very next response.created frame, which stamps
activeResponseKind for that turn. Narration is only requested while the model is
idle (hosts gate on IsBusy), so under normal ordering the next response.created
is ours.
ProtectedpendingSet when a tool result is ready while a response is active; sent on the next response.done.
Protected ReadonlyreconnectUnified I/O-free reconnect controller shared with server-side BaseRealtimeSession.
ProtectedresponseTrue while the model has a response in flight; gates narration + queues the tool result.
ProtectedtokenOptional callback installed by RealtimeSessionRuntime to refresh the provider token mid-call.
ProtectedtracksTracks requested, established, or denied on this session.
All tracks evaluated for this session (including 'unsupported' or 'denied').
Effective connection ceiling in seconds. Metadata and runtime caps narrow the driver's
provider ceiling (ClampSessionSecondsValue(DefaultCallDurationSeconds, RuntimeCallDurationSeconds, configuredCallDurationSeconds)),
never widen it.
Accessor for the client's rolling conversation memory ledger.
ProtectedDefaultDriver fallback connection ceiling in seconds when model metadata omits connectionMaxSeconds.
ProtectedDefaultEstablished tracks on this session (tracks in the 'live' state).
The frame rate (frames per second) the live inbound video track was negotiated at — the model's
ceiling or the session's request, whichever is lower — or undefined when no inbound video track
is live or none declared a rate. Anything pacing a video feed reads THIS, never a constant of its own.
AbstractIstrue while model audio is AUDIBLY playing in the browser. Distinct from
IsBusy — audio plays at realtime while generation finishes early, so the
model can be "idle" while speech is still coming out of the speaker. Hosts must
gate narration on BOTH, or queued utterances come out late and stale.
true while a model response is in flight (generation started and not yet done).
Distinct from IsAudioPlaying: generation runs ahead of playback.
Hosts use this to gate interim narration so it never interrupts a reply.
ProtectedIsTrue from the start of a reconnect cycle until it succeeds or fails. Unlike IsReconnecting, it stays true
while a planned move waits between attempts with the call live again on its old connection. A driver that would
otherwise end the call on a loss of that connection checks this and hands the loss to the running cycle
(TriggerReconnect('connection-lost')) instead.
ProtectedIsTrue while a mid-call reconnect is actively in flight.
How many concurrent inbound video streams the model accepts: the ceiling the driver declared when it
negotiated tracks, else 1 when an inbound video track is live and 0 when it is not.
A source arbiter reads this to decide whether several live video sources must be narrowed to one (with a note to the model on each switch) or passed through untouched.
ProtectedproviderDebug label used in the shared console diagnostics.
Seconds before CallDurationSeconds when the reconnect buffer window opens.
ProtectedRecoversWhether an unexpected loss of a live call's transport is handed to the shared connection-lost reconnect. On by
default for every driver that can reconnect mid-call; a driver whose transport closes benignly and whose host
reads that close as the end of the session overrides it to false.
ProtectedreleasesWhether a completed tool call CLEARS responseActive. WebRTC keeps the flag (the
provider reliably follows with response.done); websocket transports clear it as a
deadlock guard so a queued SendToolResult can never wedge if the endpoint skips
the trailing frame.
ProtectedRequiresWhether this driver requires a freshly minted single-use credential on each reconnect.
ProtectedRequiresWhether this driver's provider emits a fresh resumable handle shortly after each turnComplete
(Gemini / Vertex AI Live), so planned moves wait for that post-turn handle.
Latest recorded provider resumption handle or conversation ID.
ProtectedRuntimeOptional runtime ceiling override (e.g. Gemini's 120s ceiling when uncompressed video is active).
Narrowed against DefaultCallDurationSeconds and configuredCallDurationSeconds via ClampSessionSecondsValue.
ProtectedStateThe state to report again when ResumeOnOldConnection goes back to the old connection, or null when the
old connection has reported a state of its own since the move started (it keeps delivering events during the
attempt), which is current. 'speaking' while the turn is open, else 'listening': the turn, not the state the
move replaced, says whether the agent is answering, since a driver can leave 'speaking' without reporting it
(a tool call) and the old connection can start an answer during the attempt. A 'speaking' report opens a turn,
and one with no answer behind it would hold the next attempt until the last call. 'connected' is never
reported: a host keeps showing the move for it.
Whether the model takes inbound video on this session: the driver declared an inbound video track it supports when it
negotiated tracks, and a stream limit above zero. No video track need be live; one is added when a capture starts
(AddTrack). false before tracks are negotiated, and for a driver that does not negotiate them.
ProtectedSupportsWhether this concrete client subclass implements mid-call Reconnect.
ProtectedSupportsWhether this driver can open a relay session (ClientRealtimeSessionConfig.Transport 'relay'): connect to
MJAPI's realtime relay at the session's RelayUrl rather than to its provider. false here; a driver that speaks
its provider's protocol through the relay opts in by overriding it (the Gemini Enterprise client does).
Whether native provider resumption is trusted on the first reconnect attempt.
Adds a track to a running session, such as inbound video when the user starts their camera, resolved against
what the driver supported when it negotiated at connect. The new track is 'live', or 'unsupported' with the
reason (the model takes no such track, or no more inbound video streams). A live track with the same
direction, modality and source is returned as it is. Other tracks keep their ids.
The base re-negotiates locally, which is all a driver needs when its provider accepts the media without a setup change (Gemini Live takes video frames at any time). A driver whose wire must change overrides this and calls it.
The track, or null when the session has not negotiated tracks: before it connects, or with a driver
that does not negotiate them (only the Gemini driver does today).
Reads catalog-stamped timing and resumption settings from the typed fields of config, falling
back to config.SessionConfig (connectionMaxSeconds, sessionMaxSeconds, reconnectBufferSeconds,
trustNativeResumption), records the initial credential snapshot, and schedules proactive
credential/relay refresh.
Arms (or re-arms) the soft and hard connection-ceiling timers for the current transport socket.
CallDurationSeconds.CallDurationSeconds - ReconnectBufferSeconds and forces a reconnect; while the model is
answering it waits for the turn to end, up to shortly before CallDurationSeconds.
Both measure from the connection deadline instead when it comes before CallDurationSeconds
(connectionDeadlineAtMs: the expiry of the credential the connection opened on, on a driver whose
connections close with it).ProtectedAssertRefuses a session this driver cannot open: a relay session, unless the driver opts in through SupportsRelayTransport. Every driver calls this first in Connect, before it touches the session's media or opens a connection (driver obligation #11). The message names the provider key and never quotes the relay URL, which carries the relay's ticket.
The session the host asked this driver to open.
ProtectedattachAttaches (replacing + closing any previous) the USER-microphone meter. null detaches.
ProtectedattachAttaches (replacing + closing any previous) the AGENT-output meter. null detaches.
ProtectedbuildThe response.create frame for one spoken update — carrying the session identity, not
replacing it.
⚠️ response.instructions IS A FULL OVERRIDE OF THE SESSION SYSTEM PROMPT for that one
response, not an addition to it. That is the whole reason this method exists. The
server-side twin of this driver (openAIRealtime.ts) already states the rule and guards
the BLANK case on it — forwarding '' "would wipe the co-agent identity framing" — but the
NON-blank case was never guarded, so every caller that supplied a real direction wiped it
just as thoroughly and much less visibly.
What that cost, live: the one turn that rides this method had no persona, so the model fell back to its vendor default and opened a hiring interview with "I'm ChatGPT, your friendly voice companion" while every other turn in the same session correctly said "I'm Sam Rivera, Support Team Lead" (bizapps-caliber#397 / MJ#4591). Every commentary-style caller is exposed the same way — an opening turn, a silence check-in, a delegation narration — and none of them should have to restate an identity the session already carries.
Three cases, and the middle one is the fix:
response.create. "Respond now, under the session prompt"
(the meeting-mode bridge trigger passes ''). Matches the server twin exactly.identity, then direction. The identity leads
because it is context, and the direction is LAST because it is the thing being asked for
now — the most recent line is the one a model weights hardest.response.cancel (only when a response is actually in flight) and silences
already-generated speech via the transport's stopAudioOutput. Resets the local
response state machine (active flag, narration kind, accumulated transcript) but
PRESERVES any queued tool-result trigger: delegated work is never affected by a
floor-control cancel, and the queued trigger still fires on the cancelled response's
trailing response.done. No-op when idle or when the transport is not open.
Protected AbstractcanWhether the transport can currently deliver client events (channel/socket open).
ProtectedclearDriver hook for Disconnect: drops the stream AND the handlers. Handlers must not survive
into a later Connect on a reused instance, or a stale host callback would receive the
next session's stream.
ProtectedcloseReleases both meters — every driver calls this from its disconnect/teardown path.
AbstractConnectOpens the provider connection using the server-minted ephemeral credential and applies
config.SessionConfig once the control channel is ready — so prompt and tool authority
stay server-side even though the browser owns the socket. The payload's SHAPE is a
private pact between this driver and the same-keyed server driver that minted it (see
driver obligation #8); how it is applied is entirely driver-specific.
The server-minted client session config (provider, model, ephemeral token or relay transport, session config). A driver refuses a transport it does not speak first (see AssertTransportSupported).
The user's microphone capture stream. The caller acquires it (so IT owns the permission prompt UX); the client attaches it to the transport and stops its tracks on Disconnect.
OptionalcameraStream: MediaStreamOPTIONAL camera capture stream for a VIDEO session (the model "sees" the user). Only attached by video-capable drivers; audio-only drivers ignore it (back-compatible — existing callers and drivers need not pass or read it). The model/avatar's video comes BACK via OnRemoteVideo.
ProtectedConnectionWhether the provider closes a connection when the credential it opened on expires, even after the client holds a
newer one. false (the default) for a credential that only gates opening the connection, as OpenAI's
client_secret does. A driver returning true gets a planned move onto the refreshed credential before the old
one expires (see connectionDeadlineAtMs), instead of a drop at the expiry.
ProtectedCredentialHow many connections credential may open, or undefined for no limit (the default). A driver whose provider
counts uses on a credential overrides it, so the client re-mints before it would open a connection the
credential has no use left for, instead of spending a failed attempt finding out.
ProtectedcurrentThe session-level instructions currently in force — the co-agent's identity, persona and
standing directives, as applied by session.update.
A SEAM, not a field read, because the pact lives in a different place per transport: the
websocket subclass keeps it in sessionObject, the WebRTC driver in its own sessionConfig.
The base holds no copy of either, so it answers "no identity on record" and each transport
overrides with the one it actually applied. A transport that forgets to override loses only
the carry — buildSpokenUpdateEvent still sends the caller's direction — which is the
old behaviour rather than a new failure.
Null rather than '' for "this session carries no identity", so buildSpokenUpdateEvent
can tell "nothing to preserve" from "preserve this".
AbstractDisconnectTears down the provider connection and all client-held resources (control channel,
transport, mic tracks, audio sink) and emits a final 'closed' state (unless the
session already ended in 'error'). Safe to call more than once.
ProtectedDisposeCancels all reconnect and duration timers when the session is disconnected.
ProtecteddoSends the tool result back as a function_call_output conversation item, then triggers
a reply — immediately if the model is idle, otherwise queued until the current response
(e.g. a progress narration) finishes. Without the queueing the result's response.create
would collide with an in-flight narration and be dropped, leaving the model silent when
delegated work comes back.
ProtecteddropDrops the text and tool results held for a move that will not complete, logging how many and why.
ProtectedemitProtectedemitEmits a true barge-in interruption to the registered handler (if any).
ProtectedemitEmits the model/avatar's remote video to the registered handler (video drivers only). A driver with a plain
MediaStream passes it as it is; it is handed on as a 'stream' source.
ProtectedemitEmits a state change to the registered handler (if any) and flushes any deferred turn-boundary reconnect.
ProtectedemitEmits a tool-call request to the registered handler (if any) and records it in memory.
ProtectedemitEmits a track state change to the registered handler (if any).
ProtectedemitEmits a transcript event to the registered handler (if any) and records final normal turns in memory.
ProtectedemitEmits a usage update (token deltas for a completed response/turn) to the registered handler (if any).
ProtectedemitEmits a user-speech-onset to the registered handler (if any).
OptionalitemId: stringProtectedflushOn a turn completing, fire any queued tool-result response so the answer is spoken.
The session's current audible activity, or null when this driver attached no
meters at all (hosts then keep their turn-state-driven visuals). Sampled by hosts
per animation frame — implementations are cheap, allocation-light reads of an
AnalyserNode; no per-call provider traffic.
Returns the AGENT's remote-audio MediaStream when this driver owns a tappable
agent-audio plane (a WebRTC driver's remote track, or a PCM driver's playout output),
or null otherwise. Hosts use it to MIX the agent's voice into a browser-side recording
alongside the mic; null (drivers with no tappable plane, or before the stream lands)
degrades gracefully to mic-only capture. Hosts may still call it as client.GetRemoteMediaStream?.().
Pure read — never mutates.
ProtectedhandleHandles one base64 PCM16 audio delta. Default: no-op — on the WebRTC transport the agent's audio rides the peer connection's remote track, not data-channel frames. Websocket transports override to enqueue into their local playout engine.
ProtectedhandleDispatches a typed OpenAI-protocol server event to the appropriate behavior.
ProtectedhandleParses one raw inbound payload and dispatches it. Non-JSON frames and non-object JSON values are ignored (with the onNonJsonFrame diagnostic hook for the former).
ProtectedholdsWhether the user said something the current connection has not answered yet, with no turn open for it. The
hard-point move then waits as it does for an open turn, until shortly before the provider limit. false here: a
driver whose provider opens the turn only once the answer starts overrides it, and calls
reconnectController.NotifyIdleStateChanged() when the hold ends without a turn.
ProtectedisReturns true when the current credential is expired or within marginSeconds of expiry.
ProtectedisReturns whether the host environment reports a live network interface (navigator.onLine !== false).
Overridable by subclasses and tests so offline retry attempts do not invalidate a resumable provider handle.
ProtectedisWhether a tool result's payload requests SILENT scheduling (meaning no post-tool model reply is expected).
Subclasses that gate scheduling on model capability (e.g. GeminiRealtimeClient.supportsScheduling) can override.
Returns whether a track of the requested modality and direction is established and 'live'.
ProtectedisWhether the transport can carry the call, which a failed move's return to it needs: by default, whether it can deliver events (canSendEvents). A transport whose open state can outlast its network path overrides it.
ProtectedisProtectedisWhether the mic meter hears the user right now (always false when the driver attached no input meter).
ProtectedlogDiagnostic hook for each parsed inbound event. Default: silent.
ProtectedlogDiagnostic hook for each outbound event. Default: silent.
ProtectednegotiateNegotiates requested tracks against model capability. Audio (inbound + outbound) is the baseline floor of every realtime session; any additional requested tracks (such as channel video) are unioned onto this baseline floor.
OptionalmaxInboundVideoStreams: numberSchedules a turn-boundary reconnect (e.g. on a provider GoAway frame).
Runs immediately if the session is idle at a turn boundary, or defers until the active turn ends.
OptionaltimeLeftMs: numberRegisters the (single) error handler. Receives both fatal transport errors (which are
also followed by an 'error' state change) and non-fatal provider error frames.
Registers the (single) interruption handler.
True barge-in only: fires ONLY when USER INPUT CUT OFF ACTIVE MODEL OUTPUT — a model response in flight or audio audibly playing when the user took the floor. A user simply taking their normal turn while the model is idle is NOT an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a response is actually active or audio is playing).
On interruption, drivers must also flush locally-owned playback and report
IsAudioPlaying === false promptly (driver obligation #3). Hosts typically
use this hook to abort in-flight delegated work so a stale result is never narrated
into a conversation that has moved on.
ProtectedonDiagnostic hook for each non-JSON inbound frame. Default: silent.
ProtectedonWebRTC-only playback-buffer hooks; websocket transports never receive these frames.
ProtectedonProtectedonSends the input held during the move to the new connection, after the seed, and asks for one answer covering it
and an answer the move cut off (see seedConversationAfterMove). After super, so a result for a call
from before a reseed goes out as a late-result note.
ProtectedonThe move failed and the call is over, so the held input has no connection to go to.
ProtectedonThe move has started, so the holds it waited on are spent: the utterances belong to the connection being left, and the next connection's speech starts holds of its own.
Registers a handler invoked when the agent's remote-audio stream becomes available —
IMMEDIATELY (synchronously) if it has already landed, otherwise when the driver publishes
it (a WebRTC track typically lands AFTER Connect resolves). Lets a host attach the
agent voice to a recording that began before the stream landed. Handler errors are
isolated (logged) so a host bug never disturbs the call. Handlers are session-scoped:
clearRemoteMediaStream (called from Disconnect) drops them.
Registers the (single) remote-VIDEO handler — the model/avatar's video for a VIDEO session (a talking-head the host renders, e.g. as the agent's tile). Invoked once the provider's video is available.
The video is a MediaVideoSource: a live stream (a WebRTC track), or a player that must own the
<video> element (MSE or WebCodecs playout of encoded avatar video). Hosts show either with
AttachVideoSource.
Optional capability: audio-only drivers (the default) never emit — registering is always safe,
but hosts must not assume video arrives. A driver whose model sends video (the server driver declares an
outbound video track in its session's Capabilities.SupportedOutboundTracks, or renders an avatar,
BaseRealtimeModel.SupportsAvatarOutput) calls emitRemoteVideo when it is live.
Invoked with the remote video when it becomes available.
ProtectedonCalled when the model takes the floor (response.created), i.e. the user's turn is definitively
over. Drivers that track per-user-turn transcription state override this to reset it. No-op here.
ProtectedonHook for transports that gate on the endpoint's session.created frame. Default: no-op.
ProtectedonThe user started speaking. TRUE barge-in only when it cut off active model output (a response in flight or audio audibly playing) — a normal turn while the model is idle is NOT an interruption, so the emission is gated (base-contract rule). Transports that own their playback override to also flush the local playout queue.
Registers the (single) state-change handler. States may be re-emitted (e.g. repeated
'listening' on successive barge-ins); hosts should treat emissions as idempotent.
Registers the (single) tool-call handler. The host executes the tool and calls SendToolResult with the outcome.
Registers the track-state-change handler — invoked when tracks transition in lifecycle
(e.g. established to 'live', 'denied', or 'unsupported').
Registers the (single) transcript handler. Receives interim deltas (IsFinal: false)
and final turn transcripts (IsFinal: true) for both roles, tagged with their
RealtimeClientTranscript.Kind.
Registers the (single) usage handler.
Emissions carry token deltas for the response/turn that just completed (see
RealtimeClientUsage — deltas preferred; cumulative-only providers must convert
in the driver). Hosts accumulate and relay/persist on their own cadence (e.g. the voice
session service debounces a RelayRealtimeUsage mutation onto the co-agent prompt run).
Optional capability: drivers whose provider exposes no usage telemetry simply never emit — registering a handler is always safe, but hosts must not assume emissions arrive. See RealtimeClientUsage for per-driver availability.
Registers the single handler invoked when the provider's voice-activity detection saw
the user start to speak. Fires once per detected onset and may fire again inside one
turn after a pause; it carries no text. When the provider's event names the conversation
item this speech belongs to (OpenAI-protocol input_audio_buffer.speech_started.item_id),
itemId carries that id so the host can pair the onset with the matching user
transcript (RealtimeClientTranscript.ItemId); otherwise itemId is undefined. Unlike
OnInterruption, it fires whether or not the model was speaking.
ProtectedonEmits the user's spoken-input transcription. Default (OpenAI/HuggingFace): each
.completed frame is one final caption. Providers that STREAM the completed event
(Grok re-sends the full growing text each time) override to collapse the stream into a
single in-place-updating bubble.
OptionalitemId: stringProtectedpublishDriver hook: records the agent's remote-audio stream and fans it out to every registered
handler. Passing null (a driver with no tappable plane) just records null and
notifies nobody.
ProtectedReconnectRe-establishes the underlying provider connection mid-call, either via native session
resumption or by re-seeding from context.ConversationSeed. Concrete drivers MUST override this method.
Records a provider resumption handle or conversation ID, along with whether it is resumable now.
ProtectedrecordRecords a completed tool result into conversationMemory and clears its in-flight guard. Invoked automatically by the SendToolResult template method after doSendToolResult succeeds.
OptionaltoolName: stringProtectedrecoverHands an unexpected loss of the live call's transport (a close or error the consumer did not ask for) to the
shared reconnect controller, which retries within its recovery window and reports reconnect-failed when the
window closes. Returns false when there is no live call to save, so the caller keeps its terminal handling: the
connect has not finished (Connect reports that failure itself), the call already ended, or the driver opts out.
A loss while a reconnect cycle runs is the loss of the transport the cycle may go back to after a failed attempt
(the callers' identity checks drop events of a transport the cycle opened or already let go of). It goes to the
cycle, even on a driver that opts out of starting one, and returns true: the cycle then never returns the call
to a dead transport, and goes on as a lost connection's.
Live means the call went live and has not ended (callWentLive), not a speech state: during a tool run the
state is 'connected' (see onToolCallFrame), and a loss then is as recoverable as one mid-sentence.
ProtectedrefreshMints a fresh credential via the registered host callback, updates currentCredential, and re-arms the refresh timer.
Optionaloptions: RealtimeClientTokenRefreshOptionsEnds a track on a running session, such as inbound video when the user stops their camera: the track is
reported 'ended' and leaves AllTracks. Audio, the session's floor, cannot be removed.
Whether a track was ended.
OptionalReplaceMoves a connected session onto the microphone stream's current audio track: call it after the track changes (a device switch, or a lost device replaced by the default) or to use another stream. The connection stays up and the provider hears the new track. The driver now owns this stream as it owned the one given to Connect; the previous stream's tracks are the caller's to stop.
Mute follows the track: the new track's enabled flag decides it, so a caller replacing a
muted track carries enabled over (the LocalMediaController does).
Optional capability (driver obligation #10): a driver that can't rebind its transport leaves it undefined. Does nothing when the session isn't connected.
The stream to follow, usually the one given to Connect after the controller swapped its track.
ProtectedrequestAsks the model to speak (a tool result or typed-text reply) — immediately if it's idle,
otherwise queued until the current response finishes. An immediate trigger also
CONSUMES any queued trigger debt: every payload item is already in the conversation, so
one response.create voices everything (e.g. typed text barging in over a narration
that had tool results queued behind it).
A spoken TURN: the same response.create as the update, with the reply kept normal (MJ#4591).
Triggers ONE short spoken update with the given instructions. Marks the upcoming
response as 'narration' (flag consumed by the next response.created) so its
transcripts are emitted with Kind: 'narration' — ephemeral by contract. Sets
responseActive eagerly so a tool result landing mid-narration queues instead of
colliding.
Skips when busy (base-contract collision rule — drivers MUST queue or skip): a
response.create sent while a response is in flight would be rejected/garbled by the
provider, and narration is disposable by contract, so the update is dropped with a debug
log rather than queued to come out late and stale. Hosts SHOULD still gate on
IsBusy / IsAudioPlaying for timing quality.
The wire frame is built by buildSpokenUpdateEvent, which is where the identity rule lives — see that method for why the caller's direction is never sent on its own.
ProtectedresetResets the per-session response state machine (used on Disconnect).
ProtectedresetResets the response bookkeeping for a connection that has just replaced the old one, and returns whether an
answer was in flight (or queued) on the old one. Every count and flag here describes the old connection: a
response.create it never confirmed would otherwise make the new connection's real response.done look stale
and leave IsBusy stuck.
ProtectedresolveResolves the tool name recorded when callID was emitted, or fallbackName ?? callID.
OptionalfallbackName: stringProtectedResumeA planned move's attempt failed and the old transport can still carry the call (isTransportUsable): the
call goes on there until the next attempt. The state the move replaced is reported again, and the text and tool
results held for the move go out on it now instead of waiting for (or being dropped with) the cycle. false when
the old transport is gone too.
ProtectedseedSeeds the new connection after a move (when sendsSeed; a natively resumed one already has the history) and
notes whether the move cut off an answer the user is still owed: one was in flight on the old connection
(answerWasInFlight, from resetResponseStateForNewConnection) and the conversation ends with what it
answers (a user turn or a tool result), not counting the context notes after it. Neither a seeded nor a resumed
session replies by itself, so without this the question sits unanswered in every later seed. An answer that
completed is in memory as an assistant turn and is not asked for again.
A seeded connection knows the conversation as the seed sent it. A resumed one knows it as the provider kept it,
which includes what the old connection heard while the attempt was opening, after context.Seed was built: so
then the memory as it is now decides.
ProtectedseedSeeds bounded conversation history turns into a freshly reconnected OpenAI-protocol session.
Injects a system-role context item the model can draw on the next time it speaks, WITHOUT forcing a reply. Item creation is always safe mid-response.
NOTE: role must be 'system' — gpt-realtime (and the compatible endpoints) reject 'developer' items ("Developer messages are only supported for quicksilver sessions").
Optionaloptions: RealtimeContextNoteOptionsProtectedsendLogs (diagnostic hook) and delivers one client event when the transport is open.
Protected AbstractsendDelivers one protocol client event over the transport (only called when canSendEvents).
Injects typed text as a user-role message conversation item, then triggers a reply
through the SAME collision-safe path tool results use (requestResultResponse).
No-op when the transport isn't open.
SendText implies barge-in (base-contract rule): an active spoken response is cancelled via CancelActiveResponse before the text is injected, so the typed turn takes the floor immediately instead of waiting behind stale speech. When nothing is active the cancel is a no-op and the reply triggers immediately.
a move a result is held, like typed text, and goes out once the call is live again: on the new connection after the swap (onReconnected), or on the old one when a planned move goes back to it (ResumeOnOldConnection). Sent on now, it would be lost: a lost transport cannot carry it, and a closing one takes it along after the seed for the new connection was built. It is recorded into memory when it goes out, so it reaches the model and the seed of a later move exactly once.
OptionalSendStreams one video frame as base64-encoded image data to the model. Optional capability — drivers that support inbound video tracks implement this.
Base64-encoded image data.
OptionalmimeType: stringImage MIME type (defaults to 'image/jpeg').
OptionalsourceId: stringWhich source this frame came from. A driver whose model takes ONE inbound video
stream ignores it (the source arbiter already chose what to send); a driver whose model takes
several routes the frame to the track bound to that SourceID.
true if accepted and sent; false if dropped (throttled, unestablished, etc.).
Mutes / unmutes by toggling the mic tracks' enabled flag: the transport stays up and
streams SILENCE while muted (the provider's VAD sees a continuous stream and the un-mute
is glitch-free — the same policy across the client driver family).
ProtectedsetUpdates the client's own state view and emits the change to the host.
Registers the host callback used to mint a fresh credential before expiry or reconnect.
Protected AbstractstopSilences already-generated speech during a cancel: websocket transports flush their local playout queue; the WebRTC transport asks the provider to clear its managed output buffer.
Initiates an in-place reconnect that retries through its recovery window (REALTIME_RESUMPTION_RECOVERY_WINDOW_MS,
counted from the old connection's end for a planned move), with the rapid-reconnect circuit breaker and Attempt-2
fallback from native handle to history seeding.
Protected StaticreadReads a session pact's instructions, or null when it carries none worth preserving.
The shared OpenAI-protocol brain for realtime CLIENT drivers — the transport-agnostic middle layer between BaseRealtimeClient and the per-provider drivers.
Every OpenAI-protocol provider (OpenAI itself over WebRTC, xAI Grok Voice and self-hosted HuggingFace over websockets) shares the exact same event vocabulary and turn discipline, so this class owns — ONCE — the pieces that were previously cloned per driver:
responseActiveset onresponse.created/ cleared onresponse.done; tool-resultresponse.creates queued while a response is in flight and flushed onresponse.doneso the model ALWAYS voices delegated results.'narration'so its transcripts are emitted withKind: 'narration'(ephemeral).Transport specifics stay in subclasses through a small set of seams: canSendEvents / sendProtocolEvent (the wire), stopAudioOutput (how already-generated speech is silenced), handleAudioDeltaFrame (websocket transports enqueue PCM; WebRTC ignores — audio rides the peer connection), and per-provider behavioral overrides (onUserTranscriptFrame, onSpeechStartedFrame, releasesBusyFlagOnToolCall).