The injectable provider-connection seam (a real OpenAIRealtimeWebSocket
in production, a fake in tests).
The creating driver's static facts; defaults to xAIRealtime.DriverFacts (xAI's own static) for direct construction.
Optionalmodel: BaseRealtimeModelOptional parent driver instance.
ProtectedcallsProtectedclosedProtectedconfiguredProtectedconfiguredProtectedconfiguredProtectedconfiguredProtectedconfiguredProtected ReadonlyconversationProtected Optional ReadonlymodelProtected ReadonlypendingProtected ReadonlyreconnectProtectedresponseWhether a model response is currently in flight. Minimal response tracking that mirrors the
client driver's state machine: set on response.created (and eagerly whenever this session
sends its own response.create, so back-to-back local triggers can't race the server event),
cleared on response.done — which the API emits for every terminal status, including
cancelled after barge-in, so the flag can never stick. Consumed by
OpenAIRealtimeSession.RequestSpokenUpdate to skip (not collide with) an active
response, since the API rejects overlapping response.create requests.
Protected (not private) so compat-endpoint session subclasses can apply provider-specific robustness tweaks (e.g. HuggingFace marks a response active on the first audio delta and releases the flag when a tool call yields the floor).
ProtectedturnEffective single-connection duration in seconds, resolved via ClampSessionSecondsValue
so metadata (configuredCallDurationSeconds) narrows the driver's provider ceiling
(DefaultCallDurationSeconds) and any state-dependent RuntimeCallDurationSeconds, never widening it.
ProtectedconfigReadiness deadline in milliseconds for the deferred-config wait. Only affects consumers of WaitForConfigApplied; the deferred apply itself is not cancelled. Overridable.
Accessor for the session's provider-neutral conversation memory ledger.
ProtectedDefaultStatic driver default connection duration in seconds when metadata does not specify one.
ProtectedDefaultDriver default for whether native resumption handles are trusted when metadata does not override.
True once the consumer has closed the session.
True while an in-place reconnect is currently running.
Buffer in seconds before CallDurationSeconds at which the framework initiates a reconnect.
ProtectedRequiresWhether this session's provider emits a fresh resumable handle shortly after each turnComplete
(e.g. Gemini / Vertex AI Live), so a planned move (goAway or soft window) during a turn waits
for that post-turn handle (bounded by the deadline timer) rather than resuming from a pre-turn handle.
Latest recorded provider resumption handle, or null if none.
ProtectedRuntimeDynamic state-dependent duration cap in seconds (for example, returning 120 when
uncompressed video is active on Gemini Live, and undefined otherwise). Clamped with
configuredCallDurationSeconds via ClampSessionSecondsValue so neither a seeded 600s
catalog value nor a tighter custom metadata value (e.g., 45s) is shadowed.
Configured credential/session lifetime in seconds, when distinct from CallDurationSeconds.
Whether native resumption is trusted on this session (always false under ZeroDataRetention).
Applies the initial session config: system prompt + tools via session.update, optional initial
context as a user message. Called once by OpenAIRealtime.StartSession.
Deferral is profile-driven. On OpenAI the realtime WebSocket is NOT open when StartSession
returns — sending session.update synchronously races the handshake and the instructions (the
system prompt + tools) are silently dropped, so the model runs with NO prompt (no identity, no
companion framing). We therefore wait for the server's session.created frame — the first event once
the socket is open and the session exists, and the canonical moment to configure a realtime session —
exactly the point the browser/client-direct path applies its config. Idempotent (a re-emitted
session.created can't double-apply); the listener removes itself once it fires. Providers whose
socket accepts config immediately (xAI) set the profile flag false and send synchronously.
The session parameters.
Extracts timing and native-resumption overrides from a session configuration bag.
Arms (or re-arms) the proactive connection-duration timer for the current connection.
Closes the session and releases the underlying provider connection.
A promise that resolves once the session is fully closed.
ProtecteddispatchRoutes a provider server event to the matching contract handler. Each branch delegates to a small, single-purpose handler to keep this dispatcher flat.
Protected (not private) so OpenAI-compatible session subclasses can pre-translate legacy / beta frame aliases before delegating here.
The OpenAI realtime server event.
ProtectedDisposeCancels all reconnect and duration timers and abandons any in-flight attempt when closed by the consumer. Tool results held for the attempt are dropped: no new connection will carry them.
ProtectedemitEmits a session error to the consumer's registered OnError handler(s).
ProtectedformatFormats the fatal error message surfaced when all reconnect attempts fail.
ProtectedholdHolds a tool result that arrives while a reconnect is moving the session, and returns true when it did. Sent
then, the result would reach the connection being replaced after the attempt's seed was built, so the new
connection would have the call and never its result. A driver calls this first in SendToolResult and returns
when it is true. Once the move lands the result goes through SendToolResult again, after
callsFromBeforeReseed is rebuilt, so a reseeded connection gets it as a late-result note; a failed
reconnect or a close drops it with a warning. Outside a reconnect it returns false and changes nothing.
OptionaltaskRevision: numberClears the recorded resumption handle so subsequent reconnects fall back to history seeding.
Handles a provider-initiated connection-ending advisory (goAway / session.going_away).
Reconnects immediately if idle and ready, or waits for the turn/handle boundary up to the scaled deadline.
OptionaltimeLeftMs: numberProtectedNotifyNotifies the reconnect controller when pending tool calls drain on SendToolResult.
When expectsReply is true (default), a queued soft-window or ConnectionEnding move waits
for the model's post-tool turn (or a fresh trusted native handle) rather than reconnecting in
the gap between SendToolResult and the first model output frame.
OptionalexpectsReply: booleanMarks a turn as ended and triggers a queued soft-window or ConnectionEnding reconnect if ready.
OptionalkeepPostToolTurnWait: booleanMarks a turn as open so soft-window reconnects wait for the turn boundary.
Optional capability — registers a handler invoked when the underlying provider
connection closes WITHOUT the consumer having called IRealtimeSession.Close
(provider-side hangup, network drop). Not fired for a consumer-initiated Close() —
the caller already knows about that one.
Optional because not every provider surface exposes a close signal cheaply; callers
must feature-detect (if (session.OnClose) { ... }). An unexpected close is typically
ALSO surfaced as a Fatal IRealtimeSession.OnError, which is the signal
consumers should drive finalization from.
Invoked when the provider connection closes unexpectedly.
Registers a handler for session errors.
Fatality semantics mirror the client-side BaseRealtimeClient contract:
Fatal: true — the session is unusable (transport/socket failure, credential/token
expiry, unexpected connection loss). The consumer should finalize the session (e.g.
RealtimeSessionRunner calls Stop()) instead of idling forever on a dead socket.Fatal: false — a provider-reported, recoverable error frame; the session stays open
and the consumer should log and continue.Invoked with each RealtimeSessionError.
Registers a handler for provider-detected interruptions (barge-in).
True barge-in only: the handler fires ONLY when user speech interrupts ACTIVE model output — NOT on every user utterance. A user simply taking their normal turn while the model is idle is not an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a model response is actually in flight).
Turn detection / VAD is owned by the provider. The agent layer uses this hook to cancel
the model's current turn and to fire the cancellationToken of any in-flight
delegated agent run — a stale delegated result must never be narrated into a conversation
that has moved on.
Invoked when the provider reports a true barge-in interruption.
Registers a handler for model audio output frames (the audio media plane).
Invoked with each output audio frame as an ArrayBuffer.
ProtectedonHook invoked when a reconnect attempt succeeds.
ProtectedonHook invoked when all reconnect attempts fail or the rapid-reconnect circuit breaker trips.
Formats the error message via formatReconnectFailureMessage and emits a fatal
RealtimeSessionError with Code: code via emitSessionError.
Optionalcode: RealtimeReconnectFailureCodeProtectedonRegisters a handler for model tool-call requests.
Consumers execute the requested tool (under the session's context user) and feed the result back to the model.
Invoked with each RealtimeToolCall.
Registers a handler for transcript events (the text stream).
Consumers typically forward these to the control plane and persist them as
ConversationDetail records.
Invoked with each RealtimeTranscript (partial or final).
Registers a handler for usage/telemetry updates.
Usage is checkpointed incrementally by the agent layer (debounced onto the prompt run) so partial usage is never lost if the session is force-closed after a crash.
Invoked with each RealtimeUsage update.
turn-mode change via a partial session.update — flips server-VAD auto-response on/off without
reconnecting (e.g. re-gate a 1:1 agent to meeting mode when its room becomes multi-agent). It re-sends
the merged audio.input block (transcription, turn detection and any raw bag audio.input override)
so the partial update can't drop them. It never sends audio.output: OpenAI rejects a voice change on
a live session once audio has played.
The frame always carries audio.input.turn_detection, because flipping create_response is the
whole point. The explicit live DisableAutoResponse wins over any bag create_response: the
caller asked for this mode now. The one exception is a bag turn_detection: null, re-sent as null
in either mode so VAD stays off, as it was at start: without VAD nothing auto-responds.
ProtectedReconnectReconnects the underlying OpenAI-protocol WebSocket before connection expiry and either
resumes natively via context.ResumptionHandle (?conversation_id=<handle>) or seeds
accumulated conversation history (ConversationSeed) onto the replacement connection.
Records a provider resumption handle or conversation ID.
Optionalresumable: booleanRecords a finalized turn or tool entry in conversationMemory.
Registers the set of tools the model may call, translating them into the provider's native function-calling format.
The Core-level RealtimeToolDefinition is intentionally minimal: BaseRealtimeModel
lives in the lowest AI layer and cannot depend on the richer tool metadata defined in
higher packages (the agent layer) — doing so would create an illegal upward/circular
dependency. The agent layer is responsible for mapping its richer tool metadata down
to this Core type before calling RegisterTools, and the concrete driver maps this Core
type up to the provider's native function-calling schema.
Note that some providers (e.g. Eleven Labs) bind to a pre-declared tool set on a server-side agent configuration; for those, the driver maps these definitions onto the pre-declared tool names rather than registering arbitrary schemas at session start.
Idempotency rule: a post-start registration of a set IDENTICAL to the set supplied at connect time (via RealtimeSessionParams.Tools) MUST be a no-op. Providers that bind their tool set at connect time and cannot re-declare schemas on an open session MUST no-op (and may log) rather than degrade the conversation — e.g. by injecting schema text into the conversation as content. A genuinely DIFFERENT post-start set on such a provider is unsupported and should be surfaced as a warning, not silently mangled.
The tools to expose to the model.
Instructions for the single spoken update. Blank/empty means "respond now using
the SESSION system prompt" (the meeting-mode bridge trigger passes ''); a non-empty value is
forwarded as a per-response override carrying the session prompt ahead of it — see below.
ONE short spoken update via response.create with per-response instructions.
Collision behavior: skip. The Realtime API rejects a response.create while another
response is active, so when responseActive is set the request is dropped — interim
updates are disposable by contract (the next update or the final result supersedes them).
When sent, the flag is set eagerly (before the server's response.created echo) so two
back-to-back local triggers can't both fire.
The context note to append to the conversation.
Optionaloptions: RealtimeContextNoteOptionsa system-role conversation item (conversation.item.create) the model can draw
on the next time it speaks, WITHOUT a response.create — so no spoken reply is forced.
NOTE: the role must be 'system' — gpt-realtime rejects 'developer' items ("Developer
messages are only supported for quicksilver sessions"); same constraint the client-direct
driver hit. Item creation is always safe mid-response on OpenAI, so no collision guard is
needed here (unlike OpenAIRealtimeSession.RequestSpokenUpdate).
The CallID from the originating RealtimeToolCall.
The tool's result as a JSON-stringified string.
the tool-call loop for OpenAI: sends a conversation.item.create with a
function_call_output item carrying the tool output, then a response.create so the model
continues the turn with the result in context. While a reconnect is moving the session the
result is held and sent on the new connection (see holdToolResultDuringReconnect).
Registers the factory used to open replacement connections on duration expiry.
Initiates an in-place reconnect via the unified RealtimeReconnectController.
Resolves once the initial session config has been APPLIED (sent on the socket) — immediately
for providers that configure synchronously, or on the server's session.created frame for
deferring providers. Rejects if the transport dies (fatal error or unexpected close) or the
consumer closes the session before the config went out.
The base OpenAIRealtime.StartSession deliberately does NOT await this (OpenAI
semantics: the session handle is returned while the handshake completes). Drivers whose
contract promises "ready only after config is applied" (HuggingFace) await it in their
StartSession override.
Live realtime session for Grok Voice — the shared OpenAIRealtimeSession protocol implementation bound to XAI_REALTIME_PROFILE. Exists as a named subclass so tests and logs identify xAI sessions, and so direct construction (
new xAIRealtimeSession(connection)) keeps working exactly as it did when this was a standalone clone; a directly built session reports the facts of its own driver's static (xAIRealtime.DriverFacts).