OptionalAudioAudio encoding and sample rate format supported or negotiated for this session.
OptionalAvatarWhat became of the session's avatar request, as decided when the session opened. Its driver decides, except when
the driver has nothing to report: the session prep asked it for no avatar, or it renders no avatar and ignored the
request. The agent runtime then says why: a phone call ('phone'), no face for the model's vendor
('no-binding'), an unknown avatar ('unknown-avatar'), a model that shows none ('endpoint'), or a host that
publishes no avatar into a room ('bridged'). Absent when the session asked for no avatar (the voiced agent's video
setting is off), or its driver reported nothing about one on a model that shows avatars.
OptionalCapabilitiesCapability introspection. A small, static description of what THIS live session can do, so the
container can ask "is it safe to call X?" instead of invoking optional methods that silently no-op (or
can't be supported) on some providers — the same role IBridgeProviderFeatures plays for bridges and
BaseRealtimeModel.SupportsClientDirect plays for minting. Optional: a driver that hasn't
declared its capabilities is treated conservatively (everything unsupported). As models gain
abilities, drivers just flip a flag — no container changes.
OptionalCurrentCurrent task revision counter for cancellation and supersede-and-discard.
OptionalInputThe PCM sample rate (Hz) this model consumes on IRealtimeSession.SendInput — its audio INPUT format. Optional; consumers default to 24000 (OpenAI Realtime). Gemini Live = 16000. A server-bridged host (LiveKit/Zoom/Teams) MUST resample inbound room audio to this rate or the model receives mis-rated audio it can't parse (the symptom: the agent never responds on the bridge while the same model works client-direct, where the browser negotiates the rate itself).
OptionalOutputThe PCM sample rate (Hz) this model emits on IRealtimeSession.OnOutput — its audio OUTPUT format. Optional; consumers default to 24000 (both OpenAI and Gemini Live emit 24 kHz today).
OptionalBumpAdvances the task revision counter, causing in-flight tool results from prior revisions to be discarded.
Closes the session and releases the underlying provider connection.
A promise that resolves once the session is fully closed.
OptionalOnOptional capability — registers a handler invoked when the underlying provider
connection closes WITHOUT the consumer having called IRealtimeSession.Close
(provider-side hangup, network drop). Not fired for a consumer-initiated Close() —
the caller already knows about that one.
Optional because not every provider surface exposes a close signal cheaply; callers
must feature-detect (if (session.OnClose) { ... }). An unexpected close is typically
ALSO surfaced as a Fatal IRealtimeSession.OnError, which is the signal
consumers should drive finalization from.
Invoked when the provider connection closes unexpectedly.
Registers a handler for session errors.
Fatality semantics mirror the client-side BaseRealtimeClient contract:
Fatal: true — the session is unusable (transport/socket failure, credential/token
expiry, unexpected connection loss). The consumer should finalize the session (e.g.
RealtimeSessionRunner calls Stop()) instead of idling forever on a dead socket.Fatal: false — a provider-reported, recoverable error frame; the session stays open
and the consumer should log and continue.Invoked with each RealtimeSessionError.
Registers a handler for provider-detected interruptions (barge-in).
True barge-in only: the handler fires ONLY when user speech interrupts ACTIVE model output — NOT on every user utterance. A user simply taking their normal turn while the model is idle is not an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a model response is actually in flight).
Turn detection / VAD is owned by the provider. The agent layer uses this hook to cancel
the model's current turn and to fire the cancellationToken of any in-flight
delegated agent run — a stale delegated result must never be narrated into a conversation
that has moved on.
Invoked when the provider reports a true barge-in interruption.
Registers a handler for model audio output frames (the audio media plane).
Invoked with each output audio frame as an ArrayBuffer.
Registers a handler for model tool-call requests.
Consumers execute the requested tool (under the session's context user) and feed the result back to the model.
Invoked with each RealtimeToolCall.
Registers a handler for transcript events (the text stream).
Consumers typically forward these to the control plane and persist them as
ConversationDetail records.
Invoked with each RealtimeTranscript (partial or final).
Registers a handler for usage/telemetry updates.
Usage is checkpointed incrementally by the agent layer (debounced onto the prompt run) so partial usage is never lost if the session is force-closed after a crash.
Invoked with each RealtimeUsage update.
OptionalOnRegisters a handler for the model's video output as typed frames (RealtimeVideoFrame): pieces of
fragmented MP4, encoded chunks or images, in the order the model sent them. A driver that emits them declares an
outbound video track in RealtimeSessionCapabilities.SupportedOutboundTracks whose Encoding names the
frames' type. A Gemini Live session whose avatar was granted for a host that publishes it
(RealtimeAvatarSettings.Delivery 'room') sends the avatar's MP4 pieces here.
An fMP4 piece may carry the voice on its audio track; while it does, the voice does not also come through
OnOutput. A driver that can tell the model's answer from the video it streams between answers marks the
latter Idle (see RealtimeVideoFrame): the host plays it, but never counts it as the agent speaking.
After OnInterruption the host drops video not yet shown, and the driver drops the interrupted turn's late
frames. The seconds of video the model generated reach OnUsage (OutputTokenDetails.VideoSeconds).
Optional: audio-only drivers don't implement it; call it null-safely (session.OnVideoFrame?.(...)).
Invoked with each frame, in the order the model sent them.
OptionalOnRegisters a handler for the model's video output as untyped bytes. A driver implements this or OnVideoFrame, not both.
Invoked with each output video chunk as an ArrayBuffer.
Use OnVideoFrame, whose frames say what they are: a host can't tell an MP4 piece from an encoded chunk or an image by its bytes. Hosts still forward it. Removed in the next major version.
OptionalReconfigureOptional capability (gate on RealtimeSessionCapabilities.CanReconfigureTurnMode) —
reconfigures a live session's turn-taking without reconnecting: e.g. switch a 1:1 agent to
meeting mode (auto-response off) when its room becomes multi-agent. Providers whose runtime config is
mutable mid-socket (OpenAI: session.update) implement this and report the capability true;
providers whose turn config is fixed at connect (Gemini Live's activity detection) report false and
omit the method. The container must check the capability before calling — never blind-invoke.
The reconfiguration to apply (e.g. DisableAutoResponse).
Registers the set of tools the model may call, translating them into the provider's native function-calling format.
The Core-level RealtimeToolDefinition is intentionally minimal: BaseRealtimeModel
lives in the lowest AI layer and cannot depend on the richer tool metadata defined in
higher packages (the agent layer) — doing so would create an illegal upward/circular
dependency. The agent layer is responsible for mapping its richer tool metadata down
to this Core type before calling RegisterTools, and the concrete driver maps this Core
type up to the provider's native function-calling schema.
Note that some providers (e.g. Eleven Labs) bind to a pre-declared tool set on a server-side agent configuration; for those, the driver maps these definitions onto the pre-declared tool names rather than registering arbitrary schemas at session start.
Idempotency rule: a post-start registration of a set IDENTICAL to the set supplied at connect time (via RealtimeSessionParams.Tools) MUST be a no-op. Providers that bind their tool set at connect time and cannot re-declare schemas on an open session MUST no-op (and may log) rather than degrade the conversation — e.g. by injecting schema text into the conversation as content. A genuinely DIFFERENT post-start set on such a provider is unsupported and should be surfaced as a warning, not silently mangled.
The tools to expose to the model.
OptionalRequestOptional capability — asks the model to voice one brief interim update following the given instructions (e.g. "In one short sentence, tell the user the report agent has finished gathering data and is now drafting"). Used by the agent layer to narrate delegated-run progress while a long-running tool/agent call is still in flight.
Implementations must not collide with an in-flight model response: providers reject or garble overlapping generation requests. A driver must either queue the request until the current response completes or skip it outright — skipping is explicitly acceptable because interim updates are disposable by contract (a stale "still working…" line has no value once the real result lands; the next update or the final result supersedes it).
Like IRealtimeSession.SendContextNote, this is optional because it models provider capability: drivers whose provider cannot trigger an instructed, one-off spoken response mid-session omit the member, and callers must feature-detect before invoking.
Instructions for the single spoken update (tone, brevity, content).
true when a response was actually triggered, false when it was skipped (e.g. a response
is already in flight). A bridge that claimed the speaking floor for this turn uses this to release the
floor immediately on a skip — otherwise a skipped trigger would wedge the room until the safety timer.
void/undefined from legacy drivers is treated as "triggered" for backward compatibility.
OptionalSendOptional capability — injects background context (e.g. delegated-run progress, freshly retrieved data, or a state change the model should be aware of) into the model's conversation without forcing a spoken reply. The model simply has the note available the next time it speaks.
This is the server-side counterpart of the client-direct BaseRealtimeClient.SendContextNote
capability: in the server-bridged topology, the agent layer (e.g. a session runner observing
a delegated agent run) calls this to keep the realtime model informed of long-running work
so it can narrate naturally when asked or when it next takes the floor.
Optionality models capability, not laziness: not every provider supports injecting
conversation items into an already-open session (some only accept media frames and tool
results mid-session). Drivers that cannot inject mid-session omit the member entirely, and
callers must feature-detect (if (session.SendContextNote) { ... }) rather than assume it.
The context note to append to the conversation (plain text; the caller owns any prefixing/framing policy such as "[progress]" markers).
Sends a client media frame to the model.
Fire-and-forget: frames are streamed straight to the provider with no JSON intermediation.
RealtimeInputFrame.Kind tags the media plane: 'audio', or 'video' for a camera or
screen frame to a model that takes video (its session declares an inbound video track in
RealtimeSessionCapabilities.SupportedInboundTracks).
RealtimeInputFrame.MimeType says what format the frame is in. A driver drops a frame
it cannot send rather than sending it as something else.
The media frame, with its kind and (when known) its format.
OptionalSendMid-session instruction update (e.g. OpenAI session.instructions.append).
New or appended instructions for the ongoing session.
Send the result of an executed tool/function call back to the model so it can continue the
turn. output is the JSON-stringified tool result. Called by the agent layer after it handles
an IRealtimeSession.OnToolCall.
The CallID from the originating RealtimeToolCall, used to correlate the result.
The tool's result as a JSON-stringified string.
OptionaltaskRevision: numberOptional task revision counter from the tool call. If supplied and mismatched, stale results are discarded.
A promise that resolves once the result has been sent to the provider.
A long-lived, full-duplex session handle returned by BaseRealtimeModel.StartSession.
All
On*methods register a single handler invoked as the corresponding provider events arrive. Drivers must be written against this interface so a mock provider socket can be substituted for deterministic, network-free testing.