Member Junction
    Preparing search index...

    Live realtime session for a self-hosted HuggingFace endpoint — the shared OpenAIRealtimeSession protocol implementation bound to HUGGINGFACE_REALTIME_PROFILE, with three compat-endpoint accommodations layered on top:

    • Beta event aliases: older speech-to-speech builds emit pre-GA names (response.audio.delta, response.audio_transcript.*); these are translated to their GA equivalents before the shared dispatcher runs, so both generations of the stack work.
    • Response-active robustness: the flag is set on the first audio delta (compat stacks don't always emit response.created) and released when a tool call yields the floor (deadlock guard — the stack may not emit response.done after a function call).
    • Tool-set fingerprinting: RegisterTools no-ops an order-insensitively identical re-registration instead of re-declaring the same schemas on the live session.

    Hierarchy (View Summary)

    Index

    Constructors

    Properties

    callsFromBeforeReseed: Set<string>
    closedByConsumer: boolean
    configuredCallDurationSeconds: number
    configuredReconnectBufferSeconds: number
    configuredSessionMaxSeconds: number
    configuredTrustNativeResumption: boolean
    configuredZeroDataRetention: boolean
    conversationMemory: RealtimeConversationMemory
    pendingToolCallNames: Map<string, string>
    reconnectController: RealtimeReconnectController
    responseActive: boolean

    Whether a model response is currently in flight. Minimal response tracking that mirrors the client driver's state machine: set on response.created (and eagerly whenever this session sends its own response.create, so back-to-back local triggers can't race the server event), cleared on response.done — which the API emits for every terminal status, including cancelled after barge-in, so the flag can never stick. Consumed by OpenAIRealtimeSession.RequestSpokenUpdate to skip (not collide with) an active response, since the API rejects overlapping response.create requests.

    Protected (not private) so compat-endpoint session subclasses can apply provider-specific robustness tweaks (e.g. HuggingFace marks a response active on the first audio delta and releases the flag when a tool call yields the floor).

    turnOpen: boolean

    Accessors

    • get CallDurationSeconds(): number

      Effective single-connection duration in seconds, resolved via ClampSessionSecondsValue so metadata (configuredCallDurationSeconds) narrows the driver's provider ceiling (DefaultCallDurationSeconds) and any state-dependent RuntimeCallDurationSeconds, never widening it.

      Returns number

    • get Capabilities(): RealtimeSessionCapabilities

      Returns RealtimeSessionCapabilities

      profile-gated: only providers whose endpoint honors a live partial session.update.

    • get configReadinessTimeoutMs(): number

      Readiness deadline in milliseconds for the deferred-config wait. Only affects consumers of WaitForConfigApplied; the deferred apply itself is not cancelled. Overridable.

      Returns number

    • get ConversationMemory(): RealtimeConversationMemory

      Accessor for the session's provider-neutral conversation memory ledger.

      Returns RealtimeConversationMemory

    • get DefaultCallDurationSeconds(): number

      Static driver default connection duration in seconds when metadata does not specify one.

      Returns number

    • get DefaultTrustsNativeResumption(): boolean

      Driver default for whether native resumption handles are trusted when metadata does not override.

      Returns boolean

    • get InputSampleRate(): number

      HuggingFace speech-to-speech runs at 16 kHz for BOTH input and output (not the OpenAI-family 24 kHz default the base session leaves undefined). Declaring them lets the server-bridged audio plane resample correctly instead of feeding 24 kHz into a 16 kHz pipeline.

      Returns number

    • get IsClosedByConsumer(): boolean

      True once the consumer has closed the session.

      Returns boolean

    • get IsReconnecting(): boolean

      True while an in-place reconnect is currently running.

      Returns boolean

    • get ReconnectBufferSeconds(): number

      Buffer in seconds before CallDurationSeconds at which the framework initiates a reconnect.

      Returns number

    • get RequiresPostTurnResumptionHandle(): boolean

      Whether this session's provider emits a fresh resumable handle shortly after each turnComplete (e.g. Gemini / Vertex AI Live), so a planned move (goAway or soft window) during a turn waits for that post-turn handle (bounded by the deadline timer) rather than resuming from a pre-turn handle.

      Returns boolean

    • get ResumptionHandle(): string

      Latest recorded provider resumption handle, or null if none.

      Returns string

    • get RuntimeCallDurationSeconds(): number

      Dynamic state-dependent duration cap in seconds (for example, returning 120 when uncompressed video is active on Gemini Live, and undefined otherwise). Clamped with configuredCallDurationSeconds via ClampSessionSecondsValue so neither a seeded 600s catalog value nor a tighter custom metadata value (e.g., 45s) is shadowed.

      Returns number

    • get SessionMaxSeconds(): number

      Configured credential/session lifetime in seconds, when distinct from CallDurationSeconds.

      Returns number

    • get TrustsNativeResumption(): boolean

      Whether native resumption is trusted on this session (always false under ZeroDataRetention).

      Returns boolean

    Methods

    • Applies the initial session config: system prompt + tools via session.update, optional initial context as a user message. Called once by OpenAIRealtime.StartSession.

      Deferral is profile-driven. On OpenAI the realtime WebSocket is NOT open when StartSession returns — sending session.update synchronously races the handshake and the instructions (the system prompt + tools) are silently dropped, so the model runs with NO prompt (no identity, no companion framing). We therefore wait for the server's session.created frame — the first event once the socket is open and the session exists, and the canonical moment to configure a realtime session — exactly the point the browser/client-direct path applies its config. Idempotent (a re-emitted session.created can't double-apply); the listener removes itself once it fires. Providers whose socket accepts config immediately (xAI) set the profile flag false and send synchronously.

      Parameters

      Returns void

    • Extracts timing and native-resumption overrides from a session configuration bag.

      Parameters

      • config: Record<string, unknown>

      Returns void

    • Arms (or re-arms) the proactive connection-duration timer for the current connection.

      Returns void

    • Closes the session and releases the underlying provider connection.

      Returns Promise<void>

      A promise that resolves once the session is fully closed.

    • Cancels all reconnect and duration timers and abandons any in-flight attempt when closed by the consumer. Tool results held for the attempt are dropped: no new connection will carry them.

      Returns void

    • Holds a tool result that arrives while a reconnect is moving the session, and returns true when it did. Sent then, the result would reach the connection being replaced after the attempt's seed was built, so the new connection would have the call and never its result. A driver calls this first in SendToolResult and returns when it is true. Once the move lands the result goes through SendToolResult again, after callsFromBeforeReseed is rebuilt, so a reseeded connection gets it as a late-result note; a failed reconnect or a close drops it with a warning. Outside a reconnect it returns false and changes nothing.

      Parameters

      • callID: string
      • output: string
      • OptionaltaskRevision: number

      Returns boolean

    • Clears the recorded resumption handle so subsequent reconnects fall back to history seeding.

      Returns void

    • Handles a provider-initiated connection-ending advisory (goAway / session.going_away). Reconnects immediately if idle and ready, or waits for the turn/handle boundary up to the scaled deadline.

      Parameters

      • OptionaltimeLeftMs: number

      Returns void

    • Notifies the reconnect controller when pending tool calls drain on SendToolResult. When expectsReply is true (default), a queued soft-window or ConnectionEnding move waits for the model's post-tool turn (or a fresh trusted native handle) rather than reconnecting in the gap between SendToolResult and the first model output frame.

      Parameters

      • OptionalexpectsReply: boolean

      Returns void

    • Marks a turn as ended and triggers a queued soft-window or ConnectionEnding reconnect if ready.

      Parameters

      • OptionalkeepPostToolTurnWait: boolean

      Returns void

    • Marks a turn as open so soft-window reconnects wait for the turn boundary.

      Returns void

    • Optional capability — registers a handler invoked when the underlying provider connection closes WITHOUT the consumer having called IRealtimeSession.Close (provider-side hangup, network drop). Not fired for a consumer-initiated Close() — the caller already knows about that one.

      Optional because not every provider surface exposes a close signal cheaply; callers must feature-detect (if (session.OnClose) { ... }). An unexpected close is typically ALSO surfaced as a Fatal IRealtimeSession.OnError, which is the signal consumers should drive finalization from.

      Parameters

      • handler: () => void

        Invoked when the provider connection closes unexpectedly.

      Returns void

    • Registers a handler for session errors.

      Fatality semantics mirror the client-side BaseRealtimeClient contract:

      • Fatal: true — the session is unusable (transport/socket failure, credential/token expiry, unexpected connection loss). The consumer should finalize the session (e.g. RealtimeSessionRunner calls Stop()) instead of idling forever on a dead socket.
      • Fatal: false — a provider-reported, recoverable error frame; the session stays open and the consumer should log and continue.

      Parameters

      Returns void

    • Registers a handler for provider-detected interruptions (barge-in).

      True barge-in only: the handler fires ONLY when user speech interrupts ACTIVE model output — NOT on every user utterance. A user simply taking their normal turn while the model is idle is not an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a model response is actually in flight).

      Turn detection / VAD is owned by the provider. The agent layer uses this hook to cancel the model's current turn and to fire the cancellationToken of any in-flight delegated agent run — a stale delegated result must never be narrated into a conversation that has moved on.

      Parameters

      • handler: () => void

        Invoked when the provider reports a true barge-in interruption.

      Returns void

    • Registers a handler for model audio output frames (the audio media plane).

      Parameters

      • handler: (chunk: ArrayBuffer) => void

        Invoked with each output audio frame as an ArrayBuffer.

      Returns void

    • Hook invoked when a reconnect attempt succeeds.

      Parameters

      • _usedNativeResumption: boolean

      Returns void

    • Registers a handler for model tool-call requests.

      Consumers execute the requested tool (under the session's context user) and feed the result back to the model.

      Parameters

      Returns void

    • Registers a handler for transcript events (the text stream).

      Consumers typically forward these to the control plane and persist them as ConversationDetail records.

      Parameters

      Returns void

    • Registers a handler for usage/telemetry updates.

      Usage is checkpointed incrementally by the agent layer (debounced onto the prompt run) so partial usage is never lost if the session is force-closed after a crash.

      Parameters

      Returns void

    • Parameters

      Returns void

      turn-mode change via a partial session.update — flips server-VAD auto-response on/off without reconnecting (e.g. re-gate a 1:1 agent to meeting mode when its room becomes multi-agent). It re-sends the merged audio.input block (transcription, turn detection and any raw bag audio.input override) so the partial update can't drop them. It never sends audio.output: OpenAI rejects a voice change on a live session once audio has played.

      The frame always carries audio.input.turn_detection, because flipping create_response is the whole point. The explicit live DisableAutoResponse wins over any bag create_response: the caller asked for this mode now. The one exception is a bag turn_detection: null, re-sent as null in either mode so VAD stays off, as it was at start: without VAD nothing auto-responds.

    • Reconnects the underlying OpenAI-protocol WebSocket before connection expiry and either resumes natively via context.ResumptionHandle (?conversation_id=<handle>) or seeds accumulated conversation history (ConversationSeed) onto the replacement connection.

      Parameters

      Returns Promise<void>

    • Records a provider resumption handle or conversation ID.

      Parameters

      • handle: string
      • Optionalresumable: boolean

      Returns void

    • Parameters

      • instructions: string

        Instructions for the single spoken update. Blank/empty means "respond now using the SESSION system prompt" (the meeting-mode bridge trigger passes ''); a non-empty value is forwarded as a per-response override carrying the session prompt ahead of it — see below.

      Returns boolean

      ONE short spoken update via response.create with per-response instructions.

      Collision behavior: skip. The Realtime API rejects a response.create while another response is active, so when responseActive is set the request is dropped — interim updates are disposable by contract (the next update or the final result supersedes them). When sent, the flag is set eagerly (before the server's response.created echo) so two back-to-back local triggers can't both fire.

    • Parameters

      Returns void

      a system-role conversation item (conversation.item.create) the model can draw on the next time it speaks, WITHOUT a response.create — so no spoken reply is forced.

      NOTE: the role must be 'system' — gpt-realtime rejects 'developer' items ("Developer messages are only supported for quicksilver sessions"); same constraint the client-direct driver hit. Item creation is always safe mid-response on OpenAI, so no collision guard is needed here (unlike OpenAIRealtimeSession.RequestSpokenUpdate).

    • Parameters

      Returns void

      one audio frame to the input audio buffer. The session sends audio only, so a frame of another kind (a camera or screen frame) is dropped, never appended as audio, and reported once per kind and type. The xAI and Hugging Face sessions inherit this.

    • Parameters

      • callID: string

        The CallID from the originating RealtimeToolCall.

      • output: string

        The tool's result as a JSON-stringified string.

      Returns Promise<void>

      the tool-call loop for OpenAI: sends a conversation.item.create with a function_call_output item carrying the tool output, then a response.create so the model continues the turn with the result in context. While a reconnect is moving the session the result is held and sent on the new connection (see holdToolResultDuringReconnect).

    • Resolves once the initial session config has been APPLIED (sent on the socket) — immediately for providers that configure synchronously, or on the server's session.created frame for deferring providers. Rejects if the transport dies (fatal error or unexpected close) or the consumer closes the session before the config went out.

      The base OpenAIRealtime.StartSession deliberately does NOT await this (OpenAI semantics: the session handle is returned while the handshake completes). Drivers whose contract promises "ready only after config is applied" (HuggingFace) await it in their StartSession override.

      Returns Promise<void>