Member Junction
    Preparing search index...

    HuggingFace speech-to-speech implementation of BaseRealtimeClient: a browser-direct websocket that speaks the OpenAI-Realtime wire protocol over the shared PCM audio plane.

    Registered under the key 'huggingface' — the Provider string the server's HuggingFaceRealtime driver stamps — so hosts resolve it without referencing this class directly.

    The EphemeralToken is the full wss://<mjapi-public>/realtime-proxy?ticket=… URL (ElevenLabs-style — the credential IS the URL): the browser connects to MJAPI's realtime proxy, which tunnels transparently to the internal self-hosted endpoint. The client never learns the internal endpoint.

    Because HuggingFace speaks the OpenAI wire protocol over a websocket with a client-owned PCM plane, nearly everything lives in the shared layers (protocol brain + websocket transport in OpenAIProtocolWebSocketRealtimeClient). This class supplies only the HuggingFace specifics: the proxy-URL connect, the { session, sampleRate } server-pact parsing, the session.created readiness gate (OpenAI ordering — a session.update sent earlier is dropped), and the benign close semantics of a self-hosted proxy hop.

    Hierarchy (View Summary)

    Index

    Constructors

    Properties

    Accessors

    Methods

    AddTrack applySessionConfig ApplySessionConfigTiming ArmConnectionDurationTimer AssertTransportSupported attachInputAudioMeter attachOutputAudioMeter buildSpokenUpdateEvent CancelActiveResponse canSendEvents clearRemoteMediaStream closeAudioMeters Connect ConnectionClosesAtCredentialExpiry createMicCapture createPlayback createSocket CredentialConnectionLimit currentSessionInstructions Disconnect DisposeReconnectTimers doSendToolResult dropInputHeldDuringReconnect emitError emitInterruption emitRemoteVideo emitStateChange emitToolCall emitTrackStateChange emitTranscript emitUsage emitUserSpeechStart flushPendingResultResponse formatTransportError GetAudioActivity GetRemoteMediaStream handleAudioDeltaFrame handleEvent handleProtocolMessage handleSocketClose handleSocketError holdsHardReconnectPoint isCredentialNearExpiry isHostNetworkOnline isToolResultSilent IsTrackEstablished isTransportUsable isTurnActiveForReconnect isUserSpeaking logInboundEvent logOutboundEvent negotiateTracks NotifyConnectionEnding OnError OnInterruption onNonJsonFrame onOutputAudioBufferStarted onOutputAudioBufferStopped onReconnected onReconnectFailed onReconnecting OnRemoteMediaStream OnRemoteVideo onResponseStarted onSessionCreatedFrame onSpeechStartedFrame OnStateChange OnToolCall OnTrackStateChange OnTranscript OnUsage OnUserSpeechStart onUserTranscriptFrame openProviderSocket publishRemoteMediaStream Reconnect RecordResumptionHandle recordToolResult recoverLostTransport refreshCredentialForReconnect RemoveTrack ReplaceMicrophone requestResultResponse RequestSpokenTurn RequestSpokenUpdate resetResponseState resetResponseStateForNewConnection resolveSampleRate resolveSessionObject resolveToolName ResumeOnOldConnection seedConversationAfterMove seedConversationHistory SendContextNote sendEvent sendMicChunk sendProtocolEvent SendText SendToolResult SendVideoFrame? SetMuted setState SetTokenRefreshCallback stopAudioOutput TriggerReconnect readInstructions

    Constructors

    Properties

    activeConfig: ClientRealtimeSessionConfig = null

    The active ClientRealtimeSessionConfig, updated on credential refresh.

    activeResponseKind: "normal" | "narration" = 'normal'

    The kind of the response currently in flight. Event ordering (confirmed against the live OpenAI API): response.created → transcript deltas → *_audio_transcript.done → response.done. The transcript-done frame therefore arrives while the kind is still set, letting onAssistantDone classify the turn; response.done then resets it.

    callsFromBeforeReseed: Set<string> = ...

    Tool call IDs that were in flight when a non-native seed reconnect occurred.

    closedByConsumer: boolean = false

    True once Disconnect ran — an expected socket close must not surface as fatal.

    configuredCallDurationSeconds: number

    Duration of a single provider connection in seconds from ClientRealtimeSessionConfig.ConnectionMaxSeconds.

    configuredReconnectBufferSeconds: number = DEFAULT_REALTIME_RECONNECT_BUFFER_SECONDS

    Buffer in seconds before CallDurationSeconds expiry at which Reconnect is triggered.

    configuredSessionMaxSeconds: number

    Credential/session token lifetime in seconds from ClientRealtimeSessionConfig.SessionMaxSeconds (or MaxSessionSeconds).

    configuredTrustNativeResumption: boolean

    Optional metadata override for whether native resumption is trusted on this endpoint.

    confirmedResponseActive: boolean = false

    True while a response the provider has CONFIRMED (response.created seen, response.done not yet) is in flight. Distinct from responseActive, which is set EAGERLY before a local response.create is even sent: if the provider REJECTS that create (an error frame with no response.created), responseActive would otherwise stay stuck true forever. This flag lets onErrorFrame tell "the eager flag is a phantom for a rejected create" (no confirmed response) from "a real response is genuinely active" (e.g. a concurrent VAD turn), so it clears the phantom without disturbing a live turn. Set on response.created, cleared on response.done.

    conversationMemory: RealtimeConversationMemory = ...

    Rolling conversation memory used to re-seed provider context on reconnect.

    currentState: RealtimeClientState = 'closed'

    The client's own view of the session state — mirrors what emitStateChange last reported, EXCEPT after a tool call is emitted: the host typically shows its own busy state then, so the client silently leaves 'speaking' (no emission) to preserve the host's indicator until the result reply starts (see onToolCallFrame).

    inputAudioMeter: IRealtimeAudioMeter = null

    Meter over the USER's microphone, when the driver attached one.

    micCapture: IPcmMicCapture = null

    The mic-capture pipeline streaming PCM16 up to the provider.

    micStream: MediaStream = null

    The mic stream owned by the current connection (used by the shared SetMuted).

    outputAudioMeter: IRealtimeAudioMeter = null

    Meter over the AGENT's audio output, when the driver attached one.

    pendingAssistantText: string = ''

    Accumulates the in-flight assistant transcript across delta frames.

    pendingLocalResponseCreates: number = 0

    Count of locally-initiated response.creates whose response.created echo has not arrived yet. While non-zero, a response.done belongs to an EARLIER (typically just-cancelled) response — it must not clear the busy flag, flush the queued trigger, or flip the state for the response we just started (usage is still emitted). Consumed by response.created.

    pendingNarrationKind: boolean = false

    Set by RequestSpokenUpdate just before it sends its response.create, and CONSUMED by the very next response.created frame, which stamps activeResponseKind for that turn. Narration is only requested while the model is idle (hosts gate on IsBusy), so under normal ordering the next response.created is ours.

    pendingResultResponse: boolean = false

    Set when a tool result is ready while a response is active; sent on the next response.done.

    playback: IRealtimePcmPlayback = null

    The local playout engine (client-owned audio plane).

    reconnectController: RealtimeReconnectController = ...

    Unified I/O-free reconnect controller shared with server-side BaseRealtimeSession.

    responseActive: boolean = false

    True while the model has a response in flight; gates narration + queues the tool result.

    sessionObject: JSONObject = {}

    The wire-shaped session object applied via session.update at the readiness boundary. Protected so test subclasses can seed it without a full Connect.

    The live socket (null when disconnected). Protected so test subclasses can inspect.

    socketOpen: boolean = false

    True while the socket is OPEN — gates every send so a CONNECTING socket never throws.

    tokenRefreshCallback: RealtimeClientTokenRefreshCallback = null

    Optional callback installed by RealtimeSessionRuntime to refresh the provider token mid-call.

    tracks: RealtimeTrack[] = []

    Tracks requested, established, or denied on this session.

    Accessors

    • get CallDurationSeconds(): number

      Effective connection ceiling in seconds. Metadata and runtime caps narrow the driver's provider ceiling (ClampSessionSecondsValue(DefaultCallDurationSeconds, RuntimeCallDurationSeconds, configuredCallDurationSeconds)), never widen it.

      Returns number

    • get connectTimeoutMs(): number

      Connect-phase deadline in milliseconds, covering socket-open AND (when the provider defers) the session.created wait. On expiry Connect rejects with a fatal error instead of hanging forever against a silent endpoint. Override per driver/deployment.

      Returns number

    • get InboundVideoRate(): number

      The frame rate (frames per second) the live inbound video track was negotiated at — the model's ceiling or the session's request, whichever is lower — or undefined when no inbound video track is live or none declared a rate. Anything pacing a video feed reads THIS, never a constant of its own.

      Returns number

    • get IsAudioPlaying(): boolean

      Returns boolean

      directly from the playout engine's playhead clock — this client OWNS the output buffer, so "audibly playing" is precisely "scheduled audio extends beyond the audio context's current time".

    • get IsReconnectCycleActive(): boolean

      True from the start of a reconnect cycle until it succeeds or fails. Unlike IsReconnecting, it stays true while a planned move waits between attempts with the call live again on its old connection. A driver that would otherwise end the call on a loss of that connection checks this and hands the loss to the running cycle (TriggerReconnect('connection-lost')) instead.

      Returns boolean

    • get MaxInboundVideoStreams(): number

      How many concurrent inbound video streams the model accepts: the ceiling the driver declared when it negotiated tracks, else 1 when an inbound video track is live and 0 when it is not.

      A source arbiter reads this to decide whether several live video sources must be narrowed to one (with a note to the model on each switch) or passed through untouched.

      Returns number

    • get RecoversLostTransport(): boolean

      Returns boolean

      the proxy hop closes benignly and the host reads 'closed' as the end of the session (see handleSocketClose), so a lost socket does not start a reconnect and a socket error stays fatal. A loss during a planned move still goes to that move's reconnect.

    • get RequiresPostTurnResumptionHandle(): boolean

      Whether this driver's provider emits a fresh resumable handle shortly after each turnComplete (Gemini / Vertex AI Live), so planned moves wait for that post-turn handle.

      Returns boolean

    • get RuntimeCallDurationSeconds(): number

      Optional runtime ceiling override (e.g. Gemini's 120s ceiling when uncompressed video is active). Narrowed against DefaultCallDurationSeconds and configuredCallDurationSeconds via ClampSessionSecondsValue.

      Returns number

    • get StateToRestoreOnOldConnection(): RealtimeClientState

      The state to report again when ResumeOnOldConnection goes back to the old connection, or null when the old connection has reported a state of its own since the move started (it keeps delivering events during the attempt), which is current. 'speaking' while the turn is open, else 'listening': the turn, not the state the move replaced, says whether the agent is answering, since a driver can leave 'speaking' without reporting it (a tool call) and the old connection can start an answer during the attempt. A 'speaking' report opens a turn, and one with no answer behind it would hold the next attempt until the last call. 'connected' is never reported: a host keeps showing the move for it.

      Returns RealtimeClientState

    • get SupportsInboundVideo(): boolean

      Whether the model takes inbound video on this session: the driver declared an inbound video track it supports when it negotiated tracks, and a stream limit above zero. No video track need be live; one is added when a capture starts (AddTrack). false before tracks are negotiated, and for a driver that does not negotiate them.

      Returns boolean

    • get SupportsRelayTransport(): boolean

      Whether this driver can open a relay session (ClientRealtimeSessionConfig.Transport 'relay'): connect to MJAPI's realtime relay at the session's RelayUrl rather than to its provider. false here; a driver that speaks its provider's protocol through the relay opts in by overriding it (the Gemini Enterprise client does).

      Returns boolean

    Methods

    • Adds a track to a running session, such as inbound video when the user starts their camera, resolved against what the driver supported when it negotiated at connect. The new track is 'live', or 'unsupported' with the reason (the model takes no such track, or no more inbound video streams). A live track with the same direction, modality and source is returned as it is. Other tracks keep their ids.

      The base re-negotiates locally, which is all a driver needs when its provider accepts the media without a setup change (Gemini Live takes video frames at any time). A driver whose wire must change overrides this and calls it.

      Parameters

      Returns RealtimeTrack

      The track, or null when the session has not negotiated tracks: before it connects, or with a driver that does not negotiate them (only the Gemini driver does today).

    • Arms (or re-arms) the soft and hard connection-ceiling timers for the current transport socket.

      • Soft timer opens the turn-boundary window before CallDurationSeconds.
      • Hard timer fires at CallDurationSeconds - ReconnectBufferSeconds and forces a reconnect; while the model is answering it waits for the turn to end, up to shortly before CallDurationSeconds. Both measure from the connection deadline instead when it comes before CallDurationSeconds (connectionDeadlineAtMs: the expiry of the credential the connection opened on, on a driver whose connections close with it).

      Returns void

    • The response.create frame for one spoken update — carrying the session identity, not replacing it.

      ⚠️ response.instructions IS A FULL OVERRIDE OF THE SESSION SYSTEM PROMPT for that one response, not an addition to it. That is the whole reason this method exists. The server-side twin of this driver (openAIRealtime.ts) already states the rule and guards the BLANK case on it — forwarding '' "would wipe the co-agent identity framing" — but the NON-blank case was never guarded, so every caller that supplied a real direction wiped it just as thoroughly and much less visibly.

      What that cost, live: the one turn that rides this method had no persona, so the model fell back to its vendor default and opened a hiring interview with "I'm ChatGPT, your friendly voice companion" while every other turn in the same session correctly said "I'm Sam Rivera, Support Team Lead" (bizapps-caliber#397 / MJ#4591). Every commentary-style caller is exposed the same way — an opening turn, a silence check-in, a delegation narration — and none of them should have to restate an identity the session already carries.

      Three cases, and the middle one is the fix:

      • Blank direction → a bare response.create. "Respond now, under the session prompt" (the meeting-mode bridge trigger passes ''). Matches the server twin exactly.
      • Direction + session instructions → identity, then direction. The identity leads because it is context, and the direction is LAST because it is the thing being asked for now — the most recent line is the one a model weights hardest.
      • Direction, no session instructions → the direction alone. A session minted without instructions has no identity to lose, and prefixing a blank line would be noise.

      Parameters

      • instructions: string

      Returns OAIProtocolResponseCreateEvent

    • Sends the tool result back as a function_call_output conversation item, then triggers a reply — immediately if the model is idle, otherwise queued until the current response (e.g. a progress narration) finishes. Without the queueing the result's response.create would collide with an in-flight narration and be dropped, leaving the model silent when delegated work comes back.

      Parameters

      • callID: string
      • outputJson: string

      Returns boolean | void

    • Returns the AGENT's remote-audio MediaStream when this driver owns a tappable agent-audio plane (a WebRTC driver's remote track, or a PCM driver's playout output), or null otherwise. Hosts use it to MIX the agent's voice into a browser-side recording alongside the mic; null (drivers with no tappable plane, or before the stream lands) degrades gracefully to mic-only capture. Hosts may still call it as client.GetRemoteMediaStream?.(). Pure read — never mutates.

      Returns MediaStream

    • Returns void

      close the consumer didn't ask for is reported as a terminal 'closed' — NOT the fatal error the cloud providers surface: the self-hosted proxy hop closes benignly (ticket expiry, upstream restart), and the host treats 'closed' as the session-over signal. During a planned move the closed socket is the one the move may go back to, so the move's reconnect takes the loss and the call goes on (recoverLostTransport answers true only then).

    • Registers the (single) interruption handler.

      True barge-in only: fires ONLY when USER INPUT CUT OFF ACTIVE MODEL OUTPUT — a model response in flight or audio audibly playing when the user took the floor. A user simply taking their normal turn while the model is idle is NOT an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a response is actually active or audio is playing).

      On interruption, drivers must also flush locally-owned playback and report IsAudioPlaying === false promptly (driver obligation #3). Hosts typically use this hook to abort in-flight delegated work so a stale result is never narrated into a conversation that has moved on.

      Parameters

      • handler: () => void

      Returns void

    • Registers a handler invoked when the agent's remote-audio stream becomes available — IMMEDIATELY (synchronously) if it has already landed, otherwise when the driver publishes it (a WebRTC track typically lands AFTER Connect resolves). Lets a host attach the agent voice to a recording that began before the stream landed. Handler errors are isolated (logged) so a host bug never disturbs the call. Handlers are session-scoped: clearRemoteMediaStream (called from Disconnect) drops them.

      Parameters

      • handler: (stream: MediaStream) => void

      Returns void

    • Registers the (single) remote-VIDEO handler — the model/avatar's video for a VIDEO session (a talking-head the host renders, e.g. as the agent's tile). Invoked once the provider's video is available.

      The video is a MediaVideoSource: a live stream (a WebRTC track), or a player that must own the <video> element (MSE or WebCodecs playout of encoded avatar video). Hosts show either with AttachVideoSource.

      Optional capability: audio-only drivers (the default) never emit — registering is always safe, but hosts must not assume video arrives. A driver whose model sends video (the server driver declares an outbound video track in its session's Capabilities.SupportedOutboundTracks, or renders an avatar, BaseRealtimeModel.SupportsAvatarOutput) calls emitRemoteVideo when it is live.

      Parameters

      • handler: (video: MediaVideoSource) => void

        Invoked with the remote video when it becomes available.

      Returns void

    • Registers the single handler invoked when the provider's voice-activity detection saw the user start to speak. Fires once per detected onset and may fire again inside one turn after a pause; it carries no text. When the provider's event names the conversation item this speech belongs to (OpenAI-protocol input_audio_buffer.speech_started.item_id), itemId carries that id so the host can pair the onset with the matching user transcript (RealtimeClientTranscript.ItemId); otherwise itemId is undefined. Unlike OnInterruption, it fires whether or not the model was speaking.

      Parameters

      • handler: (itemId?: string) => void

      Returns void

    • Hands an unexpected loss of the live call's transport (a close or error the consumer did not ask for) to the shared reconnect controller, which retries within its recovery window and reports reconnect-failed when the window closes. Returns false when there is no live call to save, so the caller keeps its terminal handling: the connect has not finished (Connect reports that failure itself), the call already ended, or the driver opts out.

      A loss while a reconnect cycle runs is the loss of the transport the cycle may go back to after a failed attempt (the callers' identity checks drop events of a transport the cycle opened or already let go of). It goes to the cycle, even on a driver that opts out of starting one, and returns true: the cycle then never returns the call to a dead transport, and goes on as a lost connection's.

      Live means the call went live and has not ended (callWentLive), not a speech state: during a tool run the state is 'connected' (see onToolCallFrame), and a loss then is as recoverable as one mid-sentence.

      Parameters

      • detail: string

      Returns boolean

    • Triggers ONE short spoken update with the given instructions. Marks the upcoming response as 'narration' (flag consumed by the next response.created) so its transcripts are emitted with Kind: 'narration' — ephemeral by contract. Sets responseActive eagerly so a tool result landing mid-narration queues instead of colliding.

      Skips when busy (base-contract collision rule — drivers MUST queue or skip): a response.create sent while a response is in flight would be rejected/garbled by the provider, and narration is disposable by contract, so the update is dropped with a debug log rather than queued to come out late and stale. Hosts SHOULD still gate on IsBusy / IsAudioPlaying for timing quality.

      The wire frame is built by buildSpokenUpdateEvent, which is where the identity rule lives — see that method for why the caller's direction is never sent on its own.

      Parameters

      • instructions: string

      Returns void

    • Seeds the new connection after a move (when sendsSeed; a natively resumed one already has the history) and notes whether the move cut off an answer the user is still owed: one was in flight on the old connection (answerWasInFlight, from resetResponseStateForNewConnection) and the conversation ends with what it answers (a user turn or a tool result), not counting the context notes after it. Neither a seeded nor a resumed session replies by itself, so without this the question sits unanswered in every later seed. An answer that completed is in memory as an assistant turn and is not asked for again.

      A seeded connection knows the conversation as the seed sent it. A resumed one knows it as the provider kept it, which includes what the old connection heard while the attempt was opening, after context.Seed was built: so then the memory as it is now decides.

      Parameters

      Returns void

    • Streams one video frame as base64-encoded image data to the model. Optional capability — drivers that support inbound video tracks implement this.

      Parameters

      • base64Image: string

        Base64-encoded image data.

      • OptionalmimeType: string

        Image MIME type (defaults to 'image/jpeg').

      • OptionalsourceId: string

        Which source this frame came from. A driver whose model takes ONE inbound video stream ignores it (the source arbiter already chose what to send); a driver whose model takes several routes the frame to the track bound to that SourceID.

      Returns boolean

      true if accepted and sent; false if dropped (throttled, unestablished, etc.).