Member Junction
    Preparing search index...

    Real-time, full-duplex driver for a self-hosted HuggingFace speech-to-speech server running in OpenAI-Realtime-compatible mode. Registers as HuggingFaceRealtime and is resolved for MJ: AI Models typed Realtime (API-key env alias: AI_VENDOR_API_KEY__HuggingFaceRealtime — optional; many self-hosted endpoints are unauthenticated).

    Topologies:

    • Server-bridged (StartSession): the driver opens a raw websocket to the internal endpoint through RawRealtimeWebSocketConnection and runs the SHARED OpenAI-protocol session over it. Unlike the base driver, StartSession resolves only after the session config has been applied (the endpoint's session.created → session.update handshake) so the runner never streams audio against an unconfigured model (obligation #7).
    • Client-direct (CreateClientSession): the driver mints a one-time proxy ticket and returns a wss://<mjapi-public>/realtime-proxy?ticket=… URL. The browser opens its socket to MJAPI's proxy, which tunnels transparently to the internal endpoint (injecting any auth server-side). The internal endpoint never reaches the browser and needs no browser-facing ingress.

    No managed server object and no usage billing: the entire session config (prompt, tools, voice) is supplied per session; there is no token-usage meter, so OnUsage fires only if the compat endpoint happens to report a response.done usage block.

    Hierarchy (View Summary)

    Index

    Constructors

    Properties

    DriverFacts: Readonly<RealtimeDriverFacts> = {}

    Provider ceiling unconfirmed (self-hosted speech-to-speech stacks set their own); unknown.

    SupportsDynamicToolSet: boolean = HUGGINGFACE_REALTIME_PROFILE.supportsDynamicToolSet

    Whether this driver subclass supports dynamic, multi-tool sets projected into the realtime session (e.g. direct action invocation on realtime co-agents).

    Static capability descriptor read before a session instance exists. Default: absent / false. Subclasses that support dynamic toolsets declare public static readonly SupportsDynamicToolSet = true;.

    Accessors

    • get apiKey(): string

      Only sub-classes can access the API key

      Returns string

    • get CallDurationSeconds(): number

      Readonly getter returning the model driver's connection duration in seconds.

      Returns number

    • get DefaultCallDurationSeconds(): number

      Static driver default for a single connection's duration in seconds (for example, 600 on GeminiRealtime and GeminiEnterpriseRealtime, 3600 on OpenAIRealtime and OpenAILiveRealtime, 7200 on xAIRealtime). Defaults to DriverFacts.ProviderMaxSessionSeconds.

      Returns number

    • get OpenAI(): OpenAI

      Read-only accessor for the underlying OpenAI SDK client.

      Returns OpenAI

    • get ReconnectBufferSeconds(): number

      Buffer in seconds before CallDurationSeconds at which the framework initiates a proactive reconnect. Defaults to DEFAULT_REALTIME_RECONNECT_BUFFER_SECONDS (10s).

      Returns number

    • get RuntimeCallDurationSeconds(): number

      Dynamic runtime override hook on the model driver (returns undefined unless overridden by a subclass).

      Returns number

    • get SupportedVoices(): RealtimeVoiceOption[]

      The compat layer's TTS stage has no OpenAI-style selectable voice catalog, so the dev voice picker gets none — overriding away the OpenAI voice list this class would otherwise inherit. (A voice Config value still passes through to audio.output.voice for stacks that honor it.)

      Returns RealtimeVoiceOption[]

    • get TrustsNativeResumption(): boolean

      Whether this driver supports and trusts the provider's native session resumption on the active endpoint when a handle is present. Subclasses override as needed (e.g., true on server-side GeminiRealtime, GeminiEnterpriseRealtime, and xAIRealtime; false on OpenAIRealtime).

      Returns boolean

    Methods

    • Returns the smallest candidate that is a positive integer; see ClampSessionSecondsValue. The rule "the caller's bag wins only if smaller" lives there, once: pass the provider cap and the bag cap together.

      Parameters

      • ...candidates: readonly number[]

      Returns number

    • Resolves whether native resumption is trusted for a client-direct session minted by this driver, given the transport ('direct' vs 'relay'). Subclasses override when client-direct trust differs from server-bridged trust (e.g., GeminiRealtime is true on server v1beta keys but false on browser-direct v1alpha ephemeral tokens).

      Parameters

      Returns boolean

    • Creates the realtime connection for a model. Overridable seam for testing.

      Production returns a real OpenAIRealtimeWebSocket. Unit tests override this to return a fake IOpenAIRealtimeConnection that emits OpenAI-shaped events and captures sends.

      Parameters

      • model: string

        The provider realtime model id (e.g. gpt-realtime-2.1).

      • OptionalresumptionHandle: string

        Optional provider session/conversation handle to resume (?conversation_id=).

      Returns IOpenAIRealtimeConnection

      A connection implementing IOpenAIRealtimeConnection.

    • Optional WebRTC SDP exchange seam for drivers that support server-brokered WebRTC topologies.

      Parameters

      • _offerSdp: string

        The local SDP offer generated by the browser.

      • Optional_sessionConfig: Record<string, unknown>

        The session configuration to initialize the session with.

      Returns Promise<{ answerSdp: string; prebillSeconds: number; sessionId: string }>

      The answer SDP, provider session ID, and any prebill accounting.

    • Mints the ephemeral client secret via the provider's Realtime client-secrets API (resolved from the SDK client's base URL, so OpenAI-compatible subclasses target their own endpoint). Overridable seam for testing — unit tests return a fake response so no network call is made.

      Parameters

      • body: ClientSecretCreateParams

        The client-secret create request (carries the realtime session config).

      Returns Promise<ClientSecretCreateResponse>

      The client-secret create response (token value + expiry + echoed session).

    • Reconnect hook for server-hosted IRealtimeSession instances created by StartSession. Called by BaseRealtimeSession's duration timer (CallDurationSeconds - ReconnectBufferSeconds) or on unexpected transport close when resumable. Concrete provider drivers override this method.

      Parameters

      Returns Promise<void>

    • Resolves the effective single-connection duration in seconds from an open config bag (connectionMaxSeconds), narrowing this.CallDurationSeconds via ResolveRealtimeTiming.

      Parameters

      • Optionalconfig: Record<string, unknown>

      Returns number

    • Resolves the effective reconnect buffer in seconds from an open config bag (reconnectBufferSeconds), falling back to this.ReconnectBufferSeconds.

      Parameters

      • Optionalconfig: Record<string, unknown>

      Returns number

    • Resolves whether native session resumption is trusted for a session from an open config bag (trustNativeResumption), falling back to defaultTrust (or this.TrustsNativeResumption).

      Parameters

      • Optionalconfig: Record<string, unknown>
      • OptionaldefaultTrust: boolean

      Returns boolean

    • Builds the upstream Authorization header from the API key, or undefined when the endpoint is unauthenticated. Because the realtime resolver requires a resolvable API key for a model to be selectable, a keyless self-hosted deployment sets a sentinel value (none / self-hosted / local / n/a) for AI_VENDOR_API_KEY__HuggingFaceRealtime — treated here as "no auth" so no bogus header is ever sent upstream (consumed by the MJAPI realtime proxy; the raw browser/Node socket cannot carry headers at all).

      Returns string

    • Whether this driver's sessions can render a live avatar for the model, on the endpoint the driver serves.

      Defaults to false: a driver whose models render no avatar says nothing more, and may ignore an avatar request. A driver that renders one for some models (Gemini Enterprise) overrides it. The client-session service reads it to tell a call or a meeting that asked for an avatar "this voice model can't show an avatar" (RealtimeAvatarUnavailableReason 'endpoint'), whichever driver serves it.

      Parameters

      • _model: string

        The provider's API name for the model.

      Returns boolean

      true when sessions on this model can render a live avatar; false otherwise.

    • Builds the OpenAI-Realtime session object from the Core params:

      • instructions: the system prompt, with InitialContext folded in under a "Prior context" heading (the compat protocol has no guaranteed history-seeding channel).
      • tools: the Core tool set mapped to the OpenAI { type: 'function', … } schema.
      • audio.output.voice: from params.Config.voice, when supplied.
      • audio.input.transcription.model: from params.Config.inputTranscriptionModel, when supplied (the pipeline's STT stage transcribes the user natively; this only names a specific model if the compat layer supports selecting one).

      Parameters

      Returns JSONObject

    • Folds optional prior context into the system prompt (no guaranteed history channel).

      Parameters

      • systemPrompt: string
      • OptionalinitialContext: string

      Returns string

    • Converts an http(s) URL (or origin) into a ws(s)://host[:port] origin, dropping any path. Delegates to the shared HttpOriginToWs in @memberjunction/ai.

      Parameters

      • source: string

      Returns string