The upstream key, or a keyless sentinel (none / self-hosted / local / n/a)
for unauthenticated self-hosted endpoints. The inherited SDK client is never used for the
socket (the raw-WS adapter is), so the sentinel is harmless there too.
Static ReadonlyDriverProvider ceiling unconfirmed (self-hosted speech-to-speech stacks set their own); unknown.
Static ReadonlySupportsWhether this driver subclass supports dynamic, multi-tool sets projected into the realtime session (e.g. direct action invocation on realtime co-agents).
Static capability descriptor read before a session instance exists.
Default: absent / false. Subclasses that support dynamic toolsets declare public static readonly SupportsDynamicToolSet = true;.
ProtectedapiOnly sub-classes can access the API key
Readonly getter returning the model driver's connection duration in seconds.
Static driver default for a single connection's duration in seconds (for example, 600 on
GeminiRealtime and GeminiEnterpriseRealtime, 3600 on OpenAIRealtime and
OpenAILiveRealtime, 7200 on xAIRealtime). Defaults to DriverFacts.ProviderMaxSessionSeconds.
Read-only accessor for the underlying OpenAI SDK client.
ProtectedProfileThe HuggingFace knobs + GA feature gates the shared protocol implementation runs with.
Buffer in seconds before CallDurationSeconds at which the framework initiates a
proactive reconnect. Defaults to DEFAULT_REALTIME_RECONNECT_BUFFER_SECONDS (10s).
Dynamic runtime override hook on the model driver (returns undefined unless overridden by a subclass).
The compat layer's TTS stage has no OpenAI-style selectable voice catalog, so the dev voice
picker gets none — overriding away the OpenAI voice list this class would otherwise inherit.
(A voice Config value still passes through to audio.output.voice for stacks that honor it.)
HuggingFace supports client-direct via MJAPI's realtime proxy.
Whether this driver's sessions carry a video track in addition to audio.
false in every MJ driver.
Nothing reads it, and no driver overrides it: it is false for Gemini Live, which takes camera video
and renders avatars. What a session does with video is declared on its IRealtimeSession.Capabilities:
RealtimeSessionCapabilities.SupportedInboundTracks for video in and
RealtimeSessionCapabilities.SupportedOutboundTracks for video out, each track with its encoding. Whether a
model renders a live avatar is SupportsAvatarOutput. A driver declares video there, not here.
Whether this driver supports and trusts the provider's native session resumption on the
active endpoint when a handle is present. Subclasses override as needed (e.g., true on
server-side GeminiRealtime, GeminiEnterpriseRealtime, and xAIRealtime; false on
OpenAIRealtime).
ProtectedClampReturns the smallest candidate that is a positive integer; see ClampSessionSecondsValue. The rule "the caller's bag wins only if smaller" lives there, once: pass the provider cap and the bag cap together.
Resolves whether native resumption is trusted for a client-direct session minted by this driver,
given the transport ('direct' vs 'relay'). Subclasses override when client-direct trust
differs from server-bridged trust (e.g., GeminiRealtime is true on server v1beta keys
but false on browser-direct v1alpha ephemeral tokens).
Optional_transport: RealtimeClientTransportMints the client-direct config: a one-time proxy ticket pointing at the internal endpoint, and
the private-pact SessionConfig the same-keyed 'huggingface' client driver consumes:
{ session, sampleRate }, where session is the OpenAI-Realtime session.update payload
(server-authored prompt/tools/voice) and sampleRate drives the client's PCM audio plane.
EphemeralToken is the browser-facing wss://<mjapi-public>/realtime-proxy?ticket=… URL.
Fully overrides the base's SDK client-secret mint — a self-hosted endpoint has no
/realtime/client_secrets API, and the proxy keeps MJAPI the single ingress.
ProtectedcreateCreates the realtime connection for a model. Overridable seam for testing.
Production returns a real OpenAIRealtimeWebSocket. Unit tests override this to return a
fake IOpenAIRealtimeConnection that emits OpenAI-shaped events and captures sends.
The provider realtime model id (e.g. gpt-realtime-2.1).
OptionalresumptionHandle: stringOptional provider session/conversation handle to resume (?conversation_id=).
A connection implementing IOpenAIRealtimeConnection.
ProtectedcreateBuilds the raw-WS connection for the upstream endpoint. Overridable seam for testing — unit tests return an in-memory fake implementing IOpenAIRealtimeConnection, so no network.
NOTE: the platform-global WebSocket cannot send an Authorization header; upstream auth is
enforced by the endpoint itself or injected server-side by the MJAPI realtime proxy
(resolveUpstreamAuthHeader feeds the proxy ticket, not this socket).
The resolved ws(s)://…/v1/realtime endpoint URL.
The connection the shared session runs over.
ProtectedcreateCreates the session wrapper for a freshly-opened connection. Overridable seam so subclasses can return their own session subclass while StartSession stays shared.
The open provider connection.
The session bound to this driver's Profile.
OptionalExchangeOptional WebRTC SDP exchange seam for drivers that support server-brokered WebRTC topologies.
The local SDP offer generated by the browser.
Optional_sessionConfig: Record<string, unknown>The session configuration to initialize the session with.
The answer SDP, provider session ID, and any prebill accounting.
Internal bridge allowing BaseRealtimeSession to invoke the model driver's protected ReconnectSession.
ProtectedmintMints the ephemeral client secret via the provider's Realtime client-secrets API (resolved from the SDK client's base URL, so OpenAI-compatible subclasses target their own endpoint). Overridable seam for testing — unit tests return a fake response so no network call is made.
The client-secret create request (carries the realtime session config).
The client-secret create response (token value + expiry + echoed session).
ProtectedReconnectReconnect hook for server-hosted IRealtimeSession instances created by StartSession.
Called by BaseRealtimeSession's duration timer (CallDurationSeconds - ReconnectBufferSeconds)
or on unexpected transport close when resumable. Concrete provider drivers override this method.
Mints a fresh client credential (ephemeral token or signed URL) for an already-running client-direct session during a mid-call credential refresh or reconnect.
Defaults to delegating to CreateClientSession. Drivers where CreateClientSession
has side effects beyond credential minting (such as ElevenLabsRealtime, which provisions
a managed agent when no agentId is given) override this to mint a credential for the
existing resource in existingSessionConfig without creating duplicate vendor resources.
Optional_existingSessionConfig: JSONObjectResolves the effective single-connection duration in seconds from an open config bag
(connectionMaxSeconds), narrowing this.CallDurationSeconds via ResolveRealtimeTiming.
Optionalconfig: Record<string, unknown>ProtectedresolveResolves the browser-facing proxy ORIGIN as a ws(s)://host[:port] string through the shared
ResolveRealtimeProxyBaseWsUrl (@memberjunction/ai), which the relay drivers use too. Precedence:
params.Config.proxyBaseUrl override → MJAPI_PUBLIC_URL → GRAPHQL_BASE_URL + GRAPHQL_PORT
(the same env vars MJAPI derives its public URL from). Any path on the source URL is dropped — only
the origin is used, and REALTIME_PROXY_PATH is appended by the caller.
Resolves the effective reconnect buffer in seconds from an open config bag
(reconnectBufferSeconds), falling back to this.ReconnectBufferSeconds.
Optionalconfig: Record<string, unknown>Resolves whether native session resumption is trusted for a session from an open config bag
(trustNativeResumption), falling back to defaultTrust (or this.TrustsNativeResumption).
Optionalconfig: Record<string, unknown>OptionaldefaultTrust: booleanProtectedresolveBuilds the upstream Authorization header from the API key, or undefined when the endpoint is
unauthenticated. Because the realtime resolver requires a resolvable API key for a model to be
selectable, a keyless self-hosted deployment sets a sentinel value (none / self-hosted / local
/ n/a) for AI_VENDOR_API_KEY__HuggingFaceRealtime — treated here as "no auth" so no bogus header
is ever sent upstream (consumed by the MJAPI realtime proxy; the raw browser/Node socket cannot
carry headers at all).
ProtectedresolveResolves the internal upstream endpoint: params.Config.endpoint override → HUGGINGFACE_REALTIME_URL
env var → the local default. Self-hosted, so this is deployment config, not a hardcoded provider URL.
Opens a server-bridged session over the raw-WS adapter and — unlike the base — resolves only
after the initial config has been APPLIED (the endpoint's session.created handshake), so the
model is configured before any audio streams. Transport death during startup rejects.
Whether this driver's sessions can render a live avatar for the model, on the endpoint the driver serves.
Defaults to false: a driver whose models render no avatar says nothing more, and may ignore an avatar request. A
driver that renders one for some models (Gemini Enterprise) overrides it. The client-session service reads it to
tell a call or a meeting that asked for an avatar "this voice model can't show an avatar"
(RealtimeAvatarUnavailableReason 'endpoint'), whichever driver serves it.
The provider's API name for the model.
true when sessions on this model can render a live avatar; false otherwise.
StaticBuildBuilds the OpenAI-Realtime session object from the Core params:
instructions: the system prompt, with InitialContext folded in under a "Prior context"
heading (the compat protocol has no guaranteed history-seeding channel).tools: the Core tool set mapped to the OpenAI { type: 'function', … } schema.audio.output.voice: from params.Config.voice, when supplied.audio.input.transcription.model: from params.Config.inputTranscriptionModel, when supplied
(the pipeline's STT stage transcribes the user natively; this only names a specific model if the
compat layer supports selecting one).StaticComposeFolds optional prior context into the system prompt (no guaranteed history channel).
OptionalinitialContext: stringStaticHttpConverts an http(s) URL (or origin) into a ws(s)://host[:port] origin, dropping any path. Delegates to the
shared HttpOriginToWs in @memberjunction/ai.
StaticMapMaps a Core tool definition to an OpenAI-Realtime function tool schema.
Protected StaticreadReads a process env var (indirected so tests can stub it and non-Node runtimes don't throw).
StaticResolveResolves the PCM sample rate: params.Config.sampleRate override, else the HF-native 16 kHz default.
StaticToolCanonical, order-insensitive fingerprint of a tool set (same scheme as the AssemblyAI / Gemini / ElevenLabs realtime drivers) — used to no-op identical live re-registrations.
Real-time, full-duplex driver for a self-hosted HuggingFace speech-to-speech server running in OpenAI-Realtime-compatible mode. Registers as
HuggingFaceRealtimeand is resolved forMJ: AI ModelstypedRealtime(API-key env alias:AI_VENDOR_API_KEY__HuggingFaceRealtime— optional; many self-hosted endpoints are unauthenticated).Topologies:
StartSessionresolves only after the session config has been applied (the endpoint'ssession.created→session.updatehandshake) so the runner never streams audio against an unconfigured model (obligation #7).wss://<mjapi-public>/realtime-proxy?ticket=…URL. The browser opens its socket to MJAPI's proxy, which tunnels transparently to the internal endpoint (injecting any auth server-side). The internal endpoint never reaches the browser and needs no browser-facing ingress.No managed server object and no usage billing: the entire session config (prompt, tools, voice) is supplied per session; there is no token-usage meter, so
OnUsagefires only if the compat endpoint happens to report aresponse.doneusage block.