Self-hosted HuggingFace speech-to-speech realtime (voice) driver for MemberJunction — a BaseRealtimeModel implementation that lets an on-prem, OpenAI-Realtime-compatible voice pipeline sit side-by-side with the cloud realtime providers (OpenAI, Gemini, ElevenLabs, AssemblyAI) with no host changes.
HuggingFace's speech-to-speech stack is an open-source, cascaded VAD → STT → LLM → TTS pipeline (Silero VAD, Parakeet/Whisper ASR, any OpenAI-compatible or local LLM, Qwen3-TTS/Kokoro/ChatTTS) that can expose an OpenAI-Realtime-compatible /v1/realtime websocket. This package treats that endpoint as a MemberJunction Realtime model.
Why it's a differentiated 5th provider: it is private-by-design (audio never leaves infrastructure you own), cost-free (no per-minute cloud billing), and component-swappable (choose your own ASR/LLM/TTS). Ideal for regulated / air-gapped deployments.
Because the endpoint is self-hosted, "client-direct" would otherwise require exposing the internal box to browsers. Instead, this driver uses MJAPI's realtime proxy:
CreateClientSession mints a one-time proxy ticket (in the shared RealtimeProxyRegistry in @memberjunction/ai) pointing at the internal endpoint, and returns EphemeralToken = wss://<mjapi-public>/realtime-proxy?ticket=….HuggingFaceRealtimeClient, @memberjunction/ai-realtime-client, provider key 'huggingface') connects to that URL.RealtimeProxyServer consumes the ticket, opens the authenticated upstream leg (injecting any auth server-side), and pumps frames transparently.The internal endpoint + auth never reach the browser, and the box needs no browser-facing ingress — MJAPI stays the single ingress point. The overhead is one internal relay hop (negligible when co-located).
StartSession (server-bridged topology) connects directly to the endpoint, no proxy.
All deployment config — no mj.config.cjs changes required:
| Setting | Source | Default |
|---|---|---|
| Upstream endpoint | params.Config.endpoint → HUGGINGFACE_REALTIME_URL env |
ws://localhost:8000/v1/realtime |
| Upstream auth | AI_VENDOR_API_KEY__HuggingFaceRealtime env |
none (sentinels none/self-hosted/local/n/a ⇒ unauthenticated) |
| Browser proxy origin | params.Config.proxyBaseUrl → MJAPI_PUBLIC_URL → GRAPHQL_BASE_URL+GRAPHQL_PORT |
http://localhost:4000 |
| PCM sample rate | params.Config.sampleRate |
16000 (HuggingFace's cascade is natively 16 kHz — capture + playout both use this) |
The realtime resolver requires a resolvable API key for a model to be selectable, so a keyless endpoint should set
AI_VENDOR_API_KEY__HuggingFaceRealtime=none.
Provision the LLM backend with ≥ 32k context. MJ's realtime co-agent system prompt (identity framing + tool schemas + injected memory + conversation history) is large and burns context quickly, so the HF pipeline's LLM stage should run with a context window of at least ~32k tokens — a smaller window starves mid-conversation (the model loses the prompt/tools partway through). Confirmed during live bring-up. This is a provisioning requirement on your HF deployment, not an MJ setting.
Seeded as the HuggingFace Speech-to-Speech model (MJ: AI Models, type Realtime) under the Hugging Face vendor, DriverClass: HuggingFaceRealtime. Its PowerRank is intentionally low so it is opt-in, never the default realtime provider.
SendContextNote / RequestSpokenUpdate: native (system-role item / response.create).OnUsage: best-effort — fires only if the compat endpoint reports a response.done usage block (self-hosted has no billing meter).invoke-target-agent delegation works — provided the HF pipeline runs a tool-capable LLM backend. The in-process --llm_backend mlx-lm (and transformers) path does not emit function calls, so a co-agent on it can voice but never delegate. For delegation, run the pipeline with --llm_backend chat-completions (or responses-api) pointed at a tool-capable server (vLLM / llama.cpp) or the HF router. MJ's realtime wire is backend-agnostic — the tool call surfaces as response.function_call_arguments.done regardless. Validated live end-to-end (voice → tool call → tool result → voiced answer) against speech-to-speech v0.2.10 with a llama.cpp + Qwen2.5 backend.session.update against the GA OpenAI-Realtime schema, which requires the session object to be discriminated by type: 'realtime'. The driver stamps this automatically; without it the endpoint rejects the update and the prompt + tools are silently dropped.See the Real-Time Co-Agents Guide for the full realtime architecture.
HuggingFaceRealtime subclasses the shared OpenAIRealtime implementation (@memberjunction/ai-openai) — the self-hosted /v1/realtime endpoint speaks OpenAI's literal GA frames, so the driver is HUGGINGFACE_REALTIME_PROFILE plus the reusable RawRealtimeWebSocketConnection adapter (raw WS → the shared connection seam). HF-specific behavior preserved by the profile/subclass: ready-after-config StartSession (awaits WaitForConfigApplied, 15s readiness deadline), InitialContext folded into the prompt, beta event-name aliases, order-insensitive tool-set fingerprinting, native STT (no default transcription block), NO live turn-mode reconfigure (CanReconfigureTurnMode: false), and the MJAPI proxy-ticket client-direct topology.
The MJAPI realtime proxy supports an optional Origin allowlist — set MJ_REALTIME_PROXY_ALLOWED_ORIGINS (comma-separated origins) to reject foreign browser origins with 403 before the single-use ticket is consumed. The tunnel enforces a 15s upstream-open deadline and caps the pre-open frame buffer (512 frames, drop-oldest). NOTE: the in-memory ticket registry requires sticky routing (mint and upgrade on the same MJAPI instance) in multi-instance deployments, and upgrade requests carry no authenticated principal, so tickets — single-use, short-TTL — remain the primary guard.