ProtectedapiOnly sub-classes can access the API key
The provider-native voice ids this model can speak with (e.g. OpenAI alloy/echo/shimmer). The
model/driver is the authoritative owner of "what voices do I support", so each driver declares its
own — used to populate the dev voice picker. Default empty (a driver that hasn't declared voices
yields no picker options, falling back to the configured/default voice).
NOTE: this is the near-term, driver-owned source of truth. Long term this should move to metadata so providers that let users add their OWN voices (e.g. ElevenLabs) can be enumerated dynamically.
The supported voice ids (id + human label), or [] when none are declared.
AssemblyAI supports the client-direct topology natively: a one-time temp token minted server-side authenticates the browser's own websocket.
Whether this driver's sessions carry a video track in addition to audio — i.e. the model accepts video input (it can "see" the user's camera) and/or emits video output (a talking-head avatar / generated video), in sync with audio.
Defaults to false (audio-only — today's realtime models). Video-capable drivers (a native
multimodal realtime model, or an avatar provider) override this to true. The session's media
plane is media-tagged (IRealtimeSession.SendInput takes a RealtimeMediaKind;
IRealtimeSession.OnVideoOutput delivers video-out), so a video session reuses the entire
realtime contract — only the media frames gain a video kind. Resolution prefers a video-capable
model when an agent requests video, and degrades to audio-only otherwise.
true if sessions can carry video; false (audio-only) otherwise.
ProtectedconnectTransport seam for the server-bridged agent websocket. Production speaks the raw Voice
Agent protocol over the platform-global WebSocket (browsers / Node 22+). Resolves
once the socket is OPEN; unit tests override this to return an in-memory fake.
Mints the client-direct config: a ONE-TIME client token (connect window
ASSEMBLYAI_TOKEN_TTL_SECONDSs) plus the private pact SessionConfig the
same-keyed 'assemblyai' client driver consumes: { session, config }, where
session is the wire-shaped session.update payload (server-authored system prompt,
tools, voice, turn detection) and config passes params.Config through opaquely.
ProtectedmintREST seam: mints a ONE-TIME client token for a browser-direct session
(GET /v1/token?expires_in_seconds=… with Authorization: Bearer <api key>).
Opens a server-bridged session: connects the agent websocket authenticated with the
API key, sends the full session config as the FIRST frame (session.update — prompt,
tools, voice, turn detection), and resolves only once the server's session.ready
confirms the config is applied (driver obligation #7 — "ready only after the config is
applied"). Mic frames sent before that would be dropped by the provider, and
StartSession not resolving until ready makes that unrepresentable for consumers.
StaticBuildBuilds the session object for the session.update frame from the Core params:
system_prompt: the session system prompt, with InitialContext folded in under a
"Prior context" heading — the protocol has no history-seeding channel, so prior
conversation context rides the prompt.tools: the Core tool set mapped to the provider's { type: 'function', … } schema.params.Config keys pass through to their wire slots: voice →
output.voice, greeting (spoken on connect; omitted = no auto-greeting),
turn_detection → input.turn_detection, keyterms → input.keyterms.StaticComposeFolds optional prior context into the system prompt (no history channel exists).
OptionalinitialContext: stringStaticMapMaps a Core tool definition up to a Voice Agent function tool schema.
StaticToolCanonical, order-insensitive fingerprint of a tool set (same scheme as the Gemini and ElevenLabs realtime drivers) — used by AssemblyAIRealtimeSession.RegisterTools to no-op identical re-registrations.
Real-time, full-duplex driver for the AssemblyAI Voice Agent API (launched April 2026), implementing the Core BaseRealtimeModel primitive. Registers as
AssemblyAIRealtimeand is resolved forMJ: AI ModelstypedRealtime(API-key env alias:AI_VENDOR_API_KEY__AssemblyAIRealtime).What the provider is: a single-websocket speech-to-speech stack — Universal-3 Pro streaming ASR, server-side turn detection / barge-in, LLM reasoning, JSON-Schema tool calling, and conversational TTS — billed flat per session-hour. Unlike ElevenLabs there is no server-side agent object to manage: the ENTIRE session config (prompt, tools, voice, turn detection) is supplied per-session via the first
session.updateframe, so MJ's prompt/tool authority is native here, with no managed-agent ensure flow.Model resolution: the Voice Agent API exposes ONE endpoint and no model selection;
params.Modelis carried through to ClientRealtimeSessionConfig.Model for bookkeeping but plays no wire role.Topologies:
?token=<api key>), sends the session config, and resolves only aftersession.ready.GET /v1/token) — the browser opens its own socket with it (no API key ever leaves the server) and applies the server-built session object from theSessionConfigpact.No usage events: billing is a flat hourly session rate; the websocket reports no token usage, so sessions never emit IRealtimeSession.OnUsage. (The terminal
session.endedframe carries duration seconds, but by then the consumer has typically closed — duration accounting lives in the provider dashboard.)