Static ReadonlyDriverProvider ceiling unconfirmed; unknown until an in-repo source states it.
Static Optional ReadonlySupportsWhether this driver subclass supports dynamic, multi-tool sets projected into the realtime session (e.g. direct action invocation on realtime co-agents).
Static capability descriptor read before a session instance exists.
Default: absent / false. Subclasses that support dynamic toolsets declare public static readonly SupportsDynamicToolSet = true;.
ProtectedapiOnly sub-classes can access the API key
Readonly getter returning the model driver's connection duration in seconds.
Static driver default for a single connection's duration in seconds (for example, 600 on
GeminiRealtime and GeminiEnterpriseRealtime, 3600 on OpenAIRealtime and
OpenAILiveRealtime, 7200 on xAIRealtime). Defaults to DriverFacts.ProviderMaxSessionSeconds.
Buffer in seconds before CallDurationSeconds at which the framework initiates a
proactive reconnect. Defaults to DEFAULT_REALTIME_RECONNECT_BUFFER_SECONDS (10s).
Dynamic runtime override hook on the model driver (returns undefined unless overridden by a subclass).
The voices the AssemblyAI Voice Agent API can speak with — used to populate the voice picker.
AssemblyAI supports the client-direct topology natively: a one-time temp token minted server-side authenticates the browser's own websocket.
Whether this driver's sessions carry a video track in addition to audio.
false in every MJ driver.
Nothing reads it, and no driver overrides it: it is false for Gemini Live, which takes camera video
and renders avatars. What a session does with video is declared on its IRealtimeSession.Capabilities:
RealtimeSessionCapabilities.SupportedInboundTracks for video in and
RealtimeSessionCapabilities.SupportedOutboundTracks for video out, each track with its encoding. Whether a
model renders a live avatar is SupportsAvatarOutput. A driver declares video there, not here.
Whether this driver supports and trusts the provider's native session resumption on the
active endpoint when a handle is present. Subclasses override as needed (e.g., true on
server-side GeminiRealtime, GeminiEnterpriseRealtime, and xAIRealtime; false on
OpenAIRealtime).
ProtectedClampReturns the smallest candidate that is a positive integer; see ClampSessionSecondsValue. The rule "the caller's bag wins only if smaller" lives there, once: pass the provider cap and the bag cap together.
Resolves whether native resumption is trusted for a client-direct session minted by this driver,
given the transport ('direct' vs 'relay'). Subclasses override when client-direct trust
differs from server-bridged trust (e.g., GeminiRealtime is true on server v1beta keys
but false on browser-direct v1alpha ephemeral tokens).
Optional_transport: RealtimeClientTransportProtectedconnectTransport seam for the server-bridged agent websocket. Production speaks the raw Voice
Agent protocol over the platform-global WebSocket (browsers / Node 22+). Resolves
once the socket is OPEN; unit tests override this to return an in-memory fake.
Mints the client-direct config: a ONE-TIME client token (connect window
ASSEMBLYAI_TOKEN_TTL_SECONDSs) plus the private pact SessionConfig the
same-keyed 'assemblyai' client driver consumes: { session, config }, where
session is the wire-shaped session.update payload (server-authored system prompt,
tools, voice, turn detection) and config passes params.Config through opaquely.
OptionalExchangeOptional WebRTC SDP exchange seam for drivers that support server-brokered WebRTC topologies.
The local SDP offer generated by the browser.
Optional_sessionConfig: Record<string, unknown>The session configuration to initialize the session with.
The answer SDP, provider session ID, and any prebill accounting.
Internal bridge allowing BaseRealtimeSession to invoke the model driver's protected ReconnectSession.
ProtectedmintREST seam: mints a ONE-TIME client token for a browser-direct session
(GET /v1/token?expires_in_seconds=… with Authorization: Bearer <api key>).
ProtectedReconnectReconnect hook for server-hosted IRealtimeSession instances created by StartSession.
Called by BaseRealtimeSession's duration timer (CallDurationSeconds - ReconnectBufferSeconds)
or on unexpected transport close when resumable. Concrete provider drivers override this method.
Mints a fresh client credential (ephemeral token or signed URL) for an already-running client-direct session during a mid-call credential refresh or reconnect.
Defaults to delegating to CreateClientSession. Drivers where CreateClientSession
has side effects beyond credential minting (such as ElevenLabsRealtime, which provisions
a managed agent when no agentId is given) override this to mint a credential for the
existing resource in existingSessionConfig without creating duplicate vendor resources.
Optional_existingSessionConfig: JSONObjectResolves the effective single-connection duration in seconds from an open config bag
(connectionMaxSeconds), narrowing this.CallDurationSeconds via ResolveRealtimeTiming.
Optionalconfig: Record<string, unknown>Resolves the effective reconnect buffer in seconds from an open config bag
(reconnectBufferSeconds), falling back to this.ReconnectBufferSeconds.
Optionalconfig: Record<string, unknown>Resolves whether native session resumption is trusted for a session from an open config bag
(trustNativeResumption), falling back to defaultTrust (or this.TrustsNativeResumption).
Optionalconfig: Record<string, unknown>OptionaldefaultTrust: booleanOpens a server-bridged session: connects the agent websocket authenticated with the
API key, sends the full session config as the FIRST frame (session.update — prompt,
tools, voice, turn detection), and resolves only once the server's session.ready
confirms the config is applied (driver obligation #7 — "ready only after the config is
applied"). Mic frames sent before that would be dropped by the provider, and
StartSession not resolving until ready makes that unrepresentable for consumers.
Whether this driver's sessions can render a live avatar for the model, on the endpoint the driver serves.
Defaults to false: a driver whose models render no avatar says nothing more, and may ignore an avatar request. A
driver that renders one for some models (Gemini Enterprise) overrides it. The client-session service reads it to
tell a call or a meeting that asked for an avatar "this voice model can't show an avatar"
(RealtimeAvatarUnavailableReason 'endpoint'), whichever driver serves it.
The provider's API name for the model.
true when sessions on this model can render a live avatar; false otherwise.
StaticBuildBuilds the session object for the session.update frame from the Core params:
system_prompt: the session system prompt, with InitialContext folded in under a
"Prior context" heading — the protocol has no history-seeding channel, so prior
conversation context rides the prompt.tools: the Core tool set mapped to the provider's { type: 'function', … } schema.params.Config keys pass through to their wire slots: voice →
output.voice, firstMessage (or the legacy greeting) → greeting (spoken on connect;
omitted = no auto-greeting), turn_detection and the normalized turnDetection
(BuildAssemblyAITurnDetection) → input.turn_detection, keyterms → input.keyterms.StaticComposeFolds optional prior context into the system prompt (no history channel exists).
OptionalinitialContext: stringStaticMapMaps a Core tool definition up to a Voice Agent function tool schema.
StaticResolveReads the session's opening utterance out of the config bag — this driver's greeting wire
slot, fed from the driver-NEUTRAL firstMessage key (issue #3557) that a persona authors
without knowing which vendor will run. greeting predates the neutral name and is still
accepted so existing configs keep working.
Both keys go through the SAME trim-and-drop-blank rule the ElevenLabs driver applies, so one
authored value means the same thing on either provider. That rule is also why the neutral key
only wins when it carries something: a blank firstMessage must not suppress a valid legacy
greeting — it is an absent opening utterance, not an empty one.
The session's open config bag.
The trimmed opening utterance, or undefined when neither key carries one.
StaticToolCanonical, order-insensitive fingerprint of a tool set (same scheme as the Gemini and ElevenLabs realtime drivers) — used by AssemblyAIRealtimeSession.RegisterTools to no-op identical re-registrations.
Real-time, full-duplex driver for the AssemblyAI Voice Agent API (launched April 2026), implementing the Core BaseRealtimeModel primitive. Registers as
AssemblyAIRealtimeand is resolved forMJ: AI ModelstypedRealtime(API-key env alias:AI_VENDOR_API_KEY__AssemblyAIRealtime).What the provider is: a single-websocket speech-to-speech stack — Universal-3 Pro streaming ASR, server-side turn detection / barge-in, LLM reasoning, JSON-Schema tool calling, and conversational TTS — billed flat per session-hour. Unlike ElevenLabs there is no server-side agent object to manage: the ENTIRE session config (prompt, tools, voice, turn detection) is supplied per-session via the first
session.updateframe, so MJ's prompt/tool authority is native here, with no managed-agent ensure flow.Model resolution: the Voice Agent API exposes ONE endpoint and no model selection;
params.Modelis carried through to ClientRealtimeSessionConfig.Model for bookkeeping but plays no wire role.Topologies:
?token=<api key>), sends the session config, and resolves only aftersession.ready.GET /v1/token) — the browser opens its own socket with it (no API key ever leaves the server) and applies the server-built session object from theSessionConfigpact.No usage events: billing is a flat hourly session rate; the websocket reports no token usage, so sessions never emit IRealtimeSession.OnUsage. (The terminal
session.endedframe carries duration seconds, but by then the consumer has typically closed — duration accounting lives in the provider dashboard.)