Member Junction
    Preparing search index...

    Real-time, full-duplex driver for the AssemblyAI Voice Agent API (launched April 2026), implementing the Core BaseRealtimeModel primitive. Registers as AssemblyAIRealtime and is resolved for MJ: AI Models typed Realtime (API-key env alias: AI_VENDOR_API_KEY__AssemblyAIRealtime).

    What the provider is: a single-websocket speech-to-speech stack — Universal-3 Pro streaming ASR, server-side turn detection / barge-in, LLM reasoning, JSON-Schema tool calling, and conversational TTS — billed flat per session-hour. Unlike ElevenLabs there is no server-side agent object to manage: the ENTIRE session config (prompt, tools, voice, turn detection) is supplied per-session via the first session.update frame, so MJ's prompt/tool authority is native here, with no managed-agent ensure flow.

    Model resolution: the Voice Agent API exposes ONE endpoint and no model selection; params.Model is carried through to ClientRealtimeSessionConfig.Model for bookkeeping but plays no wire role.

    Topologies:

    • Server-bridged (StartSession): the driver opens the websocket itself (?token=<api key>), sends the session config, and resolves only after session.ready.
    • Client-direct (CreateClientSession): the driver mints a ONE-TIME, short-lived client token (GET /v1/token) — the browser opens its own socket with it (no API key ever leaves the server) and applies the server-built session object from the SessionConfig pact.

    No usage events: billing is a flat hourly session rate; the websocket reports no token usage, so sessions never emit IRealtimeSession.OnUsage. (The terminal session.ended frame carries duration seconds, but by then the consumer has typically closed — duration accounting lives in the provider dashboard.)

    Hierarchy (View Summary)

    Index

    Constructors

    Properties

    DriverFacts: Readonly<RealtimeDriverFacts> = {}

    Provider ceiling unconfirmed; unknown until an in-repo source states it.

    SupportsDynamicToolSet?: boolean

    Whether this driver subclass supports dynamic, multi-tool sets projected into the realtime session (e.g. direct action invocation on realtime co-agents).

    Static capability descriptor read before a session instance exists. Default: absent / false. Subclasses that support dynamic toolsets declare public static readonly SupportsDynamicToolSet = true;.

    Accessors

    • get apiKey(): string

      Only sub-classes can access the API key

      Returns string

    • get CallDurationSeconds(): number

      Readonly getter returning the model driver's connection duration in seconds.

      Returns number

    • get DefaultCallDurationSeconds(): number

      Static driver default for a single connection's duration in seconds (for example, 600 on GeminiRealtime and GeminiEnterpriseRealtime, 3600 on OpenAIRealtime and OpenAILiveRealtime, 7200 on xAIRealtime). Defaults to DriverFacts.ProviderMaxSessionSeconds.

      Returns number

    • get ReconnectBufferSeconds(): number

      Buffer in seconds before CallDurationSeconds at which the framework initiates a proactive reconnect. Defaults to DEFAULT_REALTIME_RECONNECT_BUFFER_SECONDS (10s).

      Returns number

    • get RuntimeCallDurationSeconds(): number

      Dynamic runtime override hook on the model driver (returns undefined unless overridden by a subclass).

      Returns number

    • get SupportsClientDirect(): boolean

      AssemblyAI supports the client-direct topology natively: a one-time temp token minted server-side authenticates the browser's own websocket.

      Returns boolean

    • get TrustsNativeResumption(): boolean

      Whether this driver supports and trusts the provider's native session resumption on the active endpoint when a handle is present. Subclasses override as needed (e.g., true on server-side GeminiRealtime, GeminiEnterpriseRealtime, and xAIRealtime; false on OpenAIRealtime).

      Returns boolean

    Methods

    • Returns the smallest candidate that is a positive integer; see ClampSessionSecondsValue. The rule "the caller's bag wins only if smaller" lives there, once: pass the provider cap and the bag cap together.

      Parameters

      • ...candidates: readonly number[]

      Returns number

    • Resolves whether native resumption is trusted for a client-direct session minted by this driver, given the transport ('direct' vs 'relay'). Subclasses override when client-direct trust differs from server-bridged trust (e.g., GeminiRealtime is true on server v1beta keys but false on browser-direct v1alpha ephemeral tokens).

      Parameters

      Returns boolean

    • Optional WebRTC SDP exchange seam for drivers that support server-brokered WebRTC topologies.

      Parameters

      • _offerSdp: string

        The local SDP offer generated by the browser.

      • Optional_sessionConfig: Record<string, unknown>

        The session configuration to initialize the session with.

      Returns Promise<{ answerSdp: string; prebillSeconds: number; sessionId: string }>

      The answer SDP, provider session ID, and any prebill accounting.

    • REST seam: mints a ONE-TIME client token for a browser-direct session (GET /v1/token?expires_in_seconds=… with Authorization: Bearer <api key>).

      Parameters

      • expiresInSeconds: number

      Returns Promise<string>

    • Reconnect hook for server-hosted IRealtimeSession instances created by StartSession. Called by BaseRealtimeSession's duration timer (CallDurationSeconds - ReconnectBufferSeconds) or on unexpected transport close when resumable. Concrete provider drivers override this method.

      Parameters

      Returns Promise<void>

    • Resolves the effective single-connection duration in seconds from an open config bag (connectionMaxSeconds), narrowing this.CallDurationSeconds via ResolveRealtimeTiming.

      Parameters

      • Optionalconfig: Record<string, unknown>

      Returns number

    • Resolves the effective reconnect buffer in seconds from an open config bag (reconnectBufferSeconds), falling back to this.ReconnectBufferSeconds.

      Parameters

      • Optionalconfig: Record<string, unknown>

      Returns number

    • Resolves whether native session resumption is trusted for a session from an open config bag (trustNativeResumption), falling back to defaultTrust (or this.TrustsNativeResumption).

      Parameters

      • Optionalconfig: Record<string, unknown>
      • OptionaldefaultTrust: boolean

      Returns boolean

    • Whether this driver's sessions can render a live avatar for the model, on the endpoint the driver serves.

      Defaults to false: a driver whose models render no avatar says nothing more, and may ignore an avatar request. A driver that renders one for some models (Gemini Enterprise) overrides it. The client-session service reads it to tell a call or a meeting that asked for an avatar "this voice model can't show an avatar" (RealtimeAvatarUnavailableReason 'endpoint'), whichever driver serves it.

      Parameters

      • _model: string

        The provider's API name for the model.

      Returns boolean

      true when sessions on this model can render a live avatar; false otherwise.

    • Builds the session object for the session.update frame from the Core params:

      • system_prompt: the session system prompt, with InitialContext folded in under a "Prior context" heading — the protocol has no history-seeding channel, so prior conversation context rides the prompt.
      • tools: the Core tool set mapped to the provider's { type: 'function', … } schema.
      • Recognized params.Config keys pass through to their wire slots: voice → output.voice, firstMessage (or the legacy greeting) → greeting (spoken on connect; omitted = no auto-greeting), turn_detection and the normalized turnDetection (BuildAssemblyAITurnDetection) → input.turn_detection, keyterms → input.keyterms.

      Parameters

      Returns JSONObject

    • Folds optional prior context into the system prompt (no history channel exists).

      Parameters

      • systemPrompt: string
      • OptionalinitialContext: string

      Returns string

    • Reads the session's opening utterance out of the config bag — this driver's greeting wire slot, fed from the driver-NEUTRAL firstMessage key (issue #3557) that a persona authors without knowing which vendor will run. greeting predates the neutral name and is still accepted so existing configs keep working.

      Both keys go through the SAME trim-and-drop-blank rule the ElevenLabs driver applies, so one authored value means the same thing on either provider. That rule is also why the neutral key only wins when it carries something: a blank firstMessage must not suppress a valid legacy greeting — it is an absent opening utterance, not an empty one.

      Parameters

      Returns string

      The trimmed opening utterance, or undefined when neither key carries one.