Member Junction
    Preparing search index...

    Eleven Realtime implementation of the BaseRealtimeModel primitive — eleven_realtime_v1_mini and eleven_realtime_v1_max.

    Eleven Realtime is a cascaded stack (ASR → LLM → TTS with ElevenLabs' turn-taking) sold as a single speech-to-speech model, and its API follows the OpenAI Realtime spec over a WebSocket. So, like xAI's Grok Voice, this driver subclasses OpenAIRealtime and inherits the whole protocol implementation — session config, tool loop, barge-in, error classification, client-direct minting — with ELEVEN_REALTIME_PROFILE gating the features.

    It is a different product from ElevenLabsRealtime, which drives the ElevenLabs Agents Platform's own protocol against a server-side agent. Here there is no agent to provision: the session carries the prompt and tools, exactly as it does for OpenAI.

    Two things differ from OpenAI:

    1. Authentication. ElevenLabs takes its API key as xi-api-key, never as a bearer token, and documents the realtime socket as authenticated by an ephemeral client secret passed where an OpenAI client would put its API key. So mintClientSecret sends xi-api-key, and the server-bridged StartSession mints a secret first and opens the socket with it — the same documented path the browser uses, rather than guessing whether the socket also accepts a raw key.
    2. Turn eagerness uses medium where MJ says auto (see MapElevenRealtimeEagerness).

    max_duration_seconds (an ElevenLabs extension: the server closes the session with code 1000 once it is exceeded) is min(provider cap, MaxSessionSeconds, Config-bag value) through the one clamp in BaseRealtimeModel. Only a positive-integer number in the bag counts; any other bag value (e.g. the string "600") is ignored AND never sent as-is. With no provider cap known, no host MaxSessionSeconds and no usable bag value the field is removed, so ElevenLabs applies its own ceiling. The bag is adjusted in ElevenRealtime.withClampedMaxDuration before the shared OpenAI payload builder runs, on both the mint (ElevenRealtime.CreateClientSession) and the server-bridged (ElevenRealtime.StartSession) paths, so no payload code is duplicated.

    Registered as ElevenRealtime; the matching MJ: AI Model Vendors rows name it as their DriverClass.

    Hierarchy (View Summary)

    Index

    Constructors

    Properties

    DriverFacts: Readonly<RealtimeDriverFacts> = ...

    The provider's session ceiling is not documented in-repo (max_duration_seconds is a per-session setting, not a ceiling), so it stays unknown.

    SupportsDynamicToolSet: boolean = ELEVEN_REALTIME_PROFILE.supportsDynamicToolSet

    Whether this driver subclass supports dynamic, multi-tool sets projected into the realtime session (e.g. direct action invocation on realtime co-agents).

    Static capability descriptor read before a session instance exists. Default: absent / false. Subclasses that support dynamic toolsets declare public static readonly SupportsDynamicToolSet = true;.

    Accessors

    • get apiKey(): string

      Only sub-classes can access the API key

      Returns string

    • get CallDurationSeconds(): number

      Readonly getter returning the model driver's connection duration in seconds.

      Returns number

    • get DefaultCallDurationSeconds(): number

      Static driver default for a single connection's duration in seconds (for example, 600 on GeminiRealtime and GeminiEnterpriseRealtime, 3600 on OpenAIRealtime and OpenAILiveRealtime, 7200 on xAIRealtime). Defaults to DriverFacts.ProviderMaxSessionSeconds.

      Returns number

    • get OpenAI(): OpenAI

      Read-only accessor for the underlying OpenAI SDK client.

      Returns OpenAI

    • get ReconnectBufferSeconds(): number

      Buffer in seconds before CallDurationSeconds at which the framework initiates a proactive reconnect. Defaults to DEFAULT_REALTIME_RECONNECT_BUFFER_SECONDS (10s).

      Returns number

    • get RuntimeCallDurationSeconds(): number

      Dynamic runtime override hook on the model driver (returns undefined unless overridden by a subclass).

      Returns number

    • get SupportsClientDirect(): boolean

      OpenAI supports the client-direct topology: the server mints a short-lived ephemeral client secret that the browser uses to open its OWN connection to OpenAI's Realtime API, while the server still controls the system prompt + tools via the returned SessionConfig.

      Returns boolean

    • get TrustsNativeResumption(): boolean

      Whether this driver supports and trusts the provider's native session resumption on the active endpoint when a handle is present. Subclasses override as needed (e.g., true on server-side GeminiRealtime, GeminiEnterpriseRealtime, and xAIRealtime; false on OpenAIRealtime).

      Returns boolean

    Methods

    • Returns the smallest candidate that is a positive integer; see ClampSessionSecondsValue. The rule "the caller's bag wins only if smaller" lives there, once: pass the provider cap and the bag cap together.

      Parameters

      • ...candidates: readonly number[]

      Returns number

    • Resolves whether native resumption is trusted for a client-direct session minted by this driver, given the transport ('direct' vs 'relay'). Subclasses override when client-direct trust differs from server-bridged trust (e.g., GeminiRealtime is true on server v1beta keys but false on browser-direct v1alpha ephemeral tokens).

      Parameters

      Returns boolean

    • Opens the realtime socket authenticated by a minted client secret. The SDK puts the secret in the openai-insecure-api-key. subprotocol — the "use the OpenAI SDK as is" path ElevenLabs documents. Overridable seam: unit tests return a fake connection.

      Parameters

      • model: string
      • clientSecret: string

      Returns IOpenAIRealtimeConnection

    • Creates the realtime connection for a model. Overridable seam for testing.

      Production returns a real OpenAIRealtimeWebSocket. Unit tests override this to return a fake IOpenAIRealtimeConnection that emits OpenAI-shaped events and captures sends.

      Parameters

      • model: string

        The provider realtime model id (e.g. gpt-realtime-2.1).

      • OptionalresumptionHandle: string

        Optional provider session/conversation handle to resume (?conversation_id=).

      Returns IOpenAIRealtimeConnection

      A connection implementing IOpenAIRealtimeConnection.

    • Optional WebRTC SDP exchange seam for drivers that support server-brokered WebRTC topologies.

      Parameters

      • _offerSdp: string

        The local SDP offer generated by the browser.

      • Optional_sessionConfig: Record<string, unknown>

        The session configuration to initialize the session with.

      Returns Promise<{ answerSdp: string; prebillSeconds: number; sessionId: string }>

      The answer SDP, provider session ID, and any prebill accounting.

    • Mints an Eleven Realtime client secret. The openai client already resolves the right URL and types the request and response; this only swaps the bearer header for xi-api-key, and reads expires_at in seconds whichever unit ElevenLabs returns it in.

      Parameters

      • body: ClientSecretCreateParams

      Returns Promise<ClientSecretCreateResponse>

    • Reconnect hook for server-hosted IRealtimeSession instances created by StartSession. Called by BaseRealtimeSession's duration timer (CallDurationSeconds - ReconnectBufferSeconds) or on unexpected transport close when resumable. Concrete provider drivers override this method.

      Parameters

      Returns Promise<void>

    • Resolves the effective single-connection duration in seconds from an open config bag (connectionMaxSeconds), narrowing this.CallDurationSeconds via ResolveRealtimeTiming.

      Parameters

      • Optionalconfig: Record<string, unknown>

      Returns number

    • Resolves the effective reconnect buffer in seconds from an open config bag (reconnectBufferSeconds), falling back to this.ReconnectBufferSeconds.

      Parameters

      • Optionalconfig: Record<string, unknown>

      Returns number

    • Resolves whether native session resumption is trusted for a session from an open config bag (trustNativeResumption), falling back to defaultTrust (or this.TrustsNativeResumption).

      Parameters

      • Optionalconfig: Record<string, unknown>
      • OptionaldefaultTrust: boolean

      Returns boolean

    • Whether this driver's sessions can render a live avatar for the model, on the endpoint the driver serves.

      Defaults to false: a driver whose models render no avatar says nothing more, and may ignore an avatar request. A driver that renders one for some models (Gemini Enterprise) overrides it. The client-session service reads it to tell a call or a meeting that asked for an avatar "this voice model can't show an avatar" (RealtimeAvatarUnavailableReason 'endpoint'), whichever driver serves it.

      Parameters

      • _model: string

        The provider's API name for the model.

      Returns boolean

      true when sessions on this model can render a live avatar; false otherwise.