Member Junction
    Preparing search index...

    ElevenLabs Agents implementation of BaseRealtimeClient: a browser-direct conversation websocket authenticated with the server-minted SIGNED URL (the EphemeralToken IS the wss://…&token=… URL — no API key ever reaches the browser).

    Registered with the ClassFactory under the key 'elevenlabs' — the Provider string the server's ElevenLabsRealtime driver stamps on its ClientRealtimeSessionConfig.

    Owns ALL ElevenLabs wire concerns (the behavioral twin of the Gemini client driver — the audio plane is client-owned over a websocket, no WebRTC):

    • Connect handshake: open the signed-URL socket → send conversation_initiation_client_data carrying the server-authored prompt override (from the SessionConfig pact) → wait for conversation_initiation_metadata → negotiate the PCM rates from user_input_audio_format / agent_output_audio_format → build the audio plane → 'listening'. The state is gated on the metadata (driver obligation #7): only then has the platform confirmed the session — including the per-session system prompt — is applied.
    • Audio in: mic PCM16 at the negotiated input rate via the shared createPcmMicCapture worklet pipeline, streamed as base64 user_audio_chunk frames (note: a bare-key frame, not a typed one — that is the wire contract).
    • Audio out: audio events decoded into the shared RealtimePcmPlayback at the negotiated output rate; IsAudioPlaying comes from its playout clock.
    • Transcripts are FINAL-only: the Agents socket emits whole-utterance user_transcript / agent_response events (no interim deltas), so every transcript this driver emits has IsFinal: true. agent_response_correction (post-barge-in truncation) re-finalizes the assistant turn with the text that was ACTUALLY spoken — hosts persisting transcripts should treat a final assistant transcript arriving right after an interruption as the authoritative replacement of the previous one.
    • Busy mapping: IsBusy is set on the first audio / agent_response of a turn (and eagerly when this client triggers a response) and cleared on agent_response_complete, on interruption, and on a client_tool_call frame (the agent has yielded the floor pending the result — deadlock guard, obligation #2).
    • Narration: ElevenLabs has no per-response-instructions channel, so RequestSpokenUpdate is EMULATED Gemini-style — the instructions ride as a user_message and the response kind is stamped 'narration' AT SEND TIME (sends are the turn triggers; there is no response.created-style confirmation frame to stamp on), then reset on the response boundary. Fidelity caveat: the instruction enters the conversation as a user turn, so the agent may occasionally reference it; hosts should phrase narration instructions accordingly.
    • Cancel: the protocol has no server-side cancel frame. CancelActiveResponse flushes the locally-owned playout queue (speech stops immediately) and marks the response inactive; residual server-side generation for a cancelled turn is simply not played. Spoken barge-in is handled by the platform's own VAD (which emits interruption), and a typed SendText takes the floor server-side as a fresh user turn.
    • No usage telemetry: the Conversational AI socket exposes no token-usage events, so this driver NEVER emits OnUsage (registering a handler is safe; it just never fires — usage accounting for ElevenLabs sessions happens platform-side).

    Hierarchy (View Summary)

    Index

    Constructors

    Properties

    Accessors

    Methods

    Constructors

    Properties

    callsFromBeforeReseed: Set<string> = ...

    Tool call IDs that were in flight when a non-native seed reconnect occurred.

    configuredCallDurationSeconds: number

    Duration of a single provider connection in seconds from ClientRealtimeSessionConfig.ConnectionMaxSeconds.

    configuredReconnectBufferSeconds: number = DEFAULT_REALTIME_RECONNECT_BUFFER_SECONDS

    Buffer in seconds before CallDurationSeconds expiry at which Reconnect is triggered.

    configuredSessionMaxSeconds: number

    Credential/session token lifetime in seconds from ClientRealtimeSessionConfig.SessionMaxSeconds (or MaxSessionSeconds).

    configuredTrustNativeResumption: boolean

    Optional metadata override for whether native resumption is trusted on this endpoint.

    conversationMemory: RealtimeConversationMemory = ...

    Rolling conversation memory used to re-seed provider context on reconnect.

    inputAudioMeter: IRealtimeAudioMeter = null

    Meter over the USER's microphone, when the driver attached one.

    outputAudioMeter: IRealtimeAudioMeter = null

    Meter over the AGENT's audio output, when the driver attached one.

    reconnectController: RealtimeReconnectController = ...

    Unified I/O-free reconnect controller shared with server-side BaseRealtimeSession.

    tokenRefreshCallback: RealtimeClientTokenRefreshCallback = null

    Optional callback installed by RealtimeSessionRuntime to refresh the provider token mid-call.

    tracks: RealtimeTrack[] = []

    Tracks requested, established, or denied on this session.

    Accessors

    • get CallDurationSeconds(): number

      Effective connection ceiling in seconds. Metadata and runtime caps narrow the driver's provider ceiling (ClampSessionSecondsValue(DefaultCallDurationSeconds, RuntimeCallDurationSeconds, configuredCallDurationSeconds)), never widen it.

      Returns number

    • get InboundVideoRate(): number

      The frame rate (frames per second) the live inbound video track was negotiated at — the model's ceiling or the session's request, whichever is lower — or undefined when no inbound video track is live or none declared a rate. Anything pacing a video feed reads THIS, never a constant of its own.

      Returns number

    • get IsAudioPlaying(): boolean

      Returns boolean

      directly from the playout engine's playhead clock — this client OWNS the output buffer, so "audibly playing" is precisely "scheduled audio extends beyond the audio context's current time".

    • get IsReconnectCycleActive(): boolean

      True from the start of a reconnect cycle until it succeeds or fails. Unlike IsReconnecting, it stays true while a planned move waits between attempts with the call live again on its old connection. A driver that would otherwise end the call on a loss of that connection checks this and hands the loss to the running cycle (TriggerReconnect('connection-lost')) instead.

      Returns boolean

    • get MaxInboundVideoStreams(): number

      How many concurrent inbound video streams the model accepts: the ceiling the driver declared when it negotiated tracks, else 1 when an inbound video track is live and 0 when it is not.

      A source arbiter reads this to decide whether several live video sources must be narrowed to one (with a note to the model on each switch) or passed through untouched.

      Returns number

    • get RequiresPostTurnResumptionHandle(): boolean

      Whether this driver's provider emits a fresh resumable handle shortly after each turnComplete (Gemini / Vertex AI Live), so planned moves wait for that post-turn handle.

      Returns boolean

    • get RuntimeCallDurationSeconds(): number

      Optional runtime ceiling override (e.g. Gemini's 120s ceiling when uncompressed video is active). Narrowed against DefaultCallDurationSeconds and configuredCallDurationSeconds via ClampSessionSecondsValue.

      Returns number

    • get StateToRestoreOnOldConnection(): RealtimeClientState

      The state to report again when ResumeOnOldConnection goes back to the old connection, or null when the old connection has reported a state of its own since the move started (it keeps delivering events during the attempt), which is current. 'speaking' while the turn is open, else 'listening': the turn, not the state the move replaced, says whether the agent is answering, since a driver can leave 'speaking' without reporting it (a tool call) and the old connection can start an answer during the attempt. A 'speaking' report opens a turn, and one with no answer behind it would hold the next attempt until the last call. 'connected' is never reported: a host keeps showing the move for it.

      Returns RealtimeClientState

    • get SupportsInboundVideo(): boolean

      Whether the model takes inbound video on this session: the driver declared an inbound video track it supports when it negotiated tracks, and a stream limit above zero. No video track need be live; one is added when a capture starts (AddTrack). false before tracks are negotiated, and for a driver that does not negotiate them.

      Returns boolean

    • get SupportsRelayTransport(): boolean

      Whether this driver can open a relay session (ClientRealtimeSessionConfig.Transport 'relay'): connect to MJAPI's realtime relay at the session's RelayUrl rather than to its provider. false here; a driver that speaks its provider's protocol through the relay opts in by overriding it (the Gemini Enterprise client does).

      Returns boolean

    Methods

    • Adds a track to a running session, such as inbound video when the user starts their camera, resolved against what the driver supported when it negotiated at connect. The new track is 'live', or 'unsupported' with the reason (the model takes no such track, or no more inbound video streams). A live track with the same direction, modality and source is returned as it is. Other tracks keep their ids.

      The base re-negotiates locally, which is all a driver needs when its provider accepts the media without a setup change (Gemini Live takes video frames at any time). A driver whose wire must change overrides this and calls it.

      Parameters

      Returns RealtimeTrack

      The track, or null when the session has not negotiated tracks: before it connects, or with a driver that does not negotiate them (only the Gemini driver does today).

    • Arms (or re-arms) the soft and hard connection-ceiling timers for the current transport socket.

      • Soft timer opens the turn-boundary window before CallDurationSeconds.
      • Hard timer fires at CallDurationSeconds - ReconnectBufferSeconds and forces a reconnect; while the model is answering it waits for the turn to end, up to shortly before CallDurationSeconds. Both measure from the connection deadline instead when it comes before CallDurationSeconds (connectionDeadlineAtMs: the expiry of the credential the connection opened on, on a driver whose connections close with it).

      Returns void

    • Returns void

      has no cancel frame — the client OWNS the audio plane, so cancelling means: flush the local playout queue (speech stops immediately), mark the in-flight response inactive, and drain queued sends (a queued tool result takes the floor next — delivery is never dropped by a cancel). Residual server-side generation for the cancelled turn is simply never played; the platform's own VAD handles SPOKEN barge-in (emitting interruption). No-op when nothing is active.

    • Feeds an executed tool's result back via client_tool_result, correlated by the platform's tool_call_id. EXACTLY-ONCE: the pending id is consumed when the result is accepted, and a duplicate (or unknown) callID is dropped with a warning. Sent immediately when idle — the platform speaks the result as the turn's continuation — otherwise queued behind the in-flight response (e.g. a progress narration) so the trigger is never lost.

      Parameters

      • callID: string
      • outputJson: string

      Returns boolean | void

    • Returns the AGENT's remote-audio MediaStream when this driver owns a tappable agent-audio plane (a WebRTC driver's remote track, or a PCM driver's playout output), or null otherwise. Hosts use it to MIX the agent's voice into a browser-side recording alongside the mic; null (drivers with no tappable plane, or before the stream lands) degrades gracefully to mic-only capture. Hosts may still call it as client.GetRemoteMediaStream?.(). Pure read — never mutates.

      Returns MediaStream

    • Whether the user said something the current connection has not answered yet, with no turn open for it. The hard-point move then waits as it does for an open turn, until shortly before the provider limit. false here: a driver whose provider opens the turn only once the answer starts overrides it, and calls reconnectController.NotifyIdleStateChanged() when the hold ends without a turn.

      Returns boolean

    • Registers the (single) interruption handler.

      True barge-in only: fires ONLY when USER INPUT CUT OFF ACTIVE MODEL OUTPUT — a model response in flight or audio audibly playing when the user took the floor. A user simply taking their normal turn while the model is idle is NOT an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a response is actually active or audio is playing).

      On interruption, drivers must also flush locally-owned playback and report IsAudioPlaying === false promptly (driver obligation #3). Hosts typically use this hook to abort in-flight delegated work so a stale result is never narrated into a conversation that has moved on.

      Parameters

      • handler: () => void

      Returns void

    • Registers a handler invoked when the agent's remote-audio stream becomes available — IMMEDIATELY (synchronously) if it has already landed, otherwise when the driver publishes it (a WebRTC track typically lands AFTER Connect resolves). Lets a host attach the agent voice to a recording that began before the stream landed. Handler errors are isolated (logged) so a host bug never disturbs the call. Handlers are session-scoped: clearRemoteMediaStream (called from Disconnect) drops them.

      Parameters

      • handler: (stream: MediaStream) => void

      Returns void

    • Registers the (single) remote-VIDEO handler — the model/avatar's video for a VIDEO session (a talking-head the host renders, e.g. as the agent's tile). Invoked once the provider's video is available.

      The video is a MediaVideoSource: a live stream (a WebRTC track), or a player that must own the <video> element (MSE or WebCodecs playout of encoded avatar video). Hosts show either with AttachVideoSource.

      Optional capability: audio-only drivers (the default) never emit — registering is always safe, but hosts must not assume video arrives. A driver whose model sends video (the server driver declares an outbound video track in its session's Capabilities.SupportedOutboundTracks, or renders an avatar, BaseRealtimeModel.SupportsAvatarOutput) calls emitRemoteVideo when it is live.

      Parameters

      • handler: (video: MediaVideoSource) => void

        Invoked with the remote video when it becomes available.

      Returns void

    • Registers the (single) usage handler.

      Emissions carry token deltas for the response/turn that just completed (see RealtimeClientUsage — deltas preferred; cumulative-only providers must convert in the driver). Hosts accumulate and relay/persist on their own cadence (e.g. the voice session service debounces a RelayRealtimeUsage mutation onto the co-agent prompt run).

      Optional capability: drivers whose provider exposes no usage telemetry simply never emit — registering a handler is always safe, but hosts must not assume emissions arrive. See RealtimeClientUsage for per-driver availability.

      Parameters

      Returns void

    • Registers the single handler invoked when the provider's voice-activity detection saw the user start to speak. Fires once per detected onset and may fire again inside one turn after a pause; it carries no text. When the provider's event names the conversation item this speech belongs to (OpenAI-protocol input_audio_buffer.speech_started.item_id), itemId carries that id so the host can pair the onset with the matching user transcript (RealtimeClientTranscript.ItemId); otherwise itemId is undefined. Unlike OnInterruption, it fires whether or not the model was speaking.

      Parameters

      • handler: (itemId?: string) => void

      Returns void

    • Triggers ONE short spoken update. EMULATED as a user_message carrying the instructions, with the resulting response stamped Kind: 'narration' at send time (reset on the response boundary) — see the class-level fidelity caveat. Queued behind any in-flight response (a user_message sent mid-response would barge in on it), per the base contract's collision rule.

      Parameters

      • instructions: string

      Returns void

    • A planned move's attempt failed while the old connection still carries the call: go back to live on it (state, held input) until the next attempt, which waits for the next idle boundary or the hard point, and return true. Return false when the old connection cannot carry the call any more; the cycle then retries on its schedule. IsReconnecting is already false when this runs, so sends go to the old connection. false here: a driver opts in by overriding it. See RealtimeReconnectControllerOptions.ReturnToOldConnection.

      Returns boolean

    • Injects typed text as a user_message (the platform's "respond to this" trigger). No-op when the session is not open.

      SendText implies barge-in (base-contract rule): an active spoken response is cancelled via CancelActiveResponse first — playback flushed, response marked inactive, queued sends drained — so the typed turn takes the floor immediately. If a drained queued send (e.g. a pending tool result) starts a new response, the text queues behind it, preserving the tool-result delivery invariant. Server-side the fresh user turn takes the floor on its own; no explicit cancel frame exists or is needed.

      Parameters

      • text: string

      Returns void

    • Feeds an executed tool's result back to the model, correlated by callID, and ensures the model SPEAKS the result as soon as possible: immediately when idle, otherwise queued behind the in-flight response (e.g. a progress narration) so the trigger is never dropped by the provider.

      Template method: dispatches to doSendToolResult first (so the wire send happens before soft-window reconnect evaluation and rejected/unknown call IDs can return false), then records the result into conversationMemory via recordToolResult.

      Parameters

      • callID: string

        The CallID from the originating RealtimeClientToolCall.

      • outputJson: string

        The tool's result as a JSON-stringified string.

      Returns void

    • Streams one video frame as base64-encoded image data to the model. Optional capability — drivers that support inbound video tracks implement this.

      Parameters

      • base64Image: string

        Base64-encoded image data.

      • OptionalmimeType: string

        Image MIME type (defaults to 'image/jpeg').

      • OptionalsourceId: string

        Which source this frame came from. A driver whose model takes ONE inbound video stream ignores it (the source arbiter already chose what to send); a driver whose model takes several routes the frame to the track bound to that SourceID.

      Returns boolean

      true if accepted and sent; false if dropped (throttled, unestablished, etc.).

    • Parses a pcm_<rate> audio-format tag from the initiation metadata. Non-PCM formats (e.g. ulaw_8000 — a telephony-only configuration) are not playable/encodable by the shared PCM pipeline; the driver warns and falls back to the platform default so the session degrades loudly rather than throwing.

      Parameters

      • format: string
      • direction: "input" | "output"

      Returns number