ProtectedapiOnly sub-classes can access the API key
The provider-native voice ids this model can speak with (e.g. OpenAI alloy/echo/shimmer). The
model/driver is the authoritative owner of "what voices do I support", so each driver declares its
own — used to populate the dev voice picker. Default empty (a driver that hasn't declared voices
yields no picker options, falling back to the configured/default voice).
NOTE: this is the near-term, driver-owned source of truth. Long term this should move to metadata so providers that let users add their OWN voices (e.g. ElevenLabs) can be enumerated dynamically.
The supported voice ids (id + human label), or [] when none are declared.
ElevenLabs supports the client-direct topology natively: the signed websocket URL is a short-lived, agent-scoped credential the browser can open directly.
Whether this driver's sessions carry a video track in addition to audio — i.e. the model accepts video input (it can "see" the user's camera) and/or emits video output (a talking-head avatar / generated video), in sync with audio.
Defaults to false (audio-only — today's realtime models). Video-capable drivers (a native
multimodal realtime model, or an avatar provider) override this to true. The session's media
plane is media-tagged (IRealtimeSession.SendInput takes a RealtimeMediaKind;
IRealtimeSession.OnVideoOutput delivers video-out), so a video session reuses the entire
realtime contract — only the media frames gain a video kind. Resolution prefers a video-capable
model when an agent requests video, and degrades to audio-only otherwise.
true if sessions can carry video; false (audio-only) otherwise.
ProtectedbuildBuilds the create/update body for the managed agent: the placeholder base prompt, the
session tool set mapped to inline CLIENT tools (expects_response: true so the agent
blocks on — and then speaks — each result; max response timeout because MJ client tools
delegate to long-running agents), and the platform-settings enablement of the
per-session agent.prompt.prompt override.
params.Config.llm (when a string) selects the agent's underlying LLM; all other
provider-level agent settings are deployment concerns (use a verbatim agent id for full
control).
Optionalconfig: JSONObjectProtectedconnectTransport seam for the server-bridged conversation websocket. Production speaks the RAW
Agents protocol over the platform-global WebSocket (browsers / Node 22+) — the SDK's
high-level Conversation wrapper owns audio devices, which a server bridge must not.
Resolves once the socket is OPEN; unit tests override this to return an in-memory fake.
ProtectedcreateREST seam: creates an agent, returning its id.
Mints the client-direct config: ensures the managed agent, fetches a signed URL
(the ephemeral credential — connect window ≈ ELEVENLABS_SIGNED_URL_TTL_MS),
and packs the private pact SessionConfig the same-keyed 'elevenlabs' client
driver consumes: { agentId, overrides, config }, where overrides is the wire-shaped
conversation_config_override carrying the server-authored system prompt and config
passes params.Config through opaquely.
ProtectedensureResolves params.Model to a concrete ElevenLabs agent id.
agent_…) is returned as-is — the deployment owns that agent's
configuration (including its tool set and override enablement).ProtectedgetREST seam: fetches an agent's full configuration.
ProtectedlistREST seam: lists agents matching a name search.
ProtectedmintREST seam: mints the signed websocket URL for an agent
(GET /v1/convai/conversation/get-signed-url?agent_id=… with the xi-api-key header —
the SDK call below is that exact request).
Opens a server-bridged conversation: ensures the agent, mints a signed URL, opens the
websocket (via the connectConversation seam), sends the
conversation_initiation_client_data frame carrying the per-session system prompt, and
resolves only once the server's conversation_initiation_metadata confirms the session
config is applied (driver obligation #7 — "ready only after the config is applied").
ProtectedupdateREST seam: PATCHes an agent's configuration.
StaticExtractExtracts the CLIENT tools from a fetched agent as Core tool definitions, for fingerprint comparison against the requested set. Non-client tools (webhook/system/mcp) are ignored — they are deployment-side additions the managed flow must not fight over.
StaticMapMaps a Core tool definition up to an ElevenLabs inline CLIENT tool config.
StaticPromptWhether the agent's platform settings enable the per-session system-prompt override.
StaticToolCanonical, order-insensitive fingerprint of a tool set (same scheme as the Gemini realtime driver). A schema round-trip through ElevenLabs that fails exact equality merely triggers a harmless idempotent PATCH.
Real-time, full-duplex driver for the ElevenLabs Agents Platform, implementing the Core BaseRealtimeModel primitive. Registers as
ElevenLabsRealtimeand is resolved forMJ: AI ModelstypedRealtime(API-key env alias:AI_VENDOR_API_KEY__ElevenLabsRealtime).Why an "agent", not a model: ElevenLabs realtime is an orchestrated STT→LLM→TTS stack exposed only through pre-configured server-side agents — there is no bare-model realtime socket. The driver hides that behind the standard realtime contract with a managed-agent strategy (ensureAgent):
params.Modelstarting withagent_→ used VERBATIM as a deployment-managed agent id.Per-session prompt authority stays with MJ: the managed agent stores only a placeholder prompt and explicitly enables
conversation_config_override.agent.prompt.prompt; every session (server-bridged or client-direct) sends the real system prompt in itsconversation_initiation_client_dataframe.Topologies:
No usage events: the Agents websocket reports no token usage; sessions never emit IRealtimeSession.OnUsage. Usage accounting for ElevenLabs realtime lives in the platform's own conversation dashboard / billing exports.