@memberjunction/ai-elevenlabs

MemberJunction AI provider for ElevenLabs. The package ships three drivers:

• ElevenLabsAudioGenerator — BaseTextToSpeech implementation for text-to-speech, voice management, and pronunciation dictionaries (documented below).
• ElevenLabsRealtime — BaseRealtimeModel driver for the ElevenLabs Agents Platform, powering realtime full-duplex voice sessions (see the next section).
• ElevenRealtime — BaseRealtimeModel driver for Eleven Realtime, ElevenLabs' OpenAI-Realtime-compatible speech-to-speech models (see Eleven Realtime).

Realtime driver — ElevenLabsRealtime (Agents Platform)

ElevenLabsRealtime (src/elevenLabsRealtime.ts, registered via @RegisterClass(BaseRealtimeModel, 'ElevenLabsRealtime')) exposes ElevenLabs' orchestrated STT→LLM→TTS Agents stack as a standard MJ realtime model, resolved for MJ: AI Models rows typed Realtime. ElevenLabsRealtimeSession is the IRealtimeSession backing the server-bridged topology; the matching browser-direct client driver (ElevenLabsRealtimeClient, ClassFactory key 'elevenlabs') ships in @memberjunction/ai-realtime-client.

Provider characteristics

Configuration

• API key env alias: AI_VENDOR_API_KEY__ElevenLabsRealtime
• RealtimeSessionParams.Config.llm (string) selects the managed agent's underlying LLM; for full provider-side control, use a verbatim agent_… id
• RealtimeSessionParams.Config.voice (string) sets the per-session ElevenLabs voice id, sent as the tts.voice_id conversation-config override on both topologies. voice is the driver-neutral key AssemblyAI and Inworld also read, and it reaches the driver from the effective config as realtime.voice.providers.elevenlabs.voice. Omit it and the agent's own configured voice is used, exactly as before. Blank or non-string values are ignored rather than sent
  • The managed agent enables this override automatically; an agent provisioned by an earlier MJ version is re-PATCHed on next use to enable it
• RealtimeSessionParams.Config.firstMessage (string) makes the agent speak first, sent as the agent.first_message conversation-config override on both topologies. Spoken VERBATIM — it is the literal opening utterance, not an instruction about how to open. Authorable on the persona as realtime.voice.default.firstMessage, from where it reaches the driver on the neutral firstMessage key. Omit it (or leave it blank) and the agent waits for the user to speak first, exactly as before
  • Same enablement rule as the voice override: written on every managed agent this driver provisions, and an agent provisioned by an earlier MJ version is re-PATCHed on next use rather than silently dropping it
  • Conversation-start behavior is not prompt-addressable on this provider: with no first_message, the agent produces no audio until it receives user audio, whatever the persona prompt says
• InitialContext is injected as a contextual_update once the session is confirmed (the protocol has no history-seeding channel)

For the full architecture (topologies, co-agent model, capability matrix across all four realtime providers), see guides/REALTIME_CO_AGENTS_GUIDE.md.

Realtime driver — ElevenRealtime (Eleven Realtime models)

ElevenRealtime (src/elevenRealtime.ts, registered via @RegisterClass(BaseRealtimeModel, 'ElevenRealtime')) drives eleven_realtime_v1_mini and eleven_realtime_v1_max. They are cascaded (speech recognition, an LLM, ElevenLabs turn-taking and voice) but sold as one model, and their API follows the OpenAI Realtime spec over a WebSocket. So the driver subclasses OpenAIRealtime from @memberjunction/ai-openai, as the Grok Voice driver does, and an OpenAIRealtimeProfile gates the features. Unlike ElevenLabsRealtime, there is no server-side agent to provision: the session carries the prompt and tools. The browser-direct client driver is ElevenRealtimeClient (ClassFactory key 'eleven-realtime') in @memberjunction/ai-realtime-client.

Configuration

• API key env alias: AI_VENDOR_API_KEY__ElevenRealtime
• max_duration_seconds (an ElevenLabs extension) is min(provider cap, MaxSessionSeconds, Config bag value): a non-number bag value is ignored and never sent, and with nothing set the field is omitted. ElevenLabs then closes the session with code 1000 when it is exceeded, which MJ reports as an unexpected close

Audio generation — ElevenLabsAudioGenerator

This driver implements BaseTextToSpeech to provide high-quality voice synthesis, voice management, and pronunciation dictionary support. It still extends the deprecated BaseAudioGenerator, so existing callers keep working. AITextToSpeechRunner in @memberjunction/ai-prompts runs it with model selection, failover and an MJ: AI Prompt Runs row.

Architecture

graph TD
    A["ElevenLabsAudioGenerator
(Provider)"] -->|extends| B["BaseAudioGenerator
(@memberjunction/ai)"]
    A -->|wraps| C["ElevenLabsClient
(elevenlabs-js SDK)"]
    C -->|provides| D["Text-to-Speech"]
    C -->|provides| E["Voice Management"]
    C -->|provides| F["Model Listing"]
    C -->|provides| G["Pronunciation
Dictionaries"]
    B -->|registered via| H["@RegisterClass"]

    style A fill:#7c5295,stroke:#563a6b,color:#fff
    style B fill:#2d6a9f,stroke:#1a4971,color:#fff
    style C fill:#2d8659,stroke:#1a5c3a,color:#fff
    style D fill:#b8762f,stroke:#8a5722,color:#fff
    style E fill:#b8762f,stroke:#8a5722,color:#fff
    style F fill:#b8762f,stroke:#8a5722,color:#fff
    style G fill:#b8762f,stroke:#8a5722,color:#fff
    style H fill:#b8762f,stroke:#8a5722,color:#fff
Copy

Features

• Text-to-Speech: High-quality voice synthesis with customizable voice settings
• Voice Management: List and browse available voices with labels and preview URLs
• Model Discovery: Query available audio models with capability metadata
• Pronunciation Dictionaries: Manage custom pronunciation dictionaries with paginated retrieval
• Streaming Audio: Audio output returned as base64-encoded buffers
• Text Normalization: Optional text normalization for improved speech output

Installation

npm install @memberjunction/ai-elevenlabs
Copy

Usage

Text-to-Speech

import { ElevenLabsAudioGenerator } from '@memberjunction/ai-elevenlabs';

const tts = new ElevenLabsAudioGenerator('your-elevenlabs-api-key');

const result = await tts.CreateSpeech({
    text: 'Hello, welcome to MemberJunction!',
    voice: 'voice-id-here',
    model_id: 'eleven_turbo_v2'
});

if (result.success) {
    // result.content contains base64-encoded audio
    // result.data contains raw Buffer
    console.log('Audio generated successfully');
}
Copy

List Voices

const voices = await tts.GetVoices();
for (const voice of voices) {
    console.log(`${voice.name} (${voice.id}): ${voice.category}`);
}
Copy

List Models

const models = await tts.GetModels();
for (const model of models) {
    console.log(`${model.name}: TTS=${model.supportsTextToSpeech}`);
}
Copy

Supported Methods

Limitations

• SpeechToText is not yet implemented

Class Registration

• ElevenLabsAudioGenerator via @RegisterClass(BaseTextToSpeech, 'ElevenLabsAudioGenerator'), and via @RegisterClass(BaseAudioGenerator, 'ElevenLabsAudioGenerator') for callers of the deprecated base
• ElevenLabsRealtime via @RegisterClass(BaseRealtimeModel, 'ElevenLabsRealtime')
• ElevenRealtime via @RegisterClass(BaseRealtimeModel, 'ElevenRealtime')

Dependencies

• @memberjunction/ai - Core AI abstractions (BaseAudioGenerator, BaseRealtimeModel)
• @memberjunction/ai-openai - The shared OpenAI Realtime protocol implementation ElevenRealtime subclasses
• openai - The SDK client and realtime socket ElevenRealtime uses against ElevenLabs' endpoints
• @memberjunction/global - Class registration
• @elevenlabs/elevenlabs-js - Official ElevenLabs SDK (REST agent management + TTS; the realtime conversation websocket is spoken raw — the SDK's high-level wrapper owns audio devices, which a server bridge must not)