@memberjunction/ai-elevenlabs

MemberJunction AI provider for ElevenLabs. The package ships two drivers:

• ElevenLabsAudioGenerator — BaseAudioGenerator implementation for text-to-speech, voice management, and pronunciation dictionaries (documented below).
• ElevenLabsRealtime — BaseRealtimeModel driver for the ElevenLabs Agents Platform, powering realtime full-duplex voice sessions (see the next section).

Realtime driver — ElevenLabsRealtime (Agents Platform)

ElevenLabsRealtime (src/elevenLabsRealtime.ts, registered via @RegisterClass(BaseRealtimeModel, 'ElevenLabsRealtime')) exposes ElevenLabs' orchestrated STT→LLM→TTS Agents stack as a standard MJ realtime model, resolved for MJ: AI Models rows typed Realtime. ElevenLabsRealtimeSession is the IRealtimeSession backing the server-bridged topology; the matching browser-direct client driver (ElevenLabsRealtimeClient, ClassFactory key 'elevenlabs') ships in @memberjunction/ai-realtime-client.

Provider characteristics

Configuration

• API key env alias: AI_VENDOR_API_KEY__ElevenLabsRealtime
• RealtimeSessionParams.Config.llm (string) selects the managed agent's underlying LLM; for full provider-side control, use a verbatim agent_… id
• RealtimeSessionParams.Config.voice (string) sets the per-session ElevenLabs voice id, sent as the tts.voice_id conversation-config override on both topologies. voice is the driver-neutral key AssemblyAI and Inworld also read, and it reaches the driver from the effective config as realtime.voice.providers.elevenlabs.voice. Omit it and the agent's own configured voice is used, exactly as before. Blank or non-string values are ignored rather than sent
  • The managed agent enables this override automatically; an agent provisioned by an earlier MJ version is re-PATCHed on next use to enable it
• RealtimeSessionParams.Config.firstMessage (string) makes the agent speak first, sent as the agent.first_message conversation-config override on both topologies. Spoken VERBATIM — it is the literal opening utterance, not an instruction about how to open. Authorable on the persona as realtime.voice.default.firstMessage, from where it reaches the driver on the neutral firstMessage key. Omit it (or leave it blank) and the agent waits for the user to speak first, exactly as before
  • Same enablement rule as the voice override: written on every managed agent this driver provisions, and an agent provisioned by an earlier MJ version is re-PATCHed on next use rather than silently dropping it
  • Conversation-start behavior is not prompt-addressable on this provider: with no first_message, the agent produces no audio until it receives user audio, whatever the persona prompt says
• InitialContext is injected as a contextual_update once the session is confirmed (the protocol has no history-seeding channel)

For the full architecture (topologies, co-agent model, capability matrix across all four realtime providers), see guides/REALTIME_CO_AGENTS_GUIDE.md.

Audio generation — ElevenLabsAudioGenerator

This driver implements the BaseAudioGenerator interface to provide high-quality voice synthesis, voice management, and pronunciation dictionary support.

Architecture

graph TD
    A["ElevenLabsAudioGenerator
(Provider)"] -->|extends| B["BaseAudioGenerator
(@memberjunction/ai)"]
    A -->|wraps| C["ElevenLabsClient
(elevenlabs-js SDK)"]
    C -->|provides| D["Text-to-Speech"]
    C -->|provides| E["Voice Management"]
    C -->|provides| F["Model Listing"]
    C -->|provides| G["Pronunciation
Dictionaries"]
    B -->|registered via| H["@RegisterClass"]

    style A fill:#7c5295,stroke:#563a6b,color:#fff
    style B fill:#2d6a9f,stroke:#1a4971,color:#fff
    style C fill:#2d8659,stroke:#1a5c3a,color:#fff
    style D fill:#b8762f,stroke:#8a5722,color:#fff
    style E fill:#b8762f,stroke:#8a5722,color:#fff
    style F fill:#b8762f,stroke:#8a5722,color:#fff
    style G fill:#b8762f,stroke:#8a5722,color:#fff
    style H fill:#b8762f,stroke:#8a5722,color:#fff
Copy

Features

• Text-to-Speech: High-quality voice synthesis with customizable voice settings
• Voice Management: List and browse available voices with labels and preview URLs
• Model Discovery: Query available audio models with capability metadata
• Pronunciation Dictionaries: Manage custom pronunciation dictionaries with paginated retrieval
• Streaming Audio: Audio output returned as base64-encoded buffers
• Text Normalization: Optional text normalization for improved speech output

Installation

npm install @memberjunction/ai-elevenlabs
Copy

Usage

Text-to-Speech

import { ElevenLabsAudioGenerator } from '@memberjunction/ai-elevenlabs';

const tts = new ElevenLabsAudioGenerator('your-elevenlabs-api-key');

const result = await tts.CreateSpeech({
    text: 'Hello, welcome to MemberJunction!',
    voice: 'voice-id-here',
    model_id: 'eleven_turbo_v2'
});

if (result.success) {
    // result.content contains base64-encoded audio
    // result.data contains raw Buffer
    console.log('Audio generated successfully');
}
Copy

List Voices

const voices = await tts.GetVoices();
for (const voice of voices) {
    console.log(`${voice.name} (${voice.id}): ${voice.category}`);
}
Copy

List Models

const models = await tts.GetModels();
for (const model of models) {
    console.log(`${model.name}: TTS=${model.supportsTextToSpeech}`);
}
Copy

Supported Methods

Limitations

• SpeechToText is not yet implemented

Class Registration

• ElevenLabsAudioGenerator via @RegisterClass(BaseAudioGenerator, 'ElevenLabsAudioGenerator')
• ElevenLabsRealtime via @RegisterClass(BaseRealtimeModel, 'ElevenLabsRealtime')

Dependencies

• @memberjunction/ai - Core AI abstractions (BaseAudioGenerator, BaseRealtimeModel)
• @memberjunction/global - Class registration
• @elevenlabs/elevenlabs-js - Official ElevenLabs SDK (REST agent management + TTS; the realtime conversation websocket is spoken raw — the SDK's high-level wrapper owns audio devices, which a server bridge must not)