Back to AI Framework Overview

@memberjunction/ai

Core abstractions and base classes for the MemberJunction AI Framework. This package defines provider-agnostic interfaces for Large Language Models, embeddings, image generation, audio, video, reranking, and more.

Zero MemberJunction dependencies beyond @memberjunction/global (which itself has no transitive dependencies). This package works in any TypeScript or JavaScript project -- no database, no metadata layer, no MJ runtime required.

Installation

npm install @memberjunction/ai
Copy

Base Classes

Every AI capability is represented by an abstract base class. Provider packages (OpenAI, Anthropic, Gemini, etc.) extend these to implement the actual API calls.

All inherit from BaseModel, which manages API key storage and provides the @RegisterClass integration point.

One additional realtime primitive lives here that is not a BaseModel capability: BaseRealtimeChannelServer — the server half of the realtime interactive-channel plugin contract (MJ: AI Agent Channels.ServerPluginClass), mirroring the client half (BaseRealtimeChannelClient in the Angular conversations package). Concrete plugins register via @RegisterClass(BaseRealtimeChannelServer, '<key>') and are resolved per session by RealtimeChannelServerHost in @memberjunction/ai-agents. See guides/REALTIME_CO_AGENTS_GUIDE.md §5.

Realtime Sessions

The server half of MJ's realtime contract lives here; the browser half is BaseRealtimeClient in @memberjunction/ai-realtime-client. The Real-Time Co-Agents Guide explains the architecture, and its §12 walks through adding a video-capable provider.

Sessions and the browser mint (baseRealtime.ts)

The model's video output

RealtimeVideoFrame (realtimeVideoOutput.ts) is one frame of a model's video, in one of three kinds, each with Data, MimeType and optional PresentationTimeMs, KeyFrame, Width and Height:

• 'fmp4': a piece of fragmented MP4 as the model sent it (Piece: 'init' or 'fragment'), which may carry the voice on its audio track (a Gemini Live avatar);
• 'chunk': one encoded frame (H.264 in Annex B form, VP8, VP9 or AV1), with a required PresentationTimeMs and KeyFrame;
• 'image': one still picture (JPEG, PNG, WebP).

PresentationTimeMs is media time on the stream's own timeline, shared with its voice, not wall-clock time. Fmp4PieceToVideoFrame(data, mimeType, init) builds an fMP4 frame from a model part: the piece from its first box, an init segment's picture size, and a fragment's time and key-frame flag from its first video sample. Bytes that open with an MP4 box are a piece whatever MIME type the part names. A server session emits frames through IRealtimeSession.OnVideoFrame; a browser driver hands them to its video player.

The fragmented MP4 reader (fmp4Reader.ts)

Pure (no DOM, no Node APIs), so the browser and the meeting bot read avatar pieces with the same code. A box whose size runs past what holds it, or a track without the boxes that identify it, reads as null: the reader never guesses.

Tracks, frames and meetings

• Tracks (realtimeTracks.ts): RealtimeTrackDescriptor (modality, direction, encoding, rate, usage basis, consent), ResolveRequestedTracks (what a session asks for, intersected with what the model supports), InboundVideoStreamsOf and InboundVideoRateOf (a session's inbound video limits from its capabilities), IsPcmAudioMimeType.
• Still frames to the model (realtimeVideoFrames.ts): RealtimeVideoFrameIntervalMs, RealtimeMinVideoFrameSpacingMs and ScaleRealtimeVideoFrame, the rate and size math the browser's frame sampler and the meeting bot share.
• Meeting attributes: realtimeAgentVision.ts (mj.agentCanSee and mj.agentWatches: AllowsAgentVision, IsAgentWatching, AgentVisionAttributes, AgentWatchesAttributes; only 'true' counts as consent) and realtimeAgentAvatar.ts (REALTIME_AGENT_AVATAR_TRACK_NAME, the agent-avatar camera track a meeting bot shows the avatar on, and mj.agentAvatar: AgentAvatarAttributes, AgentAvatarAudioOnlyAttributes, ReadAgentAvatarAttribute).
• Resumption (realtimeSessionResumption.ts): RealtimeSessionResumption moves a session to a new connection with the provider's handle when the provider says the connection is ending or it drops, with retries and a deadline.

MJAPI's realtime proxy and relay (realtimeProxyRegistry.ts)

RealtimeProxyRegistry (a BaseSingleton) is the shared state between a provider driver that mints and MJAPI, which serves:

• Proxy tickets (Issue, Consume): one-time tickets for /realtime-proxy?ticket=<id>, a byte tunnel to a self-hosted provider's internal endpoint.
• Relay sessions (IssueRelaySession): one ticket per browser session for /realtime/relay/<ticket>/…, where MJAPI holds the provider credential. A session accepts one fresh connection within REALTIME_RELAY_OPEN_WINDOW_SECONDS (300) of the mint, then only resumes that present a resumption handle the relay forwarded to it; at most REALTIME_RELAY_MAX_CONNECTIONS (10) connections; it ends after REALTIME_RELAY_SESSION_LIFETIME_SECONDS (1800) or the session's MaxSessionSeconds. Sessions live in process memory, so several MJAPI instances need sticky routing for the relay path.
• IRealtimeRelayPolicy: the provider's half of the relay, called at five points: UpstreamHeaders (on every upstream open), ReadOpenIntent (the browser's first frame: a fresh open or a resume, and whether it asks for audio only), OpeningFrames, FilterClientFrame (forward or drop each later client frame; client frames are text) and ObserveServerFrame (find resumption handles; server frames go to the browser unchanged). Gemini's is GeminiLiveRelayPolicy in @memberjunction/ai-gemini.
• URLs: BuildRealtimeRelayUrl(baseWs, id) puts the ticket in the path (a provider's browser SDK appends its own path), and ResolveRealtimeProxyBaseWsUrl(params) finds MJAPI's public websocket origin (proxyBaseUrl in the config bag, then MJAPI_PUBLIC_URL, then GRAPHQL_BASE_URL and GRAPHQL_PORT).

Catalog settings for realtime

ModelConfiguration.Realtime (the catalog's per-model and per-model-vendor JSON) carries the realtime knobs the drivers read, including TurnDetection.Coverage (RealtimeTurnCoverage: audioActivityOnly or audioActivityAndAllVideo; a driver sends only what its endpoint accepts) and Pricing.AvatarVideoOutput (a per-minute price for generated avatar video, { Price, Unit: 'Per Minute', Currency }, which prices the video on a cost line of its own).

Type Definitions

Chat Types

Embedding Types

Other Types

Utilities

Usage Examples

Chat Completion

import { ChatParams, ChatMessageRole } from "@memberjunction/ai";
import { OpenAILLM } from "@memberjunction/ai-openai";

const llm = new OpenAILLM("your-api-key");

const result = await llm.ChatCompletion({
    model: "gpt-4.1",
    messages: [
        { role: ChatMessageRole.system, content: "You are a helpful assistant." },
        { role: ChatMessageRole.user, content: "What is the capital of France?" },
    ],
    temperature: 0.7,
    maxOutputTokens: 500,
});

console.log(result.data.choices[0].message.content);
Copy

Streaming

await llm.ChatCompletion({
    model: "gpt-4.1",
    messages: [{ role: ChatMessageRole.user, content: "Explain quantum computing." }],
    streaming: true,
    streamingCallbacks: {
        OnContent: (chunk, isComplete) => process.stdout.write(chunk),
        OnComplete: (result) => console.log("\nDone!"),
        OnError: (error) => console.error("Stream error:", error),
    },
});
Copy

Parallel Completions

const paramSets = [
    { ...base, temperature: 0.3 },
    { ...base, temperature: 0.7 },
    { ...base, temperature: 1.0 },
];

const results = await llm.ChatCompletions(paramSets, {
    OnCompletion: (result, index) => console.log(`Completion ${index} done`),
    OnAllCompleted: (results) => console.log(`All ${results.length} complete`),
});
Copy

Multimodal Content

const result = await llm.ChatCompletion({
    model: "gpt-4.1",
    messages: [{
        role: ChatMessageRole.user,
        content: [
            { type: "text", content: "What is in this image?" },
            { type: "image_url", content: "data:image/png;base64,..." },
        ],
    }],
});
Copy

Text Embeddings

import { OpenAIEmbedding } from "@memberjunction/ai-openai";

const embedder = new OpenAIEmbedding("your-api-key");
const result = await embedder.EmbedText({
    model: "text-embedding-3-small",
    text: "Sample text to embed",
});
console.log(`Dimensions: ${result.vector.length}`);
Copy

API Key Resolution

import { GetAIAPIKey } from "@memberjunction/ai";

// Reads AI_VENDOR_API_KEY__OPENAILLM from environment
const key = GetAIAPIKey("OpenAILLM");

// With runtime override
const key2 = GetAIAPIKey("AnthropicLLM", [
    { driverClass: "AnthropicLLM", apiKey: "sk-ant-..." },
]);

// Restricted to the runtime keys: no environment fallback for a class the list lacks
const key3 = GetAIAPIKey("OpenAILLM", runKeys, false, 'RuntimeOnly'); // undefined unless runKeys has OpenAILLM
Copy

Implementing a New Provider

Extend the base class for the capability you want to support:

import { BaseLLM, ChatParams, ChatResult, ClassifyParams, ClassifyResult, SummarizeParams, SummarizeResult } from "@memberjunction/ai";
import { RegisterClass } from "@memberjunction/global";

@RegisterClass(BaseLLM, "MyProviderLLM")
export class MyProviderLLM extends BaseLLM {
    // Required: implement non-streaming chat
    protected async nonStreamingChatCompletion(params: ChatParams): Promise<ChatResult> {
        // Your API call here
    }

    // Required by BaseLLM but deprecated: new drivers should throw a "not supported" error
    public async ClassifyText(params: ClassifyParams): Promise<ClassifyResult> { /* ... */ }

    // Required by BaseLLM but deprecated: new drivers should throw a "not supported" error
    public async SummarizeText(params: SummarizeParams): Promise<SummarizeResult> { /* ... */ }

    // Optional: enable streaming by overriding these
    public get SupportsStreaming(): boolean { return true; }
    protected async createStreamingRequest(params: ChatParams): Promise<AsyncIterable<unknown>> { /* ... */ }
    protected processStreamingChunk(chunk: unknown): { content: string } { /* ... */ }
    protected finalizeStreamingResponse(content: string, lastChunk: unknown, usage: unknown): ChatResult { /* ... */ }
}
Copy

See the full provider list for working examples across 25+ implementations.

Dependencies

Related Packages

• AI Framework Overview -- Architecture, provider matrix, and quick start
• Providers -- All 25+ provider implementations
• Prompts -- MJ-integrated prompt template engine
• Agents -- Agent execution framework