Real-Time Co-Agents Guide

How MemberJunction delivers live, low-latency, full-duplex agents — agents you talk to the way you talk to a person, that can draw on a shared whiteboard while they speak, and that delegate real work back to the async agents you already have.

This is the developer guide for the whole stack: the model primitive, the Realtime agent type, the Realtime Co-Agent, the dual-topology session architecture, the session/channel persistence layer, the interactive-channel plugin system (with the live Whiteboard as the canonical implementation), narration, observability, and security.

Companion documents:

• plans/ai-agent-sessions.md — the original architecture plan (Part I: realtime capability, Part II: session/channel infrastructure). This guide documents what actually shipped; the plan retains the rationale and the deferred tracks.
• packages/AI/RealtimeClient/README.md — the browser-side driver package (BaseRealtimeClient + the OpenAI / Gemini / ElevenLabs / AssemblyAI client drivers and the shared PCM audio plane), its /media entry (capture, video playout, the media stage) and its /testing entry (the video conformance kit).
• packages/AI/Agents/README.md — the agent framework, including how RealtimeAgentType sits beside Loop and Flow.
• packages/Angular/Generic/whiteboard/README.md — @memberjunction/ng-whiteboard, the extracted generic whiteboard package (state engine, agent tool API, components, exports). The conversations package consumes it as a thin channel plugin.
• guides/TRANSPORT_LAYER_ARCHITECTURE_GUIDE.md — the engine → resolver → client layering the realtime resolvers follow.

---

Table of Contents

1. The Concept: Co-Agents
2. The Triple-Registry Plugin Architecture
3. Topologies: Client-Direct vs Server-Bridged
4. Session Lifecycle
5. Channels — The Heart of the System
6. Future Channel Types (Envisioned)
7. Progress Narration
8. Observability & Administration
9. Security Model
10. Known Gaps & Deferred Work
11. Audio-Reactive Call Visuals
12. Adding a Video-Capable Realtime Provider

---

1. The Concept: Co-Agents

Real-time is a complement, not a replacement

MJ agents are asynchronous by design: a request arrives, an AIAgentRun does its work over seconds to minutes, results come back. That is exactly right for substantive work, and Loop/Flow agents are untouched by the realtime feature. What async agents cannot do is hold a natural, low-latency conversation — modern realtime stacks (GPT Realtime, Gemini Live, ElevenLabs Agents, AssemblyAI Voice Agent) own the listen-reason-speak loop themselves, with provider-side voice activity detection, barge-in, and sub-second turn taking.

So instead of forcing the loop agent to drive a live call, MJ adds a third agent type — Realtime — a first-class peer of Loop and Flow (RealtimeAgentType in packages/AI/Agents/src/agent-types/realtime-agent-type.ts, registered as @RegisterClass(BaseAgentType, "RealtimeAgentType"), seeded in metadata/agent-types/ with DriverClass: "RealtimeAgentType"). A Realtime agent does not iterate a reasoning loop; it wraps a long-lived duplex model session. The marker is RealtimeAgentType.IsSessionDriven (always true); BaseAgent.isSessionDrivenAgentType() duck-types that getter and branches into executeRealtimeSession() instead of the loop.

One co-agent voices ANY agent

The first (and seeded) agent of this type is the Realtime Co-Agent (metadata/agents/.voice-co-agent.json — the agent was renamed from the original Voice Co-Agent; the seed file keeps its legacy filename, the Name field is Realtime Co-Agent). It is deliberately generic: it is the live voice for a target agent, and the target is a runtime parameter, not configuration baked into the co-agent. One Realtime Co-Agent definition fronts Sage, Query Builder, or anything else — agents gain voice by configuration, never by being rewritten.

Three design rules make this work:

1. The realtime-registered tool set is capability-gated. Dynamic-capable drivers (SupportsDynamicToolSet = true: OpenAI Realtime, OpenAI Live, xAI, Gemini Live) can project allowed direct actions from the target agent when directActions.enabled: true, executing them rapidly via ActionEngine.RunAction (with per-call timeouts) to avoid the 2–5s roundtrip of a full sub-agent run. Gemini Live (geminiRealtime.ts) takes the projected tools at connect/mint time and cannot change them mid-session (see the capability matrix in §2). For server-provisioned or self-hosted compat providers (ElevenLabs Agents, HuggingFace), toolsets are fixed or pre-provisioned at connect time, so they maintain the stable target-independent invoke-target-agent tool set (INVOKE_TARGET_AGENT_TOOL_NAME in packages/AI/Agents/src/realtime/realtime-tool-broker.ts) where all target-specific capabilities execute inside the delegated agent’s own run. This keeps the provider contract identical across targets for fixed providers — and it is exactly what let the ElevenLabs Agents driver (a pre-provisioned, server-side-agent provider) fit the same model: its managed agent is created once with the stable tool set and reused across sessions (see §2).
2. Companion framing, never impersonation. The system prompt (the Realtime Co-Agent - System Prompt row in metadata/prompts/ — its template file retains the legacy name templates/Voice Co-Agent - System Prompt.template.md) frames the model as “the live voice for the agent” — first-person, owning the work (“I’m pulling that up”), never “pretend to be X” (which makes models refuse) and never third-person (“Sage is working on it”).
3. Delegation is a normal agent run. When the co-agent invokes the target, that is a full AIAgentRun, linked to the co-agent’s run via ParentRunID and stamped with the shared AgentSessionID — the delegation tree is fully visible in standard observability. In the client-direct topology, an invoke-target-agent run also gets the browser session id as sessionID and the app-context snapshot from session start as data.appContext (the resolver keeps the snapshot in the session’s Config). So the delegated agent sees the surface context and can call the surface’s client tools. The snapshot does not follow later navigation. A scoped anonymous caller’s run executes as the system user and gets no app context, because the browser supplied the snapshot.

The co-agent resolution chain

Which Realtime-type agent voices a given target is resolved at session start by RealtimeClientSessionResolver.resolveCoAgentID() (packages/MJServer/src/resolvers/RealtimeClientSessionResolver.ts). First match wins:

Step	Source	On invalid candidate
1	Runtime coAgentId argument on StartRealtimeClientSession	Throws (fail loud — the caller asked for something specific)
2	Target agent’s AIAgent.DefaultCoAgentID (per-agent persona)	Logs a warning, falls through (metadata drift must degrade, not break calls)
3	Target agent’s type’s AIAgentType.DefaultCoAgentID (per-type default)	Logs a warning, falls through
4	The seeded Realtime Co-Agent, looked up by name (falls back to the deprecated legacy name Voice Co-Agent with a deprecation log, for deployments that haven’t re-synced the renamed seed)	Throws when both names are absent (the realtime feature is unconfigured in this deployment)

Every candidate from steps 1–3 is validated by findValidCoAgent(): it must exist in AIEngine’s cached agents, have Status = 'Active', and be of the Realtime agent type. The two DefaultCoAgentID columns were added by migration migrations/v5/V202606090930__v5.41.x__AI_Agent_Sessions_Channels.sql.

This chain is how deployments give individual agents a distinct voice persona (a different co-agent with a different prompt/voice) without touching code or the global seed.

Two further metadata layers refine which target a co-agent fronts and how it sounds/behaves — pairing rows and the effective configuration merge. Both are opt-in with zero-config defaults identical to the pre-feature behavior; see §4 — Co-agent pairing & effective configuration.

---

2. The Triple-Registry Plugin Architecture

Realtime is pluggable along three independent axes, all resolved through MJGlobal.ClassFactory + metadata. Nothing in the hosts (the session runner, the resolver, the Angular overlay) names a concrete provider or channel. The third axis — interactive channels — has two halves, one base class per side of the wire, resolved from the same registry row:

Registry	Base class	ClassFactory key comes from	Shipped implementations
Server model drivers	BaseRealtimeModel (packages/AI/Core/src/generic/baseRealtime.ts)	MJ: AI Model Vendors.DriverClass on a model of AIModelType = 'Realtime'	OpenAIRealtime (packages/AI/Providers/OpenAI/src/models/openAIRealtime.ts), GeminiRealtime (packages/AI/Providers/Gemini/src/geminiRealtime.ts), ElevenLabsRealtime (packages/AI/Providers/ElevenLabs/src/elevenLabsRealtime.ts), AssemblyAIRealtime (packages/AI/Providers/AssemblyAI/src/assemblyAIRealtime.ts), xAIRealtime (packages/AI/Providers/xAI/src/models/xAIRealtime.ts), HuggingFaceRealtime (packages/AI/Providers/HuggingFace/src/huggingFaceRealtime.ts), GeminiEnterpriseRealtime (packages/AI/Providers/Vertex/src/models/geminiEnterpriseRealtime.ts, Gemini Live on Gemini Enterprise)
Client model drivers	BaseRealtimeClient (packages/AI/RealtimeClient/src/generic/baseRealtimeClient.ts)	ClientRealtimeSessionConfig.Provider — the string the server driver stamps when it mints ('openai', 'gemini', 'gemini-enterprise', 'elevenlabs', 'assemblyai', 'xai', 'huggingface')	OpenAIRealtimeClient, GeminiRealtimeClient, GeminiEnterpriseRealtimeClient (relay sessions only), ElevenLabsRealtimeClient, AssemblyAIRealtimeClient, xAIRealtimeClient, HuggingFaceRealtimeClient (same package)
Channel plugins — client half	BaseRealtimeChannelClient (packages/RealtimeRuntime/src/channels/base-realtime-channel-client.ts)	MJ: AI Agent Channels.ClientPluginClass	RealtimeWhiteboardChannel (key 'RealtimeWhiteboardChannel') — a thin plugin over @memberjunction/ng-whiteboard
Channel plugins — server half	BaseRealtimeChannelServer (packages/AI/Core/src/generic/baseRealtimeChannelServer.ts), resolved per session by RealtimeChannelServerHost (packages/AI/Agents/src/realtime/realtime-channel-server-host.ts)	MJ: AI Agent Channels.ServerPluginClass	WhiteboardChannelServer (key 'WhiteboardChannelServer', in @memberjunction/ai-agents) — validates/canonicalizes the board’s persisted state of record

Server model drivers are resolved by BaseAgent.resolveRealtimeModel() (server-bridged) and RealtimeClientSessionService.resolveVendorAndInstantiate() (client-direct): highest-PowerRank active Realtime model → highest-Priority active vendor whose DriverClass has a resolvable API key (GetAIAPIKey, e.g. AI_VENDOR_API_KEY__OpenAIRealtime) → ClassFactory.CreateInstance<BaseRealtimeModel>(BaseRealtimeModel, driverClass, apiKey). An explicit preferredModelId bypasses ranking and fails with a specific reason rather than silently falling back (resolvePreferredRealtimeModel). A co-agent with video on (realtime.video.enabled) and no explicit model first takes the highest-ranked candidate that shows an avatar (its Video/Output modality row and its driver’s endpoint both allow it; see §12), and otherwise stays on the default model, audio only.

Client model drivers are resolved in VoiceSessionService.createRealtimeClient() by the server-reported Provider key. Channel plugins are resolved in VoiceSessionService.loadActiveChannels() from the active MJ: AI Agent Channels rows.

[!NOTE] The channel registry is the one registry with two halves: each row carries both a ClientPluginClass (resolved in the browser by VoiceSessionService) and a ServerPluginClass (resolved server-side by RealtimeChannelServerHost when SessionManager.CreateSession mints the session — one fresh instance per session, exactly mirroring the client half’s resolution: ACTIVE rows only, ClassFactory registration checked first, unregistered keys skipped with a log, never fatal). The server half’s lifecycle hooks are the durable session events — session started, channel state saved (pre-persistence, with normalization power), session closed (any provenance) — see §5 for the full contract. The registry also carries a TransportType (PubSub / WebRTC / WebSocket) for the future media-plane work.

The four-provider capability matrix

The base contracts are deliberately tolerant of provider asymmetry: several members are optional capabilities (SendContextNote / RequestSpokenUpdate / OnClose on IRealtimeSession) or may simply never fire (OnUsage). Where a provider lacks a native channel, the driver either emulates it (documented per driver) or omits it — hosts feature-detect, never assume. What actually ships, verified per driver:

Capability	OpenAI ('openai')	Gemini ('gemini')	ElevenLabs Agents ('elevenlabs')	AssemblyAI Voice Agent ('assemblyai')
Client transport	WebRTC (mic track + oai-events data channel)	WebSocket (@google/genai Live SDK)	WebSocket (raw Agents protocol)	WebSocket (raw Voice Agent protocol)
Mint style	Client secret with the full session config (instructions + tools) baked in	Ephemeral token locking the mask-safe generation subset (liveConnectConstraints)	Signed URL is the credential (wss://…&token=…, ~15-min open window); agent ensured server-side first	One-time temp token (GET /v1/token, 300 s connect window)
What the token locks	Prompt + tools + transcription (tamper-proof)	Model + generation params only — prompt/tools ride the driver-applied SessionConfig	Agent identity; the prompt rides the conversation_initiation_client_data override (enabled on the managed agent); tools are bound to the server-side agent config	Nothing beyond auth — the whole session object rides the driver-applied first session.update frame
Transcripts	Interim deltas + finals, both roles	Interim deltas + finals, both roles	Finals only, both roles; agent_response_correction re-finalizes a barged-in turn with what was actually spoken	User turns: deltas + final; agent turns: final only (post-barge-in final carries the truncated text — no correction event)
SendContextNote	Emulated (conversation-item injection without a response trigger)	Emulated (turnComplete: false client content)	Native — contextual_update, the platform’s purpose-built non-interrupting channel; sent even mid-response	Emulated via the mutable system_prompt — notes accumulate under a “Background updates” heading and the full prompt is re-sent via session.update
RequestSpokenUpdate	Native-ish (response.create with per-response instructions)	Emulated (instruction turn via realtime text — sendRealtimeInput({ text }); mid-call sendClientContent is history-seeding-only on native-audio models and never triggers generation), queued behind in-flight responses	Emulated as a user_message turn (queued; fidelity caveat — the model may reference “you asked me for an update”)	Native — reply.create carries per-response instructions (queued behind in-flight replies)
SendText (typed input)	Native user-item injection	Realtime-text turn (sendRealtimeInput({ text }) — same native-audio caveat as RequestSpokenUpdate)	Native user_message (takes the floor server-side)	Emulated via reply.create instructions — the protocol has no typed-user-input event; best-effort fidelity
Cancel (CancelActiveResponse)	Wire-level response.cancel + local flush	Local playout flush (client owns the audio plane)	No cancel frame — local playout flush; residual server generation is simply never played	No cancel frame — local flush plus suppression of residual reply.audio frames until the next reply boundary
Mid-session RegisterTools	Idempotent re-declare via session.update	Connect-bound (the StartSession params path is the only effective one)	Cannot re-declare on an open conversation — identical set no-ops, a different set warns and is ignored (next session picks it up via the ensure flow)	Native re-declare — tools is a mutable session.update field
Usage events (OnUsage)	Per-response deltas from response.done.usage	Per-turn deltas from usageMetadata	None — the Agents socket reports no token usage (platform-side accounting)	None — flat per-session-hour billing; the terminal session.ended frame carries duration seconds only
Server-side object to manage	None	None	Managed agent — ensureAgent(): find-by-name → create-if-missing → PATCH on tool-fingerprint drift / any missing per-session override enablement; cached per name + fingerprint	None — pure per-session config
Can the agent speak first?	No — the neutral firstMessage key is ignored by the driver and scrubbed before the wire	No — same: ignored, and scrubbed before the SDK spread	Yes — agent.first_message override (enabled on the managed agent). Without it the agent produces NO audio until it hears the user, whatever the prompt says (#3557)	Yes — the greeting field of the session object (fed by the neutral firstMessage key; the legacy greeting config key still works)
Audio format (client driver)	Provider-negotiated via WebRTC	PCM16 16 kHz up / 24 kHz down	PCM16 at rates negotiated from the initiation metadata (pcm_<rate>, default 16 kHz; non-PCM telephony formats degrade loudly)	PCM16 fixed 24 kHz both directions
Inbound video (camera, screen)	None	JPEG frames at the model’s rate, as the minted profile allows (Gemini 3.8 Live: one stream, 1 frame per second)	None	None
Agent video (live avatar)	None	Gemini 3.8 Live on Gemini Enterprise only ('gemini-enterprise', through MJAPI’s relay): fragmented MP4 on the socket, carrying the voice. The Developer API returns audio	None	None

Two cross-cutting notes on the websocket-audio drivers (ElevenLabs, AssemblyAI): they share the browser audio plane in packages/AI/RealtimeClient/src/audio/ (createPcmMicCapture worklet capture, RealtimePcmPlayback playhead-clock playout, pcmUtils) — IsAudioPlaying is computed from the locally-owned playout clock. And both must send session.end-style teardown correctly: AssemblyAI’s Close() sends session.end before closing the socket, because skipping it leaves the session in a billable 30-second resume hold.

Driver-author obligations (read before writing a driver)

Both base classes carry an explicit “DRIVER AUTHOR OBLIGATIONS” block in their doc headers — BaseRealtimeModel (packages/AI/Core/src/generic/baseRealtime.ts) for server drivers, with the client-side mirror on BaseRealtimeClient (packages/AI/RealtimeClient/src/generic/baseRealtimeClient.ts). Those blocks are the authoritative list (8 numbered rules on the server, 12 on the client, paid for in live debugging). Summarized:

1. Silent exit from “speaking” on tool-call emission — don’t clobber the host’s busy indicator with trailing turn frames.
2. Release the busy flag when a tool call is emitted — otherwise SendToolResult deadlocks waiting on a turn boundary that never arrives.
3. Flush playback + report IsAudioPlaying = false promptly on barge-in — stale queued audio after an interruption is a product bug.
4. Text injection must not echo a user transcript — the host owns the local echo; a synthesized user-role event renders the message twice.
5. Every tool result must EVENTUALLY be voiced and never dropped — queue the generation trigger behind an in-flight response if the provider rejects overlaps.
6. Credential expiry surfaces as a Fatal: true OnError — never idle forever on a dead socket.
7. Report “ready/listening” only after the server-built session config has been applied — early turns must not run against an unconfigured model.
8. SessionConfig is a private pact between same-keyed server and client driver halves — hosts and intermediaries treat it as an opaque blob.

The client block adds four:

9. Audio metering is a capability — a driver that owns or can tap an audio plane attaches its meters, and releases them on disconnect (see §11).
10. Follow a replaced microphone track — a device switch swaps the track inside the mic stream; rebind what used the old one (ReplaceMicrophone: replaceTrack on WebRTC senders, IPcmMicCapture.Rebind on PCM capture, a new input meter).
11. Refuse a transport you don’t speak — call AssertTransportSupported first in Connect, so a relay session (Transport: 'relay') fails clearly on a driver that only connects to its provider; a driver that speaks through MJAPI’s relay overrides SupportsRelayTransport.
12. The agent’s video keeps the voice’s rules — hand the video to the host once, only while the outbound video track is live; barge-in flushes it with the voice; a video that carries the voice never plays it twice; each generated second is reported once. See §12, and run the conformance kit, which checks each rule.

Additional contract points on IRealtimeSession: RegisterTools must be idempotent for a set identical to the connect-time set; OnInterruption fires on true barge-in only (user speech over active model output, not every utterance); SendContextNote / RequestSpokenUpdate / OnClose are optional capability members — callers feature-detect, and drivers whose provider can’t support them omit them entirely.

Recipe: add a new realtime provider

1. Server driver. In the provider’s package (packages/AI/Providers/<X>), implement BaseRealtimeModel.StartSession() returning an IRealtimeSession, and register it: @RegisterClass(BaseRealtimeModel, '<X>Realtime'). Build against an injectable connection seam (see IOpenAIRealtimeConnection, GeminiLiveSession, ElevenLabsRealtimeSocket, AssemblyAIRealtimeSocket) so unit tests can drive provider-shaped frames with no network — all four drivers ship extensive vitest suites this way.
2. Client-direct support (optional). Override SupportsClientDirect to true and implement CreateClientSession() to mint an ephemeral credential + provider-native SessionConfig, stamping a unique Provider key. The credential doesn’t have to be a token: ElevenLabs mints a signed websocket URL; AssemblyAI mints a one-time temp token via a REST call.
3. Client driver. In @memberjunction/ai-realtime-client, extend BaseRealtimeClient, register under the same Provider key (@RegisterClass(BaseRealtimeClient, '<x>')), export a Load<X>RealtimeClient() no-op, and have hosts call it (the existing Load calls live at the top of voice-session.service.ts — tree-shaking prevention). If the audio plane is a raw websocket (not WebRTC), reuse the shared src/audio/ pipeline (createPcmMicCapture + RealtimePcmPlayback) instead of writing capture/playout again.
4. Metadata. Add the MJ: AI Models row (AIModelTypeID → Realtime type) with an MJ: AI Model Vendors association carrying DriverClass: '<X>Realtime' and the provider APIName, in metadata/ai-models/; push with mj sync. Reference rows: GPT Realtime 2 (APIName gpt-realtime-2), Gemini 3.1 Flash Live (APIName gemini-3.1-flash-live-preview), ElevenLabs Agents (APIName MJ Realtime Co-Agent — see the precedent below), AssemblyAI Voice Agent (APIName voice-agent — the endpoint has no model selection; the value is bookkeeping).
5. API key. AI_VENDOR_API_KEY__<DriverClass> (or the deployment’s key-resolution path consumed by GetAIAPIKey). A session started from an agent run uses the run’s apiKeys first; under the run’s CredentialScope: 'RuntimeOnly' it uses them only, so a vendor the run has no key for is not selected.

No host code changes: the resolver ranks the new model by PowerRank/Priority, and the browser resolves the client driver by the minted Provider string.

A provider whose model sends video (a live avatar) also follows §12, Adding a Video-Capable Realtime Provider, which ends with the conformance kit.

Lessons from the ElevenLabs / AssemblyAI builds (read before driver #5)

Durable friction points the third and fourth drivers surfaced — they generalize to any future provider:

• Model/APIName doesn’t have to name a model. ElevenLabs has no bare-model realtime socket — only pre-configured server-side agents. The driver sets the precedent: RealtimeSessionParams.Model carrying the agent_ prefix is used verbatim as a deployment-managed agent id; any other value is the name of the driver-managed agent (ensureAgent(): find-by-name → create-if-missing with the session tool set + per-session prompt-override enablement → PATCH on tool-fingerprint drift; instance-cached). The seeded APIName is MJ Realtime Co-Agent. If your provider’s “thing you connect to” isn’t a model, map it through Model the same way rather than inventing config side-channels.
• Delta-less transcript providers exist. ElevenLabs emits whole-utterance finals only; AssemblyAI’s agent turns are final-only. Host code must not assume interim deltas — the caption pipeline keys off IsFinal, and that discipline is what made finals-only providers drop in cleanly.
• The post-barge-in “correction” problem. ElevenLabs re-finalizes a barged-in assistant turn via agent_response_correction (the text actually spoken); AssemblyAI’s final already reflects the truncation. The transcript contract has no Replaces marker, so a host persisting finals gets two assistant finals around an ElevenLabs barge-in (the original and the correction) with no machine-readable link — a known open contract gap (see §10). Until it’s closed, treat a final assistant transcript arriving immediately after an interruption as the authoritative replacement.
• Single-frame busy boundaries. Neither new provider has an OpenAI-style response.created/response.done envelope pair around every response. Busy state is inferred frame-by-frame (ElevenLabs: first audio/agent_response sets it, agent_response_complete clears it; AssemblyAI: reply.started/reply.done), and the narration Kind must be stamped at send time — there is no confirmation frame to stamp on. Tool-call frames still clear the busy flag (deadlock guard) without draining the queue, so a queued narration can’t wedge between a tool call and its result.
• Two-round-trip opens. Both drivers’ Connect/StartSession is a handshake, not a socket-open: open the socket → send the config frame (conversation_initiation_client_data / first session.update) → await the provider’s confirmation (conversation_initiation_metadata / session.ready) → only then build the audio plane and report 'listening' (obligation #7). On ElevenLabs the server side additionally spends REST round-trips before the socket (ensure-agent + signed-URL mint, mitigated by the fingerprint cache). Budget for this in any “time to first listening” expectations.
• Teardown can be billable. AssemblyAI holds a closed-without-session.end session in a billable 30-second resume window — Close()/Disconnect() must send the explicit end frame first.

Recipe: add a new interactive channel

See the Realtime Channels Guide for the current, complete recipe (including wrapping an existing component and publishing from an Open App); the summary below is the original short recipe.

Covered in depth in §5; the short version:

1. Subclass BaseRealtimeChannelClient<YourSurfaceComponent>, implementing the channel contract (tools + perception + surface + state of record).
2. @RegisterClass(BaseRealtimeChannelClient, '<YourClientPluginClass>') + a Load...() no-op called from a static code path (the Whiteboard’s is called from conversations.module.ts).
3. (Optional but recommended) Subclass BaseRealtimeChannelServer for the server half — at minimum a state-of-record guard like WhiteboardChannelServer’s (validate/normalize each landed save in OnChannelStateSave). @RegisterClass(BaseRealtimeChannelServer, '<YourServerPluginClass>') + a Load...Server() no-op called from a static server path (the Whiteboard’s is called from MJServer’s agentSessions/index.ts). A row whose server key is unregistered simply runs with no server plugin — it is never an error.
4. Seed an MJ: AI Agent Channels row (Name, ClientPluginClass, ServerPluginClass, TransportType, IsActive: true) in metadata/ai-agent-channels/ and push with mj sync.
5. Done — VoiceSessionService.startChannels() resolves every active registry row at session start in the browser, and RealtimeChannelServerHost resolves the same rows’ server halves when the session mints. The model gets the plugin’s tools, the overlay gets a surface tab, the server gets the lifecycle hooks. The shells carry zero channel-specific wiring.

---

3. Topologies: Client-Direct vs Server-Bridged

The same tool-execution semantics run in two transport topologies. The shared piece is RealtimeToolBroker (packages/AI/Agents/src/realtime/realtime-tool-broker.ts) — both topologies route every tool call through it so results are byte-for-byte identical: invoke-target-agent → delegate (with a broker-owned per-call AbortController — fired by barge-in on the server-bridged path, and by the explicit cancel channel on the client-direct path), everything else → the injected tool executor; failures serialize to { success: false, error } so the model can narrate failures instead of going silent.

Decision table

	Client-direct (shipped MVP, drives Explorer voice)	Server-bridged (RealtimeSessionRunner)
Provider socket lives	In the browser (OpenAI: WebRTC; Gemini: WebSocket via @google/genai Live; ElevenLabs / AssemblyAI: raw WebSocket + the shared PCM audio plane). A relay session (Gemini Enterprise) is the exception: the browser’s socket goes to MJAPI’s relay, which opens the provider’s	On the server
Audio path	Browser ↔ provider directly — frames never transit MJ (lowest latency), except on a relay session, whose frames pass through MJAPI (the provider’s frames unchanged, the browser’s through the provider’s frame policy)	Server ↔ provider; no client media transport is wired yet (the plan’s P5 media plane)
Prompt/tool authority	Server — it builds the session config; the browser applies it verbatim	Server (it owns the socket)
Tool execution	Relayed: browser → ExecuteRealtimeSessionTool mutation → RealtimeToolBroker → result JSON → browser → SendToolResult	In-process: RealtimeSessionRunner.handleToolCall → broker → IRealtimeSession.SendToolResult
Non-target tools	Channel/UI tools execute locally in the browser (never relayed); relayed unknown tools get a structured “not available” (RealtimeClientSessionService.executeNonTargetTool — action wiring is a later phase)	Agent actions execute via BaseAgent.executeRealtimeTool → ExecuteSingleAction
Entry point	StartRealtimeClientSession mutation → RealtimeClientSessionService.PrepareClientSession	AgentRunner.RunAgent on a Realtime-type agent → BaseAgent.executeRealtimeSession
Narration / context notes	Browser calls BaseRealtimeClient.SendContextNote / RequestSpokenUpdate (fully wired — see §7)	Wired: the runner pipes delegation progress into IRealtimeSession.SendContextNote and paces spoken updates through the same first-delay / spacing / digest rules as the browser (shared template via realtime-narration.ts — see §7)
Delegation cancel	Explicit-only: the overlay’s per-card ✕ → CancelRealtimeSessionTool mutation → the service’s in-flight registry aborts the run. Barge-in never aborts delegations (deliberate host policy — the narration design expects the user to keep talking while work runs)	Barge-in aborts the in-flight delegated run via RealtimeToolBroker.AbortInFlight (no human ✕ exists on this topology)
Usage telemetry	Browser accumulates OnUsage deltas → debounced (10 s) + teardown-flushed RelayRealtimeUsage mutation → accumulated onto the co-agent AIPromptRun	Runner checkpoints accumulated usage onto the long-lived AIPromptRun (debounced 5 s, crash-safe)

🚨 Single source of truth: ONE prep, the precedence cascade, the core↔host boundary

Every realtime host — native chat, LiveKit, and future Zoom/Teams/GoToMeeting/Webex — must be the same agent: same identity, personality, model/voice, delegation, and session tracking. Only media transport and host UX tools vary. The full rationale + roadmap is in plans/complete/realtime/realtime-core-host-convergence.md.

There is exactly ONE producer of realtime session prep: RealtimeClientSessionService.PrepareRealtimeSessionParams (packages/AI/Agents/src/realtime/realtime-client-session-service.ts). It resolves the model, builds the target-identity system prompt (via the single BuildRealtimeAgentFraming in realtime-tool-broker.ts — first-person AS the target, never the co-agent), the stable tool set (always incl. invoke-target-agent), voice, and memory. Do NOT build session prep in a host. The LiveKit bridge (BaseAgent.StartBridgeRealtimeSession) consumes it and differs only in opening server-side (StartSession) vs the browser mint (CreateClientSession); a bridge re-implementing prep is exactly the drift this convergence removed (guarded by __tests__/realtime-convergence-drift.test.ts).

Model + voice resolve through ONE precedence cascade (ResolveEffectiveRealtimeConfig), identical on every host:

runtime override  >  target agent's TypeConfiguration  >  co-agent config  >  agent-type default

The target layer is what lets a voiced agent (Sage, Marketing Agent, …) carry its own persisted voice/model that the shared co-agent then speaks with. The runtime-override input funnels into one slot — the native picker builds ConfigOverridesJson client-side (BuildRealtimeConfigOverridesJson), the server-bridged hosts build the identical envelope via BuildRealtimeOverridesJson — so both ride the same highest-precedence cascade layer.

Core (one place, all hosts): identity, model/voice precedence, base prompt + memory, invoke-target-agent delegation, session/run tracking. Host (varies): media transport (LiveKit audio → video) + host UX tools (whiteboard/browser/… — injected via RealtimeSessionRunnerDeps.ExtraTools/ExecuteTool, never baked into core).

Client-direct flow (the shipped path)

1. Mint. The browser (VoiceSessionService.StartVoiceSession) calls StartRealtimeClientSession(targetAgentId, conversationId?, lastSessionId?, preferredModelId?, clientToolsJson?, coAgentId?). The resolver authorizes CanRun on the target agent, resolves the co-agent (chain in §1), creates the durable AIAgentSession via SessionManager.CreateSession (storing targetAgentID server-side in the session’s Config column — accessed as Config_ on the generated entity), then RealtimeClientSessionService.PrepareClientSession resolves the model, assembles the companion system prompt (framing + co-agent prompt + target identity + history + memory via AgentMemoryContextBuilder — the same context a loop agent injects), builds the stable tool set, and asks the driver to CreateClientSession. When a lastSessionId is supplied, the resolver also loads the prior session chain’s transcript (loadPriorTranscript — ownership-checked, hidden/error rows skipped, capped at 30 newest turns / 8,000 chars / 5 chain legs, with a cycle guard) and frames it into the system prompt so the resumed model remembers the last leg. Strictly best-effort — any problem yields no hydration, never a failed start. On mint failure the just-created session is closed with CloseReason = 'Error' so nothing half-open leaks.

2. Connect. The browser resolves the client driver by the returned Provider key and calls Connect(config, micStream) — the host acquires the mic (it owns the permission UX); the driver owns the transport, applies SessionConfig verbatim once the control channel opens, and only then reports 'listening'.

3. Converse. Transcripts stream both ways; final normal turns become captions and are persisted via RelayRealtimeTranscript (a Conversation Detail stamped with AgentSessionID). Typed text rides SendText — which implies barge-in: the driver cancels any active spoken response (CancelActiveResponse, a floor-control action that never aborts server-side delegated work) before injecting the text, then routes the reply through the same collision-safe path tool results use.

4. Tools. Tool calls whose names match a registered client-tool prefix (e.g. Whiteboard_) execute locally; everything else relays to ExecuteRealtimeSessionTool, which reads the target from the session config (never from the client), threads a progress callback, nests the delegated run under the co-agent run, junction-links any artifacts the delegated run produced into the session’s conversation history (best-effort — when no session-stamped transcript turn exists yet to anchor to, a minimal hidden anchor Conversation Detail is created so the junction rows are never orphaned and no fake visible message appears), and rolls a pendingFeedbackRunID forward when an interactive target pauses AwaitingFeedback — so the user’s next answer resumes the same run (lastRunId + autoPopulateLastRunPayload).

5. Cancel. A still-working delegation card carries a ✕; clicking it calls VoiceSessionService.CancelDelegation(callId) → the CancelRealtimeSessionTool mutation → RealtimeClientSessionService.CancelInFlightDelegations, whose per-session in-flight registry aborts the matching AbortController(s); the original ExecuteRealtimeSessionTool mutation then resolves with the cancelled run’s outcome (the card was already flipped to “Cancelled by user”; the late duplicate result is suppressed). AbortedCount: 0 is a legitimate success — the work may have finished first. A callId-less sweep form (CancelInFlightDelegations()) exists for “stop everything” affordances. True barge-in never cancels delegations on this topology — it only cancels pending narration.

6. End. Teardown flushes pending channel saves and disposes plugins. It then stops the recorder (which flushes the encoder; bounded at 2 s), seals the last shards and starts the recording’s terminal request, before anything else that awaits, so a tab closed right after End still delivers the audio that was inside the encoder. Then it stops the mic and disconnects the driver (a Disconnect() that throws is logged and does not skip the close), flushes the pending usage-relay delta, and calls CloseAgentSession; SessionManager.CloseSession stamps CloseReason = 'Explicit', disconnects channel rows, and finalizes the co-agent observability runs. Only after the close does it wait for the terminal’s acknowledgement, at most 3 s (RECORDING_TERMINAL_ACK_TIMEOUT_MS), so the small, critical close never queues behind the recording. RealtimeSessionRuntime.SavingRecording$ / IsSavingRecording is true from the start of a recorded session’s teardown until that wait settles. There is no beforeunload prompt any more (#5294): the shards already on the server plus the keepalive terminal and re-sends below are what make closing the tab safe, and a crash loses at most the audio since the last 5 s tick.

   How a client-direct recording reaches the server (#5294). The browser is the only place a client-direct call can be recorded (the vendor connection never passes through MJAPI), so the recording is delivered during the call and the server builds the file:

   • Capture. RealtimeAudioRecorder (@memberjunction/ng-conversations) mixes the mic and the agent’s remote stream at 48 kHz (resampling when the browser refuses a 48 kHz AudioContext) and feeds one continuous Opus encoder for the whole call (mono, 32 kbps, 20 ms packets): WebCodecs AudioEncoder where it supports that config, else libopus WebAssembly in a Worker. The 48 kHz sample count is the recording’s clock; wall time is never used.
   • Shards. Every 5 s tick the runtime seals the packets encoded since the last tick into an MJOPUS1 shard (@memberjunction/ai-core-plus realtime-recording-shard-format.ts: header with index, StartSample, SampleCount, packet count, then length-prefixed Opus packets). The tick seals whatever it took as one shard; 500 packets (two 5 s windows, the format’s cap) and 48 KiB are caps that split a larger batch, so timer jitter never splits a tick into a full shard plus a sliver. Each shard is written to IndexedDB (mj-realtime-recording), tagged with the signed-in user who started the call, before its upload starts and deleted when the server acknowledges it or refuses it for good (400, 403, 409, 410, 413; a 401, 404, 5xx or network error keeps it for a retry, so a browser upgraded before its server keeps its audio), then sent as raw bytes to POST /realtime/recording/:agentSessionId/shard/:index (128 KiB cap). A failed upload is retried in memory (3 attempts); a shard the queue gives up on stays stored and is re-sent from IndexedDB as soon as an upload succeeds again during the call. Anything still unacknowledged is uploaded oldest-first on the next page load of any page that constructs RealtimeSessionRuntime on that origin, but only by a page signed in as the same user (at most 50 per pass, another pass when refused shards were deleted; 7-day retention). Explorer’s logout (MJAuthBase.clearClientCaches) calls ReleaseRecordingShardsAtLogout: it sends the user’s stored shards for up to 3 s, then deletes whatever of theirs is left, so a shared browser keeps no one’s call audio. IndexedDB failures are logged and never block the call.
   • Terminal. POST /realtime/recording/:agentSessionId/end?lastShardIndex=&capturedSamples= carries the current partial window (the tail, 48 KiB cap) as a fetch(..., { keepalive: true }) request. It is sent on End and, through RealtimeSessionService, on pagehide and visibilitychange→hidden (the latter with final=false: the call may go on); those handlers call the runtime’s synchronous SendRecordingTerminal(), which only copies already-encoded packets (no encoding, no IndexedDB). The terminal is dispatched first; then every shard the server has not acknowledged (on the wire, retained for a retry, or just sealed) is re-sent with keepalive, oldest first, within the 48 KiB the recording may use of the browser’s shared keepalive budget, because ordinary requests die with the page. IndexedDB writes come after. A sent tail seals its window, so an index is never re-sent with different bytes and a call that continues after the tab returns carries on at the next index; this also holds for a pagehide while End is still stopping the recorder.
   • Ledger. Both routes (packages/MJServer/src/rest/RealtimeRecordingHandler.ts) use the resolver’s owner gate plus the per-session scope check, and only write SQL: shards go into MJ: AI Agent Session Recording Shards (Data VARBINARY(MAX), unique on session + index) through guarded stored procedures. A repeat with the same SHA-256 acks; the same index with different bytes is refused, logged, and marks the session Failed. AIAgentSession tracks RecordingStatus (Pending | Assembling | Complete | Partial | Failed, NULL when not recorded), RecordingExpectedShards, RecordingCapturedSamples, RecordingGaps and RecordingAssemblyVersion. A second terminal with the same or higher counts is accepted (a higher one re-opens a finished recording); a lower one is ignored and logged. Requests are bounded by the session itself: an index at or past MAX_ASSEMBLY_SHARDS (2000), or a shard end or captured count longer than the session has existed (server-set creation to close, plus 2 minutes), gets 400 and nothing is stored; a session closed over 24 h ago answers 410.
   • Assembly. Assembly is claimed atomically (Pending/Partial → Assembling, version + 1) and RecordingAssemblyScheduler runs it off the request path. After a final terminal, with every shard present it assembles at once; with shards missing it looks again after 5 s (the browser’s keepalive re-sends follow the terminal), and if they are still missing it waits realtime.recordingAssemblyGraceSeconds (default 90) and then assembles Partial, filling each missing range with silence and listing it in RecordingGaps. AssembleRealtimeRecordingFromLedger (@memberjunction/ai-agents) feeds every packet in index order to one libopus decoder, writes recording.wav (PCM16, 48 kHz mono) and peaks.json through the existing storage path, and on Complete deletes the shard rows. A late shard for a Partial recording within 24 h of close re-runs assembly and replaces the file; after that the routes answer 410 and the file is left alone. Transient errors (SQL, storage, decoder) leave the claim for a retry; deterministic ones mark the session Failed. While the session is still open, nothing assembles sooner than the grace period except a final terminal: a hidden tab’s terminal and the shards that land after it would otherwise rewrite recording.wav on every tab switch. SessionManager.CloseSession schedules a recorded session’s assembly right away. Every assembly stamps its gaps before it overwrites recording.wav, and a re-assembly fills each missing-shard range the previous file covers from that file (a Complete deleted those rows), keeping the stored gaps as gaps. MJAIAgentSessionEntityServer.Save leaves the ledger-owned columns out of every server-side session save unless the caller set them, so a heartbeat, close or tool-call save cannot rewind them.
   • Backstop. The session janitor’s ledger sweep (RunRecordingLedgerSweep) assembles, as Partial, every Closed session 10 minutes past close (and within the 7-day lookback) that is still Pending, is Assembling with no write for recordingRecoveryAttemptTimeoutMinutes, or has shards but lost its status — the browser that never sent a terminal, or an assembler that died. It shares legacy recovery’s backoff, give-up cap and abandoned-attempt pause, and purges ledger rows past the lookback.
   • Older clients. The UploadRealtimeRecordingSegment / UploadRealtimeRecording mutations still work. A session that only has legacy seg-NNNN[.r<rate>].pcm storage shards and no RecordingFileID is still rebuilt by the janitor’s legacy sweep 10 minutes after close, concatenating the shards in order with missing indexes filled by silence and stamping recording-recovered-<epoch ms>-<random hex>.wav; that path is unchanged and never touches a session on the ledger.
   • Not covered. Server-bridged sessions record on the server (RealtimeRecordingController), and the mobile app’s RNRealtimeMediaHost has no recorder.

What an ephemeral token can lock — provider differences

All four providers keep prompt/tool authority server-side in the client-direct topology, but how much the minted credential itself enforces differs:

• OpenAI (OpenAIRealtime.CreateClientSession): the full realtime session config — instructions (system prompt), tools, input-transcription opt-in — is baked into the client secret at mint time via the client-secrets API. The browser’s session.update applies the same config; tampering buys nothing the secret didn’t already fix.
• Gemini (GeminiRealtime.CreateClientSession): the ephemeral-token API accepts only a mask-safe subset as liveConnectConstraints.config (GeminiRealtime.BuildConstraintConfig): responseModalities, speechConfig, temperature, topP, maxOutputTokens, sessionResumption — locked with lockAdditionalFields: []. systemInstruction, tools, and the transcription configs cannot be locked (their presence 400s the mint); they ride in SessionConfig and the matching client driver applies them verbatim. So on Gemini the token locks model + generation parameters, while prompt/tool integrity relies on the driver-applied config rather than a token-side lock. Token windows: ~10 minutes to open a new session, ~30 minutes total lifetime.
• ElevenLabs (ElevenLabsRealtime.CreateClientSession): the signed websocket URL is the credential (agent-scoped, ~15-minute open window; an already-open conversation continues past it). It pins the agent — and the agent’s server-side configuration carries the tool set (bound by the driver’s managed-agent ensure flow, so tools are effectively server-fixed). The per-session system prompt rides the conversation_initiation_client_data override in SessionConfig (the managed agent’s platform settings enable exactly that one field), applied by the client driver — so prompt integrity is driver-applied, tool integrity is agent-side.
• AssemblyAI (AssemblyAIRealtime.CreateClientSession): a one-time temp token (GET /v1/token, 300-second connect window, single websocket open). The token locks nothing but auth; the entire session object (prompt, tools, voice, turn detection) rides SessionConfig and is applied by the client driver as the first session.update frame — the provider confirms with session.ready before the driver reports 'listening'.
• Gemini Enterprise (GeminiEnterpriseRealtime.CreateClientSession): no token at all. Gemini Enterprise has no browser-safe Live credential, so the mint issues a relay session (Transport: 'relay', RelayUrl, an empty EphemeralToken): MJAPI holds the Google credential, sends the setup the server wrote (model, prompt, tools, voice, avatar) on every upstream connection, and forwards only realtime input, user turns and tool responses from the browser. The browser can’t change the prompt or the tools at all; from its own setup the relay reads only a resumption handle and a request for audio only. See §12.

Server-bridged flow (wired, awaiting a media plane)

BaseAgent.executeRealtimeSession (in packages/AI/Agents/src/base-agent.ts) is fully implemented: it resolves the model, creates one long-lived AIPromptRun for the session (usage is checkpointed incrementally onto it — crash-safe by design), builds RealtimeSessionRunnerDeps (delegation via ExecuteSubAgent, actions via ExecuteSingleAction, transcript persistence, debounced usage checkpoints — default 5 s, plus the DB-driven NarrationInstructionsTemplate resolved through the shared realtime-narration.ts lookup), and drives RealtimeSessionRunner.Run(). The runner passes the tool set via StartSession params (the canonical registration path — for connect-bound providers like Gemini Live, the only effective one) and wires OnTranscript / OnToolCall / OnUsage / OnInterruption (barge-in aborts the in-flight delegated run via RealtimeToolBroker.AbortInFlight and cancels pending narration) / OnError (a Fatal error finalizes the session via Stop() instead of idling on a dead socket). The runner also consumes delegation progress itself: each significant event is fed to SendContextNote?.() and paced into RequestSpokenUpdate?.() spoken digests (see §7) — both feature-detected, since they are optional capabilities.

What is not wired yet for server-bridged: a client media transport (browser audio in/out of the server socket — IRealtimeSession.SendInput / OnOutput have no client-facing pipe). The client-direct topology is the path Explorer voice uses today.

---

4. Session Lifecycle

A realtime session is a long-lived durable record backed by volatile in-memory resources (sockets, the mic, channel plugins). The lifecycle machinery keeps the durable state from drifting away from process reality.

Schema (migration V202606090930__v5.41.x__AI_Agent_Sessions_Channels.sql)

• AIAgentChannel (MJ: AI Agent Channels) — the pluggable channel-definition registry: Name (unique), ServerPluginClass, ClientPluginClass, TransportType (PubSub | WebRTC | WebSocket), ConfigSchema, IsActive. Reference data, seeded via metadata/ai-agent-channels/.
• AIAgentSession (MJ: AI Agent Sessions) — the long-lived session: AgentID (the co-agent), UserID, Status (Active | Idle | Closed), CloseReason, ConversationID, LastSessionID (resume chaining), HostInstanceID, Config (free-form JSON — authoritative targetAgentID, observability run ids, pendingFeedbackRunID), LastActiveAt, ClosedAt.
• AIAgentSessionChannel (MJ: AI Agent Session Channels) — one row per channel attached to a session: Status (Connecting | Connected | Paused | Disconnected), Config (the channel’s state of record, e.g. the serialized whiteboard), LastActiveAt, DisconnectedAt; unique per (AgentSessionID, ChannelID).
• Additive nullable FKs: AIAgentRun.AgentSessionID, ConversationDetail.AgentSessionID (the timeline/grouping keys), AIAgent.DefaultCoAgentID, AIAgentType.DefaultCoAgentID.

The column is named AgentSessionID everywhere — deliberately distinct from the per-connection GraphQL transport sessionID (a browser-generated correlation id). They are orthogonal and never merged.

SessionManager (packages/MJServer/src/agentSessions/SessionManager.ts)

Stateless per-request lifecycle manager (every method takes contextUser + request-scoped IMetadataProvider — never the global):

• CreateSession — CanRun authorization first (denial throws SessionAuthorizationError before any write), then resolve-or-create the Conversation, then persist the session Active, stamped with this host’s HostInstanceID .
• CloseSession — terminal and idempotent: Status = 'Closed', ClosedAt, CloseReason; disconnects all channel rows; finalizes the co-agent observability runs stored in the session config. An already-closed session is a no-op that keeps its original CloseReason.
• Heartbeat — coalesced: at most one LastActiveAt write per session per 3 s (HEARTBEAT_MIN_WRITE_INTERVAL_MS), so chatty channels never become a DB write storm. Reactivates Idle → Active; never reactivates Closed. Tool relays and transcript relays heartbeat implicitly.
• MarkIdle — Active → Idle when the last channel goes quiet.

CloseReason (SessionCloseReason) is the close-provenance enum the dashboards key off: 'Explicit' (user hang-up — the default), 'Janitor' (orphan reconciliation), 'Shutdown' (graceful host drain), 'Error' (failure-path teardown, e.g. a failed mint). NULL means legacy/pre-column rows.

The janitor (SessionJanitor) and host identity

HostInstance.ts gives each server process a stable identity hostname:pid:bootId (the bootId UUID is minted once at module load, so two boots of the same host are distinguishable). SessionJanitor is a BaseSingleton + IShutdownable that runs three reconciliation paths, all writing through SessionManager.CloseSession (so Record Changes captures transitions and channel rows disconnect consistently), all paging with keyset (AfterKey) pagination (200 rows/page):

1. RunStartupRecovery (once at boot) — closes Active/Idle sessions whose HostInstanceID matches this host’s hostname: prefix but a different boot — crash/redeploy orphans. CloseReason = 'Janitor'.
2. RunStalenessSweep (every 60 s by default) — closes any Active/Idle session with LastActiveAt older than 15 minutes (closeThresholdMinutes), regardless of host — catches owners that died without a clean reboot. Idempotent and safe to run concurrently on every instance.
3. RunShutdownDrain (from Shutdown() during the ShutdownRegistry drain) — closes this exact host instance’s own live sessions with CloseReason = 'Shutdown', so a clean redeploy never strands rows for the next boot’s janitor.

Continuity: lastSessionId + channel-state restore

A closed session is never reopened; resuming creates a new session chained via LastSessionID (mirroring AIAgentRun.LastRunID). The conversation is the durable thread; sessions are time-boxed episodes along it.

Continuity is real, not cosmetic — a lastSessionId on StartRealtimeClientSession triggers two restore paths:

• Channel state: loadPriorChannelStatesJson() — the prior session’s AIAgentSessionChannel.Config blobs come back as PriorChannelStatesJson (a { channelName: stateJson } map), and VoiceSessionService.applyPriorChannelStates() offers each entry to the matching plugin via BaseRealtimeChannelClient.RestoreState. The whiteboard you drew last call reappears in the next one.
• Model memory: loadPriorTranscript() walks the prior session chain (up to 5 legs, cycle-guarded) and frames the newest 30 turns / 8,000 chars of persisted captions into the new session’s system prompt, so the model remembers what was said on the previous leg(s) — not just what was drawn.

Both are strictly best-effort and ownership-checked: a prior session owned by another user, an oversized state (> 2,000,000 chars per state and in total), or a malformed payload is logged and skipped — a session start never fails because of restore.

Co-agent pairing & effective configuration

Migration V202606121100__v5.41.x__Realtime_CoAgent_Pairing_And_TypeConfig.sql adds two opt-in metadata layers on top of the co-agent resolution chain.

Pairing semantics — universal vs constrained

AIAgentPairedAgent (MJ: AI Agent Paired Agents) is an OPT-IN junction between a Realtime-type co-agent and the targets it can front (CoAgentID, TargetAgentID, IsDefault, Sequence). Pairings are never mandated:

Co-agent state	Runtime targetAgentId supplied	Behavior
Zero pairing rows (universal)	yes	Today’s behavior, untouched — fronts the supplied agent. Zero-config deployments need zero metadata.
Zero pairing rows	no	Clear error — a universal co-agent has no default target to fall back to.
Has pairing rows (constrained)	yes, in the list	Accepted (canonical row casing wins).
Has pairing rows	yes, NOT in the list	Clear structured error — the rows RESTRICT the co-agent to its prebuilt target list.
Has pairing rows	no	The IsDefault row stands in; no default row → clear error asking for an explicit target.

Resolution lives in RealtimeClientSessionResolver.resolveConstrainedTargetAgentID() (rows loaded by Sequence). Pairing is a targeting constraint, not a security boundary — CanRun on the resolved target remains the security gate, applied immediately after; a failed pairing query therefore degrades to the universal behavior (logged) rather than breaking calls. Server-side write invariants (MJAIAgentPairedAgentEntityServer.ValidateAsync in MJCoreEntitiesServer): the co-agent must be an Active agent of the Realtime type, at most one IsDefault = 1 row per co-agent, and CoAgentID ≠ TargetAgentID (defense in depth over the CHECK constraint).

The effective-configuration merge

Realtime behavior is configured in three JSON layers, deep-merged per key (later wins; plain objects merge, arrays/primitives replace):

AIAgentType.DefaultConfiguration  ←  AIAgent.TypeConfiguration  ←  runtime overrides (configOverridesJson)
(type-level defaults)                 (per-agent layer)              (per-session, authorization-gated)

The pure implementation is packages/AI/Agents/src/realtime/realtime-coagent-config.ts (DeepMergeConfigs, ParseRealtimeTypeConfiguration — tolerant, malformed layers contribute nothing — and ResolveEffectiveRealtimeConfig, which normalizes into the typed RealtimeCoAgentConfig). The canonical shape (the Realtime type’s ConfigSchema is seeded to it):

{ "realtime": {
    "modelPreference": "<AI Model name or ID>",
    "voice": { "default": { "tone": "…", "speakingStyle": "…", "voice": "<voice id>",
                            "firstMessage": "<what the agent says first, verbatim>" },
               "providers": { "openai": { "voice": "alloy" }, "elevenlabs": { "voice": "<voice id>" },
                              "assemblyai": { "voice": "…" } } },
    "allowUserModelOverride": true,
    "narration": { "paceMs": 8000 } } }

How each knob is applied at mint (RealtimeClientSessionService.PrepareClientSession, mirrored on the server-bridged path in BaseAgent):

• modelPreference (Name or ID) participates in realtime-model selection between the explicit runtime choice and the default: explicit preferredModelId (strict, fails loud, authorization-gated when deviating) → configured preference (tolerant — an unsatisfiable metadata preference logs and falls through, mirroring the chain’s metadata steps) → highest-PowerRank default. With realtime.video.enabled, the default step first looks for a model that shows an avatar (§12); both explicit choices win over that preference.
• voice.default (tone / speakingStyle) is appended to the server-built companion system prompt as a short “Voice & manner” section, right after the co-agent’s own prompt. Note these two are PROMPT-level; their siblings voice.default.voice and voice.default.firstMessage below are WIRE-level and are never folded into the prompt.
• voice.default.voice is the provider-agnostic voice id (#3530) — authored without naming a vendor and filed onto whichever driver the framework resolves, which is what lets a host carry a voice at all when it does not know (and on the default-model path cannot know) the vendor in advance. It wins the voice key over a matching voice.providers.<key> entry, while that bag still contributes its other settings. The value is still provider-native, so an alloy means nothing to ElevenLabs — and on Gemini a wrong id is now reachable, where it passes the token mint and then kills the session as the socket opens (close 1007, No matching speaker voice found) rather than degrading quietly: author this OR per-vendor pins, not both. Reach today is agent metadata plus programmatic/bridge hosts — see the picker caveat in the open-items table.
• voice.default.firstMessage is the provider-agnostic opening utterance (#3557) — what the agent says FIRST, verbatim, before the user has spoken. It resolves exactly like the agnostic voice (filed onto whichever driver the framework picks, winning the firstMessage key over a matching voice.providers.<key> entry). Authored here rather than in the persona prompt because conversation-start behaviour is not instruction-following: ElevenLabs with no first_message produces no audio at all until it receives user audio, no matter how the prompt is worded — which is why a co-agent could never greet a caller. Omit it and the agent waits for the user, as before. Honoured today by ElevenLabs (agent.first_message override) and AssemblyAI (greeting). Every other driver scrubs it and opens silently: it is registered in REALTIME_SHARED_CONFIG_KEYS, which is what makes a neutral key safe to file onto a vendor that has no use for it — unregistered, it survives each driver’s residual-bag spread and reaches the provider as an unknown session field.
• voice.providers.<provider> is matched to the resolved vendor’s DriverClass by normalized prefix (openai ↔ OpenAIRealtime, …) and merged into the session’s open Config bag — the same opaque per-driver pact every other config entry rides (a caller-supplied runtime Config wins per key). Server-bridged reach today: OpenAI spreads the bag into session.update; Gemini merges it last, consuming the neutral voice first and mapping it to speechConfig.voiceConfig.prebuiltVoiceConfig.voiceName (its LiveConnectConfig has no voice field, and the SDK’s config converter is a path allowlist that would drop it silently — issue #3721); AssemblyAI maps voice → output.voice; ElevenLabs maps voice → the tts.voice_id entry of its conversation_config_override (on BOTH topologies — its managed agent enables the tts.voiceId override so each session can carry its own voice). Client-direct reach: OpenAIRealtime.CreateClientSession does fold params.Config into the minted session — ExtractRealtimeFeatures pulls the neutral voice out of the bag and BuildAudioBlock maps it to audio.output.voice, the same block used on the server-bridged path. The resolver additionally surfaces the full resolved config as EffectiveConfigJson (and the resolved vendor as DriverClass) on StartRealtimeClientSessionResult, so client drivers can apply provider settings browser-side and can report which vendor actually spoke.
• narration.paceMs drives spoken-progress spacing: server-bridged, it feeds RealtimeSessionRunnerDeps.NarrationPaceMs (replacing the built-in 8 s spacing floor); client-direct, pacing is enforced client-side, so the value is surfaced as NarrationPaceMs on the start result.

Runtime override precedence & authorization (configOverridesJson on StartRealtimeClientSession):

Request	Gate	Outcome
Plain start (no overrides, no explicit model)	CanRun on the target only	Never touches the new gate.
Target selection (within a pairing list / universal) and coAgentId choice	CanRun only	Normal user flow — deliberately ungated.
configOverridesJson	Realtime: Advanced Session Controls authorization (hierarchy-aware, fail-closed when the seed is absent)	Unauthorized → structured rejection (never a silent ignore). Must be a JSON object.
preferredModelId equal to the metadata modelPreference	none (not a deviation)	Allowed for everyone.
preferredModelId deviating from metadata	the authorization AND effective allowUserModelOverride ≠ false (the policy blocks even authorized callers)	Denial is a structured rejection naming the reason.

Gate implementation: RealtimeClientSessionResolver.assertRuntimeOverridesAuthorized() → the pure EvaluateRuntimeOverrideAuthorization() decision (unit-tested matrix) + AuthorizationEvaluator.UserCanExecuteWithAncestors over the request provider’s cached Authorizations. The authorization is seeded in metadata/authorizations/.realtime.json.

Write-side schema enforcement: when an agent type publishes ConfigSchema, MJAIAgentEntityServer.ValidateAsync (in MJCoreEntitiesServer) validates TypeConfiguration against it with a dependency-free JSON-Schema-subset validator (json-schema-lite.ts: type / required / properties / enum / items / additionalProperties). Non-object configuration always fails; a malformed ConfigSchema only WARNS (a metadata bug on the type row must not brick agent saves).

App awareness — the app cascade layer, mint-time context, and multi-agent delegation

The co-agent is app-aware and can voice one lead while delegating to many colleagues. Three additive capabilities, all flowing through the one shared prep (RealtimeClientSessionService.PrepareRealtimeSessionParams) so client-direct and server-bridged behave identically:

1. The app cascade layer. When the session knows its ApplicationID (threaded via PrepareClientSessionInput.ApplicationID — the StartRealtimeClientSession applicationId arg client-direct, or params.data.applicationId server-bridged), the app’s Application.AgentSettings contributes a NEW cascade layer between the target agent and the runtime override:

AIAgentType.DefaultConfiguration  <  co-agent TypeConfiguration  <  target TypeConfiguration  <  APP (Application.AgentSettings.Realtime)  <  runtime override

AgentSettings.Realtime (persona / disclosure / model preference) + AgentSettings.RelevantAgents are mapped to the canonical {"realtime":{…}} shape by BuildAppRealtimeOverridesJson and merged via the extended ResolveEffectiveRealtimeConfig(typeDefault, agent, override, target?, appSettings?, dynamic?). Scalars (persona/disclosure/model) follow the per-key cascade; allowedAgents is the deliberate exception — see below.

2. Mint-time app context. PrepareClientSessionInput.AppContext (an AppContextSnapshot from @memberjunction/ai-core-plus — where the user is, what they see, the live capability manifest) is rendered by FormatAppContextNote into a CURRENT APP CONTEXT section of the companion system prompt, so the co-agent knows the user’s situation from its first word (the session-start half of client-context delivery). Client-direct passes it as the appContextJson arg; server-bridged as params.data.appContext.

2b. Continuous streaming + acting — the ClientContextChannel. The mid-session half is a headless interactive channel (ClientContextChannel, IsHeadless: 1, never mounts a tab — server half ClientContextChannelServer in @memberjunction/ai-agents, browser half in @memberjunction/ng-conversations). It does two things, both client-side in the client-direct topology (no server round-trip):

• Perception (stream). It subscribes to the host’s live app-context stream (RealtimeChannelContext.AppContext$, fed by Explorer’s MJExplorerAppComponent from NavigationService — every SetAgentContext / nav change re-publishes the AppContextSnapshot, now including a Capabilities.Tools manifest of what’s callable right now) and pushes each change as a compact SendContextNote. So as the user navigates, the co-agent’s picture stays current.
• Action — the ContextTool proxy. It declares ONE stable provider tool, ContextTool, so the tool surface never changes mid-session (connect-bound providers reject re-declaration); the live catalog of valid actions rides context. The model calls ContextTool({ action, params }); the channel routes it to RealtimeChannelContext.ExecuteClientTool, which runs the surface handler the host registered from SetAgentClientTools (mirrored into the realtime session via RealtimeSessionService.RegisterAppClientTools), and returns a structured result. This is what lets the co-agent navigate / show / do things in the app on the user’s behalf, executed in the browser. A realtime tool result is a string, so Explorer removes images (Media) from these results (WithoutMedia in explorer-app.component.ts). A surface sends an image to the voice model as a video frame, with SendVideoFrame.

The seam is boundary-clean: the generic channel never imports Explorer — it is fed AppContext$ + ExecuteClientTool through the (optional) RealtimeChannelContext members, which the RealtimeSessionService implements and Explorer drives. The same SetAgentContext/SetAgentClientTools call a dashboard already makes for the async chat agent now feeds the realtime co-agent too — one surface API, both agents.

3. Multi-agent delegation (lead + colleagues). The effective realtime.allowedAgents is a union-accumulated set (NOT array-replaced) across every cascade layer (accumulateAllowedAgents) plus the app’s RelevantAgents plus runtime/channel-registered additions, deduped by agentId (later layers win per-field). The lead’s identity stays single (the voiced target / app default agent); the union is who the lead can call. At mint, the colleagues are rendered into the framing (BuildRealtimeAgentFraming(leadName, surfaceClause, colleagues) → BuildColleaguesClause) with per-target disclosure guidance. The model names a colleague in the invoke-target-agent arguments ({ agent, request }); RealtimeClientSessionService.resolveDelegationTarget validates the name against the session’s persisted allowed union (client-direct: persisted on AIAgentSession.Config_ and read back per relayed call; server-bridged: from the live EffectiveConfig), routes to that agent, or returns a structured “not available” result naming the valid colleagues so the model self-corrects. CanRun on the resolved agent remains the security boundary — the union is an affordance filter, both apply.

Delegation disclosure (RealtimeDisclosurePolicy = 'silent' | 'mention' | 'hand-voice', default 'mention') — how the co-agent narrates a handoff: mention names it (“let me get Skip on this”), silent absorbs the result and speaks it as its own, hand-voice is reserved. Resolved per-target by GetDisclosureForTarget (per-entry override → effective default from the scalar cascade). Configurable at the type default, co-agent, app, and per-target levels.

Live session capabilities — RealtimeSessionCapabilities + Reconfigure

A live IRealtimeSession exposes a small capability surface (the realtime-session analogue of the bridge’s IBridgeProviderFeatures and of BaseRealtimeModel.SupportsClientDirect) so the container can ask “is it safe to call X?” instead of blind-invoking optional methods that no-op — or can’t be supported — on some providers. Both members are optional on the interface (@memberjunction/ai, baseRealtime.ts), so a driver that hasn’t declared them is treated conservatively (unsupported), and the 6 existing drivers compile unchanged:

• Capabilities?: RealtimeSessionCapabilities — currently { CanReconfigureTurnMode }: whether the session can change its turn-taking / auto-response mode on a live socket without reconnecting. Grow this object as providers gain runtime abilities.
• Reconfigure?(params: RealtimeReconfigureParams): void — applies a live change (e.g. { DisableAutoResponse: true }). Gate on the capability before calling. OpenAI implements it via a partial session.update and reports CanReconfigureTurnMode: true; Gemini Live’s activity detection is fixed at connect, so it reports false and omits the method.

Prime consumer: the bridge engine’s first-agent re-gating (AIBridgeEngine.ReconfigureSessionToMeeting → meeting mode when a room becomes multi-agent — see the Bridges guide §9). As models gain mid-session reconfiguration, a driver flips one flag and the container starts using it with zero container changes.

Turn moderator (multi-agent turn-taking)

When two or more agents share one realtime room (a LiveKit panel, a meeting), the hard question is who speaks next. The original answer was a per-agent regex name-match (RegexAddressedMatcher): each agent independently checked “was I addressed?” That routes direct address but nothing else — it can’t bring a relevant-but-unaddressed agent in, can’t let two agents have a productive back-and-forth, and can’t tell a useful exchange from an unproductive ping-pong loop. The turn moderator replaces it with a room-level decision.

Once per turn, a single fast LLM looks at the room roster + the recent diarized conversation and returns the ordered agent(s) who should speak next — zero, one, or several. It routes direct address and relevance (send a question to Sage and Skip when both matter), lets a productive agent↔agent discussion continue, and goes quiet on unproductive ping-pong or when nobody should speak (hand back to the human). The engine speaks the returned agents serially via the floor — never overlapping. The audio plane is untouched: every agent always hears the raw room audio; the moderator only decides when each agent is triggered to commit and speak. (It’s called a moderator, not a judge, because it brings agents in as much as it restrains them — the informal nickname is “nanny mode,” but that undersells the half that matters.)

Prompt, not agent

The moderator is RealtimeTurnModerator (packages/AI/Agents/src/realtime/realtime-turn-moderator.ts; exported as RealtimeTurnModeratorDecision, the (ctx) => Promise<string[]> function wired into the bridge engine). It is a stateless yes/who classification — no planning, tools, or sub-agents — so it runs as an AIPromptRun via AIPromptRunner, not an agent run. Each decision’s agentRunId is tied to the co-agent’s AIAgentRun, so the per-turn “who spoke and why” trail (including the moderator’s structured reason per speaker and an optional note) is fully observable through standard prompt-run logs — and those logs are exactly the data for deciding when this mechanism can be relaxed as realtime models get smarter. The prompt returns structured { speakers: [{ agent, reason }], note }; the plugin maps the returned names back to roster AgentSessionIDs (in order, de-duplicated).

Config cascade + the turnTaking shape

Two parts of the effective-config cascade carry turn-taking, and they live at different layers:

• Room-wide moderator brain — the turnTaking.moderator block on the Realtime agent type’s DefaultConfiguration (metadata/agent-types/schemas/realtime-type-default-config.json). A room has exactly one moderator.
• Per-agent participation mode — turnTaking.mode on the target agent’s TypeConfiguration: 'proactive' (default — may be brought in unaddressed when the moderator judges it relevant) or 'addressed-only' (speaks only when directly addressed by name).

Both ride the existing cascade (agent-type DefaultConfiguration ← co-agent TypeConfiguration ← target agent TypeConfiguration ← runtime override — see §4 the effective-configuration merge). The shape (in packages/AI/Agents/src/realtime/realtime-coagent-config.ts):

{ "realtime": { "turnTaking": {
    "mode": "proactive",                 // per TARGET agent (TypeConfiguration)
    "moderator": {                       // room-wide (agent-type DefaultConfiguration)
        "promptId": "<AI Prompt ID>",    // authored as @lookup:, stored as the resolved ID
        "contextWindowTurns": 30,        // diarized turns the moderator sees; clamped ≤ 50
        "maxCharsPerTurn": 240,          // per-turn clip (token savings + cacheable prefix)
        "maxConsecutiveAgentOnlyTurns": null,  // null = no cap (trust the model's progress read)
        "timeoutMs": 800,                // per-decision budget on the latency-critical path
        "onError": "silent",             // 'silent' (no one speaks) | 'addressed-only' (cheap fallback)
        "prestageOnAgentSpeech": true    // run the next decision during the prior agent's playback
} } } }

GetEffectiveModeratorConfig / GetEffectiveTurnMode are the typed accessors; absent fields fall back to REALTIME_MODERATOR_DEFAULTS. GetEffectiveModeratorConfig returns null when no promptId is configured — which the engine reads as “no moderator, fall back to the matcher.”

Serialized multi-route + pre-staging

The moderator can name several speakers; the engine queues them and drains the queue one at a time through the room floor, so a multi-agent route is heard as a clean sequence, never a pile-up. The moderator also runs on agent turns, not just human ones — that powers both agent↔agent continuation and pre-staging: when prestageOnAgentSpeech is on (the default), the next decision runs during the prior agent’s audio playback (the model emits its full response text seconds before the user finishes hearing it), so the agent→agent hand-off pays ~zero added latency. A human barge-in discards the pre-staged decision (the user changed the topic). The optional maxConsecutiveAgentOnlyTurns is a hard backstop against runaway agent-only loops; left null, the room relies on the moderator’s own progress assessment so a genuine discussion is never gated by a counter.

The model — small, fast, swappable

A small/fast LLM is the right call because this runs every turn on the latency-critical path. The seeded prompt ("Realtime: Turn Moderator", metadata/prompts/.realtime-prompts.json) is bound to GPT-OSS-120B on Cerebras. The structured output, plus putting the static roster near the top of the prompt and the variable conversation at the bottom, maximizes prefill caching. The whole mechanism is metadata-driven: swap the model binding (e.g. to Gemma on Cerebras) or retune the windows entirely in metadata, no code change — and the prestageOnAgentSpeech design means a slower model still hides most of its latency.

No-moderator fallback

When no moderator is injected into the engine — or no promptId is configured — the engine falls back to the per-agent RegexAddressedMatcher broadcast (still present): the room’s utterance is broadcast to every agent’s TurnPolicy and each independently decides “was I addressed?” The plugin itself also degrades to a crude name-contains safety net only when no prompt is configured, so a room is never left mute by a metadata gap. The engine seam (SetTurnModerator, the lookback buffer, the serialized queue, barge-in invalidation) is documented in the Bridges guide §9.

---

5. Channels — The Heart of the System

Building, wrapping, scoping or publishing a channel? The Realtime Channels Guide is the how-to: the self-describing channel contract, exposure and ReturnsChannelData, wrapping an existing Angular component with AngularComponentChannel, scoping, visual perception, publishing from an Open App, and testing. This section is the architecture those build on.

A channel is more than transport. An interactive channel is a bidirectional surface the session’s single realtime agent both perceives and acts upon. One agent per session is a hard invariant — everything every channel observes feeds that one agent, and every channel mutation comes from it.

Every interactive channel implements two directions:

1. Perception (channel state → agent context). The channel serializes compact, coalesced deltas of its state and feeds them into the live model as background context notes — replace-current-state semantics, never full snapshots, never bitmaps.
2. Action (agent → channel mutation). The channel exposes mutation tools the model can call; they execute and the result feeds back as the tool_response the model narrates.

The definition registry

MJ: AI Agent Channels rows define what channels exist (see §4 schema). At session start VoiceSessionService reads the active rows and instantiates one plugin per row via the ClassFactory (ClientPluginClass key) — one fresh instance per session, never a singleton. Registry failures degrade to “no channels”; the voice session always proceeds. The server half mirrors this exactly: when SessionManager.CreateSession mints the durable session, RealtimeChannelServerHost reads the same active rows and resolves each ServerPluginClass into one fresh BaseRealtimeChannelServer instance per session (unregistered keys skip with a log, never fatal).

The shipped channels: Whiteboard, Media, Remote Browser (all surface-bearing — they mount a tab), and ClientContextChannel (the headless app-context wire — IsHeadless: 1, no tab; perception + the ContextTool proxy, see § App awareness 2b). The registry row carries extra presentation/behavior metadata: IsHeadless (skip surface-mounting) and a UIConfig JSONType bag (DisplayName / GroupName / Color / Icon for surface channels’ tab chrome).

Per-session state of record

Each attached channel gets an AIAgentSessionChannel row. Its Config column is the channel’s state of record — written by the SaveSessionChannelState mutation through a client-side debounce (3 s, latest-state-wins, flushed at teardown with the session id captured while live so the final save lands even on the just-closed session). The model’s perception feed is derived from this state, never the source of truth. A Closed session still accepts state saves (the final flush legitimately lands after CloseAgentSession). Before persistence, each landed save routes through the channel’s server plugin (when one resolved — see below), which may validate/normalize the payload; a plugin can transform a save but can never lose or block one.

The BaseRealtimeChannelClient contract

A concrete plugin contributes everything its channel needs, so the host shell carries zero channel-specific wiring:

Member	Role
ChannelName	Must match the registry row’s Name; the persistence + restore key and the tab key
ToolNamePrefix	One prefix-routed local executor per plugin — Whiteboard_* calls run in the browser, never the server relay
GetToolDefinitions()	The client-executed tool declarations, aggregated into the session mint (clientToolsJson)
ApplyAgentTool(toolName, argsJson)	Executes one tool locally; MUST work with no surface bound (apply to the state engine, skip the UI garnish) — calls can arrive while the tab pane is collapsed
GetSurfaceComponent() / BindSurface() / UnbindSurface()	The Angular surface the overlay creates dynamically in a channel tab; the plugin wires the component’s inputs/outputs — the host treats the instance as opaque
SerializeState() / RestoreState()	The state of record + resume rehydration (tolerant by contract: bad input returns false, never throws)
RequestFocusExit()	Routed from the floating call pill back to whichever channel holds focus
Initialize(ctx) / Dispose()	Lifecycle bracket; Context is the plugin’s only line to the session

The host context (RealtimeChannelContext) supplies: SendContextNote (the perception feed), RequestSave (debounced state-of-record persistence), SaveAsArtifact (snapshot to a first-class versioned MJ: Artifacts record), SetFocusMode (the focus-layout request), and AgentName.

The server half — BaseRealtimeChannelServer

The same registry row’s ServerPluginClass resolves a server-side plugin (BaseRealtimeChannelServer in @memberjunction/ai, beside BaseRealtimeModel), hosted per session by the RealtimeChannelServerHost singleton in @memberjunction/ai-agents. Because today’s shipped topology is client-direct (channel tools execute in the browser), the server half’s contract is the durable lifecycle, not socket ownership — a pragmatic adaptation of the plan’s IAgentChannelServer (see the base class doc header for the deviation notes; socket/transport members are deferred with the unified-transport track):

Member	Role
ChannelName	Must match the registry row’s Name (the host routes by the row name and warns on a mismatch)
Initialize(ctx) / Dispose()	Lifecycle bracket; ctx is plain session facts (AgentSessionID, AgentID, UserID, ConversationID)
OnSessionStarted()	Fired right after the durable session row persists (SessionManager.CreateSession)
OnChannelStateSave(stateJson)	Fired when the client’s debounced state save lands, pre-persistence; return a replacement string to normalize the persisted state of record, or null to keep the original. Throw/garbage → original persisted — a plugin can never lose or block a save
OnSessionClosed(closeReason)	Fired from every close provenance — explicit hang-up, janitor sweeps, shutdown drain, error teardown all funnel through SessionManager.CloseSession
ToolNamePrefix	The shared prefix of this channel’s server-executed tools (e.g. MeetingControls_); the host routes any prefixed tool call back to this plugin. Default '' (no server tools)
GetServerToolDefinitions()	This channel’s server-executed tool declarations — may be runtime-computed (per session / per platform state), not only constants. Default []. The host aggregates these across a session’s live plugins (RealtimeChannelServerHost.GetSessionServerTools) and feeds them into RealtimeSessionRunner.ServerChannelTools
ExecuteServerTool(toolName, argsJson)	Executes one server-side tool; routed by ToolNamePrefix. Default returns a structured “not implemented” error (never throws) — the host additionally wraps any throw

Hard rules baked into the host: one fresh instance per session (never a singleton); every hook is wrapped in try/catch (a throwing plugin is logged and the session/persistence proceeds untouched); resolution is skip-with-log for unregistered keys; and disposal is deferred briefly after close (15 s linger) so the client’s legitimate post-close final state flush still routes through OnChannelStateSave.

The reference implementation is WhiteboardChannelServer (packages/AI/Agents/src/realtime/whiteboard-channel-server.ts, key 'WhiteboardChannelServer' — the value the Whiteboard row’s ServerPluginClass has carried since the seed): it guards the persisted state of record by validating that each landed save parses as a JSON object (flagging corrupt payloads loudly now instead of at the next resume’s RestoreState) and canonicalizing valid payloads to compact JSON. Register a new server half with @RegisterClass(BaseRealtimeChannelServer, '<key>') plus a Load...Server() no-op called from a static server path (the Whiteboard’s is invoked from MJServer’s agentSessions/index.ts).

Server-executed channel tools (Phase 2 — now wired). A server-side channel can contribute a dynamic server-executed tool vocabulary (GetServerToolDefinitions, possibly runtime-computed) plus its execution (ExecuteServerTool), namespaced by ToolNamePrefix. The RealtimeChannelServerHost aggregates every live plugin’s tools for a session (GetSessionServerTools) and routes each prefixed call back to its owner (ExecuteSessionServerTool, longest-prefix-wins). The server-bridged RealtimeSessionRunner accepts these via the new additive ServerChannelTools dep + an ExecuteServerChannelTool handler — they are registered alongside (and after) ExtraTools, and a contributed tool routes to the channel handler before the generic executor. This is purely additive: the client-direct path and runs without channels are unchanged. The reference server-only channel is MeetingControlsChannel (roster / hand-raise queue / who’s-speaking / timer → facilitator; see the Realtime Bridges guide).

Optional client surface (Phase 2). A server-only channel has no MJ client surface — BaseRealtimeChannelClient.GetSurfaceComponent() now defaults to null (HasSurface() is false), and the overlay skips a surfaceless channel’s tab while still wiring its tools + perception. A platform-native surface (e.g. a Zoom whiteboard) lives on the platform, not in MJ.

Still deferred: socket/media members (OnClientMessage, SendToClient) remain deferred with the unified-transport track.

The implicit voice/text channel

Voice itself is not a registry row — it is the session: the provider socket, live captions, the typed-input composer, and the persisted transcript. Concretely:

• Every final, normal transcript turn (both roles) becomes a caption and a Conversation Detail stamped with AgentSessionID (RelayRealtimeTranscript → persistTranscriptTurn; roles map user → 'User', everything else → 'AI'). That is the durable transcript — raw audio is not persisted.
• Typed text (VoiceSessionService.SendText) is injected as a user turn through the same collision-safe response path tool results use, then rides the same caption + relay path as speech — a typed turn behaves identically to a spoken one.
• Narration turns (Kind: 'narration') are deliberately ephemeral — never captions, never persisted (see §7).

The Whiteboard — canonical interactive channel

The whiteboard now lives in two layers, and the split is the point:

• @memberjunction/ng-whiteboard (packages/Angular/Generic/whiteboard/ — README) is the generic, reusable board: the Angular-free WhiteboardState engine, the Whiteboard_* tool API, the host/board/toolbar/zoom/popover/snapshot components, exports, the widget input bridge, and the right-click context menu. It has no MJ-metadata, AI-framework, or Router dependency — any Angular app can use it. The README there is the authoritative whiteboard reference.
• RealtimeWhiteboardChannel (packages/Angular/Generic/conversations/src/lib/components/realtime/whiteboard/whiteboard-channel.ts, registered as 'RealtimeWhiteboardChannel', seeded registry row Name: 'Whiteboard') is the thin channel plugin (~200 lines) that wires that package into a realtime session: it owns one WhiteboardState per session, declares WHITEBOARD_TOOL_DEFINITIONS to the model, routes Whiteboard_* calls to the bound RealtimeWhiteboardHostComponent (UI garnish) or the pure ApplyWhiteboardAgentTool when the pane is collapsed, pipes the host’s coalesced SceneDelta output into Context.SendContextNote, requests debounced state-of-record saves on every Changed$, and handles restore/focus/artifact snapshots. Read it before writing a new channel — it is the reference for how little a channel plugin should be.

Tool surface (prefix Whiteboard_, 11 tools): AddNote (sticky notes), AddShape (labeled rect/ellipse/diamond boxes), AddText (free-floating labels, with explicit wrap-width support), AddMarkdown (v3 — a rendered rich-text panel, ≤32,000 chars), AddHtml (v3 — an interactive sandboxed HTML widget, ≤64,000 chars), UpdateContent (v3 — rewrite an existing item’s text/markdown/HTML in place), DrawConnector (arrows between items by id or absolute points), Highlight (“pointing without touching” — a pulsing region the user dismisses with a click), MoveItem, RemoveItem, StyleItem. Every tool returns a WhiteboardToolResult { success, itemId?, summary?, error? } — the summary is what the model narrates (“Added a sticky note…”); failures are structured so the model can self-correct conversationally. Each call runs as one undo batch, so a single toast-Undo reverts a whole tool effect.

Markdown panels & sandboxed HTML widgets (v3): markdown renders through the shared sanitized mj-markdown component (no raw-HTML passthrough); HTML widgets render in an iframe sandboxed to allow-scripts only (opaque origin — no parent DOM, session, cookies, or storage access), with off-screen frames rendered as placeholders. Users can edit both in-board through the rich editors (commits flow through the cancelable ContentApplying/ContentApplied events). The input bridge (MJWhiteboard.submit(data) — whiteboard-widget-bridge.ts) is the tutoring loop: the agent draws a quiz/micro-form widget, the user answers inside it, the validated, size-capped (8,000 chars) submission flows out through postMessage → WidgetSubmitted → the channel plugin sends the model [whiteboard] the user submitted input in widget "…": {data} — and the agent reacts in voice. A right-click context menu (pure model in whiteboard-context-menu.ts) offers per-item-kind actions on the board.

Perception model: WhiteboardState is the single mutation API used by both the user’s board tools and the agent’s channel tools. Every mutation appends to a compact change journal and emits Changed$; the bound surface coalesces changes over 750 ms and emits one SceneDelta, which the plugin pipes into the model as a [whiteboard]-prefixed context note. Perception etiquette rides in the note itself: the prefix text instructs the model that board updates are background context — “do NOT comment on minor edits; only mention it if the change is significant” — so any realtime model behaves, regardless of system-prompt sync state. BuildSceneDelta(sinceToken) collapses the journal (multiple moves of one item → one moved entry) with replace-current-state semantics; BuildSceneSummary() powers the “What the agent sees” popover — the user can literally inspect the agent’s perception feed. When the user clicks Undo on an agent-action toast, the plugin tells the model: [whiteboard] user undid your last change.

Ownership / the violet system: every item carries Author: 'user' | 'agent'. The violet treatment is reserved for the agent — agent items render violet regardless of tint, the user’s pen/text palettes never offer violet, and ownership chips (“You” / the agent’s name) make authorship unambiguous on the board and in exports. Hosts can go further: the engine’s cancelable before-events (ItemAdding$ / ItemUpdating$ / ItemRemoving$) let a policy veto, e.g., the agent removing the user’s items.

Export / print (whiteboard-export.ts): BuildWhiteboardExportHtml renders one fully self-contained HTML document (inline CSS, zero external references — the documented exception to design tokens) for Download/Print; BuildWhiteboardExportSvg for SVG. Every piece of user/agent-authored text is HTML-escaped before touching the document; live widget HTML is never inlined in exports (no sandbox exists there — widgets export as placeholder cards with escaped source in a <details>).

Artifact snapshots: “Save to artifacts” snapshots the board JSON as a versioned MJ: Artifacts record via SaveSessionChannelArtifact — typed with the seeded Whiteboard artifact type (metadata/artifact-types/, ContentType: application/json, viewer DriverClass: WhiteboardArtifactViewerPlugin — the viewer component stays in the conversations package and renders through WhiteboardSnapshotComponent), best-effort linked into conversation history via a MJ: Conversation Detail Artifacts junction row against the latest session-stamped detail. On success the plugin tells the model the user saved the board, so it can reference the artifact naturally.

Restore-on-resume: RestoreState rehydrates a prior session’s board in place into the same WhiteboardState instance (LoadFromJSON), so the save subscription and any later surface binding keep pointing at one engine. Malformed payloads return false and the board starts fresh.

The Media channel’s agent media library (Collections as kits)

The Media channel (MediaChannelServer / RealtimeMediaChannel, seeded row Name: 'Media') lets the agent put images/video/audio/PDF/web on a shared surface. Beyond ad-hoc Media_ShowMedia({ url | fileId }), an agent can be given a curated, governed media kit it reasons over — built by reusing Artifacts + Collections, not a bespoke entity:

• A MJ: Collections of MJ: Artifacts IS the kit. The artifact (+ current version) describes the media — FileID → MJ: Files, MimeType, name, viewer, versioning, permissions. Bytes stream through the authenticated /media route. Nothing is duplicated.
• Agent-reasoning metadata lives per-membership on MJ: Collection Artifacts: Sequence (priority/order, pre-existing) + ContextDescription (the agent-facing “what this is / when to show it”) + Preload (eager hint). Per-membership means the same artifact can be framed differently in different kits.
• Binding: AIAgent.DefaultMediaCollectionID (FK → Collection) is the agent’s default kit; per-session resolution is runtime override > agent default > none.
• Per-session runtime override (wired). StartRealtimeClientSession accepts an optional mediaCollectionId arg — UUID-validated at the resolver boundary (IsValidUUID from @memberjunction/global; a malformed value is dropped + logged, the session still starts) and stored on the session config (RealtimeSessionConfig.mediaCollectionID → AIAgentSession.Config_). SessionManager hands the verbatim config blob to data-aware channels via RealtimeChannelServerContext.AgentSessionConfig; MediaChannelServer parses mediaCollectionID from it and passes it as the override (re-validated downstream — defense in depth). So a caller (e.g. a Praxis Protocol) can hand a session its own kit, overriding the agent default; absent ⇒ the agent default applies. Note: AgentSessionConfig (the persisted AIAgentSession.Config_) is intentionally distinct from the driver-level ClientRealtimeSessionConfig.SessionConfig “private pact” (§5.8) — same word, different layer.
• Server-side resolution, existing client tool. MediaChannelServer implements IRealtimeChannelServerDataAware (the host hands it the session contextUser + provider between Initialize and OnSessionStarted); on start it resolves the kit via agent-media-library.ts (buildAgentMediaContextNote) and SendContextNotes a manifest (fileId, type, display name, when-to-show, PRELOAD). The agent then surfaces items with the existing Media_ShowMedia({ fileId, mediaType, displayName }) — no new client tool, no client round-trips. An agent with no kit is unaffected.

The resolver (mediaTypeFromMimeType / resolveAgentMediaManifest / formatAgentMediaManifest) is exported from @memberjunction/ai-agents and unit-tested. See plans/praxis/AGENT_MEDIA_LIBRARY_PLAN.md.

Interactions between channels

The channels are not silos — they converge on the one shared model context and the one overlay shell:

• Voice narrates board actions. A Whiteboard_AddShape tool result’s summary returns through SendToolResult, and the model speaks it — the agent talks about what it just drew, in the same breath as the conversation.
• Context notes are the cross-channel bus. Whiteboard scene deltas, delegated-run progress, artifact-save notifications, and undo events all arrive as context notes in the single model conversation — the agent’s awareness of every surface lives in one place, which is exactly why the one-agent-per-session invariant exists.
• Focus mode is a shared layout protocol. Any channel can call Context.SetFocusMode(true); the overlay (realtime-session-overlay.component.ts) collapses the main call column so the surface owns the screen, with a floating call pill keeping mute/thread/end reachable; the pill’s exit routes back through RequestFocusExit() to whichever channel holds focus.
• Delegated-run artifacts surface beside channel surfaces. The overlay’s right panel is a tabbed surface (RealtimeSurfaceTabsModel): the Activity rail is always tab 1, one tab per artifact a delegated run produced (auto-opened + flash-highlighted on arrival, keyed by artifact version), and one tab per interactive channel. A delegated Sage run’s report artifact and the live whiteboard sit side by side in the same panel while the voice keeps going.

---

6. Future Channel Types (Envisioned)

[!NOTE] This section is forward-looking design discussion. Video and screen share have since shipped in a different form (see the note under each); the rest is not implemented. It is here because the channel plugin contract was explicitly shaped to make these tractable, and because “what would it take” is the best test of the abstraction. For each candidate: the perception feed shape, the mutation tool surface, transport considerations, and what the existing contract covers vs. what would need extending.

Video (camera as perception)

Shipped as the Camera channel. The user’s camera is a client-only channel (Camera, picture-in-picture by default) that the agent can ask to open; only the user turns the camera on. Its frames are sampled to JPEG and sent over the provider’s own socket at the rate the model negotiated, not over WebRTC, and VideoSourceArbiter decides which source the model sees and tells it when that changes. The agent’s own video (a live avatar) is the Avatar channel; see §12 and the Realtime Channels Guide.

• Perception: sampled camera frames (or provider-native video input — BaseRealtimeModel is deliberately modality-agnostic and IRealtimeSession.SendInput already takes raw media frames). Structured deltas don’t apply; this is the one channel class where the media plane itself is the perception feed.
• Tools: model-driven visual annotations over the video surface (Video_Highlight(x,y,label), Video_Snapshot() → an artifact) rather than mutations of the feed itself.
• Transport: WebRTC, full stop — frames are binary and latency-sensitive. The registry’s TransportType: 'WebRTC' value exists for exactly this.
• Contract fit: surface/tab/focus-mode and the tool path carry over unchanged. The gap is the perception direction: SendContextNote is text-only, so native video input needs either provider video-modality support through the driver (extending RealtimeSessionParams/client Connect) or a frame-sampling serializer that describes frames textually (cheap, lossy interim).

Screen-share (perceive the user’s screen, guide and highlight)

Shipped as the Screen Share channel, the same way as the camera: the user shares a screen, a window, a browser tab or one panel of the page (“This panel”, in Chrome and Edge), and sampled JPEG frames reach the model over its socket. The structured digest and the guidance tools below are not built.

• Perception: like video, but the richer play is hybrid — periodic frames plus a structured accessibility/DOM digest for shared app windows (“dialog open: Save Changes; focused field: Name”). Structured digests are vastly cheaper than pixels and more reliable for guidance.
• Tools: Screen_Highlight(region|elementRef), Screen_Annotate(text, anchor) — guidance overlays rendered by the channel surface, never synthetic input into the user’s apps (a deliberate security boundary).
• Transport: WebRTC for frames; the structured digest is small enough for the control plane (PubSub).
• Contract fit: tools/surface/focus are covered (a screen-share would almost always run in focus mode). Same perception gap as video for the pixel path; the digest path works today via SendContextNote.

Collaborative documents (markdown / rich text co-editing)

• Stepping stone — SHIPPED: the whiteboard’s v3 markdown panels and sandboxed HTML widgets deliver most of this value inside the existing channel today — the agent drafts a rich-text section (Whiteboard_AddMarkdown), revises it in place (Whiteboard_UpdateContent), and the user drags/edits it with the in-board editors. A dedicated doc channel remains the eventual destination for true co-editing.
• Perception: section-level structural deltas (“§2 ‘Pricing’ edited by user; +2 paragraphs”), not character-level OT/CRDT traffic — the model needs document awareness, not keystrokes. The whiteboard’s journal → coalesced-delta pattern transfers directly.
• Tools: Doc_InsertSection, Doc_ReplaceRange, Doc_Comment, Doc_ProposeEdit (a suggestion the user accepts — the doc analogue of the whiteboard’s violet ownership + undo toast).
• Transport: PubSub-class control traffic; nothing binary. State of record = the serialized doc on the session-channel row; snapshots via SaveAsArtifact exactly like the board.
• Contract fit: this is the channel the contract fits best — everything (tools, perception, state of record, restore, artifact snapshots) works without extension. The work is the editor surface and the delta serializer.

Maps / geo canvases

• Perception: viewport + feature deltas (“user panned to Chicago; zoom 11; 3 pins visible: …”) — replace-current-state, like the board.
• Tools: Map_AddPin, Map_DrawRegion, Map_FlyTo, Map_Highlight. Agent-authored features get the violet ownership treatment.
• Transport: PubSub; tiles come from the map provider, not the session.
• Contract fit: clean — a near-isomorph of the whiteboard over a geographic coordinate space.

Dashboards / live-data surfaces

• Perception: the surface pushes data-state deltas (“KPI ‘Churn’ crossed threshold; user filtered to Q3”) so the agent can voice what the user is looking at.
• Tools: Dash_SetFilter, Dash_FocusWidget, Dash_Explain(widgetId) — the last one likely combines channels: a delegated run analyzes the widget’s data while the voice narrates.
• Transport: PubSub; the data itself flows through MJ’s normal RunView/RunQuery paths, not the session.
• Contract fit: covered; the interesting question is policy — debouncing live-data churn so the perception feed doesn’t flood the context (the whiteboard’s coalescing policy generalizes).

Code editors

• Perception: file + selection + diagnostic deltas (“user opened foo.ts; cursor in resolveModel; 2 TS errors”), never whole files on every keystroke.
• Tools: Code_ProposeEdit(file, range, replacement) (rendered as an accept/reject diff — ownership semantics again), Code_Annotate, Code_NavigateTo.
• Transport: PubSub.
• Contract fit: covered structurally; the real cost is the surface (an embedded editor) and the safety policy that proposals are never auto-applied.

Form-filling / guided UI

• Perception: form-state deltas (“field ‘Tax ID’ invalid; 3 of 9 required fields complete”) — likely derivable from MJ’s metadata-driven forms automatically, which makes this uniquely cheap in MJ.
• Tools: Form_SetField, Form_Explain(field), Form_Submit (gated on user confirmation). Agent-set values get a distinct treatment until the user confirms.
• Transport: PubSub.
• Contract fit: covered. The promising angle: because MJ forms are generated from entity metadata, a generic form channel could expose any entity form to the agent without per-form code — the metadata already describes the fields, validation, and value lists the tool schemas need.

The common thread: PubSub-class structured channels (docs, maps, dashboards, code, forms) fit today’s contract as-is — they are whiteboard isomorphs over different state engines. Media-perception channels (video, screen pixels) are the ones that need a real extension: a binary perception path through the driver layer, which is exactly the boundary BaseRealtimeModel’s modality-agnostic contract and the registry’s TransportType column were designed to absorb.

---

7. Progress Narration

Delegated work takes seconds to minutes; dead air kills a voice call. The narration system keeps the wait conversational without chattering. It runs in both topologies — VoiceSessionService on the client-direct path, RealtimeSessionRunner on the server-bridged path — sharing the DB-driven instruction template through packages/AI/Agents/src/realtime/realtime-narration.ts (ResolveNarrationInstructionsTemplate / BuildServerNarrationInstructions), and is DB-tunable.

The client-direct pipeline:

1. The server threads an OnProgress callback into every delegated run (buildDelegationProgressCallback in RealtimeClientSessionResolver). Only significant steps pass the filter (prompt_execution, action_execution, subagent_execution, decision_processing); each is published on the push-status topic tagged type: 'RealtimeDelegationProgress' + agentSessionID + callID.
2. The browser correlates events to the active session, drops stale progress (events for calls no longer in inFlightCallIds — PubSub can lag the fast mutation result, and narrating “starting up” after the answer was spoken is exactly the bug this prevents).
3. Every live progress message is injected as a context note ([delegated-agent progress] …) so the model knows what’s happening even when it doesn’t speak.
4. Throttled spoken updates: the first no earlier than ~5 s into a delegation burst (FirstNarrationDelayMs), subsequent ones at ≥8 s spacing (NarrationIntervalMs) — and the spacing floor is session-global, so back-to-back tool calls can’t re-arm the faster first-update path. Floods of small updates aggregate into one digest (up to 4 distinct messages, deduped, oldest-first). If the model is busy or audio is still audibly playing, the update retries in 1.5 s — hosts must gate on both IsBusy and IsAudioPlaying, or queued utterances come out late and stale. When the result lands first, the pending narration is cancelled (it’s moot — the result is about to be spoken).
5. The spoken update itself is BaseRealtimeClient.RequestSpokenUpdate(instructions). The instruction wording is DB-driven: the Realtime Co-Agent - Progress Narration prompt (metadata/prompts/) carries the template with {{ progressMessage }}, {{ updateNumber }}, and {{ priorNarrations }} placeholders — resolved server-side at session prepare (RealtimeClientSessionService.NarrationPromptName, with a deprecated-name fallback to Voice Co-Agent - Progress Narration for un-resynced deployments) and threaded to the browser, with a built-in fallback when the deployment hasn’t synced the prompt. The template enforces strictly first-person narration (the co-agent owns the work — “I’m digging into that now”, never “Sage is analyzing”), bans repetition (prior spoken narrations are chained into the instructions), and bans generic filler.
6. Narration transcripts are tagged Kind: 'narration' by the client driver and are ephemeral by product decision: surfaced as a transient “live note” in the overlay (DelegationNarration$), never captions, never persisted as Conversation Details.

The server-bridged pipeline (B3 — RealtimeSessionRunner): the runner wraps every delegation in a narration burst (runDelegateWithNarration). Each significant progress event is fed to IRealtimeSession.SendContextNote?.() as a [delegated-agent progress] note, and spoken updates are paced by the same engine rules as the browser — first update no earlier than ~5 s into a burst (FirstNarrationDelayMs), ≥8 s session-global spacing (NarrationIntervalMs), buffered messages deduped and aggregated into one digest, pending narration cancelled on delegation completion and on barge-in. Instructions come from BuildServerNarrationInstructions — the DB template with {{ progressMessage }} / {{ updateNumber }} substituted; {{ priorNarrations }} gets a neutral note because the server cannot observe what the model actually said (those transcripts ride the provider socket — the browser host chains real prior narrations, the server-side builder is deliberately compact). Both capabilities are feature-detected: on a provider whose session lacks RequestSpokenUpdate, progress still flows as context notes; lacking both, narration is simply absent. The drivers themselves are the collision safety net (queue or skip per the RequestSpokenUpdate contract), so the runner does not gate on busy state.

Prompt seed: the instruction template is the Realtime Co-Agent - Progress Narration prompt (metadata/prompts/), resolved with a deprecated-name fallback to Voice Co-Agent - Progress Narration for deployments that haven’t re-synced the renamed seed (NARRATION_PROMPT_NAME / LEGACY_NARRATION_PROMPT_NAME in realtime-narration.ts).

---

8. Observability & Administration

Run topology

A voice session produces a standard, navigable run tree:

• A co-agent AIAgentRun (Status: 'Running', stamped AgentSessionID + ConversationID + AgentID) plus a linked AIPromptRun (stamped with both AgentRunID and AgentID, so the run shows on the agent’s own history too) plus a single MJ: AI Agent Run Steps row (StepNumber 1, StepType: 'Prompt', TargetID = the system AIPrompt, TargetLogID = the linked prompt run — so the co-agent run’s Timeline is non-empty and drills into the prompt run) are created at session start by RealtimeClientSessionService.createCoAgentObservabilityRun (best-effort — failure never blocks the call). Their ids persist in the session’s config and SessionManager.CloseSession finalizes all three via FinalizeCoAgentRun (Completed/Failed, CompletedAt, and a Success stamp), idempotently, even when the close comes from the janitor.
• Every delegated target-agent run nests under the co-agent run via ParentRunID and shares the AgentSessionID — the delegation tree groups under the session.
• The transcript is the session-stamped Conversation Detail rows; channel state is on the session-channel rows; artifacts link in via the conversation-detail junction (with a hidden anchor detail when no transcript turn exists yet — see §3).
• Usage telemetry lands on the co-agent AIPromptRun in both topologies (B7). Server-bridged: RealtimeSessionRunner checkpoints accumulated usage on a debounced cadence (default 5 s) — a crash-driven janitor close finalizes from the last-persisted values and loses nothing. Client-direct: the browser accumulates the client driver’s OnUsage token deltas and flushes them via the RelayRealtimeUsage mutation, debounced (10 s) plus once at teardown; the resolver clamps and accumulates the deltas onto TokensPrompt/TokensCompletion (recomputing TokensUsed), accepts a post-close flush, and tolerates every failure (usage relay must never break a live call). Providers without usage events (ElevenLabs, AssemblyAI) simply never emit — the run’s token counts stay zero and cost accounting lives provider-side. Still deferred: linking persisted transcript turns to those runs.

Admin surfaces

• AI Analytics dashboard → Realtime Voice (packages/Angular/Explorer/dashboards/src/AI/components/analytics/realtime/): realtime-overview.component.ts (KPI row — active sessions, sessions in window, average duration, delegated runs, janitor closes, cost; sessions-over-time; channel-usage donut; top delegated targets; recent sessions) and realtime-sessions.component.ts (the full sessions grid: Agent → Target, status pills including the persisted close cause — explicit / janitor / shutdown / error / unknown-legacy, channel icons, run counts, tokens/cost, host instance, row drill-in to the session record).
• Custom entity forms for MJ: AI Agent Sessions and MJ: AI Agent Channels (packages/Angular/Explorer/core-entity-forms/src/lib/custom/AIAgentSessions/, .../AIAgentChannels/).
• In-call developer mode: the overlay’s gear toggle reveals “Open run” links on delegation cards and an “Open session” link in the banner; clicking minimizes the call (the session stays live) and emits a navigate request the host resolves.

Session review (review-on-reopen) & resume carryover

A past session can be reopened in review mode: ConversationChatAreaComponent.OpenRealtimeSessionReview(agentSessionId) loads the session through RealtimeSessionReviewService (packages/Angular/Generic/conversations/src/lib/services/realtime-session-review.service.ts) — a stateless, tolerant batch loader that folds the session row, its persisted caption turns (MJ: Conversation Details by AgentSessionID), its delegated runs (excluding each leg’s co-agent observability run), and its saved channel states into one RealtimeSessionReview.

The loader is chain-aware: when the reviewed session was a resume (LastSessionID set), it walks the chain backwards — capped at 5 legs (MAX_REVIEW_LEGS) and 500 total caption rows (MAX_REVIEW_DETAILS, oldest legs trimmed first), cycle-guarded so a corrupt A→B→A chain can’t loop — and surfaces every leg chronologically on Review.Legs. BuildReviewThreadItems renders a divider between legs (“Session leg started · time”, carrying the previous leg’s CloseReason as a chip — explicit / janitor / shutdown / error), so a multi-call conversation arc reads as one episode. Channel state stays the latest leg’s only (earlier boards were superseded on resume). The chain’s conversation-history artifacts (the MJ: Conversation Detail Artifacts junctions, Direction Output — including ones hung off hidden anchor details) load too, and review mode registers each as an unfocused artifact tab on the surface panel.

The call overlay then renders the past session instead of a live call: a “Session review” banner badge with the lifecycle range + close-reason chip, the same thread/rail components replaying the historical turns and delegation cards, and a read-only Whiteboard tab when the session saved a parseable board state. Everything live is dead in review — no mic, no composer; the controls collapse to “Start live session” (emits RealtimeStartLiveRequest, resuming as a new session chained via lastSessionId — which restores saved channel states through PriorChannelStatesJson and hydrates the prior transcript into the new model’s prompt, see §4) and Close. On the review→live edge, the review-registered (template-based, read-only) Whiteboard tab is removed only when the live session’s channel set has no Whiteboard channel — when it has one, the live plugin re-registers the same tab key and upgrades the pane in place; review artifact tabs are deliberately kept (wanted carryover). The rule is the pure ShouldRemoveReviewWhiteboardTab helper next to RealtimeSurfaceTabsModel.RemoveTab in realtime-surface-tabs.model.ts (the Activity tab itself is irremovable by model contract; removing a focused tab falls focus back to Activity).

Deep link: the custom MJ: AI Agent Sessions form ships an “Open session review” affordance that navigates to the conversations resource with a realtimeSessionId query param; ChatConversationsResourceComponent picks it up (initial load and later OnQueryParamsChanged deliveries) and calls OpenRealtimeSessionReview once the chat area is ready.

In the conversation timeline, a session’s stamped caption rows collapse into ONE element at the session’s position rather than flooding the thread. BuildConversationTimeline in @memberjunction/conversations-runtime does the grouping; the web message list renders it as RealtimeSessionTimelineCardComponent, and the mobile thread renders its own card from the same helpers. The web card is laid out like a chat message: a call icon in the avatar column, “Voice call with agent” and the start time in the header, then a bubble with the status (Live, Ended, Timed out, Interrupted, Error), how long the call ran, the message count, the last line said, and Review call (View call while live), which opens review mode. Every label comes from the runtime’s SessionCard* helpers, so web and mobile say the same thing.

---

9. Security Model

Authorization is overwhelmingly reuse of existing primitives, plus a small number of session-specific gates:

• CanRun on the target agent gates session creation — AIAgentPermissionHelper.HasPermission(targetAgentId, user, 'run') in both SessionManager.CreateSession and RealtimeClientSessionResolver.assertCanRunTarget. The co-agent is internal orchestration; the meaningful permission is on the agent doing real work. Denial throws before any row is written.
• Runtime overrides are authorization-gated — configOverridesJson and a deviating explicit preferredModelId require the seeded Realtime: Advanced Session Controls authorization (hierarchy-aware via AuthorizationEvaluator; fail-closed when the seed is absent); a deviating model is additionally subject to the effective allowUserModelOverride policy. Denial is a structured rejection, never a silent ignore — unauthorized users still get a fully working session on the co-agent’s configured defaults. Pairing rows restrict which targets a constrained co-agent fronts, but they are a targeting constraint layered on top of CanRun, never a replacement for it (see §4).
• Session ownership on every inbound operation — every relay (ExecuteRealtimeSessionTool, RelayRealtimeTranscript, SaveSessionChannelState, SaveSessionChannelArtifact, CancelRealtimeSessionTool, RelayRealtimeUsage) loads the session and enforces UUIDsEqual(session.UserID, contextUser.ID); tool/transcript relays additionally reject Closed sessions, while state/artifact saves, the cancel channel, and usage relay accept them (the final flush legitimately lands post-close, and a cancel can legitimately race teardown). Channel-state restore and prior-transcript hydration are ownership-checked too — another user’s prior session never leaks state or words.
• The target agent cannot be swapped mid-session — it is stored server-side in the session’s config at start and read back on every relay; the browser never re-supplies it.
• Tools execute under the session’s contextUser with the request-scoped provider — a realtime model calling a tool can do exactly what the user could do, no more. The provider owning the conversation is never an authorization bypass.
• Client-declared tools grant zero server capability. clientToolsJson declarations (channel/UI tools) are declared to the model so it can call them, but the server never executes them — a relayed call for one of those names falls into the structured “not available” path. Declarations are still validated to stop config bloat: ≤16 tools, ≤64,000 chars total, per-tool shape checks (name ≤128 chars, description non-empty, schema a plain object); channel-state payloads cap at 2,000,000 chars (per state and accumulated on restore).
• Ephemeral, provider-scoped tokens — the browser never sees a long-lived provider key. OpenAI bakes the full session config (instructions + tools) into the minted client secret; Gemini token-locks the mask-safe generation subset via liveConnectConstraints + lockAdditionalFields: [] (system prompt/tools ride the driver-applied SessionConfig; see §3). Token expiry mid-session surfaces as a Fatal error so the session finalizes cleanly.
• Relay sessions hold no provider credential in the browser — a provider with no browser-safe credential (Gemini Enterprise) runs through MJAPI’s realtime relay (/realtime/relay/<ticket>/…). MJAPI keeps the Google credential and adds an unexpired OAuth token (or the key’s API key) to each upstream connection; the provider’s frame policy writes the setup and drops browser frames outside its allowlist, so a tampered client can’t change the prompt, tools or avatar. The relay URL carries the session’s ticket, so it is a credential: it is never logged or put in an error. A ticket allows one fresh connection within 5 minutes of the mint, then only resumes that present a resumption handle the relay forwarded to that session, at most 10 connections in all, for at most 30 minutes (or the session’s MaxSessionSeconds). MJ_REALTIME_PROXY_ALLOWED_ORIGINS applies to the relay too. Relay sessions live in one MJAPI process’s memory, so several instances need sticky routing for /realtime/relay. See §12.
• XSS-safe exports — every user/agent-authored string in whiteboard HTML/SVG exports is HTML-escaped before touching the document, and live widget HTML is never inlined in exports (no sandbox exists there).
• Sandboxed agent-authored HTML — whiteboard HTML widgets run in allow-scripts-only iframes (opaque origin — no parent DOM, session, cookies, or storage); the only outbound channel is the validated, size-capped MJWhiteboard.submit postMessage bridge, gated on a tracked-frame source check. Markdown panels render through the sanitized shared markdown component. See the ng-whiteboard README for the full sandbox rationale.
• Ticket-gated WebRTC SDP broker router (/realtime/sdp-exchange) — enabled by default in MJServer (realtime.enabled: true), the SDP proxy exchange router mounts out of the box for zero-friction realtime voice and audio interactions. It is guarded by single-use ticket consumption via RealtimeProxyRegistry: inbound WebRTC offers cannot establish sessions without an ephemeral ticket issued during authenticated session creation. Operators who run an API server strictly as a headless backend without realtime media capabilities can explicitly disable mounting the route via MJ_REALTIME_ENABLED=false or realtime: { enabled: false } in mj.config.cjs.
• Deferred: channel-grained permissions (today: if you can run the agent, you can use its channels), multi-party sessions (AIAgentSession.UserID is singular by design), and audio retention (off — the text transcript is the durable record).

---

10. Known Gaps & Deferred Work

Honest ledger of what is not done on this branch, so nobody reads aspiration into the docs above. (Items previously listed here and since shipped — the ElevenLabs and AssemblyAI drivers, server-bridged narration consumption, client-direct usage relay, prior-session transcript hydration, the delegation cancel channel, the collapsed session block in the conversation timeline — have been pruned; they are documented in the sections above.)

Item	Status
Server-bridged client media plane (browser audio ↔ server socket, the plan’s P5 / WebRTC transport)	Not built — client-direct is the shipped audio path; server-bridged runs are fully wired up to the provider socket but have no client media transport
Transcript Replaces marker	Shipped — RealtimeClientTranscript.ReplacesPrevious is the machine-readable correction marker: the ElevenLabs driver stamps it on agent_response_correction, the overlay replaces the caption in place, and RelayRealtimeTranscript(replacesPrevious) UPDATES the persisted turn instead of appending (insert fallback when no prior turn exists). The server-bridged RealtimeTranscript type can adopt the same field when that path needs it
Provider session.resume windows	AssemblyAI shipped — the client driver captures session_id from session.ready and, on an unexpected socket drop, makes ONE reattach inside the provider’s 30-second window (session.resume first frame; 'connecting' shown while reattaching; mic worklet + playout engine survive; a failed/second drop falls through to the pre-existing fatal path). Gemini shipped — client and server sessions move to a new connection with Google’s resumption handle when Google announces the connection is ending (goAway; on Vertex AI it came about 9 minutes into a connection, with 30 s left) and after an unexpected drop; a relay session resumes through the same relay URL, and an avatar call keeps its video element and last frame across the move (checked live on Gemini Enterprise). A separate limit is the context window (about 15 minutes for audio, about 2 for audio plus video): every Gemini session gets sliding-window context compression by default (GeminiRealtime.DefaultContextWindowCompression), which resumption does not replace. Zero-data-retention sessions get no resumption. MJ’s own chain resume (LastSessionID) is unchanged and complementary
Transcript-turn → co-agent-run linkage	Deferred (noted in RelayRealtimeTranscript’s doc comment) — turns are session-stamped but not linked to the observability runs
Chat conversation-history hydration into the companion prompt	MVP gap — the resolver passes ConversationMessages: [] (the service supports it). Distinct from the shipped prior-session transcript hydration on resume (§4)
Unified session transport (SessionEnvelope / ISessionTransport)	Independent track per the plan — voice deliberately does not depend on it
Video channel / video modality	Shipped — user video in (the Camera and Screen Share channels, and channel surfaces, through VideoSourceArbiter) and the agent’s video out (live avatars on Gemini Enterprise, in calls and LiveKit meetings; see §12). What is checked live and what isn’t: plans/realtime/video-avatar-status.md
Server-side channel plugin execution (ServerPluginClass consumption)	Shipped with scope notes — BaseRealtimeChannelServer (in @memberjunction/ai) is resolved per session by RealtimeChannelServerHost from the ACTIVE registry rows and wired into the durable lifecycle: session start (SessionManager.CreateSession), pre-persistence channel-state saves (SaveSessionChannelState), and session close from every provenance incl. the janitor (SessionManager.CloseSession); WhiteboardChannelServer ships as the reference (state-of-record validation/canonicalization). Not covered: socket/media members (deferred with the unified-transport track) and server-tool contribution to RealtimeSessionRunner (documented TODO — no code path gives the server-bridged runner per-session channel instances yet). See §5
Non-target server tools on the client-direct relay (action wiring through executeNonTargetTool)	Later phase — structured “not available” today; the server-bridged path already executes actions
Channel-grained permissions, multi-party sessions, audio retention/consent	Out of scope for this iteration (see plan)
RealtimeClientSessionService ↔ BaseAgent prepare-logic duplication	Known debt — the service’s doc header calls for a future shared RealtimeSessionPreparer; until then the two are kept in sync intentionally (the shared realtime-narration.ts and realtime-coagent-config.ts modules are the first extracted pieces — both paths consume the same effective-config merge)
Agent speaks first (realtime.voice.default.firstMessage)	Shipped (#3557) for ElevenLabs (agent.first_message override, enabled on the managed agent — agents provisioned before the change are re-PATCHed on next use rather than silently dropping it) and AssemblyAI (greeting). Remaining gap: OpenAI and Gemini read the neutral firstMessage key nowhere, so a persona authoring one still opens silently on those vendors.
Per-driver voice plumbing for the effective config (realtime.voice.default.voice / realtime.voice.providers)	Shipped for authoring (#3530) — a voice is authored PROVIDER-AGNOSTICALLY as realtime.voice.default.voice and filed onto whichever vendor resolves, so a runtime pick is no longer OpenAI-only; both override builders emit that shape and GetProviderVoiceSettings resolves it after the model is chosen. Provider-matched bags still merge into the open Config bag (OpenAI spread, AssemblyAI/Inworld output.voice, HuggingFace audio.output.voice, ElevenLabs voice → tts.voice_id on both topologies, Gemini voice → speechConfig.voiceConfig.prebuiltVoiceConfig.voiceName on both topologies — mask-safe, so the ephemeral-token mint locks it, #3721), and OpenAIRealtime.CreateClientSession does fold params.Config (via ExtractRealtimeFeatures → BuildAudioBlock). Remaining gap: only OpenAIRealtime, xAIRealtime and HuggingFaceRealtime declare SupportedVoices (HF returns none), and the picker’s dropdown is gated on that list — so for ElevenLabs/Gemini/Inworld/AssemblyAI the voice is reachable via agent metadata and programmatic hosts, but the picker still offers nothing.

---

11. Audio-Reactive Call Visuals (Audio Activity Metering)

The call overlay is a PROGRESSIVE-DISCLOSURE console (see realtime-disclosure.ts + ng-conversations’ README): a first-ever call is pure audio — a breathing hero orb and three controls, with the caption thread / composer dock / surface panel / gear unlocking by level (0–4) as the user acts or across sessions via a per-user UserInfoEngine milestones ratchet (mj.realtimeVoice.uxMilestones.v1; the gear’s Simple/Standard/Pro/Auto density control is the escape hatch). Content never flips the console open — the ONE auto-reveal is a channel’s first agent activity (ChannelActivity$), which opens the surface panel as a peek with that channel’s tab focused; finished artifacts arrive as unfocused, glowing tabs.

That pure-audio hero orb doesn’t act out speech — it reacts to the actual waveform, vibrating like a speaker cone with the agent’s voice and flipping color when the user talks. Three layers, each degrading gracefully:

The capability contract (client drivers)

BaseRealtimeClient.GetAudioActivity(): RealtimeAudioActivity | null is a capability, not an obligation (driver obligation #9): per-direction RMS levels (0..1) plus 9 normalized frequency bins, or null for any direction the driver couldn’t meter — full null when no meters exist at all. Drivers attach meters through the protected attachInputAudioMeter / attachOutputAudioMeter hooks and MUST release them on disconnect (closeAudioMeters()).

Metering sources per driver (all four currently meter BOTH directions):

Driver	Agent audio (output)	User mic (input)
OpenAI (WebRTC)	RealtimeAudioMeter.ForStream on the remote track’s stream (attached in ontrack)	ForStream on the mic stream at Connect
Gemini / ElevenLabs / AssemblyAI (client-owned audio)	RealtimePcmPlayback.CreateMeter() — an AnalyserNode on the playout engine’s master gain	ForStream on the mic stream

RealtimeAudioMeter (in packages/AI/RealtimeClient/src/audio/audioMeter.ts) is defensive by contract: factories return null where Web Audio is unavailable (tests, SSR), and the DSP math (ComputeRmsLevel, BucketizeFrequencyData) is exported pure for unit testing.

The sampling loop (call overlay)

VoiceSessionService.GetAudioActivity() is a plain passthrough; the overlay runs ONE requestAnimationFrame loop outside Angular that samples it, runs the frame through RealtimeAudioVisualSmoother (attack/decay envelopes + direction hysteresis — see realtime-audio-visuals.ts in ng-conversations), and writes CSS custom properties (--voice-out, --voice-in, --eq-1..9) plus data-audio-live / data-voice-dir attributes directly on the rendered overlay element. Zero change detection per frame.

The render gate (CSS)

All audio-reactive rules key off [data-audio-live='true']: orb scale follows the smoothed output envelope, the EQ bars become a true spectrum, and [data-voice-dir='user'] recolors orb + bars green (the established “user audio” palette). When a driver attaches no meters the attribute stays false and the original turn-state keyframe animations remain in charge — the fallback is the pre-existing behavior, not a degraded new one.

---

12. Adding a Video-Capable Realtime Provider

A realtime model that sends video (a live avatar, today Gemini 3.8 Live on Gemini Enterprise) uses the same contracts as voice: the video is one more track, and each layer has a few more rules. The UI and the runtime show the agent’s video without knowing which provider sent it: the call through the Avatar channel, a meeting through the bot’s camera track.

The checklist:

1. An endpoint profile says, per endpoint and model, whether the model renders an avatar, in what format, and whether the video carries the voice.
2. The server driver grants or refuses the session’s avatar request when it builds the connect config, reports the decision, and (on a server-side session) emits the video as typed frames.
3. The browser driver plays the video through VideoPlayout, hands it to the host once, and follows driver obligations #3, #11 and #12.
4. Catalog rows: a model-vendor row for the driver, the model’s Video/Output modality row, a price for the video, and the avatars as persona Video bindings.
5. The conformance kit passes (@memberjunction/ai-realtime-client/testing).

The reference implementation is Gemini Enterprise: GeminiEnterpriseRealtime (packages/AI/Providers/Vertex/src/models/geminiEnterpriseRealtime.ts) on the server, which extends GeminiRealtime, and GeminiEnterpriseRealtimeClient ('gemini-enterprise') in the browser, which extends GeminiRealtimeClient.

How a request becomes video on screen:

voiced agent's persona (Video binding) ─► RealtimeSessionParams.Avatar (RealtimeAvatarSettings)
  ─► server driver: grant (connect config + pact block) or refuse (one log line + AvatarStatus reason)
  call:    mint ─► browser driver ─► VideoPlayout ─► emitRemoteVideo ─► runtime AgentVideo$ ─► Avatar channel tile
  meeting: StartSession (Delivery 'room') ─► OnVideoFrame ─► bridge 'video-out' ─► bot decodes ─► 'agent-avatar' camera track

Endpoint profile

What a model renders depends on the endpoint serving it, not on the model alone: gemini-3.8-live renders an avatar on Gemini Enterprise and returns audio on the Gemini Developer API. Keep these facts in a data table in the provider’s package, as Gemini does in geminiLiveProfiles.ts:

• ResolveGeminiLiveProfile(model, endpoint) returns the model’s row plus what it renders on that endpoint (GEMINI_LIVE_ENDPOINT_OVERLAYS): SupportsAvatarOutput, AvatarOutputEncoding (the MIME type with codecs, 'video/mp4; codecs="avc1.42c01f, mp4a.40.2"' for Gemini) and AvatarAudioMuxed (the MP4 carries the voice). An overlay matches one model row exactly, so a longer model id (gemini-3.8-live-extended-thinking) never inherits a shorter one’s avatar.
• What the endpoint accepts whatever the model (GEMINI_LIVE_ENDPOINT_PROFILES): its AcceptedTurnCoverages. Gemini Enterprise refuses TURN_INCLUDES_AUDIO_ACTIVITY_AND_ALL_VIDEO at setup, so it accepts only audioActivityOnly; the Developer API accepts both. A configured value the endpoint refuses is sent as one it accepts, with one log line naming both. A catalog value never costs the user the session.
• Override BaseRealtimeModel.SupportsAvatarOutput(model) from the profile. The model choice, the persona picker and the call’s notice all read it.
• MJ_GEMINI_LIVE_MODEL_ALIASES (<model id>=<known model id>, comma-separated) gives a model id the table doesn’t know the profile of one it does, for when Google names a model differently from the table.

Server driver

The avatar request. RealtimeSessionParams.Avatar (RealtimeAvatarSettings) carries AvatarID (the vendor’s avatar id), Kind ('preset' or 'custom'), Resolution, Background, PersonaName, Source and Delivery. Apply it last in the connect config, after the config bag, so the bag can’t add or keep video on its own:

• Grant it only where the profile says the model renders avatars. Gemini’s applyAvatarOutput sets the VIDEO response modality and avatarConfig.avatarName, with videoBitrateBps from MJ_GEMINI_AVATAR_VIDEO_BITRATE_BPS (2,000,000 by default; 0 leaves the field out). A VIDEO modality or an avatarConfig that came from the config bag is removed.
• Anywhere else the session runs audio only and logs one line with a RealtimeAvatarUnavailableReason: endpoint (the model renders none here), bridged (a server-side session whose host can’t publish video), custom-disabled (custom likenesses are off), unknown-avatar. Never fail the mint over an avatar.
• Report the decision as RealtimeAvatarStatus (Requested, Granted, Reason): on ClientRealtimeSessionConfig.AvatarStatus for a browser session, on IRealtimeSession.AvatarStatus for a server-side one. The server merges in its own reasons (no-binding, unknown-avatar, and endpoint when the model’s Video/Output row turns video off) and the browser adds host and browser; the call and the meeting room turn the reason into one “Audio only” notice.
• Tell the browser driver through the private pact. Gemini adds an avatar block (output, encoding, audioMuxed) to the minted SessionConfig only when the avatar is granted.

Server-side sessions (meetings, phone calls) grant an avatar only when the host publishes video into a room (Delivery: 'room'); otherwise the reason is bridged. A granted session:

• declares an outbound video track in Capabilities.SupportedOutboundTracks whose Encoding names its frames’ type;
• emits the video through IRealtimeSession.OnVideoFrame as RealtimeVideoFrames, in the order the model sent them (the untyped OnVideoOutput is deprecated);
• while the video carries the voice, sends no PCM for that turn through OnOutput;
• after OnInterruption, drops the interrupted turn’s late frames;
• reports the seconds of video it generated through OnUsage (OutputTokenDetails.VideoSeconds, an amount for the update).

Frames. RealtimeVideoFrame (packages/AI/Core/src/generic/realtimeVideoOutput.ts) has three kinds, each with Data, MimeType, and optional PresentationTimeMs, KeyFrame, Width and Height:

Kind	What it holds	Notes
'fmp4'	A piece of fragmented MP4: Piece is 'init' (ftyp + moov) or 'fragment' (moof + mdat)	May carry the voice on its audio track. Fmp4PieceToVideoFrame(data, mimeType, init) builds one from a model part, with the fragment’s time and key-frame flag read from the boxes
'chunk'	One encoded frame: H.264 in Annex B form (with the WebCodecs codec string in the MIME type’s codecs parameter), VP8, VP9 or AV1	PresentationTimeMs and KeyFrame are required. No voice
'image'	One still picture: JPEG, PNG or WebP	Every image is a key frame. No voice

PresentationTimeMs is media time on the stream’s own timeline, the same timeline as its voice, not wall-clock time. A new stream (an init segment, or a key frame whose time goes backwards) may restart it.

Browser driver

Driver obligations #11 and #12 on BaseRealtimeClient are the browser half’s rules; the conformance kit checks #12.

• Refuse a transport you don’t speak (#11). Call AssertTransportSupported(config) first in Connect. A driver that speaks its provider’s protocol through MJAPI’s relay overrides SupportsRelayTransport to true.
• Offer the video track only when it can play. Add an outbound video track to the supported tracks when the pact grants an avatar and VideoPlayout.IsSupported(encoding) says this browser can play it. The host asks for the track: the runtime adds the tracks its channels sink to the session config’s requestedTracks, and the Avatar channel sinks outbound video. When the track isn’t live, connect audio only and log one line: host when the host asked for no agent video, browser when this browser can’t play it.
• Create the player through an overridable seam. Gemini calls CreateVideoPlayout(options); tests and the kit replace it with a recorder. Hand the player’s Source to the host once per session with emitRemoteVideo, at connect, so the tile is up before the agent speaks. OnRemoteVideo takes one handler, which the runtime owns (RealtimeSessionRuntime.AgentVideo$).
• Route model output by MIME type. Video goes to the player as RealtimeVideoFrames, in order (VideoPlayout.Append). Gemini also treats a part whose bytes open with an MP4 box as video, whatever type it names. PCM goes to the voice. Anything else is dropped and reported once: unknown data never plays as audio.
• Turns. Call EndOfTurn() when the model has generated the whole turn (Gemini: generationComplete, with turnComplete as the fallback), so the video plays to its end and holds the last frame. On barge-in, Flush() the video with the voice (obligation #3) and drop the cut turn’s late parts until its turn completes. After a resume, keep the same player and let a turn the drop cut off play out what arrived; the next connection’s init segment appends to the same player.
• The voice. When the video carries the voice, don’t also play that turn’s PCM: the turn’s first video part flushes the PCM the turn queued, and later PCM in the turn is dropped. Route the player’s element audio into the PCM playback’s Web Audio graph (VideoPlayoutOptions.OnElementAttached → RealtimePcmPlayback.ConnectMediaElement), so the call’s meter and recording carry it, and count the player’s IsPlaying in IsAudioPlaying.
• Lip sync for chunks and images. When the voice and the frames share a media timeline, queue each PCM chunk at its media time (IRealtimePcmPlayback.Enqueue(pcm16, mediaTimeMs)) and pass the playback as VideoPlayoutOptions.Clock; restart both timelines together. MSE needs no clock: an MP4 that carries the voice keeps face and voice together on its own timestamps.
• Usage. Report each second of generated video once (OutputTokenDetails.VideoSeconds on OnUsage), counted from the video’s own durations (Fmp4VideoSeconds for fMP4); Gemini sends them in updates of their own at each turn boundary. An interrupted turn counts what arrived before the interruption; video after the turn’s generation ended (idle frames) does not count; a flush never subtracts.
• Disconnect releases the player.

Playout. VideoPlayout (in /media) plays any RealtimeVideoFrame into the host’s <video> element through a decoder chosen per frame type from VideoFrameDecoderRegistry:

Decoder	Plays	How
'mse-fmp4'	video/mp4 fragmented MP4	Media Source Extensions (ManagedMediaSource on iOS Safari 17.1+); unmuted when the MP4 carries the voice
'webcodecs'	video/h264, video/vp8, video/vp9, video/av1 chunks	A WebCodecs VideoDecoder, drawn into a MediaStream the element shows
'image'	image/jpeg, image/png, image/webp	createImageBitmap, drawn the same way

A frame type no decoder plays is dropped and reported once ('no-decoder'); a decoder that gives up (Failed) is replaced by the next that can play the type. Register another decoder with VideoFrameDecoderRegistry.Instance.Register({ Name, Kind, Priority, CanPlay, Create }); CanPlay must answer at once. With a clock, the chunk and image decoders show each frame when the voice reaches its PresentationTimeMs: a frame more than 500 ms behind the voice is dropped, one ahead waits, one more than 2 s ahead is dropped, and before the clock reads anything frames wait up to 500 ms for the voice, then go by their own times.

Browser auth: direct or relay

A browser session reaches the provider one of two ways (ClientRealtimeSessionConfig.Transport):

• 'direct' (absent means direct): the browser opens the provider’s socket with a short-lived credential in EphemeralToken, as on the Gemini Developer API.
• 'relay': the provider has no browser-safe credential (Gemini Enterprise), so the browser connects to MJAPI’s relay at RelayUrl and MJAPI holds the credential. The server driver issues a relay session (RealtimeProxyRegistry.Instance.IssueRelaySession({ UpstreamUrl, Policy, UserID, DriverClass, MaxSessionSeconds })) and returns Transport: 'relay', the URL from BuildRealtimeRelayUrl(ResolveRealtimeProxyBaseWsUrl(params), ticket.ID) and an empty EphemeralToken. The ticket is in the URL’s path, so the URL is a credential: never log it or put it in an error.

A relay needs an IRealtimeRelayPolicy (in @memberjunction/ai) for the provider’s protocol, kept in the provider’s package. MJAPI owns the sockets and calls the policy at five points: UpstreamHeaders on every upstream open (so a short-lived OAuth token is current for each connection), ReadOpenIntent on the browser’s first frame (a fresh open, or a resume with a handle, and whether it asks for audio only), OpeningFrames before anything else goes upstream, FilterClientFrame on each later client frame (client frames are text; binary ones are dropped first), and ObserveServerFrame to find resumption handles in server frames, which go to the browser unchanged. GeminiLiveRelayPolicy sends the setup the server wrote (BuildGeminiLiveSetup) on every connection, reads only a resumption handle and an audio-only request from the browser’s setup, and forwards only realtime input, user turns and tool responses. Widening what passes lets a browser change the agent mid-session, so it is a security change.

What MJAPI enforces for a relay session: one fresh connection within 5 minutes of the mint, then only resumes that present a handle the relay forwarded to that session; at most 10 connections; an end at 30 minutes or MaxSessionSeconds; the MJ_REALTIME_PROXY_ALLOWED_ORIGINS Origin allowlist, with refusals logged. At that end the relay closes the browser leg with REALTIME_RELAY_TIME_LIMIT_CLOSE_CODE (4000); the Gemini client then ends the call with a time-limit error instead of resuming, the runtime keeps it on LastSessionError, and DescribeCallEnd words it (“Time limit reached” in the call bar; “This call reached its time limit. Start a new call to keep talking.” in the widget, mj-error code time-limit, and on the phone). Relay sessions live in one MJAPI process’s memory, so a deployment with several instances needs sticky routing for /realtime/relay, and restarting MJAPI ends every relayed call on it (the client’s resume finds no session).

Usage and pricing

• Where usage lands. A browser session’s client driver reports OnUsage; the runtime relays it with RelayRealtimeUsage, whose usageDetailsJson carries the per-modality detail blocks, and the co-agent prompt run keeps them in ModelSpecificResponseDetails.RealtimeUsage. Stored output video seconds are capped at the run’s elapsed time plus 30 s. A server-side session’s OnUsage is recorded on its co-agent prompt run by RealtimeClientSessionService.WireBridgeRealtimeSession (every 10 s and once more before the run is finalized).
• Per-modality detail. Report the provider’s split (InputTokenDetails, OutputTokenDetails: text, audio, image and video tokens) and the generated video seconds. Gemini reports usageMetadata per turn, not as running totals, and MJ adds the updates up; a short avatar reply is about 20,000 to 25,000 response tokens, almost all VIDEO.
• The price. Give the model-vendor row a video price in ModelConfiguration.Realtime.Pricing.AvatarVideoOutput ({ Price, Unit, Currency }), with Unit 'Per 1M Tokens' or 'Per Minute'. When the run is priced, the output tokens the provider counted as video (per 1M tokens) or the stored output video seconds (per minute) become a line of their own, the video tokens leave the cost row’s output bucket, and the split is written under CostLines in ModelSpecificResponseDetails. Without a usable price the run prices as before. Gemini 3.8 Live on Vertex AI is seeded at $1.00 per 1M video tokens, Google’s list price: Google bills avatar video only while the avatar speaks, and the stored seconds also count the idle avatar between answers, so a per-minute price would overcharge.

Catalog rows

• The model-vendor row. Add an MJ: AI Model Vendors row with the server driver’s DriverClass and the provider’s APIName. Realtime vendor selection walks a model’s Active vendor rows by Priority and takes the first whose driver has a key, so a row whose key is missing is skipped. Gemini 3.8 Live has two: Google (GeminiRealtime, Priority 0) and Vertex AI (GeminiEnterpriseRealtime, Priority 1, Realtime.TurnDetection.Coverage audioActivityOnly, and the video price). A co-agent session then follows whether it will show an avatar: when the first keyed row renders avatars and the session won’t show one (video off, a phone call, an app without agent video), another keyed row that renders none is taken, and the other way round. So a deployment with both keys runs avatar sessions on Vertex AI and audio-only ones on Google; with only AI_VENDOR_API_KEY__GeminiEnterpriseRealtime it runs both on Vertex AI.
• The Video/Output modality row. A model shows an avatar only when its MJ: AI Model Modalities Video/Output row allows it and the driver’s endpoint profile renders one (RealtimeModelShowsAvatar in @memberjunction/ai-agents). IsSupported false turns video off for the model on every vendor (reason endpoint); no row leaves it to the profile. Modality rows belong to a model, not to a model-vendor pair, which is why the endpoint stays the driver’s fact.
• The video preference. A co-agent with realtime.video.enabled and no explicit model (no preferredModelId and no realtime.modelPreference) takes the first candidate that shows an avatar: models by PowerRank, each model’s keyed vendors by Priority, and only drivers that support client-direct sessions. When none does, it keeps the model it would have taken, audio only, with one log line. A phone call, or a meeting whose host can’t publish video, skips the preference.
• Avatars are personas. An avatar is an MJ: AI Persona Vendors row with Modality=Video on the vendor: APIName is the vendor’s avatar id (Google’s “Ben” preset for the example persona “Ben”), tuned by VendorSettings.Avatar (Kind, Resolution, Background). The persona’s Audio binding on the same vendor is its voice. A model that lists its personas explicitly (Gemini 3.8 Live does) also needs an MJ: AI Model Personas row for each. A session’s avatar resolves from realtime.video.avatarId (only when it names an Active Video binding on the vendor), else the voiced agent’s personas (default first), else the co-agent’s; never from the model’s own persona. The avatar persona’s voice becomes the session’s voice unless the call picked one, and the voice picker marks a voice “Comes with an avatar” only when the model shows avatars. Kind: 'custom' is refused (custom-disabled) until custom likenesses have a governance design.

Meetings

1. Before the model session opens, the LiveKit room coordinator asks the native room module whether this host’s bot can publish an avatar (describeAvatarVideo: @livekit/rtc-node’s video classes, and ffmpeg 4.2 or later with the H.264 and AAC decoders, from MJ_FFMPEG_PATH or the PATH). Only then does the session’s avatar request carry Delivery: 'room' (PrepareClientSessionInput.AvatarDelivery), so a host that can’t show the avatar never asks the model for it.
2. The server session emits 'fmp4' frames through OnVideoFrame; the bridge engine sends them on video-out with their MIME type; LiveKitBridge.SendMedia passes video/mp4 frames to the bot’s avatar outlet (publishAvatarMedia).
3. The bot reads the pieces with Core’s fMP4 reader, decodes them in two ffmpeg child processes, plays the voice on its audio track and shows each frame on a camera track named agent-avatar when the voice reaches the frame’s timestamp. The track is published at the first frame, and the last frame stays between turns and after a barge-in.
4. A decoder that keeps failing, or a room that refuses the track, takes the avatar down: the bot unpublishes the camera, sets its mj.agentAvatar attribute to audio-only:<reason> (decoder-missing, decoder-failed, publish-failed), and the engine replaces the model session with an audio-only one. The room shows the agent-avatar track as the agent’s tile, with the “AI-generated video” label, and one notice when the avatar can’t be shown.

Only fMP4 reaches a meeting today: any other video-out frame is dropped, with one log line per session. A raw-frame provider’s avatar shows in calls, not in meetings, until the bot decodes chunks and images.

Run the conformance kit

@memberjunction/ai-realtime-client/testing checks a client driver against the rules above. It needs no test framework, no provider SDK and no driver, so a provider in its own package runs it: RunRealtimeVideoConformance(driverFactory) resolves to one result per check, and ListRealtimeVideoConformanceChecks(driverFactory) gives a test runner one test per check.

You write a harness (IRealtimeVideoConformanceHarness):

• Name and Traits (below).
• Mint(grant): the ClientRealtimeSessionConfig your server mints, with or without an avatar. The kit adds the host’s track request.
• CreateClient(media): your real client driver with a fake transport and its creation seams wired to the kit’s recorders (media.CreateVideoPlayer for the video player, media.Voice for the PCM playback). Register no handlers on it. A second call starts over with a new client.
• Connect(client, config, microphone): connect, and open whatever the fake transport needs, until the session listens.
• The model’s side of the wire: SendVideo(frame) (a RealtimeVideoFrame of your kind, put on your wire as the provider would), SendVoice(pcm16, mediaTimeMs?), TurnComplete(), Interrupted(); optionally SendPart(mimeType, data), GenerationComplete(), Resume(), RefusePlayback(), AskedProviderForVideo() and Dispose(). A check that needs a method you don’t have is skipped with the reason.

Keep harness construction free of side effects: the factory is called once per check, and once more to read the traits.

Then the test file:

import { ListRealtimeVideoConformanceChecks } from '@memberjunction/ai-realtime-client/testing';

for (const check of ListRealtimeVideoConformanceChecks(() => new MyProviderHarness())) {
    (check.SkipReason ? it.skip : it)(`${check.Id} ${check.Title}`, () => check.Run());
}

Traits (RealtimeVideoConformanceTraits):

Trait	Values	Meaning
AgentVideo	'playout', 'stream', 'none'	The video plays through a player the driver creates (the kit records it), arrives as a live MediaStream (WebRTC), or never comes (audio only)
GrantsAvatar	boolean	Whether the provider’s server can grant an avatar at all
Voice	'pcm', 'transport'	The driver plays PCM itself, or the transport plays the voice (WebRTC)
VideoCanCarryVoice	boolean	The video can carry the voice, so the kit checks it never plays twice
VideoUsage	boolean	The driver reports generated video seconds
VideoFrameKind	'fmp4' (default), 'chunk', 'image'	The form the model’s video comes in
TimedVoice	boolean (default false)	Voice and video share one media timeline, and the driver passes its playback as the player’s clock
EnvironmentSkips	{ [checkId]: reason }	Checks this test environment can’t run faithfully; reported skipped

The checks:

Id	Check	Runs when
VC01	An audio-only session hands over no agent video and still works	always
VC02	In an audio-only session, a video part never plays as audio	the harness has SendPart
VC03	A granted avatar is handed over once, and its track is live	agent video and a grant
VC04	A granted avatar the host doesn’t show stays audio only	agent video and a grant
VC05	A granted avatar this browser can’t play stays audio only	a player, a grant and RefusePlayback
VC06	The avatar’s video goes to its player in order, never to the voice	a player and a grant
VC07	A part of an unknown type is dropped and reported once	a player, a grant and SendPart
VC08	Barge-in stops the video and the voice at once	a player and a grant
VC09	Media for an interrupted turn never plays	a player and a grant
VC10	A turn’s end lets its video play out	a player and a grant
VC11	A video that carries the voice never plays it twice	a player, a grant, video that can carry the voice, and PCM voice
VC12	Generated video seconds are reported, each once	a player, a grant and video usage
VC13	An interrupted turn counts what arrived; idle video doesn’t count	a player, a grant and video usage
VC14	A resumed session keeps its avatar	a player, a grant and Resume
VC15	Disconnect releases the video	a player and a grant
VF01	Frames reach the player typed, as the model sent them	a player and a grant
VF02	A timed voice drives the video’s clock	a player, a grant and TimedVoice

The kit checks the driver’s facts. The notice reasons endpoint, host and browser are decided by the server’s mint and the runtime, and inbound video (camera and screen to the model) has its own tests. src/__tests__/video-conformance/ runs the kit against Gemini on the Developer API and on Gemini Enterprise (through the real web SDK over a fake relay socket), OpenAI over WebRTC, and a synthetic raw-frame provider built only on the package’s public API (synthetic-video-client.ts), which is the worked example for a new provider.

Which drivers send video. realtime-video-driver-audit.test.ts keeps the client table true: it fails when a driver is registered without a row, or a row’s video columns stop matching the code.

Client key	Class	Agent video	Inbound video
'openai'	OpenAIRealtimeClient	none	none
'openai-live', 'OpenAILiveRealtime'	OpenAILiveClient	none	none
'gemini'	GeminiRealtimeClient	when granted (the Developer API grants none)	per the minted profile
'gemini-enterprise'	GeminiEnterpriseRealtimeClient	when granted	per the minted profile
'elevenlabs'	ElevenLabsRealtimeClient	none	none
'assemblyai'	AssemblyAIRealtimeClient	none	none
'xai'	xAIRealtimeClient	none	none
'huggingface'	HuggingFaceRealtimeClient	none	none

On the server, only the Gemini drivers declare outbound video, and only for a granted avatar: GeminiRealtime on the Developer API never renders one, and GeminiEnterpriseRealtime does for gemini-3.8-live. Every other server driver keeps SupportsAvatarOutput false and declares no outbound video track. The UI follows: the Avatar channel is open in every call but stays out of sight until video arrives, the “AI-generated video” label shows only while a tile shows an avatar, and an agent that asked for an avatar on an audio-only model gets one “Audio only” notice.

---

Reference Map

Concern	Where
Model primitive + driver obligations	packages/AI/Core/src/generic/baseRealtime.ts
Channel plugin contract — server half (+ host + reference impl)	packages/AI/Core/src/generic/baseRealtimeChannelServer.ts, packages/AI/Agents/src/realtime/realtime-channel-server-host.ts, packages/AI/Agents/src/realtime/whiteboard-channel-server.ts
Server drivers	packages/AI/Providers/OpenAI/src/models/openAIRealtime.ts, packages/AI/Providers/Gemini/src/geminiRealtime.ts, packages/AI/Providers/ElevenLabs/src/elevenLabsRealtime.ts, packages/AI/Providers/AssemblyAI/src/assemblyAIRealtime.ts
Browser drivers + shared PCM audio plane	packages/AI/RealtimeClient/ (README) — src/drivers/*, src/audio/*
Realtime video: frames, fMP4 reader, avatar settings and status	packages/AI/Core/src/generic/realtimeVideoOutput.ts, fmp4Reader.ts, baseRealtime.ts (RealtimeAvatarSettings, RealtimeAvatarStatus)
Browser media (capture, VideoPlayout and its decoders, the playback clock, the media stage) and the video conformance kit	packages/AI/RealtimeClient/src/media/, src/testing/
Gemini Live profiles, setup writer and relay policy; Gemini Enterprise driver	packages/AI/Providers/Gemini/src/geminiLiveProfiles.ts, geminiLiveSetup.ts, geminiLiveRelayPolicy.ts; packages/AI/Providers/Vertex/src/models/geminiEnterpriseRealtime.ts
MJAPI realtime relay	packages/MJServer/src/realtimeProxy/ (RealtimeProxyServer.ts, RealtimeRelayTunnel.ts); sessions in RealtimeProxyRegistry (packages/AI/Core/src/generic/realtimeProxyRegistry.ts)
Avatar choice and the video model preference	packages/AI/Agents/src/realtime/realtime-avatar-resolution.ts, realtime-video-output-gate.ts, realtime-client-session-service.ts
Agent type / runner / broker / client-session service / shared narration / memory builder	packages/AI/Agents/src/agent-types/realtime-agent-type.ts, src/realtime/* (incl. realtime-narration.ts), src/agent-memory-context-builder.ts
Session records, janitor, host identity	packages/MJServer/src/agentSessions/
GraphQL resolvers	packages/MJServer/src/resolvers/RealtimeClientSessionResolver.ts, AgentSessionResolver.ts
Browser orchestration + overlay + channels + session review	packages/Angular/Generic/conversations/src/lib/services/voice-session.service.ts, src/lib/services/realtime-session-review.service.ts, src/lib/components/realtime/**
Generic whiteboard (engine, tools, surfaces, exports, widget bridge)	packages/Angular/Generic/whiteboard/ (README)
Admin dashboard	packages/Angular/Explorer/dashboards/src/AI/components/analytics/realtime/
Session / channel custom entity forms (incl. the review deep link)	packages/Angular/Explorer/core-entity-forms/src/lib/custom/AIAgentSessions/, .../AIAgentChannels/
Schema	migrations/v5/V202606090930__v5.41.x__AI_Agent_Sessions_Channels.sql
Seeds	metadata/agents/.voice-co-agent.json (Realtime Co-Agent), metadata/prompts/.voice-co-agent-*.json (Realtime Co-Agent prompts), metadata/ai-agent-channels/, metadata/ai-model-types/, metadata/ai-models/ (realtime rows incl. ElevenLabs Agents + AssemblyAI Voice Agent), metadata/agent-types/ (Realtime), metadata/artifact-types/ (Whiteboard)

Session tuning, cost attribution & resilience (QA-hardening wave, v5.49)

realtime.session — provider-session tuning knobs

The effective co-agent config gained a session sub-section that flows into the driver’s open Config bag on both topologies (same cascade voice rides; the runtime bag wins per key):

{
  "realtime": {
    "session": {
      "effortLevel": 85,                 // MJ-normalized (ChatParams.effortLevel vocabulary: 1–100 or named)
      "parallelToolCalls": true,
      "mcpTools": [{ "type": "mcp", "server_label": "kb", "server_url": "https://…", "require_approval": "never" }],
      "inputTranscriptionModel": "whisper-1",
      "transcriptionLanguages": ["en-US"]  // BCP-47 caption-language hint (Gemini); unset = auto-detect
    }
  }
}

GetSessionTuningSettings projects the section onto the flat bag keys; each driver translates a key to its native wire field only when its profile confirms support and scrubs it otherwise — a shared config is safe on every provider (Gemini additionally warns on scrubbed foreign keys via REALTIME_SHARED_CONFIG_KEYS). Grok’s GA gates are OFF pending xAI docs; flipping them is a one-line profile change.

transcriptionLanguages matters for Gemini Live, whose captions on BOTH sides come from its own transcription streams. Unset, Gemini guesses each utterance’s language, and a call held in English can show captions in another script (Arabic, Tamil, Korean) even though the audio is fine. The codes are a hint, not a lock: speech in another language is still captioned in that language. The OpenAI family scrubs the key, since the agent’s captions there are the model’s own text.

Turn detection — catalog-driven, per model (ModelConfiguration)

Turn detection is configurable through the same pact, with an extra base layer sourced from the model catalog (see plans/model-configuration.md). The ModelConfiguration JSONType column exists on MJ: AI Model Types < MJ: AI Models < MJ: AI Model Vendors (inherit-with-override, deep-merged per key by AIEngine.GetEffectiveModelConfiguration), and its Realtime.TurnDetection section is folded into the session Config bag as the normalized turnDetection key on both topologies. Full precedence, lowest to highest:

profile hard default
  < ModelConfiguration cascade (type < model < model-vendor)   ← the catalog declares what the model supports
  < realtime.session.turnDetection (agent/app config cascade)  ← agents/apps refine it
  < runtime configOverridesJson (raw audio.input.turn_detection — auth-gated escape hatch, unchanged)

The normalized shape is { Mode: 'default' | 'serverVad' | 'semanticVad' | 'native', Eagerness?, Threshold?, SilenceDurationMs? }. Each OpenAI-protocol profile declares its supportedTurnModes and maps the vocabulary via MapNormalizedTurnDetection; an unsupported mode is diag-logged and falls back to the profile default — a shared catalog never rejects a session. 'native' is the deliberate forward slot: when a provider documents a smarter full-duplex turn mode (e.g. for the Grok Voice Think Fast family), its profile adds 'native' + the mapping once, and the catalog opts models in via metadata with no further driver changes. Meeting mode (disableAutoResponse) composes on top of whatever mode wins — create_response/interrupt_response always reflect the bridge’s floor control, and a live Reconfigure rebuilds the session’s actual mode (it no longer hardcodes server_vad). GPT Realtime 2.1 / 2.1-mini are seeded to semanticVad in metadata/ai-models/.ai-models.json; delete that seed to return to the provider default.

Turn coverage is the same section’s Coverage: what a turn carries as input, audioActivityOnly or audioActivityAndAllVideo (every video frame since the last turn; more input tokens). Gemini states it on every session rather than inherit a default, sending audioActivityOnly when nothing is configured. Gemini 3.8 Live and Extended Thinking ask for all video on the Gemini Developer API. Gemini Enterprise accepts only audioActivityOnly, so its Gemini 3.8 Live row asks for that, and the driver sends it, with one log line, for any session there that asks for all video.

Multi-channel cost attribution

RealtimeUsage now carries per-modality token detail (InputTokenDetails/OutputTokenDetails: text/audio/image/video/cached, plus the seconds of video the model generated, OutputTokenDetails.VideoSeconds). The OpenAI driver maps the GA response.done detail blocks, the Gemini drivers map usageMetadata’s per-modality counts (VIDEO included), the session runner accumulates them field-wise across turns, and the checkpoint persists the detail as JSON on the realtime AIPromptRun.Result — so audio-vs-text pricing (audio-in ~8× text-in on GPT Realtime 2.1) is attributable instead of blended. A model-vendor row with an avatar video price (ModelConfiguration.Realtime.Pricing.AvatarVideoOutput) prices the generated video on a line of its own; see §12.

Resilience & lifecycle

• Connect/readiness deadlines everywhere — client WS drivers reject an awaited Connect on a silent endpoint (15s, overridable), and the server session’s WaitForConfigApplied has a matching readiness deadline; no path can hang forever.
• Cancellation — the session runner observes the chained agent cancellation signal (RealtimeSessionRunnerDeps.AbortSignal): an abort stops + finalizes (usage flushed) instead of waiting on the janitor.
• Bounded transport reconnect — a fatal socket drop triggers up to MaxTransportReconnects (default 1) fresh-session reconnects with rewired handlers + a context note; usage/transcripts span the reconnect; exhausted budget finalizes.
• MCP approval — with no approval UX yet, an mcp_approval_request is auto-DENIED (correct correlation id) so the model voices the refusal instead of dead-air blocking; declare servers require_approval: 'never' to skip the round-trip.
• Barge-in floor rule — on a TRUE user barge-in, queued tool-result auto-triggers are dropped (the result item is already in the conversation; the user’s next turn voices it) — the model never speaks over a user who just took the floor.
• Proxy hardening (HF client-direct) — optional MJ_REALTIME_PROXY_ALLOWED_ORIGINS origin allowlist (403 before ticket consumption; the realtime relay applies the same list, and both log each refused origin), 15s upstream-open deadline, bounded pre-open frame buffer. Multi-instance deployments need sticky routing for the in-memory ticket registry.