Real-Time Co-Agents Guide

How MemberJunction delivers live, low-latency, full-duplex agents — agents you talk to the way you talk to a person, that can draw on a shared whiteboard while they speak, and that delegate real work back to the async agents you already have.

This is the developer guide for the whole stack: the model primitive, the Realtime agent type, the Realtime Co-Agent, the dual-topology session architecture, the session/channel persistence layer, the interactive-channel plugin system (with the live Whiteboard as the canonical implementation), narration, observability, and security.

Companion documents:

• plans/ai-agent-sessions.md — the original architecture plan (Part I: realtime capability, Part II: session/channel infrastructure). This guide documents what actually shipped; the plan retains the rationale and the deferred tracks.
• packages/AI/RealtimeClient/README.md — the browser-side driver package (BaseRealtimeClient + the OpenAI / Gemini / ElevenLabs / AssemblyAI client drivers and the shared PCM audio plane).
• packages/AI/Agents/README.md — the agent framework, including how RealtimeAgentType sits beside Loop and Flow.
• packages/Angular/Generic/whiteboard/README.md — @memberjunction/ng-whiteboard, the extracted generic whiteboard package (state engine, agent tool API, components, exports). The conversations package consumes it as a thin channel plugin.
• guides/TRANSPORT_LAYER_ARCHITECTURE_GUIDE.md — the engine → resolver → client layering the realtime resolvers follow.

---

Table of Contents

1. The Concept: Co-Agents
2. The Triple-Registry Plugin Architecture
3. Topologies: Client-Direct vs Server-Bridged
4. Session Lifecycle
5. Channels — The Heart of the System
6. Future Channel Types (Envisioned)
7. Progress Narration
8. Observability & Administration
9. Security Model
10. Known Gaps & Deferred Work
11. Audio-Reactive Call Visuals

---

1. The Concept: Co-Agents

Real-time is a complement, not a replacement

MJ agents are asynchronous by design: a request arrives, an AIAgentRun does its work over seconds to minutes, results come back. That is exactly right for substantive work, and Loop/Flow agents are untouched by the realtime feature. What async agents cannot do is hold a natural, low-latency conversation — modern realtime stacks (GPT Realtime, Gemini Live, ElevenLabs Agents, AssemblyAI Voice Agent) own the listen-reason-speak loop themselves, with provider-side voice activity detection, barge-in, and sub-second turn taking.

So instead of forcing the loop agent to drive a live call, MJ adds a third agent type — Realtime — a first-class peer of Loop and Flow (RealtimeAgentType in packages/AI/Agents/src/agent-types/realtime-agent-type.ts, registered as @RegisterClass(BaseAgentType, "RealtimeAgentType"), seeded in metadata/agent-types/ with DriverClass: "RealtimeAgentType"). A Realtime agent does not iterate a reasoning loop; it wraps a long-lived duplex model session. The marker is RealtimeAgentType.IsSessionDriven (always true); BaseAgent.isSessionDrivenAgentType() duck-types that getter and branches into executeRealtimeSession() instead of the loop.

One co-agent voices ANY agent

The first (and seeded) agent of this type is the Realtime Co-Agent (metadata/agents/.voice-co-agent.json — the agent was renamed from the original Voice Co-Agent; the seed file keeps its legacy filename, the Name field is Realtime Co-Agent). It is deliberately generic: it is the live voice for a target agent, and the target is a runtime parameter, not configuration baked into the co-agent. One Realtime Co-Agent definition fronts Sage, Query Builder, or anything else — agents gain voice by configuration, never by being rewritten.

Three design rules make this work:

1. The realtime-registered tool set is stable and target-independent. Every voice session registers the single invoke-target-agent tool (INVOKE_TARGET_AGENT_TOOL_NAME in packages/AI/Agents/src/realtime/realtime-tool-broker.ts) plus fixed UI/channel tools. The target is an argument inside that one tool; target-specific capabilities execute inside the delegated agent’s own run and are never registered on the provider socket. This keeps the provider contract identical across targets — and it is exactly what let the ElevenLabs Agents driver (a pre-provisioned, server-side-agent provider) fit the same model: its managed agent is created once with the stable tool set and reused across sessions (see §2).
2. Companion framing, never impersonation. The system prompt (the Realtime Co-Agent - System Prompt row in metadata/prompts/ — its template file retains the legacy name templates/Voice Co-Agent - System Prompt.template.md) frames the model as “the live voice for the agent” — first-person, owning the work (“I’m pulling that up”), never “pretend to be X” (which makes models refuse) and never third-person (“Sage is working on it”).
3. Delegation is a normal agent run. When the co-agent invokes the target, that is a full AIAgentRun, linked to the co-agent’s run via ParentRunID and stamped with the shared AgentSessionID — the delegation tree is fully visible in standard observability.

The co-agent resolution chain

Which Realtime-type agent voices a given target is resolved at session start by RealtimeClientSessionResolver.resolveCoAgentID() (packages/MJServer/src/resolvers/RealtimeClientSessionResolver.ts). First match wins:

Step	Source	On invalid candidate
1	Runtime coAgentId argument on StartRealtimeClientSession	Throws (fail loud — the caller asked for something specific)
2	Target agent’s AIAgent.DefaultCoAgentID (per-agent persona)	Logs a warning, falls through (metadata drift must degrade, not break calls)
3	Target agent’s type’s AIAgentType.DefaultCoAgentID (per-type default)	Logs a warning, falls through
4	The seeded Realtime Co-Agent, looked up by name (falls back to the deprecated legacy name Voice Co-Agent with a deprecation log, for deployments that haven’t re-synced the renamed seed)	Throws when both names are absent (the realtime feature is unconfigured in this deployment)

Every candidate from steps 1–3 is validated by findValidCoAgent(): it must exist in AIEngine’s cached agents, have Status = 'Active', and be of the Realtime agent type. The two DefaultCoAgentID columns were added by migration migrations/v5/V202606090930__v5.41.x__AI_Agent_Sessions_Channels.sql.

This chain is how deployments give individual agents a distinct voice persona (a different co-agent with a different prompt/voice) without touching code or the global seed.

Two further metadata layers refine which target a co-agent fronts and how it sounds/behaves — pairing rows and the effective configuration merge. Both are opt-in with zero-config defaults identical to the pre-feature behavior; see §4 — Co-agent pairing & effective configuration.

---

2. The Triple-Registry Plugin Architecture

Realtime is pluggable along three independent axes, all resolved through MJGlobal.ClassFactory + metadata. Nothing in the hosts (the session runner, the resolver, the Angular overlay) names a concrete provider or channel. The third axis — interactive channels — has two halves, one base class per side of the wire, resolved from the same registry row:

Registry	Base class	ClassFactory key comes from	Shipped implementations
Server model drivers	BaseRealtimeModel (packages/AI/Core/src/generic/baseRealtime.ts)	MJ: AI Model Vendors.DriverClass on a model of AIModelType = 'Realtime'	OpenAIRealtime (packages/AI/Providers/OpenAI/src/models/openAIRealtime.ts), GeminiRealtime (packages/AI/Providers/Gemini/src/geminiRealtime.ts), ElevenLabsRealtime (packages/AI/Providers/ElevenLabs/src/elevenLabsRealtime.ts), AssemblyAIRealtime (packages/AI/Providers/AssemblyAI/src/assemblyAIRealtime.ts)
Client model drivers	BaseRealtimeClient (packages/AI/RealtimeClient/src/generic/baseRealtimeClient.ts)	ClientRealtimeSessionConfig.Provider — the string the server driver stamps when it mints ('openai', 'gemini', 'elevenlabs', 'assemblyai')	OpenAIRealtimeClient, GeminiRealtimeClient, ElevenLabsRealtimeClient, AssemblyAIRealtimeClient (same package)
Channel plugins — client half	BaseRealtimeChannelClient (packages/Angular/Generic/conversations/src/lib/components/realtime/channels/base-realtime-channel-client.ts)	MJ: AI Agent Channels.ClientPluginClass	RealtimeWhiteboardChannel (key 'RealtimeWhiteboardChannel') — a thin plugin over @memberjunction/ng-whiteboard
Channel plugins — server half	BaseRealtimeChannelServer (packages/AI/Core/src/generic/baseRealtimeChannelServer.ts), resolved per session by RealtimeChannelServerHost (packages/AI/Agents/src/realtime/realtime-channel-server-host.ts)	MJ: AI Agent Channels.ServerPluginClass	WhiteboardChannelServer (key 'WhiteboardChannelServer', in @memberjunction/ai-agents) — validates/canonicalizes the board’s persisted state of record

Server model drivers are resolved by BaseAgent.resolveRealtimeModel() (server-bridged) and RealtimeClientSessionService.resolveVendorAndInstantiate() (client-direct): highest-PowerRank active Realtime model → highest-Priority active vendor whose DriverClass has a resolvable API key (GetAIAPIKey, e.g. AI_VENDOR_API_KEY__OpenAIRealtime) → ClassFactory.CreateInstance<BaseRealtimeModel>(BaseRealtimeModel, driverClass, apiKey). An explicit preferredModelId bypasses ranking and fails with a specific reason rather than silently falling back (resolvePreferredRealtimeModel).

Client model drivers are resolved in VoiceSessionService.createRealtimeClient() by the server-reported Provider key. Channel plugins are resolved in VoiceSessionService.loadActiveChannels() from the active MJ: AI Agent Channels rows.

[!NOTE] The channel registry is the one registry with two halves: each row carries both a ClientPluginClass (resolved in the browser by VoiceSessionService) and a ServerPluginClass (resolved server-side by RealtimeChannelServerHost when SessionManager.CreateSession mints the session — one fresh instance per session, exactly mirroring the client half’s resolution: ACTIVE rows only, ClassFactory registration checked first, unregistered keys skipped with a log, never fatal). The server half’s lifecycle hooks are the durable session events — session started, channel state saved (pre-persistence, with normalization power), session closed (any provenance) — see §5 for the full contract. The registry also carries a TransportType (PubSub / WebRTC / WebSocket) for the future media-plane work.

The four-provider capability matrix

The base contracts are deliberately tolerant of provider asymmetry: several members are optional capabilities (SendContextNote / RequestSpokenUpdate / OnClose on IRealtimeSession) or may simply never fire (OnUsage). Where a provider lacks a native channel, the driver either emulates it (documented per driver) or omits it — hosts feature-detect, never assume. What actually ships, verified per driver:

Capability	OpenAI ('openai')	Gemini ('gemini')	ElevenLabs Agents ('elevenlabs')	AssemblyAI Voice Agent ('assemblyai')
Client transport	WebRTC (mic track + oai-events data channel)	WebSocket (@google/genai Live SDK)	WebSocket (raw Agents protocol)	WebSocket (raw Voice Agent protocol)
Mint style	Client secret with the full session config (instructions + tools) baked in	Ephemeral token locking the mask-safe generation subset (liveConnectConstraints)	Signed URL is the credential (wss://…&token=…, ~15-min open window); agent ensured server-side first	One-time temp token (GET /v1/token, 300 s connect window)
What the token locks	Prompt + tools + transcription (tamper-proof)	Model + generation params only — prompt/tools ride the driver-applied SessionConfig	Agent identity; the prompt rides the conversation_initiation_client_data override (enabled on the managed agent); tools are bound to the server-side agent config	Nothing beyond auth — the whole session object rides the driver-applied first session.update frame
Transcripts	Interim deltas + finals, both roles	Interim deltas + finals, both roles	Finals only, both roles; agent_response_correction re-finalizes a barged-in turn with what was actually spoken	User turns: deltas + final; agent turns: final only (post-barge-in final carries the truncated text — no correction event)
SendContextNote	Emulated (conversation-item injection without a response trigger)	Emulated (turnComplete: false client content)	Native — contextual_update, the platform’s purpose-built non-interrupting channel; sent even mid-response	Emulated via the mutable system_prompt — notes accumulate under a “Background updates” heading and the full prompt is re-sent via session.update
RequestSpokenUpdate	Native-ish (response.create with per-response instructions)	Emulated (instruction turn via realtime text — sendRealtimeInput({ text }); mid-call sendClientContent is history-seeding-only on native-audio models and never triggers generation), queued behind in-flight responses	Emulated as a user_message turn (queued; fidelity caveat — the model may reference “you asked me for an update”)	Native — reply.create carries per-response instructions (queued behind in-flight replies)
SendText (typed input)	Native user-item injection	Realtime-text turn (sendRealtimeInput({ text }) — same native-audio caveat as RequestSpokenUpdate)	Native user_message (takes the floor server-side)	Emulated via reply.create instructions — the protocol has no typed-user-input event; best-effort fidelity
Cancel (CancelActiveResponse)	Wire-level response.cancel + local flush	Local playout flush (client owns the audio plane)	No cancel frame — local playout flush; residual server generation is simply never played	No cancel frame — local flush plus suppression of residual reply.audio frames until the next reply boundary
Mid-session RegisterTools	Idempotent re-declare via session.update	Connect-bound (the StartSession params path is the only effective one)	Cannot re-declare on an open conversation — identical set no-ops, a different set warns and is ignored (next session picks it up via the ensure flow)	Native re-declare — tools is a mutable session.update field
Usage events (OnUsage)	Per-response deltas from response.done.usage	Per-turn deltas from usageMetadata	None — the Agents socket reports no token usage (platform-side accounting)	None — flat per-session-hour billing; the terminal session.ended frame carries duration seconds only
Server-side object to manage	None	None	Managed agent — ensureAgent(): find-by-name → create-if-missing → PATCH on tool-fingerprint drift / missing prompt-override enablement; cached per name + fingerprint	None — pure per-session config
Audio format (client driver)	Provider-negotiated via WebRTC	PCM16 16 kHz up / 24 kHz down	PCM16 at rates negotiated from the initiation metadata (pcm_<rate>, default 16 kHz; non-PCM telephony formats degrade loudly)	PCM16 fixed 24 kHz both directions

Two cross-cutting notes on the websocket-audio drivers (ElevenLabs, AssemblyAI): they share the browser audio plane in packages/AI/RealtimeClient/src/audio/ (createPcmMicCapture worklet capture, RealtimePcmPlayback playhead-clock playout, pcmUtils) — IsAudioPlaying is computed from the locally-owned playout clock. And both must send session.end-style teardown correctly: AssemblyAI’s Close() sends session.end before closing the socket, because skipping it leaves the session in a billable 30-second resume hold.

Driver-author obligations (read before writing a driver)

Both base classes carry an explicit “DRIVER AUTHOR OBLIGATIONS” block in their doc headers — BaseRealtimeModel (packages/AI/Core/src/generic/baseRealtime.ts) for server drivers, with the client-side mirror on BaseRealtimeClient (packages/AI/RealtimeClient/src/generic/baseRealtimeClient.ts). Those blocks are the authoritative list (8 numbered rules each, paid for in live debugging). Summarized:

1. Silent exit from “speaking” on tool-call emission — don’t clobber the host’s busy indicator with trailing turn frames.
2. Release the busy flag when a tool call is emitted — otherwise SendToolResult deadlocks waiting on a turn boundary that never arrives.
3. Flush playback + report IsAudioPlaying = false promptly on barge-in — stale queued audio after an interruption is a product bug.
4. Text injection must not echo a user transcript — the host owns the local echo; a synthesized user-role event renders the message twice.
5. Every tool result must EVENTUALLY be voiced and never dropped — queue the generation trigger behind an in-flight response if the provider rejects overlaps.
6. Credential expiry surfaces as a Fatal: true OnError — never idle forever on a dead socket.
7. Report “ready/listening” only after the server-built session config has been applied — early turns must not run against an unconfigured model.
8. SessionConfig is a private pact between same-keyed server and client driver halves — hosts and intermediaries treat it as an opaque blob.

Additional contract points on IRealtimeSession: RegisterTools must be idempotent for a set identical to the connect-time set; OnInterruption fires on true barge-in only (user speech over active model output, not every utterance); SendContextNote / RequestSpokenUpdate / OnClose are optional capability members — callers feature-detect, and drivers whose provider can’t support them omit them entirely.

Recipe: add a new realtime provider

1. Server driver. In the provider’s package (packages/AI/Providers/<X>), implement BaseRealtimeModel.StartSession() returning an IRealtimeSession, and register it: @RegisterClass(BaseRealtimeModel, '<X>Realtime'). Build against an injectable connection seam (see IOpenAIRealtimeConnection, GeminiLiveSession, ElevenLabsRealtimeSocket, AssemblyAIRealtimeSocket) so unit tests can drive provider-shaped frames with no network — all four drivers ship extensive vitest suites this way.
2. Client-direct support (optional). Override SupportsClientDirect to true and implement CreateClientSession() to mint an ephemeral credential + provider-native SessionConfig, stamping a unique Provider key. The credential doesn’t have to be a token: ElevenLabs mints a signed websocket URL; AssemblyAI mints a one-time temp token via a REST call.
3. Client driver. In @memberjunction/ai-realtime-client, extend BaseRealtimeClient, register under the same Provider key (@RegisterClass(BaseRealtimeClient, '<x>')), export a Load<X>RealtimeClient() no-op, and have hosts call it (the existing Load calls live at the top of voice-session.service.ts — tree-shaking prevention). If the audio plane is a raw websocket (not WebRTC), reuse the shared src/audio/ pipeline (createPcmMicCapture + RealtimePcmPlayback) instead of writing capture/playout again.
4. Metadata. Add the MJ: AI Models row (AIModelTypeID → Realtime type) with an MJ: AI Model Vendors association carrying DriverClass: '<X>Realtime' and the provider APIName, in metadata/ai-models/; push with mj sync. Reference rows: GPT Realtime 2 (APIName gpt-realtime-2), Gemini 3.1 Flash Live (APIName gemini-3.1-flash-live-preview), ElevenLabs Agents (APIName MJ Realtime Co-Agent — see the precedent below), AssemblyAI Voice Agent (APIName voice-agent — the endpoint has no model selection; the value is bookkeeping).
5. API key. AI_VENDOR_API_KEY__<DriverClass> (or the deployment’s key-resolution path consumed by GetAIAPIKey).

No host code changes: the resolver ranks the new model by PowerRank/Priority, and the browser resolves the client driver by the minted Provider string.

Lessons from the ElevenLabs / AssemblyAI builds (read before driver #5)

Durable friction points the third and fourth drivers surfaced — they generalize to any future provider:

• Model/APIName doesn’t have to name a model. ElevenLabs has no bare-model realtime socket — only pre-configured server-side agents. The driver sets the precedent: RealtimeSessionParams.Model carrying the agent_ prefix is used verbatim as a deployment-managed agent id; any other value is the name of the driver-managed agent (ensureAgent(): find-by-name → create-if-missing with the session tool set + per-session prompt-override enablement → PATCH on tool-fingerprint drift; instance-cached). The seeded APIName is MJ Realtime Co-Agent. If your provider’s “thing you connect to” isn’t a model, map it through Model the same way rather than inventing config side-channels.
• Delta-less transcript providers exist. ElevenLabs emits whole-utterance finals only; AssemblyAI’s agent turns are final-only. Host code must not assume interim deltas — the caption pipeline keys off IsFinal, and that discipline is what made finals-only providers drop in cleanly.
• The post-barge-in “correction” problem. ElevenLabs re-finalizes a barged-in assistant turn via agent_response_correction (the text actually spoken); AssemblyAI’s final already reflects the truncation. The transcript contract has no Replaces marker, so a host persisting finals gets two assistant finals around an ElevenLabs barge-in (the original and the correction) with no machine-readable link — a known open contract gap (see §10). Until it’s closed, treat a final assistant transcript arriving immediately after an interruption as the authoritative replacement.
• Single-frame busy boundaries. Neither new provider has an OpenAI-style response.created/response.done envelope pair around every response. Busy state is inferred frame-by-frame (ElevenLabs: first audio/agent_response sets it, agent_response_complete clears it; AssemblyAI: reply.started/reply.done), and the narration Kind must be stamped at send time — there is no confirmation frame to stamp on. Tool-call frames still clear the busy flag (deadlock guard) without draining the queue, so a queued narration can’t wedge between a tool call and its result.
• Two-round-trip opens. Both drivers’ Connect/StartSession is a handshake, not a socket-open: open the socket → send the config frame (conversation_initiation_client_data / first session.update) → await the provider’s confirmation (conversation_initiation_metadata / session.ready) → only then build the audio plane and report 'listening' (obligation #7). On ElevenLabs the server side additionally spends REST round-trips before the socket (ensure-agent + signed-URL mint, mitigated by the fingerprint cache). Budget for this in any “time to first listening” expectations.
• Teardown can be billable. AssemblyAI holds a closed-without-session.end session in a billable 30-second resume window — Close()/Disconnect() must send the explicit end frame first.

Recipe: add a new interactive channel

Covered in depth in §5; the short version:

1. Subclass BaseRealtimeChannelClient<YourSurfaceComponent>, implementing the channel contract (tools + perception + surface + state of record).
2. @RegisterClass(BaseRealtimeChannelClient, '<YourClientPluginClass>') + a Load...() no-op called from a static code path (the Whiteboard’s is called from conversations.module.ts).
3. (Optional but recommended) Subclass BaseRealtimeChannelServer for the server half — at minimum a state-of-record guard like WhiteboardChannelServer’s (validate/normalize each landed save in OnChannelStateSave). @RegisterClass(BaseRealtimeChannelServer, '<YourServerPluginClass>') + a Load...Server() no-op called from a static server path (the Whiteboard’s is called from MJServer’s agentSessions/index.ts). A row whose server key is unregistered simply runs with no server plugin — it is never an error.
4. Seed an MJ: AI Agent Channels row (Name, ClientPluginClass, ServerPluginClass, TransportType, IsActive: true) in metadata/ai-agent-channels/ and push with mj sync.
5. Done — VoiceSessionService.startChannels() resolves every active registry row at session start in the browser, and RealtimeChannelServerHost resolves the same rows’ server halves when the session mints. The model gets the plugin’s tools, the overlay gets a surface tab, the server gets the lifecycle hooks. The shells carry zero channel-specific wiring.

---

3. Topologies: Client-Direct vs Server-Bridged

The same tool-execution semantics run in two transport topologies. The shared piece is RealtimeToolBroker (packages/AI/Agents/src/realtime/realtime-tool-broker.ts) — both topologies route every tool call through it so results are byte-for-byte identical: invoke-target-agent → delegate (with a broker-owned per-call AbortController — fired by barge-in on the server-bridged path, and by the explicit cancel channel on the client-direct path), everything else → the injected tool executor; failures serialize to { success: false, error } so the model can narrate failures instead of going silent.

Decision table

	Client-direct (shipped MVP, drives Explorer voice)	Server-bridged (RealtimeSessionRunner)
Provider socket lives	In the browser (OpenAI: WebRTC; Gemini: WebSocket via @google/genai Live; ElevenLabs / AssemblyAI: raw WebSocket + the shared PCM audio plane)	On the server
Audio path	Browser ↔ provider directly — frames never transit MJ (lowest latency)	Server ↔ provider; no client media transport is wired yet (the plan’s P5 media plane)
Prompt/tool authority	Server — it builds the session config; the browser applies it verbatim	Server (it owns the socket)
Tool execution	Relayed: browser → ExecuteRealtimeSessionTool mutation → RealtimeToolBroker → result JSON → browser → SendToolResult	In-process: RealtimeSessionRunner.handleToolCall → broker → IRealtimeSession.SendToolResult
Non-target tools	Channel/UI tools execute locally in the browser (never relayed); relayed unknown tools get a structured “not available” (RealtimeClientSessionService.executeNonTargetTool — action wiring is a later phase)	Agent actions execute via BaseAgent.executeRealtimeTool → ExecuteSingleAction
Entry point	StartRealtimeClientSession mutation → RealtimeClientSessionService.PrepareClientSession	AgentRunner.RunAgent on a Realtime-type agent → BaseAgent.executeRealtimeSession
Narration / context notes	Browser calls BaseRealtimeClient.SendContextNote / RequestSpokenUpdate (fully wired — see §7)	Wired: the runner pipes delegation progress into IRealtimeSession.SendContextNote and paces spoken updates through the same first-delay / spacing / digest rules as the browser (shared template via realtime-narration.ts — see §7)
Delegation cancel	Explicit-only: the overlay’s per-card ✕ → CancelRealtimeSessionTool mutation → the service’s in-flight registry aborts the run. Barge-in never aborts delegations (deliberate host policy — the narration design expects the user to keep talking while work runs)	Barge-in aborts the in-flight delegated run via RealtimeToolBroker.AbortInFlight (no human ✕ exists on this topology)
Usage telemetry	Browser accumulates OnUsage deltas → debounced (10 s) + teardown-flushed RelayRealtimeUsage mutation → accumulated onto the co-agent AIPromptRun	Runner checkpoints accumulated usage onto the long-lived AIPromptRun (debounced 5 s, crash-safe)

🚨 Single source of truth: ONE prep, the precedence cascade, the core↔host boundary

Every realtime host — native chat, LiveKit, and future Zoom/Teams/GoToMeeting/Webex — must be the same agent: same identity, personality, model/voice, delegation, and session tracking. Only media transport and host UX tools vary. The full rationale + roadmap is in plans/realtime/realtime-core-host-convergence.md.

There is exactly ONE producer of realtime session prep: RealtimeClientSessionService.PrepareRealtimeSessionParams (packages/AI/Agents/src/realtime/realtime-client-session-service.ts). It resolves the model, builds the target-identity system prompt (via the single BuildRealtimeAgentFraming in realtime-tool-broker.ts — first-person AS the target, never the co-agent), the stable tool set (always incl. invoke-target-agent), voice, and memory. Do NOT build session prep in a host. The LiveKit bridge (BaseAgent.StartBridgeRealtimeSession) consumes it and differs only in opening server-side (StartSession) vs the browser mint (CreateClientSession); a bridge re-implementing prep is exactly the drift this convergence removed (guarded by __tests__/realtime-convergence-drift.test.ts).

Model + voice resolve through ONE precedence cascade (ResolveEffectiveRealtimeConfig), identical on every host:

runtime override  >  target agent's TypeConfiguration  >  co-agent config  >  agent-type default

The target layer is what lets a voiced agent (Sage, Marketing Agent, …) carry its own persisted voice/model that the shared co-agent then speaks with. The runtime-override input funnels into one slot — the native picker builds ConfigOverridesJson client-side (BuildRealtimeConfigOverridesJson), the server-bridged hosts build the identical envelope via BuildRealtimeOverridesJson — so both ride the same highest-precedence cascade layer.

Core (one place, all hosts): identity, model/voice precedence, base prompt + memory, invoke-target-agent delegation, session/run tracking. Host (varies): media transport (LiveKit audio → video) + host UX tools (whiteboard/browser/… — injected via RealtimeSessionRunnerDeps.ExtraTools/ExecuteTool, never baked into core).

Client-direct flow (the shipped path)

1. Mint. The browser (VoiceSessionService.StartVoiceSession) calls StartRealtimeClientSession(targetAgentId, conversationId?, lastSessionId?, preferredModelId?, clientToolsJson?, coAgentId?). The resolver authorizes CanRun on the target agent, resolves the co-agent (chain in §1), creates the durable AIAgentSession via SessionManager.CreateSession (storing targetAgentID server-side in the session’s Config column — accessed as Config_ on the generated entity), then RealtimeClientSessionService.PrepareClientSession resolves the model, assembles the companion system prompt (framing + co-agent prompt + target identity + history + memory via AgentMemoryContextBuilder — the same context a loop agent injects), builds the stable tool set, and asks the driver to CreateClientSession. When a lastSessionId is supplied, the resolver also loads the prior session chain’s transcript (loadPriorTranscript — ownership-checked, hidden/error rows skipped, capped at 30 newest turns / 8,000 chars / 5 chain legs, with a cycle guard) and frames it into the system prompt so the resumed model remembers the last leg. Strictly best-effort — any problem yields no hydration, never a failed start. On mint failure the just-created session is closed with CloseReason = 'Error' so nothing half-open leaks.
2. Connect. The browser resolves the client driver by the returned Provider key and calls Connect(config, micStream) — the host acquires the mic (it owns the permission UX); the driver owns the transport, applies SessionConfig verbatim once the control channel opens, and only then reports 'listening'.
3. Converse. Transcripts stream both ways; final normal turns become captions and are persisted via RelayRealtimeTranscript (a Conversation Detail stamped with AgentSessionID). Typed text rides SendText — which implies barge-in: the driver cancels any active spoken response (CancelActiveResponse, a floor-control action that never aborts server-side delegated work) before injecting the text, then routes the reply through the same collision-safe path tool results use.
4. Tools. Tool calls whose names match a registered client-tool prefix (e.g. Whiteboard_) execute locally; everything else relays to ExecuteRealtimeSessionTool, which reads the target from the session config (never from the client), threads a progress callback, nests the delegated run under the co-agent run, junction-links any artifacts the delegated run produced into the session’s conversation history (best-effort — when no session-stamped transcript turn exists yet to anchor to, a minimal hidden anchor Conversation Detail is created so the junction rows are never orphaned and no fake visible message appears), and rolls a pendingFeedbackRunID forward when an interactive target pauses AwaitingFeedback — so the user’s next answer resumes the same run (lastRunId + autoPopulateLastRunPayload).
5. Cancel. A still-working delegation card carries a ✕; clicking it calls VoiceSessionService.CancelDelegation(callId) → the CancelRealtimeSessionTool mutation → RealtimeClientSessionService.CancelInFlightDelegations, whose per-session in-flight registry aborts the matching AbortController(s); the original ExecuteRealtimeSessionTool mutation then resolves with the cancelled run’s outcome (the card was already flipped to “Cancelled by user”; the late duplicate result is suppressed). AbortedCount: 0 is a legitimate success — the work may have finished first. A callId-less sweep form (CancelInFlightDelegations()) exists for “stop everything” affordances. True barge-in never cancels delegations on this topology — it only cancels pending narration.
6. End. Teardown flushes pending channel saves and the pending usage-relay delta, disposes plugins, stops the mic, disconnects the driver, and calls CloseAgentSession; SessionManager.CloseSession stamps CloseReason = 'Explicit', disconnects channel rows, and finalizes the co-agent observability runs.

What an ephemeral token can lock — provider differences

All four providers keep prompt/tool authority server-side in the client-direct topology, but how much the minted credential itself enforces differs:

• OpenAI (OpenAIRealtime.CreateClientSession): the full realtime session config — instructions (system prompt), tools, input-transcription opt-in — is baked into the client secret at mint time via the client-secrets API. The browser’s session.update applies the same config; tampering buys nothing the secret didn’t already fix.
• Gemini (GeminiRealtime.CreateClientSession): the ephemeral-token API accepts only a mask-safe subset as liveConnectConstraints.config (GeminiRealtime.BuildConstraintConfig): responseModalities, speechConfig, temperature, topP, maxOutputTokens, sessionResumption — locked with lockAdditionalFields: []. systemInstruction, tools, and the transcription configs cannot be locked (their presence 400s the mint); they ride in SessionConfig and the matching client driver applies them verbatim. So on Gemini the token locks model + generation parameters, while prompt/tool integrity relies on the driver-applied config rather than a token-side lock. Token windows: ~10 minutes to open a new session, ~30 minutes total lifetime.
• ElevenLabs (ElevenLabsRealtime.CreateClientSession): the signed websocket URL is the credential (agent-scoped, ~15-minute open window; an already-open conversation continues past it). It pins the agent — and the agent’s server-side configuration carries the tool set (bound by the driver’s managed-agent ensure flow, so tools are effectively server-fixed). The per-session system prompt rides the conversation_initiation_client_data override in SessionConfig (the managed agent’s platform settings enable exactly that one field), applied by the client driver — so prompt integrity is driver-applied, tool integrity is agent-side.
• AssemblyAI (AssemblyAIRealtime.CreateClientSession): a one-time temp token (GET /v1/token, 300-second connect window, single websocket open). The token locks nothing but auth; the entire session object (prompt, tools, voice, turn detection) rides SessionConfig and is applied by the client driver as the first session.update frame — the provider confirms with session.ready before the driver reports 'listening'.

Server-bridged flow (wired, awaiting a media plane)

BaseAgent.executeRealtimeSession (in packages/AI/Agents/src/base-agent.ts) is fully implemented: it resolves the model, creates one long-lived AIPromptRun for the session (usage is checkpointed incrementally onto it — crash-safe by design), builds RealtimeSessionRunnerDeps (delegation via ExecuteSubAgent, actions via ExecuteSingleAction, transcript persistence, debounced usage checkpoints — default 5 s, plus the DB-driven NarrationInstructionsTemplate resolved through the shared realtime-narration.ts lookup), and drives RealtimeSessionRunner.Run(). The runner passes the tool set via StartSession params (the canonical registration path — for connect-bound providers like Gemini Live, the only effective one) and wires OnTranscript / OnToolCall / OnUsage / OnInterruption (barge-in aborts the in-flight delegated run via RealtimeToolBroker.AbortInFlight and cancels pending narration) / OnError (a Fatal error finalizes the session via Stop() instead of idling on a dead socket). The runner also consumes delegation progress itself: each significant event is fed to SendContextNote?.() and paced into RequestSpokenUpdate?.() spoken digests (see §7) — both feature-detected, since they are optional capabilities.

What is not wired yet for server-bridged: a client media transport (browser audio in/out of the server socket — IRealtimeSession.SendInput / OnOutput have no client-facing pipe). The client-direct topology is the path Explorer voice uses today.

---

4. Session Lifecycle

A realtime session is a long-lived durable record backed by volatile in-memory resources (sockets, the mic, channel plugins). The lifecycle machinery keeps the durable state from drifting away from process reality.

Schema (migration V202606090930__v5.41.x__AI_Agent_Sessions_Channels.sql)

• AIAgentChannel (MJ: AI Agent Channels) — the pluggable channel-definition registry: Name (unique), ServerPluginClass, ClientPluginClass, TransportType (PubSub | WebRTC | WebSocket), ConfigSchema, IsActive. Reference data, seeded via metadata/ai-agent-channels/.
• AIAgentSession (MJ: AI Agent Sessions) — the long-lived session: AgentID (the co-agent), UserID, Status (Active | Idle | Closed), CloseReason, ConversationID, LastSessionID (resume chaining), HostInstanceID, Config (free-form JSON — authoritative targetAgentID, observability run ids, pendingFeedbackRunID), LastActiveAt, ClosedAt.
• AIAgentSessionChannel (MJ: AI Agent Session Channels) — one row per channel attached to a session: Status (Connecting | Connected | Paused | Disconnected), Config (the channel’s state of record, e.g. the serialized whiteboard), LastActiveAt, DisconnectedAt; unique per (AgentSessionID, ChannelID).
• Additive nullable FKs: AIAgentRun.AgentSessionID, ConversationDetail.AgentSessionID (the timeline/grouping keys), AIAgent.DefaultCoAgentID, AIAgentType.DefaultCoAgentID.

The column is named AgentSessionID everywhere — deliberately distinct from the per-connection GraphQL transport sessionID (a browser-generated correlation id). They are orthogonal and never merged.

SessionManager (packages/MJServer/src/agentSessions/SessionManager.ts)

Stateless per-request lifecycle manager (every method takes contextUser + request-scoped IMetadataProvider — never the global):

• CreateSession — CanRun authorization first (denial throws SessionAuthorizationError before any write), then resolve-or-create the Conversation, then persist the session Active, stamped with this host’s HostInstanceID .
• CloseSession — terminal and idempotent: Status = 'Closed', ClosedAt, CloseReason; disconnects all channel rows; finalizes the co-agent observability runs stored in the session config. An already-closed session is a no-op that keeps its original CloseReason.
• Heartbeat — coalesced: at most one LastActiveAt write per session per 3 s (HEARTBEAT_MIN_WRITE_INTERVAL_MS), so chatty channels never become a DB write storm. Reactivates Idle → Active; never reactivates Closed. Tool relays and transcript relays heartbeat implicitly.
• MarkIdle — Active → Idle when the last channel goes quiet.

CloseReason (SessionCloseReason) is the close-provenance enum the dashboards key off: 'Explicit' (user hang-up — the default), 'Janitor' (orphan reconciliation), 'Shutdown' (graceful host drain), 'Error' (failure-path teardown, e.g. a failed mint). NULL means legacy/pre-column rows.

The janitor (SessionJanitor) and host identity

HostInstance.ts gives each server process a stable identity hostname:pid:bootId (the bootId UUID is minted once at module load, so two boots of the same host are distinguishable). SessionJanitor is a BaseSingleton + IShutdownable that runs three reconciliation paths, all writing through SessionManager.CloseSession (so Record Changes captures transitions and channel rows disconnect consistently), all paging with keyset (AfterKey) pagination (200 rows/page):

1. RunStartupRecovery (once at boot) — closes Active/Idle sessions whose HostInstanceID matches this host’s hostname: prefix but a different boot — crash/redeploy orphans. CloseReason = 'Janitor'.
2. RunStalenessSweep (every 60 s by default) — closes any Active/Idle session with LastActiveAt older than 15 minutes (closeThresholdMinutes), regardless of host — catches owners that died without a clean reboot. Idempotent and safe to run concurrently on every instance.
3. RunShutdownDrain (from Shutdown() during the ShutdownRegistry drain) — closes this exact host instance’s own live sessions with CloseReason = 'Shutdown', so a clean redeploy never strands rows for the next boot’s janitor.

Continuity: lastSessionId + channel-state restore

A closed session is never reopened; resuming creates a new session chained via LastSessionID (mirroring AIAgentRun.LastRunID). The conversation is the durable thread; sessions are time-boxed episodes along it.

Continuity is real, not cosmetic — a lastSessionId on StartRealtimeClientSession triggers two restore paths:

• Channel state: loadPriorChannelStatesJson() — the prior session’s AIAgentSessionChannel.Config blobs come back as PriorChannelStatesJson (a { channelName: stateJson } map), and VoiceSessionService.applyPriorChannelStates() offers each entry to the matching plugin via BaseRealtimeChannelClient.RestoreState. The whiteboard you drew last call reappears in the next one.
• Model memory: loadPriorTranscript() walks the prior session chain (up to 5 legs, cycle-guarded) and frames the newest 30 turns / 8,000 chars of persisted captions into the new session’s system prompt, so the model remembers what was said on the previous leg(s) — not just what was drawn.

Both are strictly best-effort and ownership-checked: a prior session owned by another user, an oversized state (> 2,000,000 chars per state and in total), or a malformed payload is logged and skipped — a session start never fails because of restore.

Co-agent pairing & effective configuration

Migration V202606121100__v5.41.x__Realtime_CoAgent_Pairing_And_TypeConfig.sql adds two opt-in metadata layers on top of the co-agent resolution chain.

Pairing semantics — universal vs constrained

AIAgentPairedAgent (MJ: AI Agent Paired Agents) is an OPT-IN junction between a Realtime-type co-agent and the targets it can front (CoAgentID, TargetAgentID, IsDefault, Sequence). Pairings are never mandated:

Co-agent state	Runtime targetAgentId supplied	Behavior
Zero pairing rows (universal)	yes	Today’s behavior, untouched — fronts the supplied agent. Zero-config deployments need zero metadata.
Zero pairing rows	no	Clear error — a universal co-agent has no default target to fall back to.
Has pairing rows (constrained)	yes, in the list	Accepted (canonical row casing wins).
Has pairing rows	yes, NOT in the list	Clear structured error — the rows RESTRICT the co-agent to its prebuilt target list.
Has pairing rows	no	The IsDefault row stands in; no default row → clear error asking for an explicit target.

Resolution lives in RealtimeClientSessionResolver.resolveConstrainedTargetAgentID() (rows loaded by Sequence). Pairing is a targeting constraint, not a security boundary — CanRun on the resolved target remains the security gate, applied immediately after; a failed pairing query therefore degrades to the universal behavior (logged) rather than breaking calls. Server-side write invariants (MJAIAgentPairedAgentEntityServer.ValidateAsync in MJCoreEntitiesServer): the co-agent must be an Active agent of the Realtime type, at most one IsDefault = 1 row per co-agent, and CoAgentID ≠ TargetAgentID (defense in depth over the CHECK constraint).

The effective-configuration merge

Realtime behavior is configured in three JSON layers, deep-merged per key (later wins; plain objects merge, arrays/primitives replace):

AIAgentType.DefaultConfiguration  ←  AIAgent.TypeConfiguration  ←  runtime overrides (configOverridesJson)
(type-level defaults)                 (per-agent layer)              (per-session, authorization-gated)

The pure implementation is packages/AI/Agents/src/realtime/realtime-coagent-config.ts (DeepMergeConfigs, ParseRealtimeTypeConfiguration — tolerant, malformed layers contribute nothing — and ResolveEffectiveRealtimeConfig, which normalizes into the typed RealtimeCoAgentConfig). The canonical shape (the Realtime type’s ConfigSchema is seeded to it):

{ "realtime": {
    "modelPreference": "<AI Model name or ID>",
    "voice": { "default": { "tone": "…", "speakingStyle": "…" },
               "providers": { "openai": { "voice": "alloy" }, "elevenlabs": { "voiceId": "…" },
                              "gemini": { "voice": "…" }, "assemblyai": { "voice": "…" } } },
    "allowUserModelOverride": true,
    "narration": { "paceMs": 8000 } } }

How each knob is applied at mint (RealtimeClientSessionService.PrepareClientSession, mirrored on the server-bridged path in BaseAgent):

• modelPreference (Name or ID) participates in realtime-model selection between the explicit runtime choice and the default: explicit preferredModelId (strict, fails loud, authorization-gated when deviating) → configured preference (tolerant — an unsatisfiable metadata preference logs and falls through, mirroring the chain’s metadata steps) → highest-PowerRank default.
• voice.default (tone / speakingStyle) is appended to the server-built companion system prompt as a short “Voice & manner” section, right after the co-agent’s own prompt.
• voice.providers.<provider> is matched to the resolved vendor’s DriverClass by normalized prefix (openai ↔ OpenAIRealtime, …) and merged into the session’s open Config bag — the same opaque per-driver pact every other config entry rides (a caller-supplied runtime Config wins per key). Server-bridged reach today: OpenAI spreads the bag into session.update; Gemini merges it last; AssemblyAI maps voice → output.voice. ElevenLabs provisions voice on its managed agent and does not consume a per-session bag (per-driver TODO). Client-direct reach: OpenAIRealtime.CreateClientSession does not yet fold params.Config into the minted session (per-driver TODO) — the resolver instead surfaces the full resolved config as EffectiveConfigJson on StartRealtimeClientSessionResult so client drivers can apply provider voice settings browser-side.
• narration.paceMs drives spoken-progress spacing: server-bridged, it feeds RealtimeSessionRunnerDeps.NarrationPaceMs (replacing the built-in 8 s spacing floor); client-direct, pacing is enforced client-side, so the value is surfaced as NarrationPaceMs on the start result.

Runtime override precedence & authorization (configOverridesJson on StartRealtimeClientSession):

Request	Gate	Outcome
Plain start (no overrides, no explicit model)	CanRun on the target only	Never touches the new gate.
Target selection (within a pairing list / universal) and coAgentId choice	CanRun only	Normal user flow — deliberately ungated.
configOverridesJson	Realtime: Advanced Session Controls authorization (hierarchy-aware, fail-closed when the seed is absent)	Unauthorized → structured rejection (never a silent ignore). Must be a JSON object.
preferredModelId equal to the metadata modelPreference	none (not a deviation)	Allowed for everyone.
preferredModelId deviating from metadata	the authorization AND effective allowUserModelOverride ≠ false (the policy blocks even authorized callers)	Denial is a structured rejection naming the reason.

Gate implementation: RealtimeClientSessionResolver.assertRuntimeOverridesAuthorized() → the pure EvaluateRuntimeOverrideAuthorization() decision (unit-tested matrix) + AuthorizationEvaluator.UserCanExecuteWithAncestors over the request provider’s cached Authorizations. The authorization is seeded in metadata/authorizations/.realtime.json.

Write-side schema enforcement: when an agent type publishes ConfigSchema, MJAIAgentEntityServer.ValidateAsync (in MJCoreEntitiesServer) validates TypeConfiguration against it with a dependency-free JSON-Schema-subset validator (json-schema-lite.ts: type / required / properties / enum / items / additionalProperties). Non-object configuration always fails; a malformed ConfigSchema only WARNS (a metadata bug on the type row must not brick agent saves).

App awareness — the app cascade layer, mint-time context, and multi-agent delegation

The co-agent is app-aware and can voice one lead while delegating to many colleagues. Three additive capabilities, all flowing through the one shared prep (RealtimeClientSessionService.PrepareRealtimeSessionParams) so client-direct and server-bridged behave identically:

1. The app cascade layer. When the session knows its ApplicationID (threaded via PrepareClientSessionInput.ApplicationID — the StartRealtimeClientSession applicationId arg client-direct, or params.data.applicationId server-bridged), the app’s Application.AgentSettings contributes a NEW cascade layer between the target agent and the runtime override:

AIAgentType.DefaultConfiguration  <  co-agent TypeConfiguration  <  target TypeConfiguration  <  APP (Application.AgentSettings.Realtime)  <  runtime override

AgentSettings.Realtime (persona / disclosure / model preference) + AgentSettings.RelevantAgents are mapped to the canonical {"realtime":{…}} shape by BuildAppRealtimeOverridesJson and merged via the extended ResolveEffectiveRealtimeConfig(typeDefault, agent, override, target?, appSettings?, dynamic?). Scalars (persona/disclosure/model) follow the per-key cascade; allowedAgents is the deliberate exception — see below.

2. Mint-time app context. PrepareClientSessionInput.AppContext (an AppContextSnapshot from @memberjunction/ai-core-plus — where the user is, what they see, the live capability manifest) is rendered by FormatAppContextNote into a CURRENT APP CONTEXT section of the companion system prompt, so the co-agent knows the user’s situation from its first word (the session-start half of client-context delivery). Client-direct passes it as the appContextJson arg; server-bridged as params.data.appContext.

2b. Continuous streaming + acting — the ClientContextChannel. The mid-session half is a headless interactive channel (ClientContextChannel, IsHeadless: 1, never mounts a tab — server half ClientContextChannelServer in @memberjunction/ai-agents, browser half in @memberjunction/ng-conversations). It does two things, both client-side in the client-direct topology (no server round-trip):

• Perception (stream). It subscribes to the host’s live app-context stream (RealtimeChannelContext.AppContext$, fed by Explorer’s MJExplorerAppComponent from NavigationService — every SetAgentContext / nav change re-publishes the AppContextSnapshot, now including a Capabilities.Tools manifest of what’s callable right now) and pushes each change as a compact SendContextNote. So as the user navigates, the co-agent’s picture stays current.
• Action — the ContextTool proxy. It declares ONE stable provider tool, ContextTool, so the tool surface never changes mid-session (connect-bound providers reject re-declaration); the live catalog of valid actions rides context. The model calls ContextTool({ action, params }); the channel routes it to RealtimeChannelContext.ExecuteClientTool, which runs the surface handler the host registered from SetAgentClientTools (mirrored into the realtime session via RealtimeSessionService.RegisterAppClientTools), and returns a structured result. This is what lets the co-agent navigate / show / do things in the app on the user’s behalf, executed in the browser.

The seam is boundary-clean: the generic channel never imports Explorer — it is fed AppContext$ + ExecuteClientTool through the (optional) RealtimeChannelContext members, which the RealtimeSessionService implements and Explorer drives. The same SetAgentContext/SetAgentClientTools call a dashboard already makes for the async chat agent now feeds the realtime co-agent too — one surface API, both agents.

3. Multi-agent delegation (lead + colleagues). The effective realtime.allowedAgents is a union-accumulated set (NOT array-replaced) across every cascade layer (accumulateAllowedAgents) plus the app’s RelevantAgents plus runtime/channel-registered additions, deduped by agentId (later layers win per-field). The lead’s identity stays single (the voiced target / app default agent); the union is who the lead can call. At mint, the colleagues are rendered into the framing (BuildRealtimeAgentFraming(leadName, surfaceClause, colleagues) → BuildColleaguesClause) with per-target disclosure guidance. The model names a colleague in the invoke-target-agent arguments ({ agent, request }); RealtimeClientSessionService.resolveDelegationTarget validates the name against the session’s persisted allowed union (client-direct: persisted on AIAgentSession.Config_ and read back per relayed call; server-bridged: from the live EffectiveConfig), routes to that agent, or returns a structured “not available” result naming the valid colleagues so the model self-corrects. CanRun on the resolved agent remains the security boundary — the union is an affordance filter, both apply.

Delegation disclosure (RealtimeDisclosurePolicy = 'silent' | 'mention' | 'hand-voice', default 'mention') — how the co-agent narrates a handoff: mention names it (“let me get Skip on this”), silent absorbs the result and speaks it as its own, hand-voice is reserved. Resolved per-target by GetDisclosureForTarget (per-entry override → effective default from the scalar cascade). Configurable at the type default, co-agent, app, and per-target levels.

Live session capabilities — RealtimeSessionCapabilities + Reconfigure

A live IRealtimeSession exposes a small capability surface (the realtime-session analogue of the bridge’s IBridgeProviderFeatures and of BaseRealtimeModel.SupportsClientDirect) so the container can ask “is it safe to call X?” instead of blind-invoking optional methods that no-op — or can’t be supported — on some providers. Both members are optional on the interface (@memberjunction/ai, baseRealtime.ts), so a driver that hasn’t declared them is treated conservatively (unsupported), and the 6 existing drivers compile unchanged:

• Capabilities?: RealtimeSessionCapabilities — currently { CanReconfigureTurnMode }: whether the session can change its turn-taking / auto-response mode on a live socket without reconnecting. Grow this object as providers gain runtime abilities.
• Reconfigure?(params: RealtimeReconfigureParams): void — applies a live change (e.g. { DisableAutoResponse: true }). Gate on the capability before calling. OpenAI implements it via a partial session.update and reports CanReconfigureTurnMode: true; Gemini Live’s activity detection is fixed at connect, so it reports false and omits the method.

Prime consumer: the bridge engine’s first-agent re-gating (AIBridgeEngine.ReconfigureSessionToMeeting → meeting mode when a room becomes multi-agent — see the Bridges guide §9). As models gain mid-session reconfiguration, a driver flips one flag and the container starts using it with zero container changes.

Turn moderator (multi-agent turn-taking)

When two or more agents share one realtime room (a LiveKit panel, a meeting), the hard question is who speaks next. The original answer was a per-agent regex name-match (RegexAddressedMatcher): each agent independently checked “was I addressed?” That routes direct address but nothing else — it can’t bring a relevant-but-unaddressed agent in, can’t let two agents have a productive back-and-forth, and can’t tell a useful exchange from an unproductive ping-pong loop. The turn moderator replaces it with a room-level decision.

Once per turn, a single fast LLM looks at the room roster + the recent diarized conversation and returns the ordered agent(s) who should speak next — zero, one, or several. It routes direct address and relevance (send a question to Sage and Skip when both matter), lets a productive agent↔agent discussion continue, and goes quiet on unproductive ping-pong or when nobody should speak (hand back to the human). The engine speaks the returned agents serially via the floor — never overlapping. The audio plane is untouched: every agent always hears the raw room audio; the moderator only decides when each agent is triggered to commit and speak. (It’s called a moderator, not a judge, because it brings agents in as much as it restrains them — the informal nickname is “nanny mode,” but that undersells the half that matters.)

Prompt, not agent

The moderator is RealtimeTurnModerator (packages/AI/Agents/src/realtime/realtime-turn-moderator.ts; exported as RealtimeTurnModeratorDecision, the (ctx) => Promise<string[]> function wired into the bridge engine). It is a stateless yes/who classification — no planning, tools, or sub-agents — so it runs as an AIPromptRun via AIPromptRunner, not an agent run. Each decision’s agentRunId is tied to the co-agent’s AIAgentRun, so the per-turn “who spoke and why” trail (including the moderator’s structured reason per speaker and an optional note) is fully observable through standard prompt-run logs — and those logs are exactly the data for deciding when this mechanism can be relaxed as realtime models get smarter. The prompt returns structured { speakers: [{ agent, reason }], note }; the plugin maps the returned names back to roster AgentSessionIDs (in order, de-duplicated).

Config cascade + the turnTaking shape

Two parts of the effective-config cascade carry turn-taking, and they live at different layers:

• Room-wide moderator brain — the turnTaking.moderator block on the Realtime agent type’s DefaultConfiguration (metadata/agent-types/schemas/realtime-type-default-config.json). A room has exactly one moderator.
• Per-agent participation mode — turnTaking.mode on the target agent’s TypeConfiguration: 'proactive' (default — may be brought in unaddressed when the moderator judges it relevant) or 'addressed-only' (speaks only when directly addressed by name).

Both ride the existing cascade (agent-type DefaultConfiguration ← co-agent TypeConfiguration ← target agent TypeConfiguration ← runtime override — see §4 the effective-configuration merge). The shape (in packages/AI/Agents/src/realtime/realtime-coagent-config.ts):

{ "realtime": { "turnTaking": {
    "mode": "proactive",                 // per TARGET agent (TypeConfiguration)
    "moderator": {                       // room-wide (agent-type DefaultConfiguration)
        "promptId": "<AI Prompt ID>",    // authored as @lookup:, stored as the resolved ID
        "contextWindowTurns": 30,        // diarized turns the moderator sees; clamped ≤ 50
        "maxCharsPerTurn": 240,          // per-turn clip (token savings + cacheable prefix)
        "maxConsecutiveAgentOnlyTurns": null,  // null = no cap (trust the model's progress read)
        "timeoutMs": 800,                // per-decision budget on the latency-critical path
        "onError": "silent",             // 'silent' (no one speaks) | 'addressed-only' (cheap fallback)
        "prestageOnAgentSpeech": true    // run the next decision during the prior agent's playback
} } } }

GetEffectiveModeratorConfig / GetEffectiveTurnMode are the typed accessors; absent fields fall back to REALTIME_MODERATOR_DEFAULTS. GetEffectiveModeratorConfig returns null when no promptId is configured — which the engine reads as “no moderator, fall back to the matcher.”

Serialized multi-route + pre-staging

The moderator can name several speakers; the engine queues them and drains the queue one at a time through the room floor, so a multi-agent route is heard as a clean sequence, never a pile-up. The moderator also runs on agent turns, not just human ones — that powers both agent↔agent continuation and pre-staging: when prestageOnAgentSpeech is on (the default), the next decision runs during the prior agent’s audio playback (the model emits its full response text seconds before the user finishes hearing it), so the agent→agent hand-off pays ~zero added latency. A human barge-in discards the pre-staged decision (the user changed the topic). The optional maxConsecutiveAgentOnlyTurns is a hard backstop against runaway agent-only loops; left null, the room relies on the moderator’s own progress assessment so a genuine discussion is never gated by a counter.

The model — small, fast, swappable

A small/fast LLM is the right call because this runs every turn on the latency-critical path. The seeded prompt ("Realtime: Turn Moderator", metadata/prompts/.realtime-prompts.json) is bound to GPT-OSS-120B on Cerebras. The structured output, plus putting the static roster near the top of the prompt and the variable conversation at the bottom, maximizes prefill caching. The whole mechanism is metadata-driven: swap the model binding (e.g. to Gemma on Cerebras) or retune the windows entirely in metadata, no code change — and the prestageOnAgentSpeech design means a slower model still hides most of its latency.

No-moderator fallback

When no moderator is injected into the engine — or no promptId is configured — the engine falls back to the per-agent RegexAddressedMatcher broadcast (still present): the room’s utterance is broadcast to every agent’s TurnPolicy and each independently decides “was I addressed?” The plugin itself also degrades to a crude name-contains safety net only when no prompt is configured, so a room is never left mute by a metadata gap. The engine seam (SetTurnModerator, the lookback buffer, the serialized queue, barge-in invalidation) is documented in the Bridges guide §9.

---

5. Channels — The Heart of the System

A channel is more than transport. An interactive channel is a bidirectional surface the session’s single realtime agent both perceives and acts upon. One agent per session is a hard invariant — everything every channel observes feeds that one agent, and every channel mutation comes from it.

Every interactive channel implements two directions:

1. Perception (channel state → agent context). The channel serializes compact, coalesced deltas of its state and feeds them into the live model as background context notes — replace-current-state semantics, never full snapshots, never bitmaps.
2. Action (agent → channel mutation). The channel exposes mutation tools the model can call; they execute and the result feeds back as the tool_response the model narrates.

The definition registry

MJ: AI Agent Channels rows define what channels exist (see §4 schema). At session start VoiceSessionService reads the active rows and instantiates one plugin per row via the ClassFactory (ClientPluginClass key) — one fresh instance per session, never a singleton. Registry failures degrade to “no channels”; the voice session always proceeds. The server half mirrors this exactly: when SessionManager.CreateSession mints the durable session, RealtimeChannelServerHost reads the same active rows and resolves each ServerPluginClass into one fresh BaseRealtimeChannelServer instance per session (unregistered keys skip with a log, never fatal).

The shipped channels: Whiteboard, Media, Remote Browser (all surface-bearing — they mount a tab), and ClientContextChannel (the headless app-context wire — IsHeadless: 1, no tab; perception + the ContextTool proxy, see § App awareness 2b). The registry row carries extra presentation/behavior metadata: IsHeadless (skip surface-mounting) and a UIConfig JSONType bag (DisplayName / GroupName / Color / Icon for surface channels’ tab chrome).

Per-session state of record

Each attached channel gets an AIAgentSessionChannel row. Its Config column is the channel’s state of record — written by the SaveSessionChannelState mutation through a client-side debounce (3 s, latest-state-wins, flushed at teardown with the session id captured while live so the final save lands even on the just-closed session). The model’s perception feed is derived from this state, never the source of truth. A Closed session still accepts state saves (the final flush legitimately lands after CloseAgentSession). Before persistence, each landed save routes through the channel’s server plugin (when one resolved — see below), which may validate/normalize the payload; a plugin can transform a save but can never lose or block one.

The BaseRealtimeChannelClient contract

A concrete plugin contributes everything its channel needs, so the host shell carries zero channel-specific wiring:

Member	Role
ChannelName	Must match the registry row’s Name; the persistence + restore key and the tab key
ToolNamePrefix	One prefix-routed local executor per plugin — Whiteboard_* calls run in the browser, never the server relay
GetToolDefinitions()	The client-executed tool declarations, aggregated into the session mint (clientToolsJson)
ApplyAgentTool(toolName, argsJson)	Executes one tool locally; MUST work with no surface bound (apply to the state engine, skip the UI garnish) — calls can arrive while the tab pane is collapsed
GetSurfaceComponent() / BindSurface() / UnbindSurface()	The Angular surface the overlay creates dynamically in a channel tab; the plugin wires the component’s inputs/outputs — the host treats the instance as opaque
SerializeState() / RestoreState()	The state of record + resume rehydration (tolerant by contract: bad input returns false, never throws)
RequestFocusExit()	Routed from the floating call pill back to whichever channel holds focus
Initialize(ctx) / Dispose()	Lifecycle bracket; Context is the plugin’s only line to the session

The host context (RealtimeChannelContext) supplies: SendContextNote (the perception feed), RequestSave (debounced state-of-record persistence), SaveAsArtifact (snapshot to a first-class versioned MJ: Artifacts record), SetFocusMode (the focus-layout request), and AgentName.

The server half — BaseRealtimeChannelServer

The same registry row’s ServerPluginClass resolves a server-side plugin (BaseRealtimeChannelServer in @memberjunction/ai, beside BaseRealtimeModel), hosted per session by the RealtimeChannelServerHost singleton in @memberjunction/ai-agents. Because today’s shipped topology is client-direct (channel tools execute in the browser), the server half’s contract is the durable lifecycle, not socket ownership — a pragmatic adaptation of the plan’s IAgentChannelServer (see the base class doc header for the deviation notes; socket/transport members are deferred with the unified-transport track):

Member	Role
ChannelName	Must match the registry row’s Name (the host routes by the row name and warns on a mismatch)
Initialize(ctx) / Dispose()	Lifecycle bracket; ctx is plain session facts (AgentSessionID, AgentID, UserID, ConversationID)
OnSessionStarted()	Fired right after the durable session row persists (SessionManager.CreateSession)
OnChannelStateSave(stateJson)	Fired when the client’s debounced state save lands, pre-persistence; return a replacement string to normalize the persisted state of record, or null to keep the original. Throw/garbage → original persisted — a plugin can never lose or block a save
OnSessionClosed(closeReason)	Fired from every close provenance — explicit hang-up, janitor sweeps, shutdown drain, error teardown all funnel through SessionManager.CloseSession
ToolNamePrefix	The shared prefix of this channel’s server-executed tools (e.g. MeetingControls_); the host routes any prefixed tool call back to this plugin. Default '' (no server tools)
GetServerToolDefinitions()	This channel’s server-executed tool declarations — may be runtime-computed (per session / per platform state), not only constants. Default []. The host aggregates these across a session’s live plugins (RealtimeChannelServerHost.GetSessionServerTools) and feeds them into RealtimeSessionRunner.ServerChannelTools
ExecuteServerTool(toolName, argsJson)	Executes one server-side tool; routed by ToolNamePrefix. Default returns a structured “not implemented” error (never throws) — the host additionally wraps any throw

Hard rules baked into the host: one fresh instance per session (never a singleton); every hook is wrapped in try/catch (a throwing plugin is logged and the session/persistence proceeds untouched); resolution is skip-with-log for unregistered keys; and disposal is deferred briefly after close (15 s linger) so the client’s legitimate post-close final state flush still routes through OnChannelStateSave.

The reference implementation is WhiteboardChannelServer (packages/AI/Agents/src/realtime/whiteboard-channel-server.ts, key 'WhiteboardChannelServer' — the value the Whiteboard row’s ServerPluginClass has carried since the seed): it guards the persisted state of record by validating that each landed save parses as a JSON object (flagging corrupt payloads loudly now instead of at the next resume’s RestoreState) and canonicalizing valid payloads to compact JSON. Register a new server half with @RegisterClass(BaseRealtimeChannelServer, '<key>') plus a Load...Server() no-op called from a static server path (the Whiteboard’s is invoked from MJServer’s agentSessions/index.ts).

Server-executed channel tools (Phase 2 — now wired). A server-side channel can contribute a dynamic server-executed tool vocabulary (GetServerToolDefinitions, possibly runtime-computed) plus its execution (ExecuteServerTool), namespaced by ToolNamePrefix. The RealtimeChannelServerHost aggregates every live plugin’s tools for a session (GetSessionServerTools) and routes each prefixed call back to its owner (ExecuteSessionServerTool, longest-prefix-wins). The server-bridged RealtimeSessionRunner accepts these via the new additive ServerChannelTools dep + an ExecuteServerChannelTool handler — they are registered alongside (and after) ExtraTools, and a contributed tool routes to the channel handler before the generic executor. This is purely additive: the client-direct path and runs without channels are unchanged. The reference server-only channel is MeetingControlsChannel (roster / hand-raise queue / who’s-speaking / timer → facilitator; see the Realtime Bridges guide).

Optional client surface (Phase 2). A server-only channel has no MJ client surface — BaseRealtimeChannelClient.GetSurfaceComponent() now defaults to null (HasSurface() is false), and the overlay skips a surfaceless channel’s tab while still wiring its tools + perception. A platform-native surface (e.g. a Zoom whiteboard) lives on the platform, not in MJ.

Still deferred: socket/media members (OnClientMessage, SendToClient) remain deferred with the unified-transport track.

The implicit voice/text channel

Voice itself is not a registry row — it is the session: the provider socket, live captions, the typed-input composer, and the persisted transcript. Concretely:

• Every final, normal transcript turn (both roles) becomes a caption and a Conversation Detail stamped with AgentSessionID (RelayRealtimeTranscript → persistTranscriptTurn; roles map user → 'User', everything else → 'AI'). That is the durable transcript — raw audio is not persisted.
• Typed text (VoiceSessionService.SendText) is injected as a user turn through the same collision-safe response path tool results use, then rides the same caption + relay path as speech — a typed turn behaves identically to a spoken one.
• Narration turns (Kind: 'narration') are deliberately ephemeral — never captions, never persisted (see §7).

The Whiteboard — canonical interactive channel

The whiteboard now lives in two layers, and the split is the point:

• @memberjunction/ng-whiteboard (packages/Angular/Generic/whiteboard/ — README) is the generic, reusable board: the Angular-free WhiteboardState engine, the Whiteboard_* tool API, the host/board/toolbar/zoom/popover/snapshot components, exports, the widget input bridge, and the right-click context menu. It has no MJ-metadata, AI-framework, or Router dependency — any Angular app can use it. The README there is the authoritative whiteboard reference.
• RealtimeWhiteboardChannel (packages/Angular/Generic/conversations/src/lib/components/realtime/whiteboard/whiteboard-channel.ts, registered as 'RealtimeWhiteboardChannel', seeded registry row Name: 'Whiteboard') is the thin channel plugin (~200 lines) that wires that package into a realtime session: it owns one WhiteboardState per session, declares WHITEBOARD_TOOL_DEFINITIONS to the model, routes Whiteboard_* calls to the bound RealtimeWhiteboardHostComponent (UI garnish) or the pure ApplyWhiteboardAgentTool when the pane is collapsed, pipes the host’s coalesced SceneDelta output into Context.SendContextNote, requests debounced state-of-record saves on every Changed$, and handles restore/focus/artifact snapshots. Read it before writing a new channel — it is the reference for how little a channel plugin should be.

Tool surface (prefix Whiteboard_, 11 tools): AddNote (sticky notes), AddShape (labeled rect/ellipse/diamond boxes), AddText (free-floating labels, with explicit wrap-width support), AddMarkdown (v3 — a rendered rich-text panel, ≤32,000 chars), AddHtml (v3 — an interactive sandboxed HTML widget, ≤64,000 chars), UpdateContent (v3 — rewrite an existing item’s text/markdown/HTML in place), DrawConnector (arrows between items by id or absolute points), Highlight (“pointing without touching” — a pulsing region the user dismisses with a click), MoveItem, RemoveItem, StyleItem. Every tool returns a WhiteboardToolResult { success, itemId?, summary?, error? } — the summary is what the model narrates (“Added a sticky note…”); failures are structured so the model can self-correct conversationally. Each call runs as one undo batch, so a single toast-Undo reverts a whole tool effect.

Markdown panels & sandboxed HTML widgets (v3): markdown renders through the shared sanitized mj-markdown component (no raw-HTML passthrough); HTML widgets render in an iframe sandboxed to allow-scripts only (opaque origin — no parent DOM, session, cookies, or storage access), with off-screen frames rendered as placeholders. Users can edit both in-board through the rich editors (commits flow through the cancelable ContentApplying/ContentApplied events). The input bridge (MJWhiteboard.submit(data) — whiteboard-widget-bridge.ts) is the tutoring loop: the agent draws a quiz/micro-form widget, the user answers inside it, the validated, size-capped (8,000 chars) submission flows out through postMessage → WidgetSubmitted → the channel plugin sends the model [whiteboard] the user submitted input in widget "…": {data} — and the agent reacts in voice. A right-click context menu (pure model in whiteboard-context-menu.ts) offers per-item-kind actions on the board.

Perception model: WhiteboardState is the single mutation API used by both the user’s board tools and the agent’s channel tools. Every mutation appends to a compact change journal and emits Changed$; the bound surface coalesces changes over 750 ms and emits one SceneDelta, which the plugin pipes into the model as a [whiteboard]-prefixed context note. Perception etiquette rides in the note itself: the prefix text instructs the model that board updates are background context — “do NOT comment on minor edits; only mention it if the change is significant” — so any realtime model behaves, regardless of system-prompt sync state. BuildSceneDelta(sinceToken) collapses the journal (multiple moves of one item → one moved entry) with replace-current-state semantics; BuildSceneSummary() powers the “What the agent sees” popover — the user can literally inspect the agent’s perception feed. When the user clicks Undo on an agent-action toast, the plugin tells the model: [whiteboard] user undid your last change.

Ownership / the violet system: every item carries Author: 'user' | 'agent'. The violet treatment is reserved for the agent — agent items render violet regardless of tint, the user’s pen/text palettes never offer violet, and ownership chips (“You” / the agent’s name) make authorship unambiguous on the board and in exports. Hosts can go further: the engine’s cancelable before-events (ItemAdding$ / ItemUpdating$ / ItemRemoving$) let a policy veto, e.g., the agent removing the user’s items.

Export / print (whiteboard-export.ts): BuildWhiteboardExportHtml renders one fully self-contained HTML document (inline CSS, zero external references — the documented exception to design tokens) for Download/Print; BuildWhiteboardExportSvg for SVG. Every piece of user/agent-authored text is HTML-escaped before touching the document; live widget HTML is never inlined in exports (no sandbox exists there — widgets export as placeholder cards with escaped source in a <details>).

Artifact snapshots: “Save to artifacts” snapshots the board JSON as a versioned MJ: Artifacts record via SaveSessionChannelArtifact — typed with the seeded Whiteboard artifact type (metadata/artifact-types/, ContentType: application/json, viewer DriverClass: WhiteboardArtifactViewerPlugin — the viewer component stays in the conversations package and renders through WhiteboardSnapshotComponent), best-effort linked into conversation history via a MJ: Conversation Detail Artifacts junction row against the latest session-stamped detail. On success the plugin tells the model the user saved the board, so it can reference the artifact naturally.

Restore-on-resume: RestoreState rehydrates a prior session’s board in place into the same WhiteboardState instance (LoadFromJSON), so the save subscription and any later surface binding keep pointing at one engine. Malformed payloads return false and the board starts fresh.

The Media channel’s agent media library (Collections as kits)

The Media channel (MediaChannelServer / RealtimeMediaChannel, seeded row Name: 'Media') lets the agent put images/video/audio/PDF/web on a shared surface. Beyond ad-hoc Media_ShowMedia({ url | fileId }), an agent can be given a curated, governed media kit it reasons over — built by reusing Artifacts + Collections, not a bespoke entity:

• A MJ: Collections of MJ: Artifacts IS the kit. The artifact (+ current version) describes the media — FileID → MJ: Files, MimeType, name, viewer, versioning, permissions. Bytes stream through the authenticated /media route. Nothing is duplicated.
• Agent-reasoning metadata lives per-membership on MJ: Collection Artifacts: Sequence (priority/order, pre-existing) + ContextDescription (the agent-facing “what this is / when to show it”) + Preload (eager hint). Per-membership means the same artifact can be framed differently in different kits.
• Binding: AIAgent.DefaultMediaCollectionID (FK → Collection) is the agent’s default kit; per-session resolution is runtime override > agent default > none.
• Per-session runtime override (wired). StartRealtimeClientSession accepts an optional mediaCollectionId arg — UUID-validated at the resolver boundary (IsValidUUID from @memberjunction/global; a malformed value is dropped + logged, the session still starts) and stored on the session config (RealtimeSessionConfig.mediaCollectionID → AIAgentSession.Config_). SessionManager hands the verbatim config blob to data-aware channels via RealtimeChannelServerContext.AgentSessionConfig; MediaChannelServer parses mediaCollectionID from it and passes it as the override (re-validated downstream — defense in depth). So a caller (e.g. a Praxis Protocol) can hand a session its own kit, overriding the agent default; absent ⇒ the agent default applies. Note: AgentSessionConfig (the persisted AIAgentSession.Config_) is intentionally distinct from the driver-level ClientRealtimeSessionConfig.SessionConfig “private pact” (§5.8) — same word, different layer.
• Server-side resolution, existing client tool. MediaChannelServer implements IRealtimeChannelServerDataAware (the host hands it the session contextUser + provider between Initialize and OnSessionStarted); on start it resolves the kit via agent-media-library.ts (buildAgentMediaContextNote) and SendContextNotes a manifest (fileId, type, display name, when-to-show, PRELOAD). The agent then surfaces items with the existing Media_ShowMedia({ fileId, mediaType, displayName }) — no new client tool, no client round-trips. An agent with no kit is unaffected.

The resolver (mediaTypeFromMimeType / resolveAgentMediaManifest / formatAgentMediaManifest) is exported from @memberjunction/ai-agents and unit-tested. See plans/praxis/AGENT_MEDIA_LIBRARY_PLAN.md.

Interactions between channels

The channels are not silos — they converge on the one shared model context and the one overlay shell:

• Voice narrates board actions. A Whiteboard_AddShape tool result’s summary returns through SendToolResult, and the model speaks it — the agent talks about what it just drew, in the same breath as the conversation.
• Context notes are the cross-channel bus. Whiteboard scene deltas, delegated-run progress, artifact-save notifications, and undo events all arrive as context notes in the single model conversation — the agent’s awareness of every surface lives in one place, which is exactly why the one-agent-per-session invariant exists.
• Focus mode is a shared layout protocol. Any channel can call Context.SetFocusMode(true); the overlay (realtime-session-overlay.component.ts) collapses the main call column so the surface owns the screen, with a floating call pill keeping mute/thread/end reachable; the pill’s exit routes back through RequestFocusExit() to whichever channel holds focus.
• Delegated-run artifacts surface beside channel surfaces. The overlay’s right panel is a tabbed surface (RealtimeSurfaceTabsModel): the Activity rail is always tab 1, one tab per artifact a delegated run produced (auto-opened + flash-highlighted on arrival, keyed by artifact version), and one tab per interactive channel. A delegated Sage run’s report artifact and the live whiteboard sit side by side in the same panel while the voice keeps going.

---

6. Future Channel Types (Envisioned)

[!NOTE] Everything in this section is forward-looking design discussion — none of it is implemented. It is here because the channel plugin contract was explicitly shaped to make these tractable, and because “what would it take” is the best test of the abstraction. For each candidate: the perception feed shape, the mutation tool surface, transport considerations, and what the existing contract covers vs. what would need extending.

Video (camera as perception)

• Perception: sampled camera frames (or provider-native video input — BaseRealtimeModel is deliberately modality-agnostic and IRealtimeSession.SendInput already takes raw media frames). Structured deltas don’t apply; this is the one channel class where the media plane itself is the perception feed.
• Tools: model-driven visual annotations over the video surface (Video_Highlight(x,y,label), Video_Snapshot() → an artifact) rather than mutations of the feed itself.
• Transport: WebRTC, full stop — frames are binary and latency-sensitive. The registry’s TransportType: 'WebRTC' value exists for exactly this.
• Contract fit: surface/tab/focus-mode and the tool path carry over unchanged. The gap is the perception direction: SendContextNote is text-only, so native video input needs either provider video-modality support through the driver (extending RealtimeSessionParams/client Connect) or a frame-sampling serializer that describes frames textually (cheap, lossy interim).

Screen-share (perceive the user’s screen, guide and highlight)

• Perception: like video, but the richer play is hybrid — periodic frames plus a structured accessibility/DOM digest for shared app windows (“dialog open: Save Changes; focused field: Name”). Structured digests are vastly cheaper than pixels and more reliable for guidance.
• Tools: Screen_Highlight(region|elementRef), Screen_Annotate(text, anchor) — guidance overlays rendered by the channel surface, never synthetic input into the user’s apps (a deliberate security boundary).
• Transport: WebRTC for frames; the structured digest is small enough for the control plane (PubSub).
• Contract fit: tools/surface/focus are covered (a screen-share would almost always run in focus mode). Same perception gap as video for the pixel path; the digest path works today via SendContextNote.

Collaborative documents (markdown / rich text co-editing)

• Stepping stone — SHIPPED: the whiteboard’s v3 markdown panels and sandboxed HTML widgets deliver most of this value inside the existing channel today — the agent drafts a rich-text section (Whiteboard_AddMarkdown), revises it in place (Whiteboard_UpdateContent), and the user drags/edits it with the in-board editors. A dedicated doc channel remains the eventual destination for true co-editing.
• Perception: section-level structural deltas (“§2 ‘Pricing’ edited by user; +2 paragraphs”), not character-level OT/CRDT traffic — the model needs document awareness, not keystrokes. The whiteboard’s journal → coalesced-delta pattern transfers directly.
• Tools: Doc_InsertSection, Doc_ReplaceRange, Doc_Comment, Doc_ProposeEdit (a suggestion the user accepts — the doc analogue of the whiteboard’s violet ownership + undo toast).
• Transport: PubSub-class control traffic; nothing binary. State of record = the serialized doc on the session-channel row; snapshots via SaveAsArtifact exactly like the board.
• Contract fit: this is the channel the contract fits best — everything (tools, perception, state of record, restore, artifact snapshots) works without extension. The work is the editor surface and the delta serializer.

Maps / geo canvases

• Perception: viewport + feature deltas (“user panned to Chicago; zoom 11; 3 pins visible: …”) — replace-current-state, like the board.
• Tools: Map_AddPin, Map_DrawRegion, Map_FlyTo, Map_Highlight. Agent-authored features get the violet ownership treatment.
• Transport: PubSub; tiles come from the map provider, not the session.
• Contract fit: clean — a near-isomorph of the whiteboard over a geographic coordinate space.

Dashboards / live-data surfaces

• Perception: the surface pushes data-state deltas (“KPI ‘Churn’ crossed threshold; user filtered to Q3”) so the agent can voice what the user is looking at.
• Tools: Dash_SetFilter, Dash_FocusWidget, Dash_Explain(widgetId) — the last one likely combines channels: a delegated run analyzes the widget’s data while the voice narrates.
• Transport: PubSub; the data itself flows through MJ’s normal RunView/RunQuery paths, not the session.
• Contract fit: covered; the interesting question is policy — debouncing live-data churn so the perception feed doesn’t flood the context (the whiteboard’s coalescing policy generalizes).

Code editors

• Perception: file + selection + diagnostic deltas (“user opened foo.ts; cursor in resolveModel; 2 TS errors”), never whole files on every keystroke.
• Tools: Code_ProposeEdit(file, range, replacement) (rendered as an accept/reject diff — ownership semantics again), Code_Annotate, Code_NavigateTo.
• Transport: PubSub.
• Contract fit: covered structurally; the real cost is the surface (an embedded editor) and the safety policy that proposals are never auto-applied.

Form-filling / guided UI

• Perception: form-state deltas (“field ‘Tax ID’ invalid; 3 of 9 required fields complete”) — likely derivable from MJ’s metadata-driven forms automatically, which makes this uniquely cheap in MJ.
• Tools: Form_SetField, Form_Explain(field), Form_Submit (gated on user confirmation). Agent-set values get a distinct treatment until the user confirms.
• Transport: PubSub.
• Contract fit: covered. The promising angle: because MJ forms are generated from entity metadata, a generic form channel could expose any entity form to the agent without per-form code — the metadata already describes the fields, validation, and value lists the tool schemas need.

The common thread: PubSub-class structured channels (docs, maps, dashboards, code, forms) fit today’s contract as-is — they are whiteboard isomorphs over different state engines. Media-perception channels (video, screen pixels) are the ones that need a real extension: a binary perception path through the driver layer, which is exactly the boundary BaseRealtimeModel’s modality-agnostic contract and the registry’s TransportType column were designed to absorb.

---

7. Progress Narration

Delegated work takes seconds to minutes; dead air kills a voice call. The narration system keeps the wait conversational without chattering. It runs in both topologies — VoiceSessionService on the client-direct path, RealtimeSessionRunner on the server-bridged path — sharing the DB-driven instruction template through packages/AI/Agents/src/realtime/realtime-narration.ts (ResolveNarrationInstructionsTemplate / BuildServerNarrationInstructions), and is DB-tunable.

The client-direct pipeline:

1. The server threads an OnProgress callback into every delegated run (buildDelegationProgressCallback in RealtimeClientSessionResolver). Only significant steps pass the filter (prompt_execution, action_execution, subagent_execution, decision_processing); each is published on the push-status topic tagged type: 'RealtimeDelegationProgress' + agentSessionID + callID.
2. The browser correlates events to the active session, drops stale progress (events for calls no longer in inFlightCallIds — PubSub can lag the fast mutation result, and narrating “starting up” after the answer was spoken is exactly the bug this prevents).
3. Every live progress message is injected as a context note ([delegated-agent progress] …) so the model knows what’s happening even when it doesn’t speak.
4. Throttled spoken updates: the first no earlier than ~5 s into a delegation burst (FirstNarrationDelayMs), subsequent ones at ≥8 s spacing (NarrationIntervalMs) — and the spacing floor is session-global, so back-to-back tool calls can’t re-arm the faster first-update path. Floods of small updates aggregate into one digest (up to 4 distinct messages, deduped, oldest-first). If the model is busy or audio is still audibly playing, the update retries in 1.5 s — hosts must gate on both IsBusy and IsAudioPlaying, or queued utterances come out late and stale. When the result lands first, the pending narration is cancelled (it’s moot — the result is about to be spoken).
5. The spoken update itself is BaseRealtimeClient.RequestSpokenUpdate(instructions). The instruction wording is DB-driven: the Realtime Co-Agent - Progress Narration prompt (metadata/prompts/) carries the template with {{ progressMessage }}, {{ updateNumber }}, and {{ priorNarrations }} placeholders — resolved server-side at session prepare (RealtimeClientSessionService.NarrationPromptName, with a deprecated-name fallback to Voice Co-Agent - Progress Narration for un-resynced deployments) and threaded to the browser, with a built-in fallback when the deployment hasn’t synced the prompt. The template enforces strictly first-person narration (the co-agent owns the work — “I’m digging into that now”, never “Sage is analyzing”), bans repetition (prior spoken narrations are chained into the instructions), and bans generic filler.
6. Narration transcripts are tagged Kind: 'narration' by the client driver and are ephemeral by product decision: surfaced as a transient “live note” in the overlay (DelegationNarration$), never captions, never persisted as Conversation Details.

The server-bridged pipeline (B3 — RealtimeSessionRunner): the runner wraps every delegation in a narration burst (runDelegateWithNarration). Each significant progress event is fed to IRealtimeSession.SendContextNote?.() as a [delegated-agent progress] note, and spoken updates are paced by the same engine rules as the browser — first update no earlier than ~5 s into a burst (FirstNarrationDelayMs), ≥8 s session-global spacing (NarrationIntervalMs), buffered messages deduped and aggregated into one digest, pending narration cancelled on delegation completion and on barge-in. Instructions come from BuildServerNarrationInstructions — the DB template with {{ progressMessage }} / {{ updateNumber }} substituted; {{ priorNarrations }} gets a neutral note because the server cannot observe what the model actually said (those transcripts ride the provider socket — the browser host chains real prior narrations, the server-side builder is deliberately compact). Both capabilities are feature-detected: on a provider whose session lacks RequestSpokenUpdate, progress still flows as context notes; lacking both, narration is simply absent. The drivers themselves are the collision safety net (queue or skip per the RequestSpokenUpdate contract), so the runner does not gate on busy state.

Prompt seed: the instruction template is the Realtime Co-Agent - Progress Narration prompt (metadata/prompts/), resolved with a deprecated-name fallback to Voice Co-Agent - Progress Narration for deployments that haven’t re-synced the renamed seed (NARRATION_PROMPT_NAME / LEGACY_NARRATION_PROMPT_NAME in realtime-narration.ts).

---

8. Observability & Administration

Run topology

A voice session produces a standard, navigable run tree:

• A co-agent AIAgentRun (Status: 'Running', stamped AgentSessionID + ConversationID + AgentID) plus a linked AIPromptRun (stamped with both AgentRunID and AgentID, so the run shows on the agent’s own history too) plus a single MJ: AI Agent Run Steps row (StepNumber 1, StepType: 'Prompt', TargetID = the system AIPrompt, TargetLogID = the linked prompt run — so the co-agent run’s Timeline is non-empty and drills into the prompt run) are created at session start by RealtimeClientSessionService.createCoAgentObservabilityRun (best-effort — failure never blocks the call). Their ids persist in the session’s config and SessionManager.CloseSession finalizes all three via FinalizeCoAgentRun (Completed/Failed, CompletedAt, and a Success stamp), idempotently, even when the close comes from the janitor.
• Every delegated target-agent run nests under the co-agent run via ParentRunID and shares the AgentSessionID — the delegation tree groups under the session.
• The transcript is the session-stamped Conversation Detail rows; channel state is on the session-channel rows; artifacts link in via the conversation-detail junction (with a hidden anchor detail when no transcript turn exists yet — see §3).
• Usage telemetry lands on the co-agent AIPromptRun in both topologies (B7). Server-bridged: RealtimeSessionRunner checkpoints accumulated usage on a debounced cadence (default 5 s) — a crash-driven janitor close finalizes from the last-persisted values and loses nothing. Client-direct: the browser accumulates the client driver’s OnUsage token deltas and flushes them via the RelayRealtimeUsage mutation, debounced (10 s) plus once at teardown; the resolver clamps and accumulates the deltas onto TokensPrompt/TokensCompletion (recomputing TokensUsed), accepts a post-close flush, and tolerates every failure (usage relay must never break a live call). Providers without usage events (ElevenLabs, AssemblyAI) simply never emit — the run’s token counts stay zero and cost accounting lives provider-side. Still deferred: linking persisted transcript turns to those runs.

Admin surfaces

• AI Analytics dashboard → Realtime Voice (packages/Angular/Explorer/dashboards/src/AI/components/analytics/realtime/): realtime-overview.component.ts (KPI row — active sessions, sessions in window, average duration, delegated runs, janitor closes, cost; sessions-over-time; channel-usage donut; top delegated targets; recent sessions) and realtime-sessions.component.ts (the full sessions grid: Agent → Target, status pills including the persisted close cause — explicit / janitor / shutdown / error / unknown-legacy, channel icons, run counts, tokens/cost, host instance, row drill-in to the session record).
• Custom entity forms for MJ: AI Agent Sessions and MJ: AI Agent Channels (packages/Angular/Explorer/core-entity-forms/src/lib/custom/AIAgentSessions/, .../AIAgentChannels/).
• In-call developer mode: the overlay’s gear toggle reveals “Open run” links on delegation cards and an “Open session” link in the banner; clicking minimizes the call (the session stays live) and emits a navigate request the host resolves.

Session review (review-on-reopen) & resume carryover

A past session can be reopened in review mode: ConversationChatAreaComponent.OpenRealtimeSessionReview(agentSessionId) loads the session through RealtimeSessionReviewService (packages/Angular/Generic/conversations/src/lib/services/realtime-session-review.service.ts) — a stateless, tolerant batch loader that folds the session row, its persisted caption turns (MJ: Conversation Details by AgentSessionID), its delegated runs (excluding each leg’s co-agent observability run), and its saved channel states into one RealtimeSessionReview.

The loader is chain-aware: when the reviewed session was a resume (LastSessionID set), it walks the chain backwards — capped at 5 legs (MAX_REVIEW_LEGS) and 500 total caption rows (MAX_REVIEW_DETAILS, oldest legs trimmed first), cycle-guarded so a corrupt A→B→A chain can’t loop — and surfaces every leg chronologically on Review.Legs. BuildReviewThreadItems renders a divider between legs (“Session leg started · time”, carrying the previous leg’s CloseReason as a chip — explicit / janitor / shutdown / error), so a multi-call conversation arc reads as one episode. Channel state stays the latest leg’s only (earlier boards were superseded on resume). The chain’s conversation-history artifacts (the MJ: Conversation Detail Artifacts junctions, Direction Output — including ones hung off hidden anchor details) load too, and review mode registers each as an unfocused artifact tab on the surface panel.

The call overlay then renders the past session instead of a live call: a “Session review” banner badge with the lifecycle range + close-reason chip, the same thread/rail components replaying the historical turns and delegation cards, and a read-only Whiteboard tab when the session saved a parseable board state. Everything live is dead in review — no mic, no composer; the controls collapse to “Start live session” (emits RealtimeStartLiveRequest, resuming as a new session chained via lastSessionId — which restores saved channel states through PriorChannelStatesJson and hydrates the prior transcript into the new model’s prompt, see §4) and Close. On the review→live edge, the review-registered (template-based, read-only) Whiteboard tab is removed only when the live session’s channel set has no Whiteboard channel — when it has one, the live plugin re-registers the same tab key and upgrades the pane in place; review artifact tabs are deliberately kept (wanted carryover). The rule is the pure ShouldRemoveReviewWhiteboardTab helper next to RealtimeSurfaceTabsModel.RemoveTab in realtime-surface-tabs.model.ts (the Activity tab itself is irremovable by model contract; removing a focused tab falls focus back to Activity).

Deep link: the custom MJ: AI Agent Sessions form ships an “Open session review” affordance that navigates to the conversations resource with a realtimeSessionId query param; ChatConversationsResourceComponent picks it up (initial load and later OnQueryParamsChanged deliveries) and calls OpenRealtimeSessionReview once the chat area is ready.

The plan’s complementary timeline treatment — the conversation rendering a completed session as a single collapsed session block (”🎙 Voice session · 14 min · 6 exchanges · 2 agent runs”) inline in the message history — remains a UX direction; today the session’s turns appear as ordinary conversation messages, and review mode is the way to replay an episode as an episode.

---

9. Security Model

Authorization is overwhelmingly reuse of existing primitives, plus a small number of session-specific gates:

• CanRun on the target agent gates session creation — AIAgentPermissionHelper.HasPermission(targetAgentId, user, 'run') in both SessionManager.CreateSession and RealtimeClientSessionResolver.assertCanRunTarget. The co-agent is internal orchestration; the meaningful permission is on the agent doing real work. Denial throws before any row is written.
• Runtime overrides are authorization-gated — configOverridesJson and a deviating explicit preferredModelId require the seeded Realtime: Advanced Session Controls authorization (hierarchy-aware via AuthorizationEvaluator; fail-closed when the seed is absent); a deviating model is additionally subject to the effective allowUserModelOverride policy. Denial is a structured rejection, never a silent ignore — unauthorized users still get a fully working session on the co-agent’s configured defaults. Pairing rows restrict which targets a constrained co-agent fronts, but they are a targeting constraint layered on top of CanRun, never a replacement for it (see §4).
• Session ownership on every inbound operation — every relay (ExecuteRealtimeSessionTool, RelayRealtimeTranscript, SaveSessionChannelState, SaveSessionChannelArtifact, CancelRealtimeSessionTool, RelayRealtimeUsage) loads the session and enforces UUIDsEqual(session.UserID, contextUser.ID); tool/transcript relays additionally reject Closed sessions, while state/artifact saves, the cancel channel, and usage relay accept them (the final flush legitimately lands post-close, and a cancel can legitimately race teardown). Channel-state restore and prior-transcript hydration are ownership-checked too — another user’s prior session never leaks state or words.
• The target agent cannot be swapped mid-session — it is stored server-side in the session’s config at start and read back on every relay; the browser never re-supplies it.
• Tools execute under the session’s contextUser with the request-scoped provider — a realtime model calling a tool can do exactly what the user could do, no more. The provider owning the conversation is never an authorization bypass.
• Client-declared tools grant zero server capability. clientToolsJson declarations (channel/UI tools) are declared to the model so it can call them, but the server never executes them — a relayed call for one of those names falls into the structured “not available” path. Declarations are still validated to stop config bloat: ≤16 tools, ≤64,000 chars total, per-tool shape checks (name ≤128 chars, description non-empty, schema a plain object); channel-state payloads cap at 2,000,000 chars (per state and accumulated on restore).
• Ephemeral, provider-scoped tokens — the browser never sees a long-lived provider key. OpenAI bakes the full session config (instructions + tools) into the minted client secret; Gemini token-locks the mask-safe generation subset via liveConnectConstraints + lockAdditionalFields: [] (system prompt/tools ride the driver-applied SessionConfig; see §3). Token expiry mid-session surfaces as a Fatal error so the session finalizes cleanly.
• XSS-safe exports — every user/agent-authored string in whiteboard HTML/SVG exports is HTML-escaped before touching the document, and live widget HTML is never inlined in exports (no sandbox exists there).
• Sandboxed agent-authored HTML — whiteboard HTML widgets run in allow-scripts-only iframes (opaque origin — no parent DOM, session, cookies, or storage); the only outbound channel is the validated, size-capped MJWhiteboard.submit postMessage bridge, gated on a tracked-frame source check. Markdown panels render through the sanitized shared markdown component. See the ng-whiteboard README for the full sandbox rationale.
• Deferred: channel-grained permissions (today: if you can run the agent, you can use its channels), multi-party sessions (AIAgentSession.UserID is singular by design), and audio retention (off — the text transcript is the durable record).

---

10. Known Gaps & Deferred Work

Honest ledger of what is not done on this branch, so nobody reads aspiration into the docs above. (Items previously listed here and since shipped — the ElevenLabs and AssemblyAI drivers, server-bridged narration consumption, client-direct usage relay, prior-session transcript hydration, the delegation cancel channel — have been pruned; they are documented in the sections above.)

Item	Status
Server-bridged client media plane (browser audio ↔ server socket, the plan’s P5 / WebRTC transport)	Not built — client-direct is the shipped audio path; server-bridged runs are fully wired up to the provider socket but have no client media transport
Transcript Replaces marker	Shipped — RealtimeClientTranscript.ReplacesPrevious is the machine-readable correction marker: the ElevenLabs driver stamps it on agent_response_correction, the overlay replaces the caption in place, and RelayRealtimeTranscript(replacesPrevious) UPDATES the persisted turn instead of appending (insert fallback when no prior turn exists). The server-bridged RealtimeTranscript type can adopt the same field when that path needs it
Provider session.resume windows	AssemblyAI shipped — the client driver captures session_id from session.ready and, on an unexpected socket drop, makes ONE reattach inside the provider’s 30-second window (session.resume first frame; 'connecting' shown while reattaching; mic worklet + playout engine survive; a failed/second drop falls through to the pre-existing fatal path). Gemini’s sessionResumption token remains unused — it needs server-mint participation (documented TODO); MJ’s own chain resume (LastSessionID) is unchanged and complementary
Transcript-turn → co-agent-run linkage	Deferred (noted in RelayRealtimeTranscript’s doc comment) — turns are session-stamped but not linked to the observability runs
Chat conversation-history hydration into the companion prompt	MVP gap — the resolver passes ConversationMessages: [] (the service supports it). Distinct from the shipped prior-session transcript hydration on resume (§4)
Collapsed session block inline in the conversation timeline	UX direction only — session review mode (see §8) ships the replay; the inline collapsed-block rendering in message history is not built
Unified session transport (SessionEnvelope / ISessionTransport)	Independent track per the plan — voice deliberately does not depend on it
Video channel / video modality	Deferred — BaseRealtimeModel’s contract anticipates it
Server-side channel plugin execution (ServerPluginClass consumption)	Shipped with scope notes — BaseRealtimeChannelServer (in @memberjunction/ai) is resolved per session by RealtimeChannelServerHost from the ACTIVE registry rows and wired into the durable lifecycle: session start (SessionManager.CreateSession), pre-persistence channel-state saves (SaveSessionChannelState), and session close from every provenance incl. the janitor (SessionManager.CloseSession); WhiteboardChannelServer ships as the reference (state-of-record validation/canonicalization). Not covered: socket/media members (deferred with the unified-transport track) and server-tool contribution to RealtimeSessionRunner (documented TODO — no code path gives the server-bridged runner per-session channel instances yet). See §5
Non-target server tools on the client-direct relay (action wiring through executeNonTargetTool)	Later phase — structured “not available” today; the server-bridged path already executes actions
Channel-grained permissions, multi-party sessions, audio retention/consent	Out of scope for this iteration (see plan)
RealtimeClientSessionService ↔ BaseAgent prepare-logic duplication	Known debt — the service’s doc header calls for a future shared RealtimeSessionPreparer; until then the two are kept in sync intentionally (the shared realtime-narration.ts and realtime-coagent-config.ts modules are the first extracted pieces — both paths consume the same effective-config merge)
Per-driver voice plumbing for the effective config (realtime.voice.providers)	Partially shipped — provider-matched settings flow into the open Config bag (server-bridged: OpenAI spread, Gemini merge-last, AssemblyAI voice mapping). TODOs: OpenAIRealtime.CreateClientSession does not fold params.Config into the minted client-direct session (clients apply EffectiveConfigJson instead); the ElevenLabs server driver provisions voice on its managed agent and consumes no per-session bag

---

11. Audio-Reactive Call Visuals (Audio Activity Metering)

The call overlay is a PROGRESSIVE-DISCLOSURE console (see realtime-disclosure.ts + ng-conversations’ README): a first-ever call is pure audio — a breathing hero orb and three controls, with the caption thread / composer dock / surface panel / gear unlocking by level (0–4) as the user acts or across sessions via a per-user UserInfoEngine milestones ratchet (mj.realtimeVoice.uxMilestones.v1; the gear’s Simple/Standard/Pro/Auto density control is the escape hatch). Content never flips the console open — the ONE auto-reveal is a channel’s first agent activity (ChannelActivity$), which opens the surface panel as a peek with that channel’s tab focused; finished artifacts arrive as unfocused, glowing tabs.

That pure-audio hero orb doesn’t act out speech — it reacts to the actual waveform, vibrating like a speaker cone with the agent’s voice and flipping color when the user talks. Three layers, each degrading gracefully:

The capability contract (client drivers)

BaseRealtimeClient.GetAudioActivity(): RealtimeAudioActivity | null is a capability, not an obligation (driver obligation #9): per-direction RMS levels (0..1) plus 9 normalized frequency bins, or null for any direction the driver couldn’t meter — full null when no meters exist at all. Drivers attach meters through the protected attachInputAudioMeter / attachOutputAudioMeter hooks and MUST release them on disconnect (closeAudioMeters()).

Metering sources per driver (all four currently meter BOTH directions):

Driver	Agent audio (output)	User mic (input)
OpenAI (WebRTC)	RealtimeAudioMeter.ForStream on the remote track’s stream (attached in ontrack)	ForStream on the mic stream at Connect
Gemini / ElevenLabs / AssemblyAI (client-owned audio)	RealtimePcmPlayback.CreateMeter() — an AnalyserNode on the playout engine’s master gain	ForStream on the mic stream

RealtimeAudioMeter (in packages/AI/RealtimeClient/src/audio/audioMeter.ts) is defensive by contract: factories return null where Web Audio is unavailable (tests, SSR), and the DSP math (ComputeRmsLevel, BucketizeFrequencyData) is exported pure for unit testing.

The sampling loop (call overlay)

VoiceSessionService.GetAudioActivity() is a plain passthrough; the overlay runs ONE requestAnimationFrame loop outside Angular that samples it, runs the frame through RealtimeAudioVisualSmoother (attack/decay envelopes + direction hysteresis — see realtime-audio-visuals.ts in ng-conversations), and writes CSS custom properties (--voice-out, --voice-in, --eq-1..9) plus data-audio-live / data-voice-dir attributes directly on the rendered overlay element. Zero change detection per frame.

The render gate (CSS)

All audio-reactive rules key off [data-audio-live='true']: orb scale follows the smoothed output envelope, the EQ bars become a true spectrum, and [data-voice-dir='user'] recolors orb + bars green (the established “user audio” palette). When a driver attaches no meters the attribute stays false and the original turn-state keyframe animations remain in charge — the fallback is the pre-existing behavior, not a degraded new one.

---

Reference Map

Concern	Where
Model primitive + driver obligations	packages/AI/Core/src/generic/baseRealtime.ts
Channel plugin contract — server half (+ host + reference impl)	packages/AI/Core/src/generic/baseRealtimeChannelServer.ts, packages/AI/Agents/src/realtime/realtime-channel-server-host.ts, packages/AI/Agents/src/realtime/whiteboard-channel-server.ts
Server drivers	packages/AI/Providers/OpenAI/src/models/openAIRealtime.ts, packages/AI/Providers/Gemini/src/geminiRealtime.ts, packages/AI/Providers/ElevenLabs/src/elevenLabsRealtime.ts, packages/AI/Providers/AssemblyAI/src/assemblyAIRealtime.ts
Browser drivers + shared PCM audio plane	packages/AI/RealtimeClient/ (README) — src/drivers/*, src/audio/*
Agent type / runner / broker / client-session service / shared narration / memory builder	packages/AI/Agents/src/agent-types/realtime-agent-type.ts, src/realtime/* (incl. realtime-narration.ts), src/agent-memory-context-builder.ts
Session records, janitor, host identity	packages/MJServer/src/agentSessions/
GraphQL resolvers	packages/MJServer/src/resolvers/RealtimeClientSessionResolver.ts, AgentSessionResolver.ts
Browser orchestration + overlay + channels + session review	packages/Angular/Generic/conversations/src/lib/services/voice-session.service.ts, src/lib/services/realtime-session-review.service.ts, src/lib/components/realtime/**
Generic whiteboard (engine, tools, surfaces, exports, widget bridge)	packages/Angular/Generic/whiteboard/ (README)
Admin dashboard	packages/Angular/Explorer/dashboards/src/AI/components/analytics/realtime/
Session / channel custom entity forms (incl. the review deep link)	packages/Angular/Explorer/core-entity-forms/src/lib/custom/AIAgentSessions/, .../AIAgentChannels/
Schema	migrations/v5/V202606090930__v5.41.x__AI_Agent_Sessions_Channels.sql
Seeds	metadata/agents/.voice-co-agent.json (Realtime Co-Agent), metadata/prompts/.voice-co-agent-*.json (Realtime Co-Agent prompts), metadata/ai-agent-channels/, metadata/ai-model-types/, metadata/ai-models/ (realtime rows incl. ElevenLabs Agents + AssemblyAI Voice Agent), metadata/agent-types/ (Realtime), metadata/artifact-types/ (Whiteboard)

Session tuning, cost attribution & resilience (QA-hardening wave, v5.49)

realtime.session — provider-session tuning knobs

The effective co-agent config gained a session sub-section that flows into the driver’s open Config bag on both topologies (same cascade voice rides; the runtime bag wins per key):

{
  "realtime": {
    "session": {
      "effortLevel": 85,                 // MJ-normalized (ChatParams.effortLevel vocabulary: 1–100 or named)
      "parallelToolCalls": true,
      "mcpTools": [{ "type": "mcp", "server_label": "kb", "server_url": "https://…", "require_approval": "never" }],
      "inputTranscriptionModel": "whisper-1"
    }
  }
}

GetSessionTuningSettings projects the section onto the flat bag keys; each driver translates a key to its native wire field only when its profile confirms support and scrubs it otherwise — a shared config is safe on every provider (Gemini additionally warns on scrubbed foreign keys via REALTIME_SHARED_CONFIG_KEYS). Grok’s GA gates are OFF pending xAI docs; flipping them is a one-line profile change.

Multi-channel cost attribution

RealtimeUsage now carries per-modality token detail (InputTokenDetails/OutputTokenDetails: text/audio/image/cached). The OpenAI driver maps the GA response.done detail blocks, the session runner accumulates them field-wise across turns, and the checkpoint persists the detail as JSON on the realtime AIPromptRun.Result — so audio-vs-text pricing (audio-in ~8× text-in on GPT Realtime 2.1) is attributable instead of blended.

Resilience & lifecycle

• Connect/readiness deadlines everywhere — client WS drivers reject an awaited Connect on a silent endpoint (15s, overridable), and the server session’s WaitForConfigApplied has a matching readiness deadline; no path can hang forever.
• Cancellation — the session runner observes the chained agent cancellation signal (RealtimeSessionRunnerDeps.AbortSignal): an abort stops + finalizes (usage flushed) instead of waiting on the janitor.
• Bounded transport reconnect — a fatal socket drop triggers up to MaxTransportReconnects (default 1) fresh-session reconnects with rewired handlers + a context note; usage/transcripts span the reconnect; exhausted budget finalizes.
• MCP approval — with no approval UX yet, an mcp_approval_request is auto-DENIED (correct correlation id) so the model voices the refusal instead of dead-air blocking; declare servers require_approval: 'never' to skip the round-trip.
• Barge-in floor rule — on a TRUE user barge-in, queued tool-result auto-triggers are dropped (the result item is already in the conversation; the user’s next turn voices it) — the model never speaks over a user who just took the floor.
• Proxy hardening (HF client-direct) — optional MJ_REALTIME_PROXY_ALLOWED_ORIGINS origin allowlist (403 before ticket consumption), 15s upstream-open deadline, bounded pre-open frame buffer. Multi-instance deployments need sticky routing for the in-memory ticket registry.