Realtime Bridges Guide

How MemberJunction connects its one realtime agent engine to the outside world — Zoom/Teams/ Slack/Meet/Webex/Discord meetings and Twilio/Vonage/RingCentral telephony — through a single, pluggable media-transport seam.

There is exactly one realtime agent engine (the Realtime agent type, the co-agent resolution chain, tools, narration, channels, transcript relay, the AIAgentSession lifecycle — all documented in the Real-Time Co-Agents Guide). A bridge is a pluggable media transport that fills that engine’s long-deferred server-bridged seam from an external endpoint. We do not build a second realtime stack; a bridge is just a driver that supplies the media plane.

Nothing in this architecture is audio-specific. Audio is the first track lit up because the realtime models are there today, but the seam carries typed media tracks (audio / video / screen, each inbound and outbound). When realtime models gain full-duplex video, the same bridges already carry the video tracks with zero re-architecture.

Companion documents:

• /plans/realtime/realtime-bridges-architecture.md — the full architecture (mermaid, ERD, all phases, channels, multi-party, turn-taking). This guide is the developer-facing distillation of what has shipped; the plan retains the full rationale and the planned/deferred tracks.
• /guides/REALTIME_CO_AGENTS_GUIDE.md — the realtime engine the bridge plugs into (the Realtime agent type, IRealtimeSession, channels, sessions, the janitor pattern bridges copy).
• packages/AI/BridgeBase/README.md — the base package (@memberjunction/ai-bridge-base).
• packages/AI/Bridge/README.md — the server package (@memberjunction/ai-bridge-server).

---

Table of Contents

1. What a Bridge Is
2. Architecture
3. The Capability Model
4. How to Add a New Bridge Driver
5. The Transport Seam
6. Turn-Taking
7. Entity Invariants
8. Join Methods & Agent Identity (Planned)
9. Multi-party (LiveKit + multiple agents)
10. Roadmap / Phase Status

---

1. What a Bridge Is

A bridge is a media transport + (planned) channel contributor that connects the realtime agent engine to an external endpoint. Concretely it is a driver class (LoopbackBridge today; ZoomBridge, TwilioBridge, … as drivers land) that:

• carries bidirectional media — audio, video, screen, full duplex; nothing is audio-specific; and
• (planned, Phase 2) contributes channels — the platform’s native surfaces (hand-raise, roster, whiteboard) routed through MJ’s existing channel plane.

The single biggest design win is reuse. The realtime session contract IRealtimeSession (in @memberjunction/ai) is already media-agnostic — its SendInput(chunk) is what the agent hears and OnOutput(handler) is what the agent says — but it had no client-facing pipe. A bridge is that pipe.

🚨 A bridge is transport + host UX tools ONLY — it does NOT build the agent. Identity (the agent speaks first-person AS the target), the model/voice precedence cascade, the base prompt + memory, the invoke-target-agent delegation, and session/run tracking all come from the one core producer, RealtimeClientSessionService.PrepareRealtimeSessionParams, which the bridge consumes (see BaseAgent.StartBridgeRealtimeSession). A bridge that re-implements any of that is drift — exactly what the convergence in plans/realtime/realtime-core-host-convergence.md removed (and what ai-agents/src/__tests__/realtime-convergence-drift.test.ts guards). What a bridge DOES own: media transport (audio/video/screen) and any host-specific UX tools, injected via RealtimeSessionRunnerDeps.ExtraTools/ExecuteTool — never baked into core. So the agent is the same agent (identity, voice, delegation, tracking) whether you reach it in native chat, LiveKit, or Zoom/Teams; only how it’s heard and what surfaces it can touch differ. See the boundary table in the Co-Agents Guide §3.

The transport seam

external endpoint                 the bridge driver                the realtime session
(Zoom / Twilio / …)               (BaseRealtimeBridge)             (IRealtimeSession)
        │   inbound media   ──▶   OnMedia(frame)         ──▶   SendInput(chunk)   "the agent hears"
        │   outbound media  ◀──   SendMedia(track,frame) ◀──   OnOutput(chunk)    "the agent says"

The engine wires those two hooks once, generically (see §5). Meetings and telephony are two families of the same abstraction — and because the session is the existing AIAgentSession, a bridged session reviews in the existing console with zero new persistence code.

No new “session” entity

The session is the existing AIAgentSession. A bridge is an attachment to it (the AIAgentSessionBridge row), exactly parallel to how a channel attaches to a session. One session record, one lifecycle, one transcript path — whether the media plane is a browser, a Zoom room, or a phone line.

A bridge is also a channel contributor (forward reference)

Beyond media, a bridge is also a contributor of channels — the platform’s native tool + signaling vocabulary (hand-raise, roster, native whiteboard) routed through MJ’s existing channel plane. That server-side channel plane is planned (Phase 2) and is not in the shipped code; see §10. This guide documents the media transport + capability + turn-taking + entity layers that have shipped (Phase 0/1).

---

2. Architecture

The engine pair — AIBridgeEngineBase / AIBridgeEngine

The bridge plane mirrors AIEngineBase / AIEngine exactly.

graph TD
    subgraph Meta["Metadata (CodeGen entities, seeded via mj-sync)"]
        P["MJ: AI Bridge Providers<br/>registry + SupportedFeatures"]
        ID["MJ: AI Bridge Agent Identities<br/>agent's address / phone #"]
        PC["MJ: AI Bridge Provider Channels<br/>provider → channel junction"]
        SB["MJ: AI Agent Session Bridges<br/>session ↔ transport binding"]
        PART["MJ: AI Agent Session Bridge Participants<br/>who's on the call"]
    end
    subgraph Base["@memberjunction/ai-bridge-base (metadata only, no execution)"]
        EB["AIBridgeEngineBase<br/>(BaseEngine) — caches providers,<br/>identities, provider-channels"]
        BBP["BaseRealtimeBridge (abstract)<br/>media contract + capability-gated methods"]
        TT["TurnTakingPolicy<br/>passive / active / hybrid"]
    end
    subgraph Server["@memberjunction/ai-bridge-server (coordination + execution)"]
        ENG["AIBridgeEngine<br/>composes the base — lifecycle,<br/>host affinity, janitor, transport seam"]
        LB["LoopbackBridge<br/>(in-memory test driver)"]
    end
    RS["IRealtimeSession<br/>(@memberjunction/ai — injected, never constructed)"]

    Meta --> EB
    EB -.composed by.-> ENG
    BBP -.implemented by.-> LB
    TT --> ENG
    ENG <-->|"transport seam<br/>SendInput / OnOutput"| RS
    ENG -->|"ClassFactory.CreateInstance(BaseRealtimeBridge, DriverClass)"| LB

    style Base fill:#1f3a5f,stroke:#264f7a,color:#fff
    style Server fill:#5a3d7a,stroke:#7c5295,color:#fff
    style Meta fill:#2d5016,stroke:#1a5c3a,color:#fff

AIBridgeEngineBase (@memberjunction/ai-bridge-base) — a BaseEngine subclass that caches the three bridge reference datasets and exposes synchronous resolution helpers over them. No execution. Because the datasets are small reference tables, every lookup filters the in-memory array client-side (no RunViews). It is @RegisterForStartup() and reactive via the inherited BaseEngine event integration. Key helpers:

Helper	Purpose
ProviderByName(name) / ProviderByDriverClass(driverClass)	Resolve a provider row (case-insensitive, trim-tolerant).
IdentityByValue(value, providerId?)	Route an inbound invite/call (addressed to a mailbox/DID) to the owning agent. Active identities only.
IdentitiesForAgent(agentId, providerId?)	The agent’s active identities (which surfaces it is reachable on).
ChannelsForProvider(id) / DefaultChannelsForProvider(id)	The channels a provider contributes / auto-attaches, ordered by Sequence.

AIBridgeEngine (@memberjunction/ai-bridge-server) — the server tier that composes the base (it does not extend it) and adds all coordination + execution: the transport seam, the bot-session lifecycle state machine, host affinity + the janitor, participant tracking, and turn-taking integration.

Why composition, not inheritance

AIBridgeEngineBase is a BaseEngine singleton keyed by class name. If AIBridgeEngine inherited from it, the startup manager would instantiate two separate engines (the base AND the subclass), each loading and event-subscribing its own copy of the same caches. By composing — AIBridgeEngine reads through protected get Base() => AIBridgeEngineBase.Instance and delegates every metadata accessor — there is exactly one BaseEngine cache, warmed once. This is the identical pattern AIEngine uses over AIEngineBase. From the architecture’s source:

@RegisterForStartup({ deferred: true, deferredDelay: 15000, description: 'Server-side AI Bridge Engine' })
export class AIBridgeEngine extends BaseSingleton<AIBridgeEngine> implements IStartupSink {
    /** The single underlying base instance whose BaseEngine cache holds the registry. */
    protected get Base(): AIBridgeEngineBase {
        return AIBridgeEngineBase.Instance;
    }
    public async HandleStartup(contextUser?, provider?) { await this.Config(false, contextUser, provider); }
    public async Config(forceRefresh?, contextUser?, provider?) { await this.Base.Config(forceRefresh, contextUser, provider); }
    // ...every metadata read delegates: ProviderByName, IdentityByValue, ChannelsForProvider, …
}

The BaseRealtimeBridge driver family

BaseRealtimeBridge is the abstract driver — a sibling to BaseRealtimeModel in the realtime stack. Where a realtime model driver wraps a speech-to-speech provider, a bridge driver wraps a media transport. Both self-register with the MJ ClassFactory and are resolved by metadata key. The base class is abstract and unregistered; the engine resolves a concrete driver via MJGlobal.Instance.ClassFactory.CreateInstance(BaseRealtimeBridge, provider.DriverClass). See §4.

The five entities

All five are CodeGen-generated, named with the MJ: prefix, and seeded via mj-sync metadata (not SQL INSERTs). The migration created the tables; CodeGen produced the entity classes in @memberjunction/core-entities.

erDiagram
    AIAgent ||--o{ AIBridgeAgentIdentity : "reachable as"
    AIBridgeProvider ||--o{ AIBridgeAgentIdentity : "on platform"
    AIBridgeProvider ||--o{ AIAgentSessionBridge : "transports via"
    AIBridgeProvider ||--o{ AIBridgeProviderChannel : "contributes"
    AIAgentChannel ||--o{ AIBridgeProviderChannel : "as channel"
    AIAgentSession ||--o| AIAgentSessionBridge : "bridged by"
    AIAgentSessionBridge ||--o{ AIAgentSessionBridgeParticipant : "has"
    User ||--o{ AIAgentSessionBridgeParticipant : "is (if matched)"

Entity (name)	Generated class	Role
MJ: AI Bridge Providers	MJAIBridgeProviderEntity	The platform registry. Holds Name, DriverClass (the ClassFactory key), Status, and SupportedFeatures (the typed JSON capability blob, accessed via SupportedFeaturesObject).
MJ: AI Bridge Agent Identities	MJAIBridgeAgentIdentityEntity	Which agents are reachable where — IdentityType (Email / PhoneNumber / AccountID) + IdentityValue, per agent, per provider.
MJ: AI Bridge Provider Channels	MJAIBridgeProviderChannelEntity	The provider→AIAgentChannel junction (IsDefault, Sequence) — which channels a provider contributes (consumed by the planned Phase 2 channel plane).
MJ: AI Agent Session Bridges	MJAIAgentSessionBridgeEntity	The transport attachment to an AIAgentSession. The lifecycle state machine (Status), direction, join method, turn mode, platform handles, HostInstanceID.
MJ: AI Agent Session Bridge Participants	MJAIAgentSessionBridgeParticipantEntity	One row per participant on the call (the diarization map), including the agent bot itself (IsAgent = 1).

Telephony reuses these same five tables with no schema change: BridgeType = 'Telephony', IdentityType = 'PhoneNumber', and a SupportedFeatures blob with { DTMF, OutboundDial, CallTransfer } on. That is the proof the model is genuinely unified.

---

3. The Capability Model

Every bridge is 100% generic and capability-gated. A platform’s capabilities are declared in MJ: AI Bridge Providers.SupportedFeatures — a single NVARCHAR(MAX) JSON column strongly typed via the IBridgeProviderFeatures interface (MJ’s JSONType system; CodeGen emits the typed SupportedFeaturesObject accessor). One JSON column instead of ~16 BIT columns keeps the table simple and lets new features be added without a schema migration — just extend the interface.

IBridgeProviderFeatures

The shipped interface (metadata/entities/JSONType-interfaces/IBridgeProviderFeatures.ts) holds transport/media concerns only. All flags are optional — an omitted flag means not supported.

export interface IBridgeProviderFeatures {
    // Join methods
    OnDemandJoin?: boolean; ScheduledJoin?: boolean; InviteJoin?: boolean;
    NativeInvite?: boolean; InboundRouting?: boolean; OutboundDial?: boolean;
    // Media tracks (directional — nothing here is audio-specific)
    AudioIn?: boolean;  AudioOut?: boolean;
    VideoIn?: boolean;  VideoOut?: boolean;
    ScreenIn?: boolean; ScreenOut?: boolean;
    // Signals & telephony
    SpeakerDiarization?: boolean; DTMF?: boolean; CallTransfer?: boolean; Recording?: boolean;
}

Interactive surfaces (hand-raise, in-meeting chat, native whiteboard) are deliberately not features here — they are channels the bridge contributes (planned Phase 2). This interface is transport/media capabilities only.

In bridge code, IBridgeProviderFeatures is available as a documented type alias from @memberjunction/ai-bridge-base (it aliases the generated MJAIBridgeProviderEntity_IBridgeProviderFeatures so drivers needn’t import the verbose generated name; the canonical source remains @memberjunction/core-entities).

The two-layer gating rule (defense in depth)

flowchart TD
    A["Engine wants SendDTMF()"] --> B{"provider.SupportedFeaturesObject.DTMF?"}
    B -->|"false / omitted (metadata)"| C["skip — never call the driver"]
    B -->|"true"| D["driver.SendDTMF()"]
    D --> E{"driver overrode it?"}
    E -->|"no — base rejects with<br/>BridgeCapabilityNotSupportedError"| F["loud failure: the flag lied"]
    E -->|"yes"| G["RequireFeature('DTMF') re-asserts, then sends"]

    style C fill:#2d5016,stroke:#1a5c3a,color:#fff
    style F fill:#7a1f1f,stroke:#a02a2a,color:#fff
    style G fill:#2d5016,stroke:#1a5c3a,color:#fff

1. The engine checks the flag first and never calls a driver method whose feature is off. For example, participant tracking is gated before the driver is ever touched:

   private wireParticipantTracking(active: ActiveBridgeSession): void {
       if (active.Provider.SupportedFeaturesObject?.SpeakerDiarization !== true) {
           return; // provider metadata says no roster — never call the driver.
       }
       try { active.Bridge.OnParticipantChange(/* … */); }
       catch (err) { /* driver didn't implement it despite the flag — log + continue */ }
   }

2. The driver’s virtual methods throw by default (BridgeCapabilityNotSupportedError). This is the second layer: a driver that has not overridden a method — or a metadata flag that lied — fails loudly rather than silently degrading. An overriding driver re-asserts the flag at the top of its method via the protected RequireFeature(flag) helper, so even a direct (engine-bypassing) caller cannot run an action the metadata says is off.

BridgeCapabilityNotSupportedError carries both FeatureName and ProviderName, so the loud failure points at exactly which capability flag lied (or which driver method is missing) for which platform.

---

4. How to Add a New Bridge Driver

This is the most important section. Adding a platform is a minimal subclass — the base + engine carry all the generic weight (session wiring, frame normalization, turn-taking, participant bookkeeping, lifecycle, the janitor). You implement only the irreducibly platform-specific primitives.

Step 1 — Subclass BaseRealtimeBridge and register it

import { RegisterClass } from '@memberjunction/global';
import { BaseRealtimeBridge } from '@memberjunction/ai-bridge-base';

@RegisterClass(BaseRealtimeBridge, 'ZoomBridge')   // ← the DriverClass key the provider row resolves
export class ZoomBridge extends BaseRealtimeBridge {
    // ...
}

The 'ZoomBridge' string is the ClassFactory key. A MJ: AI Bridge Providers row whose DriverClass = 'ZoomBridge' resolves to this class. (Add a tree-shaking-prevention loader and call it from your package entry point, exactly as LoadLoopbackBridge does, so bundlers don’t eliminate the dynamically-registered class.)

Step 2 — Implement the four abstract methods (every bridge MUST)

public async Connect(ctx: RealtimeBridgeContext): Promise<BridgeConnectResult> {
    this.applyContext(ctx);                       // capture features + provider name FIRST
    // … join the meeting / place or accept the call …
    return { BotParticipantId, ExternalConnectionId };   // engine persists these onto the bridge row
}

public async Disconnect(reason: BridgeDisconnectReason): Promise<void> {
    // leave / hang up cleanly. `reason` lets you shortcut a graceful handshake on 'Error'/'Shutdown'.
}

public SendMedia(track: BridgeMediaTrackKind, frame: BridgeMediaFrame): void {
    // outbound: the agent's voice/video/screen INTO the meeting (fed from IRealtimeSession.OnOutput).
    // Audio is just one track — `frame.Track` selects which.
}

public OnMedia(handler: (frame: BridgeMediaFrame) => void): void {
    // inbound: what the agent HEARS/SEES (routed to IRealtimeSession.SendInput).
    // When the platform diarizes, stamp `frame.SpeakerLabel` so turn-taking can attribute speech.
}

applyContext(ctx) (a protected helper) captures the provider’s SupportedFeatures and name onto the instance so RequireFeature and capability-error messages work. Call it first in Connect.

Step 3 — Override only the capability-gated virtuals your platform supports

Each optional method throws BridgeCapabilityNotSupportedError by default. Override the ones your platform offers, and re-assert the flag at the top with RequireFeature:

Method	Gating flag	Family
GetParticipants() / OnParticipantChange(handler)	SpeakerDiarization	roster + diarization mapping
SendDTMF(digits) / OnDTMF(handler)	DTMF	telephony
TransferCall(target)	CallTransfer	telephony
StartRecording()	Recording	recording (subject to consent handling upstream)

Video/screen tracks need no extra methods — they ride the same SendMedia / OnMedia calls, gated by the directional flags (VideoIn/Out, ScreenIn/Out).

Step 4 — Set the matching SupportedFeatures flags in the provider seed

The driver advertises what it overrode; the provider’s SupportedFeatures JSON must match — and is validated server-side (§7). Seed the provider row (with its capability blob) via mj-sync metadata, never a SQL INSERT.

Worked minimal example — LoopbackBridge

The shipped LoopbackBridge (@memberjunction/ai-bridge-server) is the canonical minimal driver. It proves the transport seam round-trips media with no platform: every outbound frame is echoed straight back inbound (re-stamped audio-out → audio-in), and it overrides the diarization-gated roster methods to return a single synthetic agent participant.

@RegisterClass(BaseRealtimeBridge, LOOPBACK_BRIDGE_DRIVER_CLASS)   // 'LoopbackBridge'
export class LoopbackBridge extends BaseRealtimeBridge {
    public async Connect(ctx: RealtimeBridgeContext): Promise<BridgeConnectResult> {
        this.applyContext(ctx);
        this.connected = true;
        this.participantHandler?.([LoopbackBridge.AGENT_PARTICIPANT]);   // emit initial roster
        return { BotParticipantId: 'loopback-agent', ExternalConnectionId: `loopback:${ctx.Address || 'in-memory'}` };
    }

    public SendMedia(track: BridgeMediaTrackKind, frame: BridgeMediaFrame): void {
        if (!this.connected) return;                                     // disconnected → drop
        this.Sent.push(frame);
        this.mediaHandler?.({ ...frame, Track: this.toInboundTrack(track) });   // echo outbound → inbound
    }

    public OnMedia(handler: (frame: BridgeMediaFrame) => void): void { this.mediaHandler = handler; }

    public override async GetParticipants(): Promise<BridgeParticipantInfo[]> {
        this.RequireFeature('SpeakerDiarization');                       // defense-in-depth re-assert
        return [LoopbackBridge.AGENT_PARTICIPANT];
    }

    public override OnParticipantChange(handler: (p: BridgeParticipantInfo[]) => void): void {
        this.RequireFeature('SpeakerDiarization');
        this.participantHandler = handler;
    }
    // SendDTMF / TransferCall / StartRecording stay un-overridden → still throw NotSupported.
}

Telephony/recording features stay off, so those virtual base methods keep throwing — the loopback driver documents the capability-gating contract by example. A real driver (e.g. ZoomBridge, planned Phase 3) lives in its own package or the server package and overrides the diarization + (for telephony, planned Phase 6) DTMF/transfer methods, translating between the platform’s media SDK and BridgeMediaFrame.

Media-track translation & diarization

A real driver translates the platform’s media SDK frames to/from BridgeMediaFrame:

• Payload: populate exactly one of Bytes (raw ArrayBuffer, the fast in-process path) or Base64 (for transports that can only carry text). The engine’s seam unwraps either.
• Track + direction: stamp Track (audio-in … screen-out). Direction is from the agent’s POV: *-in flows from the endpoint to the agent; *-out from the agent to the endpoint.
• Diarization: when the platform labels speakers, stamp SpeakerLabel on inbound frames and emit BridgeParticipantInfo rosters via OnParticipantChange. The engine upserts those to AIAgentSessionBridgeParticipant rows and turn-taking uses the labels.

---

5. The Transport Seam

The transport seam is the single piece of code that completes the realtime engine’s deferred server-bridged media transport — the “SendInput/OnOutput have no client-facing pipe” gap. The bridge is that pipe. It is wired in AIBridgeEngine.StartBridgeSession → wireTransportSeam.

Injection — the engine never constructs the model

The realtime session is injected into StartBridgeSession, never constructed by the engine. This keeps the only coupling the seam itself (to the IRealtimeSession interface), keeps the model’s lifecycle owned by the agent/session layer above, and makes the whole engine unit-testable with a mock session + the LoopbackBridge.

export interface StartBridgeSessionParams {
    AgentSessionID: string;              // the existing AIAgentSession this bridge attaches to
    Provider: MJAIBridgeProviderEntity;  // DriverClass resolves the driver
    RealtimeSession: IRealtimeSession;   // ← INJECTED — the engine wires its media plane, never builds it
    Address: string;                     // join URL/ID (meetings) or phone number (telephony)
    Direction?: 'Inbound' | 'Outbound';  // default 'Outbound'
    JoinMethod?: /* … */;                // default 'OnDemand'
    TurnMode?: BridgeTurnMode;           // default 'Passive'
    TurnMatcher?: IAddressedMatcher; TurnScorer?: IWorthSayingScorer; TurnTuning?: /* … */;
    ContextUser?: UserInfo;
    MetadataProvider?: IMetadataProvider;
}

Wiring the two directions

private wireTransportSeam(active: ActiveBridgeSession): void {
    const { Bridge, RealtimeSession } = active;

    // Inbound: endpoint media → the agent HEARS it.
    Bridge.OnMedia((frame: BridgeMediaFrame) => {
        const chunk = this.frameToArrayBuffer(frame);   // unwrap Bytes (fast) or decode Base64
        if (chunk) RealtimeSession.SendInput(chunk);
    });

    // Outbound: the agent SPEAKS → into the meeting/call.
    RealtimeSession.OnOutput((chunk: ArrayBuffer) => {
        const track: BridgeMediaTrackKind = 'audio-out';
        Bridge.SendMedia(track, this.arrayBufferToFrame(chunk, track));
    });
}

Audio is the first track lit up, so the outbound direction is wired as audio-out; video/screen ride the same SendMedia method once the model emits them. The seam’s frameToArrayBuffer prefers the binary Bytes payload and decodes Base64 only when that’s all the frame carries (environment-agnostic — Node Buffer server-side, atob fallback).

Lifecycle state machine

StartBridgeSession drives the AIAgentSessionBridge.Status state machine and stamps HostInstanceID for node affinity:

stateDiagram-v2
    [*] --> Pending: createBridgeRow (HostInstanceID stamped)
    Pending --> Connecting: transitionStatus + driver resolved via ClassFactory
    Connecting --> Connected: driver.Connect succeeds → handles persisted, seam wired
    Connecting --> Failed: resolution or Connect throws → failBridgeRow
    Connected --> Disconnected: StopBridgeSession (driver.Disconnect + CloseReason)
    Failed --> [*]
    Disconnected --> [*]

The flow, from the source:

1. createBridgeRow — insert the AIAgentSessionBridge as Pending, stamped with this host.
2. resolveDriver — ClassFactory.CreateInstance(BaseRealtimeBridge, provider.DriverClass). (It first checks GetRegistration exists, because CreateInstance falls back to the abstract base when no driver is registered — returning an unusable instance otherwise.)
3. transitionStatus → 'Connecting', build the RealtimeBridgeContext, call driver.Connect.
4. On success: persist ExternalConnectionID + BotParticipantID, stamp ConnectedAt, transitionStatus → 'Connected', then wireTransportSeam + wireParticipantTracking + wireTurnTaking, and register the live ActiveBridgeSession in-memory.
5. On failure: failBridgeRow stamps Failed / CloseReason = 'Error' / DisconnectedAt, and rethrows.

StopBridgeSession(id, reason, …) disconnects the driver (tolerant of teardown errors) and transitions the row to Disconnected with the CloseReason. Both start and stop are idempotent — stopping a session this process no longer holds still reconciles the durable row (markBridgeDisconnected treats an already-terminal row as success).

HostInstanceID + the janitor

Every bridge row is stamped with HostInstanceID (from an injected IHostInstanceIdentity — the host app supplies the real one; DefaultHostInstanceIdentity is the standalone fallback). A crash or redeploy vaporizes the in-memory driver sockets but leaves durable rows reading Connected forever. ReconcileOrphans(contextUser, provider) force-closes Connecting/Connected bridges left by a prior boot of this host (matching hostname prefix, differing instance id) with CloseReason = 'Janitor' — mirroring the realtime SessionJanitor. The scheduling (run-once-at-boot

• periodic sweep) is intentionally a host-provided hook, so this engine package carries no timer/IO of its own; the host’s janitor calls ReconcileOrphans.

---

5b. The Channel Plane — a bridge contributes channels

A bridge is not only a media pipe; it is also a channel contributor (§4b of the architecture). AIBridgeEngine wires the Phase 2 server-side channel plane (RealtimeChannelServerHost + MeetingControlsChannelServer, both in @memberjunction/ai-agents) to a bridged session generically — without the bridge-server engine depending on the heavy agents package.

The additive seam — IBridgeChannelHost + GetMeetingControlsEventSource

Two additive pieces make this decoupled and back-compatible:

• StartBridgeSessionParams.ChannelHost?: IBridgeChannelHost — an optional, session-scoped view of the channel plane (StartSessionChannels / GetSessionServerTools / ExecuteSessionServerTool / CloseSessionChannels), declared in @memberjunction/ai-bridge-base. The runner-constructing layer (which already depends on @memberjunction/ai-agents) supplies a thin adapter binding these to RealtimeChannelServerHost. When omitted, an IRealtimeSession-only caller works exactly as before — no channel wiring.
• BaseRealtimeBridge.GetMeetingControlsEventSource(): IBridgeMeetingControlsEventSource | null — a driver returns a facilitator event source (roster / hand-raise / speaking / mute) when it has one (default null). This is not a throwing capability gate — most drivers legitimately contribute none. ZoomBridge returns one when SpeakerDiarization is on; the LoopbackBridge exposes a simple one for tests.

What wireChannelPlane does (in StartBridgeSession)

private async wireChannelPlane(active: ActiveBridgeSession): Promise<void> {
    const host = active.ChannelHost;
    if (!host) return;                                       // back-compat: no channel plane
    const meetingControls = active.Bridge.GetMeetingControlsEventSource();
    const sink = active.RealtimeSession.SendContextNote      // perception sink (provider-gated)
        ? (t: string) => active.RealtimeSession.SendContextNote?.(t) : undefined;

    await host.StartSessionChannels(active.AgentSessionID, meetingControls, sink);
    active.ServerChannelTools = host.GetSessionServerTools(active.AgentSessionID);   // → runner.ServerChannelTools
    active.ExecuteServerChannelTool = (name, args) =>                                // → runner.ExecuteServerChannelTool
        host.ExecuteSessionServerTool(active.AgentSessionID, name, args);
}

The contributed ServerChannelTools + ExecuteServerChannelTool are surfaced on the returned ActiveBridgeSession, so the runner-constructing layer registers them on RealtimeSessionRunner. The realtime session’s SendContextNote is fed into the channel context so channel perception (a hand went up, who is speaking, time remaining) reaches the model. Channel teardown runs on every stop path via closeChannelPlane. The whole wiring is failure-tolerant: a channel-host error is logged and never breaks the live media plane.

The Zoom driver realizes this end-to-end: ZoomMeetingControlsEventSource adapts the IZoomMeetingSdk roster / hand-raise / speaking / mute into the channel’s event source, so an agent on a Zoom call gains the facilitator tool vocabulary (MeetingControls_RaiseHand, _CallOnParticipant, _MuteParticipant, _SetTimer) + perception with zero platform code in the channel.

---

6. Turn-Taking

Turn-taking is generic and platform-agnostic — it operates on the diarized transcript stream the bridge provides, lives in the base package, and is identical for Zoom or Twilio. The agent always hears (inbound media always reaches the model so it has context); the policy gates only generation — whether the agent speaks, posts to chat, or stays silent. All three modes ship together.

TurnTakingPolicy is pure: every external dependency (the addressed-matcher, the “worth saying” scorer, and the clock) is injected, so there is no real time and no platform code in it — fully unit-testable and deterministic.

Mode	Behavior
Passive (default)	Generate ONLY when the agent is addressed (name/mention detection via the injected IAddressedMatcher; RegexAddressedMatcher is the shipped default). Two passive agents in one room never loop — neither speaks unless called on.
Active	A “worth saying” scorer that fires only in silence windows (never barging over a live speaker), throttled so it can’t dominate. Speaks when score ≥ threshold AND the silence window elapsed AND not throttled.
Hybrid	Passive voice PLUS a chat hand-raise — speak when addressed, but post to chat (the social-cost-free “raise hand”) when it has something to add and chat is available. Degrades to plain passive when chat is unavailable.

const policy = new TurnTakingPolicy({
    Mode: 'Passive',
    Matcher: new RegexAddressedMatcher(['Sage', 'the assistant']),
    // Active/Hybrid tuning (all optional): Scorer, SilenceWindowMs (1500), ThrottleMs (15000),
    // ScoreThreshold (0.7), Now (Date.now — injectable for deterministic tests).
});

const decision = policy.EvaluateTurn({
    Segment: { Text: 'Hey Sage, what do you think?', IsAgent: false, EndMs: Date.now() },
    HumanSpeaking: false,   // Active never fires while a human speaks
    ChatAvailable: false,   // Hybrid degrades to passive when false
});
// → { Action: 'Speak', Reason: 'Agent was addressed (passive).' }

EvaluateTurn returns { Action: 'Speak' | 'PostToChat' | 'Silent', Reason }. The Reason string is for observability and tests. Agent-spoken segments (IsAgent: true) are never treated as addressing the agent — that avoids self-trigger loops.

How the engine feeds the policy

AIBridgeEngine.wireTurnTaking subscribes the realtime session’s transcript stream and feeds only completed human turns through the policy, then acts on the decision:

private wireTurnTaking(active: ActiveBridgeSession): void {
    active.RealtimeSession.OnTranscript((t: RealtimeTranscript) => {
        if (t.Role !== 'user' || !t.IsFinal) return;   // ignore the agent's own speech + partials
        const segment: TurnTranscriptSegment = { Text: t.Text, IsAgent: false, EndMs: Date.now() };
        const decision = active.TurnPolicy.EvaluateTurn({ Segment: segment });
        this.applyTurnDecision(active, decision.Action);
    });
}

applyTurnDecision acts behind the injected IRealtimeSession: Speak requests a spoken update (via RequestSpokenUpdate when the provider supports it; otherwise the inbound media already reached the model and it responds on its own turn). PostToChat is a documented hook for the Phase 2 bridge-chat channel (the wiring is complete; a chat-capable channel consumes it once the server-side channel plane lands). Silent does nothing.

---

7. Entity Invariants

Each of the five entities has a server-side *EntityServer subclass (in MJCoreEntitiesServer) that enforces cross-field / cross-record invariants the DB can’t express, via ValidateAsync (DefaultSkipAsyncValidation overridden to false). Each invariant is a pure, separately-exported, unit-tested helper, and each gates on the fields it reads (so unrelated edits don’t re-pay the cost or resurface a pre-existing quirk on an untouched field).

Entity server	Invariants enforced
MJAIBridgeProviderEntityServer	(1) SupportedFeatures is well-formed — parses as a JSON object whose every key is a known IBridgeProviderFeatures flag and every value is a boolean. Typos (OutbondDial) and non-booleans ("true", 1) are rejected with precise messages (ValidateSupportedFeaturesJson). (2) DriverClass is non-empty — it’s the ClassFactory key; a blank value silently fails resolution (ValidateDriverClass). Both are intra-row → no DB round trips.
MJAIBridgeAgentIdentityEntityServer	(1) (ProviderID, IdentityValue) is unique (case-insensitive, cross-record RunView self-excluded on update) — two agents reachable at the same address on the same provider would make inbound routing ambiguous. (2) IdentityValue shape matches IdentityType — Email looks like an email, PhoneNumber is E.164-ish digits; AccountID is opaque (ValidateIdentityValueFormat).
MJAIBridgeProviderChannelEntityServer	(ProviderID, ChannelID) is unique — defense-in-depth + friendly message over the DB UQ_AIBridgeProviderChannel constraint (turns a raw SQL unique violation into a clear, field-attributed validation message).
MJAIAgentSessionBridgeEntityServer	Cross-field coherence the DB CHECKs can’t express: (1) Outbound needs a target — Direction='Outbound' requires an Address or ExternalConnectionID. (2) Connected ⇒ ConnectedAt set; terminal ⇒ DisconnectedAt set. (3) CloseReason only on terminal status (a reason on an active bridge is a hard failure; a terminal bridge missing a reason is a warning, so the janitor/error paths aren’t blocked from persisting). All intra-row → no DB round trips.
MJAIAgentSessionBridgeParticipantEntityServer	At most one IsAgent = 1 participant per bridge — a bridge is one agent’s connection into one room (multiple agents = multiple bridges, §9); two bot participants would confuse diarization and the self-echo gate. The DB can’t express a filtered-unique index, so a cross-record RunView (gated on dirty, self-excluded) enforces it.

---

8. Join Methods & Agent Identity

There are several ways an agent gets onto an endpoint, each gated by a SupportedFeatures join flag and recorded in AIAgentSessionBridge.JoinMethod:

• On-demand (OnDemandJoin) — a user pastes a join URL and sends the agent. The shipped default JoinMethod is 'OnDemand'.
• Scheduled (ScheduledJoin) — a Scheduled bridge fires at the meeting start time (see the ScheduledBridgeRunner below).
• Invite / Calendar (InviteJoin) — the headline UX, now built (Phase 4). The agent gets a real mailbox/calendar identity in the customer’s own tenant (AIBridgeAgentIdentity with IdentityType = 'Email'). Organizers invite sage@customer.com like a person; the CalendarWatcher matches the invite to the identity and creates a Scheduled bridge that joins at start. The same identity model serves inbound telephony (a call to the agent’s DID routes to it via IdentityByValue).
• Native marketplace inclusion (NativeInvite) and in-meeting command — planned, per-platform.

Because the tenant-resident identity generalizes to “agent presence,” the same AIBridgeAgentIdentity row is the seam for future email and calendar surfaces, and existing tenant governance (retention, DLP, eDiscovery, offboarding) applies to the agent automatically.

8a. Invite & Calendar Joins (built — @memberjunction/ai-bridge-server)

The “invite the agent like a person” pipeline has four collaborating pieces, all injectable seams so the whole flow is unit-testable with no network/DB:

flowchart LR
    CAL["Calendar (Graph / Google)"] -->|ICalendarSource| W["CalendarWatcher.Sweep()"]
    W -->|new invite, agent is attendee| B["Scheduled AIAgentSessionBridge<br/>JoinMethod=Invite · Status=Scheduled<br/>ScheduledStartTime=start · Address=joinUrl"]
    B -->|ScheduledStartTime <= now| R["ScheduledBridgeRunner.RunDueBridges()"]
    R -->|host builds IRealtimeSession| E["AIBridgeEngine.StartBridgeSession()"]
    PROV["IAgentIdentityProvisioner"] -.provisions.-> ID["AIBridgeAgentIdentity (tenant mailbox)"]
    ID -.watched by.-> W

    style B fill:#2d5016,stroke:#1a5c3a,color:#fff
    style E fill:#5a3d7a,stroke:#7c5295,color:#fff

1. CalendarWatcher (calendar-watcher.ts) — for each active IdentityType='Email' identity, polls its calendar via an injected ICalendarSource, and for each new invite where the identity is an attendee creates a Scheduled AIAgentSessionBridge (+ a linked Idle AIAgentSession). It is idempotent (dedupes on the invite’s ExternalEventID, stamped onto ExternalConnectionID), tolerant (a failing identity/source is logged and the sweep continues), and exposes only Sweep() — no timer; the host schedules the polling loop (matching the engine janitor). It ignores non-attendee invites, invites with no join URL, past-start invites, and invites whose join URL resolves to no active provider.

2. ICalendarSource (calendar-source.ts) — the per-provider calendar-API seam: ListUpcomingInvites(identityValue, sinceCursor) returning normalized { ExternalEventID, JoinUrl, StartTime, Attendees, Organizer }. Ships with GraphCalendarSource and GoogleCalendarSource stubs that throw CalendarSourceNotBoundError until production binds the real Microsoft Graph (/me/events / app-level calendarView delta) and Google Calendar (events.list + syncToken) APIs. The real-API binding is a documented seam (like the bridge SDK seams) — see the TODO(production) in each adapter.

3. ResolveProviderFromJoinUrl(url, providers) (join-url-resolver.ts) — a pure helper mapping a join URL to the matching active AIBridgeProvider by hostname (zoom.us → Zoom, teams.microsoft.com → Teams, meet.google.com → Google Meet, Webex, Slack, Discord, RingCentral, …). Uses a strict domain-suffix test, so zoom.us.attacker.com and evil.com/zoom.us/... do not match. ClassifyJoinUrl(url) returns just the platform; both return null for unknown/malformed input.

4. ScheduledBridgeRunner (scheduled-bridge-runner.ts) — a documented host hook (like the janitor’s ReconcileOrphans): RunDueBridges() loads Scheduled bridges with ScheduledStartTime <= now and calls AIBridgeEngine.StartBridgeSession for each. The IRealtimeSession construction stays the host’s job — the runner asks a host-supplied ScheduledBridgeSessionFactory to build the start params, keeping the transport seam clean exactly like StartBridgeSession. The engine owns the Scheduled → Connecting → Connected/Failed transition, so a started/failed bridge drops off the next pass automatically (idempotent); a factory that returns null leaves the bridge Scheduled for later.

5. IAgentIdentityProvisioner (identity-provisioner.ts) — a thin documented seam to provision a tenant mailbox/calendar (or telephony DID) identity for an agent: Provision(request) returns the concrete { IdentityValue, Configuration }, which ApplyProvisionResultToIdentity maps onto a MJAIBridgeAgentIdentityEntity (the caller saves it). StubAgentIdentityProvisioner throws IdentityProvisionerNotBoundError until production wires Microsoft Graph admin / Google Workspace Admin SDK / a telephony carrier API. The schema already holds the result — this is the seam + the TODO(production), deliberately not over-built.

Host wiring (production): schedule CalendarWatcher.Sweep() and ScheduledBridgeRunner.RunDueBridges() on the host’s timer/scheduler, bind a real ICalendarSource + IAgentIdentityProvisioner, and supply a ScheduledBridgeSessionFactory that constructs the IRealtimeSession + RealtimeSessionRunner for a due bridge. Everything below the host is platform-free and tested with mocks.

---

9. Multi-party (LiveKit + multiple agents)

Emergent property of the bridge — shipped (Phase 7 core). Multi-party is not a separate build. The conferencing platform (or the MJ-native LiveKit room) is the shared room — it already does the SFU, the mixing, the multi-party media plane.

• Multiple humans? The platform already provides it — MJ builds nothing.
• Multiple agents? Each agent is an independent bridge connection (its own bot, its own AIAgentSession) into the same room. Two agents joining one room literally hear each other — each one’s voice is part of “everyone else” in the other’s inbound mix. No transcript-relay hack, no mixer in MJ — the room already mixes.

LiveKit — the MJ-native room (@memberjunction/ai-bridge-livekit)

Every other bridge connects out to a 3rd-party platform. LiveKit is the opposite: a self-hosted WebRTC SFU that MJ runs itself — the “MJ-native room” for an MJ-owned multi-party experience (e.g. embedded in Explorer). The architecture treats a Zoom meeting and a LiveKit room identically — both are multi-party media transports — so LiveKit is “another bridge, not a special build.”

• LiveKitBridge (@RegisterClass(BaseRealtimeBridge, 'LiveKitBridge')) is an ordinary BaseRealtimeBridge driver behind an injectable ILiveKitRoomSdk seam (production binds livekit-server-sdk + a room client; tests inject FakeLiveKitRoomSdk — 24 tests, no network).
• connect(roomUrl, token) joins the room as a bot participant with an MJ-minted token. LiveKit does the full media set — audio/video/screen in+out — and per-participant audio tracks, so diarization is native (no extra mixer). Chat rides the LiveKit data channel (sendDataMessage). It contributes a Meeting Controls facilitator channel exactly like Zoom.
• Capabilities: on-demand join, A/V/screen in+out, diarization, data-channel chat, room-admin mute. No scheduled/invite/telephony — those gated base methods throw BridgeCapabilityNotSupportedError.

Running LiveKit: local dev vs production (Cloud). The LiveKit SFU runs alongside MJAPI — MJAPI only mints join tokens (LIVEKIT_URL / LIVEKIT_API_KEY / LIVEKIT_API_SECRET), it never proxies media. For local dev, run a throwaway server — Docker for browser-only testing, or native livekit-server --config docker/livekit/livekit.yaml for the agent-bot path on macOS (Docker NAT breaks the bot’s WebRTC media there). For production / real-world deployments, use LiveKit Cloud — drop its wss://…livekit.cloud URL + key + secret into .env, no server to run. The canonical how-to (both options, the macOS gotcha, env vars) lives in docker/livekit/README.md.

Inter-agent speaking discipline — MultiAgentRoomCoordinator

The one genuinely new problem is keeping multiple agents from talking over each other or looping. The MultiAgentRoomCoordinator (@memberjunction/ai-bridge-server, pure + synchronous, 20 tests) adds floor arbitration on top of passive turn-taking:

• An agent calls CanTakeFloor(roomId, agentSessionId) before generating speech → true only if no other agent holds the floor. TakeFloor claims it, ReleaseFloor frees it, IsFloorHolder lets a bumped agent learn it must yield. At most one agent speaks at a time across agents.
• A designated facilitator agent (typically the one running the Meeting Controls channel) may override the floor (FacilitatorOverride) to cut in and call on a specific agent. Designate via RegisterRoomParticipant(roomId, id, /*isFacilitator*/ true) or SetFacilitator(...) at runtime.
• Loop-safety = two guards. Passive turn-taking (§6) means an agent speaks only when addressed by name, so two passive agents never start a loop; the floor coordinator means even if both policies fired, only one speaks. The combination makes two agents in a room loop-safe and non-overlapping.
• Distinct rooms are fully isolated; single-agent rooms impose no contention.

It is wired through the engine’s additive hooks — AIBridgeEngine.RegisterRoomParticipant(roomId, agentSessionId, isFacilitator?) / UnregisterRoomParticipant(...) — keyed on the shared external room id (the driver’s ExternalConnectionID). Every bridged session registers on connect and unregisters on teardown, so a single-agent room is a 1-member room whose floor is always free (no contention) and a 2+- member room takes turns. The engine’s Speak turn-decision (applyTurnDecision) now runs the floor internally: TakeFloor → denied ⇒ stay silent; granted ⇒ armFloorHold (a floorMaxHoldMs safety release) then trigger speech; the floor releases on the agent’s own final assistant transcript, the safety timer, or teardown.

Meeting mode — “hear everything, speak when addressed”

The actual fix for agents reacting to all room audio (spiral / “picked up the last conversation”): in a multi-agent room each agent’s model auto-response is disabled and the bridge becomes the sole speech trigger, gated by the addressing matcher. The lever is a host-neutral Config.disableAutoResponse flag each driver translates to its own mechanism — OpenAI → turn_detection.create_response=false (keeps VAD/transcription, suppresses the auto-reply); Gemini → realtimeInputConfig.automaticActivityDetection. disabled=true + manual activityStart/activityEnd turns (it has no detect-but-don’t-respond lever). The LiveKitAgentRoomCoordinator detects the multi-agent transition and starts the 2nd+ agent in meeting mode (RegexAddressedMatcher on its names + a meeting-discipline prompt clause). Gemini’s manual-activity turn boundaries are unit-tested at the send-shape level but await live validation in a busy room.

Retroactive re-gating — capability-gated (IRealtimeSession.Reconfigure)

When a room becomes multi-agent, the agents already in it are switched into meeting mode too, if their provider can reconfigure a live socket. This is gated by the realtime-session capability surface (RealtimeSessionCapabilities.CanReconfigureTurnMode, see the Co-Agents guide §4): OpenAI reports true and implements Reconfigure({DisableAutoResponse:true}) via a partial session.update (+ a SendContextNote so the agent knows it’s now in a meeting); Gemini reports false and is left conversational — no dead-method call. Engine: AIBridgeEngine.ReconfigureSessionToMeeting(bridgeId, matcher); coordinator re-gates each existing roster agent.

Turn moderator — room-level “who speaks next” (engine seam)

Per-agent addressing (the RegexAddressedMatcher broadcast above) routes direct address but can’t bring a relevant-but-unaddressed agent in, can’t let two agents have a productive exchange, and can’t distinguish a useful back-and-forth from an unproductive loop. The turn moderator is an optional engine seam that replaces the matcher with a single room-level decision per turn. The LLM specifics (it’s a prompt not an agent run, the config cascade, the model, proactive vs addressed-only) live in the Co-Agents guide §4; this is the engine-side mechanics.

• The seam. AIBridgeEngine.SetTurnModerator(moderator) injects a TurnModerator — (ctx: TurnModeratorContext) => Promise<string[]> returning the ordered AgentSessionIDs to trigger (0+). It is analogous to SetTranscriptSink / SetSessionRunFinalizer: the engine stays framework-agnostic (it knows only the function signature + the TurnModeratorContext / ModeratorRosterAgent / ModeratorLookbackTurn shapes it exports); the @memberjunction/ai-agents RealtimeTurnModerator supplies the implementation, wired at MJServer schema-build time (RealtimeBridgeResolver calls SetTurnModerator(RealtimeTurnModeratorDecision)).
• Broadcast → moderator → serialized queue. In a multi-agent room dispatchUserTurn branches: when a moderator is set it routes the turn through runRoomModerator (one decision) instead of broadcasting to each agent’s matcher. runRoomModerator builds the context from the live roster + diarized lookback, calls the moderator, filters the result to valid roster sessions, sets the room’s ordered roomSpeakerQueue, and drainSpeakerQueue triggers speakers one at a time through the floor (MultiAgentRoomCoordinator) — a multi-agent route is heard as a clean sequence, never overlap. Per-room re-entrancy is guarded (roomModeratorBusy); the floor-release path drains the next queued speaker.
• Lookback buffer. The engine keeps a bounded per-room diarized lookback (roomLookback, capped at ROOM_LOOKBACK_MAX_TURNS = 50, each turn clipped) appended on every final transcript via recordRoomTurn. That buffer is the moderator’s conversation window (the moderator narrows it further to its configured contextWindowTurns). The engine also tracks roomConsecutiveAgentTurns (reset on any human turn) for the optional agent-only-loop backstop.
• Pre-stage on agent speech. The moderator also runs on agent turns (onAgentTurnCompleted), not just human ones — powering both agent↔agent continuation and pre-staging: because it fires on the agent’s FINAL TEXT while that agent’s audio is still playing out, the next decision’s latency is hidden behind playback.
• Barge-in invalidation. A human barge-in clears the room’s staged speaker queue (clearRoomModeratorState(roomKey, /*full*/ false)) so a pre-staged agent→agent hand-off is discarded when the user changes the topic; teardown clears the full moderator state (lookback + queue + counter).
• Matcher fallback. When no moderator is injected, dispatchUserTurn keeps using the existing per-agent RegexAddressedMatcher broadcast — single-agent rooms and no-moderator deployments are entirely unaffected.

Unified room transcript + per-speaker diarization

The engine elects one scribe per room (the first agent; re-elected on departure) and feeds its FINAL transcripts to a registered BridgeTranscriptSink. The app-layer sink (CreateBridgeRoomTranscriptSink, @memberjunction/ai-agents) writes one MJ: Conversations of Type='Meeting Room' per room (linked to the Meet application by name, scoped out of normal chat) + one MJ: Conversation Details per utterance. Diarization: the model transcript has no speaker label, so the engine tracks LastInboundSpeaker from the inbound frame’s SpeakerLabel (which the drivers populate; see §4 media-track translation) and stamps it on each User line — so a heard line resolves to the actual participant (human or another agent), not a generic “User”. UserID resolution (vs. display name) is the remaining refinement.

Surfacing: the Voice Transcripts nav item in the AI Analytics dashboard (@memberjunction/ng-dashboards, AnalyticsRealtimeTranscriptsComponent + realtime-transcripts-data.ts) is a master-detail, diarized browser over these meeting transcripts.

Session reaping — auto-leave + idle sweep + run finalization

Three same-process safeguards keep abandoned sessions from lingering Connected with a Running co-agent run: (1) auto-leave — once a human has been seen and the roster drops to agents-only, an emptyGraceMs timer (cancelled on re-join) stops the session; (2) idle/TTL sweep — SweepStaleSessions (self-scheduled from HandleStartup) reaps sessions idle past idleTtlMs or over maxDurationMs, the same-process complement to the prior-boot ReconcileOrphans; (3) run finalization on every teardown path — a registered finalizer (called from markBridgeDisconnected) closes the co-agent run even on orphan/cross-host reaps. All thresholds are overridable via AIBridgeEngine.ConfigureSessionTimings({...}).

⚠️ Single-node room state (a multi-node hardening constraint)

The room-scoped coordination state — the MultiAgentRoomCoordinator floor, the transcript scribe election, and the room roster (membership for meeting-mode detection + re-gating) — all live in-memory per engine instance, i.e. per MJAPI node. The design therefore assumes all agents in a given room land on the same host. Single-node deployments are unaffected.

In a multi-node deployment behind a load balancer, two “add agent to room” requests can hit different nodes, splitting the state: two scribes could both write the transcript, the floor wouldn’t arbitrate across hosts, and meeting-mode detection (which counts roster members on this node) could miss agents on another node. Bridge rows already carry HostInstanceID (used by the prior-boot janitor), so the hardening paths are: (a) room-affinity routing — pin all of a room’s agents to one node, or (b) shared room state — move the floor/scribe/roster to a shared store (DB or Redis). Until one of those lands, treat single-node-per-room as a deployment invariant.

Echo / self-audio

A bot must not hear its own output or it would react to itself. Conferencing platforms (and the LiveKit SFU) exclude a participant’s own published audio from that participant’s inbound mix, so the bridge driver never feeds the agent its own voice (LiveKitBridge documents this). Where a platform does not exclude own-audio, the bridge driver must gate it before OnMedia. The MultiAgentRoomCoordinator itself never touches media — it operates purely on floor state. The BridgeParticipantInfo.IsAgent flag also lets multi-agent turn-taking exclude agents from “a human addressed me” detection.

---

10. Roadmap / Phase Status

This guide documents shipped code (Phase 0 and Phase 1 core). Everything else is planned — honestly marked here and in the architecture plan.

Phase	Deliverable	Status
0 — Transport seam	BaseRealtimeBridge media contract wired to IRealtimeSession (OnMedia → SendInput, OnOutput → SendMedia); AIBridgeEngine ↔ injected IRealtimeSession; LoopbackBridge proving an in-memory round-trip.	✅ Shipped
1 — Schema + engines	The 5-entity migration → CodeGen; AIBridgeEngineBase (cache) + AIBridgeEngine (composition, lifecycle, host registry, janitor ReconcileOrphans); capability-gated BaseRealtimeBridge + BridgeCapabilityNotSupportedError; TurnTakingPolicy; the *EntityServer invariants.	✅ Shipped (provider seed metadata + the Realtime: Advanced Bridge Controls authorization pending)
2 — Server-side channel plane	Dynamic tool-definition contribution into the runner’s ServerChannelTools / ExecuteServerChannelTool, optional client surface, the Meeting Controls channel (roster / hand-raise queue / who’s-speaking / timer → facilitator). The bridge ↔ channel-plane integration (AIBridgeEngine wires a session’s channels generically via the optional IBridgeChannelHost + BaseRealtimeBridge.GetMeetingControlsEventSource) is shipped.	✅ Shipped
3 — Zoom meeting bridge	ZoomBridge (@memberjunction/ai-bridge-zoom): on-demand + scheduled join, audio in/out, diarized participants, participant mute + chat, and a Meeting Controls facilitator surface — all behind an injectable IZoomMeetingSdk seam (real-SDK binding is a deployment TODO via SetSdkFactory). First real platform, built + unit-tested with no network.	✅ Shipped (real Zoom SDK adapter binding pending)
4 — Invite/calendar joins + agent identity	CalendarWatcher (“invite the agent like a person”) over an injectable ICalendarSource (Graph/Google stubs), ResolveProviderFromJoinUrl, ScheduledBridgeRunner (a janitor-style host hook starting due Scheduled bridges), and the IAgentIdentityProvisioner seam — all platform-free + unit-tested. Shared with inbound telephony.	✅ Shipped (real calendar-API + provisioning admin-API binding pending — documented TODO(production) seams)
5 — Teams → Meet → Webex → Slack → Discord	One minimal subclass per platform on the proven base.	🔜 Planned
6 — Telephony: RingCentral → Twilio → Vonage	BaseTelephonyBridge — outbound dial + inbound DID routing, DTMF, transfer. Same engine/session/turn-taking.	🔜 Planned
7 — Multi-party	LiveKitBridge (@memberjunction/ai-bridge-livekit) — the self-hosted MJ-native room (full A/V/screen, native per-participant diarization, data-channel chat) behind an injectable ILiveKitRoomSdk seam; MultiAgentRoomCoordinator (@memberjunction/ai-bridge-server) — floor arbitration (one agent speaks at a time) + facilitator override, loop-safe with passive turn-taking; additive AIBridgeEngine.RegisterRoomParticipant/UnregisterRoomParticipant hooks. Built + unit-tested with no network.	✅ Shipped (core) (real LiveKit SDK adapter binding pending; runner-layer floor wiring around generation pending)
8 — Remote Browser channel	RemoteBrowserChannel + ContainerRunner (any Playwright Docker image), screen-track out, control arbiter.	🔜 Planned
9 — Native marketplace inclusion	Per-platform “add the agent” apps.	🔜 Planned
UI — Realtime dashboard + forms	A Realtime section in the AI dashboard + custom *Extended forms + an AIAgent Realtime panel.	🔜 Planned

graph LR
    P0["0 · Transport seam ✅"] --> P1["1 · Schema + engines ✅"]
    P1 --> P2["2 · Channel plane"]
    P2 --> P3["3 · Zoom ✅"]
    P3 --> P4["4 · Invite/identity ✅"]
    P3 --> P5["5 · Teams·Meet·Webex·Slack·Discord"]
    P4 --> P6["6 · Telephony"]
    P5 --> P7["7 · Multi-party + LiveKit ✅"]
    P6 --> P7
    P3 --> P8["8 · Remote Browser"]

    style P0 fill:#2d5016,stroke:#1a5c3a,color:#fff
    style P1 fill:#2d5016,stroke:#1a5c3a,color:#fff
    style P2 fill:#2d5016,stroke:#1a5c3a,color:#fff
    style P3 fill:#2d5016,stroke:#1a5c3a,color:#fff
    style P4 fill:#2d5016,stroke:#1a5c3a,color:#fff
    style P7 fill:#2d5016,stroke:#1a5c3a,color:#fff

---

Reference Map

Concern	Where
Abstract driver, capability gating	packages/AI/BridgeBase/src/base-realtime-bridge.ts
Metadata cache engine	packages/AI/BridgeBase/src/ai-bridge-engine-base.ts
Media-track types	packages/AI/BridgeBase/src/media-tracks.ts
Turn-taking policy	packages/AI/BridgeBase/src/turn-taking-policy.ts
Capability error	packages/AI/BridgeBase/src/capability-errors.ts
Server engine + transport seam + channel-plane wiring + janitor	packages/AI/Bridge/src/ai-bridge-engine.ts
Bridge ↔ channel-plane seam (IBridgeChannelHost, IBridgeMeetingControlsEventSource)	packages/AI/BridgeBase/src/channel-plane.ts
Worked minimal driver example	packages/AI/Bridge/src/loopback-bridge.ts
Calendar watcher (invite → scheduled bridge)	packages/AI/Bridge/src/calendar-watcher.ts
Calendar-API seam + Graph/Google stubs	packages/AI/Bridge/src/calendar-source.ts
Join-URL → provider resolution (pure)	packages/AI/Bridge/src/join-url-resolver.ts
Scheduled-bridge runner (host hook)	packages/AI/Bridge/src/scheduled-bridge-runner.ts
Agent identity provisioning seam	packages/AI/Bridge/src/identity-provisioner.ts
Zoom driver (first real platform)	packages/AI/Providers/BridgeZoom/src/zoom-bridge.ts, zoom-sdk.ts, zoom-meeting-controls.ts
LiveKit driver (MJ-native room)	packages/AI/RealtimeBridge/Providers/LiveKit/src/livekit-bridge.ts, livekit-sdk.ts, livekit-meeting-controls.ts
Multi-agent floor arbitration (one speaker at a time + facilitator)	packages/AI/Bridge/src/multi-agent-room-coordinator.ts (engine hooks: AIBridgeEngine.RegisterRoomParticipant / UnregisterRoomParticipant / RoomCoordinator)
Turn-moderator engine seam (SetTurnModerator, lookback, serialized queue, pre-stage, barge-in invalidation)	packages/AI/RealtimeBridge/Server/src/ai-bridge-engine.ts
Turn-moderator LLM plugin (prompt-not-agent, config cascade, name→session mapping)	packages/AI/Agents/src/realtime/realtime-turn-moderator.ts, realtime-coagent-config.ts
Server channel host + Meeting Controls channel	packages/AI/Agents/src/realtime/realtime-channel-server-host.ts, meeting-controls-channel-server.ts
Capability interface	metadata/entities/JSONType-interfaces/IBridgeProviderFeatures.ts
Entity invariants	packages/MJCoreEntitiesServer/src/custom/MJAIBridge*EntityServer.server.ts, MJAIAgentSessionBridge*EntityServer.server.ts
Generated entities	@memberjunction/core-entities (MJAIBridgeProviderEntity, …)
Full architecture + roadmap	plans/realtime/realtime-bridges-architecture.md