ReadonlyActive$True while a session is open (mic button active, overlay shown).
ReadonlyActiveThe session's ACTIVE interactive-channel plugins, resolved from the MJ: AI Agent Channels registry at session start (one instance per session, per channel). Emits
[] before a session starts and after teardown. The overlay subscribes to register
one surface tab per plugin — it never knows any concrete channel type.
ReadonlyAgentDisplay name of the agent the active session fronts (set at session start).
ReadonlyAppObservable of the live app-context snapshot (see _appContext$).
ReadonlyCaptions$Live captions for both sides of the conversation.
ReadonlyChannelFires with the channel PLUGIN every time the agent ACTS on that channel (a tool call was routed to its local executor — e.g. the agent drew on the whiteboard). The overlay uses the FIRST emission per channel to auto-reveal + focus the channel's surface tab, so the user discovers the surface the moment the agent starts using it. Finer-grained than SessionStarted$/SessionEnded$ (per tool call, not per session).
ReadonlyChannelChannel requests to enter / leave the FOCUS layout (see
RealtimeChannelFocusEvent). Fired when a plugin calls its host context's
SetFocusMode — e.g. the whiteboard's "Focus board" toggle.
ReadonlyConnectionCurrent connection / turn state.
ReadonlyDelegationEPHEMERAL spoken progress narrations (see RealtimeDelegationNarration). These are deliberately kept OUT of Captions$ and never relayed/persisted — the overlay renders them as a transient "live note" near the active working card.
ReadonlyDelegationProgress updates from a delegated agent run (e.g. Sage) while the realtime model waits on it. The future overlay subscribes to render a "working" card; the model also narrates these aloud.
ReadonlyDelegationTerminal result of a delegation, so the overlay can complete the working card with real content.
ReadonlyMinimized$True while the active call overlay is MINIMIZED to the host's floating "on call" pill
(e.g. after a dev link navigated away). The mic and session stay fully live — this is
pure presentation state, reset to false at session start and teardown.
ReadonlyModelDisplay name of the realtime MODEL the active session runs on (server-reported at session
start, e.g. "GPT Realtime 2"). null before a session starts / when the server didn't report
one. The overlay banner shows it subtly next to the agent identity.
ReadonlySessionFired EXACTLY ONCE per session as teardown begins, with the prior
agentSessionId (so subscribers can correlate against SessionStarted$'s
sessionId) and the client-distinguishable reason — 'explicit' when the
user called EndRealtimeSession, 'error' when teardown ran from a catch
block. Server-side close paths (janitor, shutdown) do NOT propagate here —
they happen out-of-process and have no client push channel today.
ReadonlySessionFired EXACTLY ONCE per session after both agentSessionId is set AND the
realtime client is connected. Carries the server-issued sessionId and the
ChannelName of each plugin resolved at session mint. Consumed by
RealtimeSessionsAdapter (in this package) to feed
@memberjunction/conversations-runtime's SessionsObserver.
Why this exists separately from Active$: Active$ flips true BEFORE
mintSession resolves, so agentSessionId is still null at that moment.
Subscribers correlating (Active$, agentSessionId) would race; this event
removes the race.
Synchronous access to the session's active interactive-channel plugins.
Synchronous access to the display name of the agent the active session fronts.
The first final user utterance of the session (naming seed); null before the user speaks.
True when a session is currently open.
Synchronous access to the minimized presentation state.
Metadata provider used for the GraphQL relay mutations. Falls back to the global default when unset (single-provider apps see no change).
When the active/last session CREATED its conversation (started without one), the new conversation's id — the host uses it to refresh the cached list, conditionally select it on close, and auto-name it. Null when the session joined an existing conversation.
The ChannelNames the agent has used (acted on) at least once this session. The overlay
uses this to register a channel's surface tab only after it has come into play. A fresh
Set snapshot so callers can't mutate the service's tracking.
Cancels ONE in-flight delegated tool call — the overlay's per-card ✕ affordance.
EXPLICIT USER INTENT ONLY (deliberate host policy): true barge-in never aborts
delegations — the narration design expects the user to talk while delegated work runs.
Calls the CancelRealtimeSessionTool mutation (ownership-gated server-side); when the
server reports it aborted the run, the card is flipped immediately to a FAILED
"Cancelled by user" result and the eventual late result from the aborted run is
suppressed (see emitDelegationResult).
true when the server aborted the in-flight run; false when there was
nothing to cancel (the work finished first — its real result is already racing in)
or the mutation failed (logged, never thrown).
Cancels EVERY in-flight delegated tool call for the active session (callId-less form of
the CancelRealtimeSessionTool mutation). Exposed for host policies that need a
sweep-cancel (e.g. an explicit "stop everything" affordance) — NOT wired to barge-in,
by the same deliberate policy as CancelDelegation.
The number of in-flight runs the server aborted (0 when nothing was tracked in flight client-side, nothing was in flight server-side, or the mutation failed).
End the active session: stop the mic, tear down the provider connection, and close the server-side agent session. Safe to call when no session is active.
The active client's current audio activity (per-direction RMS levels + spectrum
bins), or null when no session is live or the driver attached no audio meters.
Sampled by the overlay's animation-frame loop to drive the audio-reactive orb/EQ —
a cheap analyser read, never provider traffic.
Whether the agent has used (acted on) the named channel at least once this session.
Replaces the set of host-registered surface client tools the realtime ContextTool can execute.
The host calls this at session start and whenever the active surface's tool set changes (the
continuous-capability half of client-context delivery). Passing [] clears them.
The current surface client tools (name + handler). Descriptions/schemas ride the app-context manifest separately; only the executable handler is needed here.
Registers a handler for CLIENT-EXECUTED UI tools whose names start with toolNamePrefix
(e.g. 'Whiteboard_' → all Whiteboard_* calls). Matching tool calls execute LOCALLY via
the handler — they are never relayed to the server — and the handler's result JSON is sent
back to the model as the tool_response. Re-registering the same prefix replaces the
handler. The registry is cleared at session teardown.
Persists an interactive channel's state of record (e.g. the whiteboard's serialized scene)
onto the session's MJ: AI Agent Session Channels row via SaveSessionChannelState.
The channel definition name (e.g. 'Whiteboard').
The serialized channel state.
OptionalagentSessionId: string | nullOptional EXPLICIT session id. The debounced channel-save pipeline
captures the id while the session is live and passes it here, so the final teardown
flush still lands on the just-closed session. Falls back to the active session's id;
returns false when neither is available.
Whether the server persisted the state. Failures are logged, never thrown — channel persistence is best-effort and must not disturb the live call.
Feeds a background context note into the live model (no spoken reply is requested) — the perception channel interactive surfaces use (e.g. the whiteboard's coalesced scene deltas). No-op when no session is live.
Inject a typed message into the live session as a user turn.
Decomposed into two steps, each mirroring an existing voice path so the typed turn behaves identically to a spoken one:
No-op when no session is open / the control channel isn't ready, or when the text is empty.
Minimizes / restores the active call overlay (host renders the floating pill while minimized). Presentation-only — the live audio session is untouched.
Start a client-direct voice session fronting targetAgentId.
The agent the Realtime Co-Agent voices on behalf of.
OptionalconversationId: string | nullOptional existing conversation to bind + seed context from.
OptionallastSessionId: string | nullOptional prior session to chain to (resume / continuation).
OptionalagentName: string | nullOptional display name of the target agent — resolved by the caller (which knows the conversation's routing context) and surfaced on AgentName$ so ANY host (composer trigger, chat-area overlay) can render it without re-resolving.
OptionalpreferredModelId: string | nullOptional EXPLICIT realtime model choice (MJ: AI Models.ID). When
set, the server uses exactly that model and FAILS with a clear reason if it can't (no
silent fallback). Omit for the server's automatic (highest-PowerRank) selection.
OptionalclientTools: RealtimeToolDefinition[] | nullOptional EXTRA client-executed UI tool declarations to expose to the realtime model alongside the server's stable tool set and the interactive-channel tools (which are aggregated automatically from the registry-resolved plugins — see ActiveChannels$). The server only DECLARES these — execution stays in the browser via handlers registered with RegisterClientToolHandler. This is an extension point for hosts with bespoke (non-channel) UI tools; most callers omit it.
OptionalcoAgentId: string | nullOptional EXPLICIT co-agent choice (MJ: AI Agents.ID of an Active,
Realtime-type agent) — the highest-precedence step of the server's co-agent resolution
chain. When set, the server uses exactly that co-agent and FAILS with a clear reason if
it can't (no silent fallback). Omit to let server metadata drive the choice: the target
agent's DefaultCoAgentID, then the type-level AIAgentCoAgent default row, then the global Realtime Co-Agent.
OptionalconfigOverridesJson: string | nullOptional JSON payload of SESSION CONFIG overrides (e.g.
{"realtime":{"modelPreference":"<modelId>"}}), forwarded verbatim on the mint
mutation. The server enforces the Realtime: Advanced Session Controls
authorization on any overrides — hosts only populate this from authorization-gated
pickers, and never synthesize overrides beyond what the user explicitly chose.
Omit/null for the server's defaults (today's behavior).
OptionalrecordingConsent: boolean | nullOptional EXPLICIT "record this call" consent for THIS session. When
true, the browser records a mic + agent-audio mix and uploads it at session end. When
omitted/null, the per-user persisted preference (mj.realtimeVoice.recordingConsent.v1
via UserInfoEngine) is read as the default. false never records.
OptionalmediaCollectionId: string | nullOptional per-session media-kit override (MJ: Collections.ID). When set,
the server-side Media channel resolves THIS collection as the agent's media kit for the session,
taking precedence over the agent's DefaultMediaCollectionID. The server UUID-validates it
(malformed ⇒ ignored, the agent default applies). Omit/null to use the agent default kit.
OptionalapplicationId: string | nullOptionalappContext: AppContextSnapshot | nullMute / unmute the local microphone track. Returns the new muted state.
Removes the handler registered for toolNamePrefix (no-op when none is registered).
Push an updated app-context snapshot mid-session (the continuous-streaming half of client-context
delivery). The host (Explorer) calls this when the user navigates / the active surface's state or
capability manifest changes; the ClientContextChannel turns the delta into a SendContextNote.
No-op semantics when no session is live — the channel simply re-reads on next start.
The latest app-context snapshot (or null to clear).
Drives a client-direct real-time voice session: the browser mints an ephemeral token from the MJ server, then connects DIRECTLY to the realtime provider. Audio frames never transit the MJ server (low latency); only tool calls and final transcripts are relayed back to MJ over GraphQL.
This service is PROVIDER-AGNOSTIC policy/orchestration. All provider wire concerns (transport, event translation, the response state machine, narration-kind tagging, playback tracking) live in a BaseRealtimeClient driver resolved through the MJ ClassFactory by the server-reported
Providerkey (e.g.'openai'→OpenAIRealtimeClient). Future providers (Gemini Live, …) snap in by registering a new driver — this service does not change.The Realtime Co-Agent (server-side) fronts the conversation's current agent — the server bakes the companion instructions + tool set into
SessionConfigJson, which the client driver applies verbatim.Lifecycle: StartRealtimeSession → live duplex → EndRealtimeSession.