the platform's media capabilities. The runtime never touches navigator,
Blob or FileReader itself — microphone acquisition and audio capture/encoding
are the host's, because both are platform-specific (permission UX, which
encoder the platform offers).
ReadonlyActive$True while a session is open (mic button active, overlay shown).
ReadonlyActiveThe session's ACTIVE interactive-channel plugins, resolved from the MJ: AI Agent Channels registry at session start (one instance per session, per channel). Emits
[] before a session starts and after teardown. The overlay subscribes to register
one surface tab per plugin — it never knows any concrete channel type.
ReadonlyAgentDisplay name of the agent the active session fronts (set at session start).
ReadonlyAgentThe agent's video (an avatar) while the model sends it: a live stream, or a player that owns the <video> element.
null until the video arrives, for a model that sends none, and outside a call. The driver hands it over once it is
live, and a newer one replaces it. A host shows it with AttachVideoSource; a channel that sinks outbound video
follows it through its context, and counts as used once it arrives, so the host shows that channel's surface.
ReadonlyAppObservable of the live app-context snapshot (see _appContext$).
ReadonlyAvatarWhy the call shows no avatar the agent asked for: set once per call, after it connects, when the model renders none,
the persona has no face for it, the app shows no agent video (host) or the browser can't play it (browser); with
no Reason when the mint gave one this version doesn't know, such as a newer server's. null when there is nothing
to say (no avatar asked for, the avatar shows, or the mint reported no status) and after the call ends. A provider
resume inside the call does not set it again. A host turns the reason into words once, for example "Audio only: this
voice model can't show an avatar".
ReadonlyCaptions$Live captions for both sides of the conversation.
ReadonlyCaptureWhich captures the call offers, now and on every change. A capture is offered while the call is connected, the model takes inbound video, the host can open it (a camera controller; screen sharing) and the call's policy admits it (its channel is in the call and may show the agent pixels). A host shows its Camera and Share controls from this. Neither is offered outside a call.
ReadonlyCaptures$The user's camera and screen share: off, starting, on (with the stream to show the user) or failed (with the reason). Both off outside a session. Start and stop them with StartCamera, StartScreenShare, StopCamera and StopScreenShare; while one is on, it is also a source on VideoSources$.
ReadonlyChannelFires with the channel PLUGIN every time the agent ACTS on that channel (a tool call was routed to its local executor — e.g. the agent drew on the whiteboard). The overlay uses the FIRST emission per channel to auto-reveal + focus the channel's surface tab, so the user discovers the surface the moment the agent starts using it. Finer-grained than SessionStarted$/SessionEnded$ (per tool call, not per session).
Loads channel code on demand. Called, with the ClientPluginClass keys of every channel about to be
resolved (the registry's rows plus the host's declarations), after the registry is read and before any
plugin is built — so a host can register a heavy channel's class only when the session could use it.
It must resolve once the classes it knows are registered; a rejection is logged, never fatal.
ReadonlyChannelChannel requests to enter / leave the FOCUS layout (see
RealtimeChannelFocusEvent). Fired when a plugin calls its host context's
SetFocusMode — e.g. the whiteboard's "Focus board" toggle.
ReadonlyConnectionCurrent connection / turn state.
ReadonlyDelegationEPHEMERAL spoken progress narrations (see RealtimeDelegationNarration). These are deliberately kept OUT of Captions$ and never relayed/persisted — the overlay renders them as a transient "live note" near the active working card.
ReadonlyDelegationProgress updates from a delegated agent run (e.g. Sage) while the realtime model waits on it. The future overlay subscribes to render a "working" card; the model also narrates these aloud.
ReadonlyDelegationTerminal result of a delegation, so the overlay can complete the working card with real content.
What the agent should SAY the moment the user's identity is verified, or null (the default) to
say nothing — the agent is still told, silently, through a context note. A string is used as the
instruction verbatim; a function receives the verified payload (e.g. to greet by name) and returns
the instruction, or null to stay silent for this verification.
Protected Readonlymediathe platform's media capabilities. The runtime never touches navigator,
Blob or FileReader itself — microphone acquisition and audio capture/encoding
are the host's, because both are platform-specific (permission UX, which
encoder the platform offers).
ReadonlyMicrophone$The call's microphone, now and on every change: the one in use and the ones the user can move it to with
SwitchMicrophone, as the host's camera-and-microphone controller lists them. Offered from the moment the call
connects until it ends, and kept current as devices come and go. When the call loses its microphone (the one in use
fails and so does the fallback to the default, or the last one goes away), its Failure says why: the agent hears
nothing until the user picks a microphone, which opens it again. REALTIME_MICROPHONE_NONE outside a call and
on a host without a controller, which opens the microphone itself (IRealtimeMediaHost.AcquireMicrophone).
ReadonlyMinimized$True while the active call overlay is MINIMIZED to the host's floating "on call" pill
(e.g. after a dev link navigated away). The mic and session stay fully live — this is
pure presentation state, reset to false at session start and teardown.
ReadonlyModelDisplay name of the realtime MODEL the active session runs on (server-reported at session
start, e.g. "GPT Realtime 2"). null before a session starts / when the server didn't report
one. The overlay banner shows it subtly next to the agent identity.
ReadonlyResuming$Whether the call is resuming: moving to a new connection with its provider mid-call. true from a 'connecting' the
driver reports while the call is live, until the call is live again ('listening', 'speaking' or 'thinking'),
fails ('error') or ends ('closed'). A call's first connect is not a resume, so it is false until the call has
been live, and outside a call. Gemini Live resumes when Google ends a connection (goAway) and after a dropped
socket, and the agent's video can stop for a few seconds meanwhile: a host can show that as a pause rather than as
the agent leaving. It changes before ConnectionState$ does, so a resume's 'connecting' arrives with it
already true.
ReadonlySavingTrue while the end-of-call recording is being saved: from the first statement of a teardown that holds a recorder until that teardown settles — which includes waiting (at most RECORDING_TERMINAL_ACK_TIMEOUT_MS) for the server to acknowledge the recording's terminal request. False at construction and for unrecorded sessions. Exists for #5195; the runtime is platform-free, so hosts own the reaction (a "saving" indicator, a disabled navigation button, etc.).
ReadonlySessionFired EXACTLY ONCE per session as teardown begins, with the prior
agentSessionId (so subscribers can correlate against SessionStarted$'s
sessionId) and the client-distinguishable reason — 'explicit' when the
user called EndRealtimeSession, 'error' when teardown ran from a catch
block. Server-side close paths (janitor, shutdown) do NOT propagate here —
they happen out-of-process and have no client push channel today.
ReadonlySessionEvery event the server publishes to THIS session, for the session's life (identity verification today; apps add their own types). Hot, with no replay — subscribe before the session starts, once, and it keeps working across sessions. The runtime has already acted on the events it understands (see handleIdentityVerified) by the time one reaches here; this stream is for the host UI.
Silent when the session's provider has no subscription transport (see createSessionEventSource).
ReadonlySessionFired EXACTLY ONCE per session after both agentSessionId is set AND the
realtime client is connected. Carries the server-issued sessionId and the
ChannelName of each plugin resolved at session mint. Consumed by
RealtimeSessionsAdapter (in this package) to feed
@memberjunction/conversations-runtime's SessionsObserver.
Why this exists separately from Active$: Active$ flips true BEFORE
mintSession resolves, so agentSessionId is still null at that moment.
Subscribers correlating (Active$, agentSessionId) would race; this event
removes the race.
ReadonlyThoughtModel-authored thought / reasoning narrations (see RealtimeThoughtNarration). These are reasoning summaries author-emitted during extended thinking, separate from spoken progress updates.
ReadonlyVideoThe video sources the agent is perceiving or could perceive right now (a whiteboard, a remote browser, a shared screen), with whether each is on and whether its frames are reaching the model. Empty when no session is live or nothing has offered a frame source. This is what the "agent can see" control renders; toggle one with SetVideoSourceEnabled.
Synchronous access to the session's active interactive-channel plugins.
The on-demand channels that are in the session but not open yet — what the agent can open
through ContextTool. A fresh array; empty before a session starts and after teardown.
The active BaseRealtimeClient driving the media plane, or null when not connected.
Synchronous access to the display name of the agent the active session fronts.
Synchronous access to AvatarNotice$'s current value.
The store the user's per-channel exposure choices are currently kept in (so a host can wrap it).
The first final user utterance of the session (naming seed); null before the user speaks.
True when a session is currently open.
Synchronous access to the minimized presentation state.
Synchronous read of SavingRecording$.
The fatal error the call's client reported, or null when it has reported none.
Read it when ConnectionState$ reports 'error' on a call that was live, to tell a call that reached its time
limit (Code 'time-limit'; DescribeCallEnd words it) from a dropped connection. Each start clears it before
it reports 'connecting'.
Why the latest session start failed, or null when it has not failed or none has run.
Read it when ConnectionState$ reports 'error', to tell a denied microphone apart from
a provider or backend failure and show copy the user can act on. Each start clears it before it
reports 'connecting', so it never describes an earlier start.
How this runtime mints a session — the stock StartRealtimeClientSession mutation unless a host
installs its own IRealtimeSessionLauncher (a guest-session exchange, a deployment-specific
mutation). Everything after the mint — channel scoping, the driver, transcripts, teardown — is
unchanged. Assign null to restore the default.
Metadata provider used for the GraphQL relay mutations. Falls back to the global default when unset (single-provider apps see no change).
When the active/last session CREATED its conversation (started without one), the new conversation's id — the host uses it to refresh the cached list, conditionally select it on close, and auto-name it. Null when the session joined an existing conversation.
Synchronous access to SessionDeadline$.
The client's copy of the session's absolute deadline (null when none is known). A host that
knows the session's cap at start sets it with SetSessionDeadline; verification extends it
from the server's identity.verified payload. The SERVER enforces the real deadline — this exists
so the host can end the call gracefully first and show a countdown. It only ever moves later by
the server's word, so nothing a client does can extend the real one.
The ChannelNames the agent has used (acted on) at least once this session. The overlay
uses this to register a channel's surface tab only after it has come into play. A fresh
Set snapshot so callers can't mutate the service's tracking.
Use BuildClientConfig.
Builds the client-direct session config the realtime client connects with.
Aggregates the tracks active channels source and sink under REQUESTED_TRACKS_SESSION_KEY so the
driver can negotiate them (e.g., inbound video for Whiteboard / RemoteBrowser, outbound video for a
channel that shows the agent's video). A track the model does not support resolves to 'unsupported'.
The session's transport and relay URL go to the driver as minted (an unknown transport is left out: direct).
Cancels ONE in-flight delegated tool call — the overlay's per-card ✕ affordance.
EXPLICIT USER INTENT ONLY (deliberate host policy): true barge-in never aborts
delegations — the narration design expects the user to talk while delegated work runs.
Calls the CancelRealtimeSessionTool mutation (ownership-gated server-side); when the
server reports it aborted the run, the card is flipped immediately to a FAILED
"Cancelled by user" result and the eventual late result from the aborted run is
suppressed (see emitDelegationResult).
true when the server aborted the in-flight run; false when there was
nothing to cancel (the work finished first — its real result is already racing in)
or the mutation failed (logged, never thrown).
Cancels EVERY in-flight delegated tool call for the active session (callId-less form of
the CancelRealtimeSessionTool mutation). Exposed for host policies that need a
sweep-cancel (e.g. an explicit "stop everything" affordance) — NOT wired to barge-in,
by the same deliberate policy as CancelDelegation.
The number of in-flight runs the server aborted (0 when nothing was tracked in flight client-side, nothing was in flight server-side, or the mutation failed).
Removes every owner's tools.
The user checked the camera and turned it on: the agent sees it from now on, and later camera starts in this call skip the check. Returns the camera's state; unless the camera is waiting for its check (see StartCamera), nothing changes.
ProtectedcreateBuilds the durable local shard store. The default is IndexedDB where the host has it and a no-op store elsewhere (Node, some React Native runtimes). Called once per runtime. Override to supply another store.
ProtectedcreateBuilds the transport for shard uploads and terminal requests. The default posts to MJAPI's REST recording routes, derived from the session's GraphQL provider configuration, with credentials read per request so a refreshed token is used. Returns null — shards then wait in the local store — when the provider has no URL (a test double, a provider that is not GraphQL, or one not configured yet). Override to supply another transport.
ProtectedcreateBuilds the transport that reads the session's events. The default rides the session's GraphQL
provider; it returns null — "no session events here" — for a provider with no subscription
support (a test double, a host on another transport), which the runtime tolerates. Override to
supply another transport.
Re-uploads shards an earlier, interrupted call left in the local store (crash, closed tab, dead network), each under its OWN session id and index, oldest first, at most MAX_DRAINED_SHARDS_PER_LOAD per runtime. A shard is deleted only on the server's ack.
Runs once per runtime instance, re-armed whenever a recording ends (which may leave shards behind for the next session start on the same page). When there is no transport yet (no provider configuration, e.g. before login) it returns 0 WITHOUT marking itself done, so a later call — session start — drains instead of the shards being skipped for this page. Concurrent calls share one run. Never rejects.
the number of shards the server acknowledged in this call.
End the active session: stop the mic, tear down the provider connection, and close the server-side agent session. Safe to call when no session is active.
The active client's current audio activity (per-direction RMS levels + spectrum
bins), or null when no session is live or the driver attached no audio meters.
Sampled by the overlay's animation-frame loop to drive the audio-reactive orb/EQ —
a cheap analyser read, never provider traffic.
How much of a channel the model may perceive right now: the channel's ceiling, lowered by the server's
policy and the user's choice. null when the channel is not in this session.
The channel's key (case-insensitive).
The resolved behavior of an in-session channel (display, exposure ceiling, config, what put it
in the session), or null when the channel is not in this session. Hosts and overlays read the
resolved display from here rather than from the channel's code default.
The channel's key (case-insensitive).
Whether the agent has used (acted on) the named channel at least once this session.
ProtectedhostDeclares which provider keys this host can actually carry audio for.
The server resolves a realtime model by rank across every configured vendor, so it can legitimately return a provider whose client driver this host cannot run. A browser can run all of them; React Native can run the WebRTC ones but not those needing a Web Audio PCM plane. Connecting anyway gets as far as constructing the driver's playback engine and then throws — a crash, where the honest answer is "this workspace's voice provider is not one this app can use".
The default accepts everything, so existing hosts are unaffected. Override to narrow it.
True when a session is open or starting for conversationId. Ids are compared without
regard to case.
Checks whether a media track is established on the active realtime client.
Opens (and seeds) a channel from the HOST — the same path the agent's open action takes: the channel
is mounted if it was only advertised, its Inputs schema validates inputs, and it announces itself to
the model. A channel that is already open is re-seeded. Never throws; a refusal is a structured result.
The channel to open (case-insensitive).
Seed inputs, validated against the channel's descriptor.
Registers the tools ONE source of the host can run for the realtime ContextTool, replacing only
that source's previous set — other owners' tools are untouched. A host with two sources (Explorer's
always-available globals and the active surface's tools) registers each under its own owner key, so
refreshing the surface's tools can neither drop the globals nor leave the previous surface's tools
behind. Passing [] clears just that owner. When two owners register the same tool name, the
later-registered owner wins.
Calling it with one argument (the original form) registers under DEFAULT_APP_TOOL_OWNER, so a host with a single source behaves exactly as before.
The owner's complete current set (name + handler; description/schema optional).
A stable key naming the source (e.g. 'explorer.surface'). Defaults to a shared owner.
Registers a handler for CLIENT-EXECUTED UI tools whose names start with toolNamePrefix
(e.g. 'Whiteboard_' → all Whiteboard_* calls). Matching tool calls execute LOCALLY via
the handler — they are never relayed to the server — and the handler's result JSON is sent
back to the model as the tool_response. Re-registering the same prefix replaces the
handler. The registry is cleared at session teardown.
Asks the live model to SPEAK FIRST — before the human has said anything.
Every other path into the model's voice reacts to something: the human spoke, or a channel reported input. A host that needs the agent to open the conversation (an interviewer greeting a candidate, a guide introducing a task) had no way to ask for that, so the session connected and both sides waited for the other. The instructions are what to say, in the host's words — the model still speaks in its own voice and persona.
Returns whether the request was DELIVERED, which is the one way this deliberately differs from
SendContextNote beside it. A context note that is dropped costs the model a little
perception; an opening line that is dropped is a session that sits in silence, and the host
needs to be able to tell the two apart. false means no session was live (or the instructions
were empty) — usually a host that asked before the connection reached a speaking state, which
it can then retry.
Persists an interactive channel's state of record (e.g. the whiteboard's serialized scene)
onto the session's MJ: AI Agent Session Channels row via SaveSessionChannelState.
The channel definition name (e.g. 'Whiteboard').
The serialized channel state.
OptionalagentSessionId: string | nullOptional EXPLICIT session id. The debounced channel-save pipeline
captures the id while the session is live and passes it here, so the final teardown
flush still lands on the just-closed session. Falls back to the active session's id;
returns false when neither is available.
Whether the server persisted the state. Failures are logged, never thrown — channel persistence is best-effort and must not disturb the live call.
Picks the video source the agent sees: what the "agent can see" control calls when more sources are on than the
model takes. The pick beats every other rule (a camera or screen share the user started, the surface the user is
looking at) until it is cleared, or its source is turned off or leaves. The pick shows on VideoSources$
(Picked), and the model is told when what it sees changes.
The source's id (from VideoSources$), or null to let the call decide again.
false with no live session, or for a source that is not there or is turned off.
Feeds a background context note into the live model (no spoken reply is requested) — the perception channel interactive surfaces use (e.g. the whiteboard's coalesced scene deltas). No-op when no session is live.
Tells the server, right now, how much of the recording exists — for a host's pagehide and
visibilitychange → hidden handlers, where the page may be gone a moment later. Synchronous:
it seals the pending partial shard and fires the keepalive terminal with it, then re-sends
with keepalive whatever the server has not acknowledged yet (see
dispatchRecordingTerminal), and returns without waiting. Never stops the recorder: after
a hidden the call may well continue, and later audio simply starts the next shard index (the
tail is sealed, so no index is ever re-sent with different bytes). Also works while End is still
stopping the recorder, which is exactly when a host that does not await End navigates away.
No-op when nothing is recording, there is no session id or no transport, or nothing was
captured. Never throws.
Deliberately opens NO IndexedDB transaction inside the handler: a pagehide handler gets a few
milliseconds, and a store write started there could not complete during unload anyway. The
shards it sealed are written to the store from a setTimeout(0) task after the handler returns:
on a page that is merely hidden (the call goes on) that keeps the durable copy until the server
acknowledges it; on a page that is unloading the task just never runs.
Optionaloptions: RecordingTerminalOptionsInject a typed message into the live session as a user turn.
Decomposed into two steps, each mirroring an existing voice path so the typed turn behaves identically to a spoken one:
No-op when no session is open / the control channel isn't ready, or when the text is empty.
Relays a video frame straight to the underlying realtime client, if a session is live.
It bypasses the session's VideoSourceArbiter: the frame overrides the source the arbiter picked, and the
model is not told what it is looking at. Channels no longer send through it (their context's SendVideoFrame goes
through the arbiter); a host that shows the model something registers a source there instead, as
ChannelInboundVideoBridge does.
OptionalmimeType: stringtrue when the client accepted and sent the frame; false when no session is live, the
client cannot take video frames, or the client dropped the frame.
Replaces the store the user's per-channel exposure choices are kept in. The default persists them in
the signed-in user's settings (mj.realtime.visualPerception.v1) and keeps them in memory for an
anonymous principal or a connect-only embed; a host that remembers them somewhere else supplies its own.
Tells the session which channel's surface the user is looking at (null for none, e.g. the activity tab).
When the model can see only one video source and several are live, the one the user is looking at is the one
it sees (after an explicit pick, and after a camera or screen share the user started). Safe to call before the
session is live: it applies when the connection comes up.
The focused channel's key, as it appears on VideoSources$ entries' ChannelKey.
Minimizes / restores the active call overlay (host renders the floating pill while minimized). Presentation-only — the live audio session is untouched.
Sets the baseline deadline a host knows at session start (null clears). See SessionDeadline$.
The user's own choice of how much of a channel the agent may perceive. Takes effect immediately
(frames and notes the new level forbids stop, and the model is told), and is remembered per channel key
for next time. The level can only LOWER what the server's policy allows; choosing 'pixels' for a channel
the agent's policy capped at 'state' changes nothing.
The channel's key (case-insensitive).
The user's choice; undefined clears it (back to what policy allows).
false when the channel is not in this session (nothing is applied, but the choice is still remembered).
Turns the agent's view of one video source on or off — what the "agent can see" control calls.
A source that belongs to a channel goes through SetUserChannelExposure, so the choice is remembered per channel and the channel itself tells the model. Any other source (a camera or screen share that is not a channel) is switched at the arbiter, which tells the model.
The source's id (from VideoSources$).
Whether the agent may see it.
false when there is no such source.
Starts the user's camera and shows it to the agent. Call it from the user's click: it may ask for camera permission. Resolves with the camera's state; a failure (no session, a model that takes no video, a refused permission) is a state with a message, never a throw. While the camera is on, it is a source on VideoSources$, and it stops by itself if the device goes away.
When the host shows a camera check (RealtimeSessionStartOptions.CameraCheck), the call's first start
resolves still starting, with Checking set and the stream for the host to preview. The agent sees the camera
once the host calls ConfirmCamera; StopCamera is the user's "not now".
OptionaldeviceId: stringThe camera to open; the system default when absent.
Start a client-direct voice session fronting targetAgentId.
The agent the Realtime Co-Agent voices on behalf of.
OptionalconversationId: string | nullOptional existing conversation to bind + seed context from.
OptionallastSessionId: string | nullOptional prior session to chain to (resume / continuation).
OptionalagentName: string | nullOptional display name of the target agent — resolved by the caller (which knows the conversation's routing context) and surfaced on AgentName$ so ANY host (composer trigger, chat-area overlay) can render it without re-resolving.
OptionalpreferredModelId: string | nullOptional EXPLICIT realtime model choice (MJ: AI Models.ID). When
set, the server uses exactly that model and FAILS with a clear reason if it can't (no
silent fallback). Omit for the server's automatic (highest-PowerRank) selection.
OptionalclientTools: RealtimeToolDefinition[] | nullOptional EXTRA client-executed UI tool declarations to expose to the realtime model alongside the server's stable tool set and the interactive-channel tools (which are aggregated automatically from the registry-resolved plugins — see ActiveChannels$). The server only DECLARES these — execution stays in the browser via handlers registered with RegisterClientToolHandler. This is an extension point for hosts with bespoke (non-channel) UI tools; most callers omit it.
OptionalcoAgentId: string | nullOptional EXPLICIT co-agent choice (MJ: AI Agents.ID of an Active,
Realtime-type agent) — the highest-precedence step of the server's co-agent resolution
chain. When set, the server uses exactly that co-agent and FAILS with a clear reason if
it can't (no silent fallback). Omit to let server metadata drive the choice: the target
agent's DefaultCoAgentID, then the type-level AIAgentCoAgent default row, then the global Realtime Co-Agent.
OptionalconfigOverridesJson: string | nullOptional JSON payload of SESSION CONFIG overrides (e.g.
{"realtime":{"modelPreference":"<modelId>"}}), forwarded verbatim on the mint
mutation. The server enforces the Realtime: Advanced Session Controls
authorization on any overrides — hosts only populate this from authorization-gated
pickers, and never synthesize overrides beyond what the user explicitly chose.
Omit/null for the server's defaults (today's behavior).
OptionalrecordingConsent: boolean | nullOptional EXPLICIT "record this call" consent for THIS session. When
true, the browser records a mic + agent-audio mix and uploads it as 5 s Opus shards during the call. When
omitted/null, the per-user persisted preference (mj.realtimeVoice.recordingConsent.v1
via UserInfoEngine) is read as the default. false never records.
OptionalmediaCollectionId: string | nullOptional per-session media-kit override (MJ: Collections.ID). When set,
the server-side Media channel resolves THIS collection as the agent's media kit for the session,
taking precedence over the agent's DefaultMediaCollectionID. The server UUID-validates it
(malformed ⇒ ignored, the agent default applies). Omit/null to use the agent default kit.
OptionalapplicationId: string | nullOptional application the session runs in (sources the app config cascade,
including Application.AgentSettings.Realtime.Channels).
OptionalappContext: AppContextSnapshot | nullOptional live app-context snapshot injected into the companion prompt at mint.
Optionaloptions: RealtimeSessionStartOptions | nullOptional per-start extras — chiefly the channels the HOST brings (RealtimeSessionStartOptions.HostChannels), the way a connect-only embed with no registry gets channels at all, and the fork the session starts on (RealtimeSessionStartOptions.ConversationBranchID; omitted is Main, where its transcript is written).
Run a session the HOST has already minted itself — the second half of
StartRealtimeSession, without the StartRealtimeClientSession mutation.
For hosts that must mint through their own server surface because they attach per-session context the stock mutation cannot carry (e.g. an interview persona baked into the companion prompt). They call their own mutation, shape the reply into a StartRealtimeClientSessionResult, and hand it here: driver resolution, the ephemeral-token connect, tool/transcript relays, recording, connection state and teardown are all identical to the all-in-one path — there is exactly one implementation of the run half.
NOTE: the interactive-channel plugins are NOT started on this path. Their tool sets must be declared to the model AT MINT, which happened on the host's side — so a host that wants channels owns that half too.
The minted session — the same ten fields the StartRealtimeClientSession
mutation returns. EphemeralToken and Provider are what actually open the call; a relay
session opens with its Transport and RelayUrl instead, so a host must pass those on.
Optionaloptions: RealtimeSessionRunOptionsHost-side inputs the result cannot carry; see RealtimeSessionRunOptions. Every field defaults exactly as its StartRealtimeSession counterpart does.
Asks the user for a screen, window or browser tab (or one panel of the page) and shows it to the agent. Call it from the user's click: the browser's picker needs one. Resolves with the share's state; a failure (no session, a model that takes no video, a closed picker, a host that cannot share) is a state with a message. The share also stops when the user ends it from the browser's own bar.
Optionaloptions: DisplayCaptureOptionsWhat the picker offers first, or the panel to share and its name (PanelLabel), which the share's
state carries and the agent sees as "
Stops the user's camera. Safe to call when it is off.
Stops the user's screen share. Safe to call when nothing is shared.
Moves the user's camera to another device while it is open (being checked or on): one of the camera's Devices on
Captures$. The stream stays the same. Resolves with the camera's state; nothing changes while the camera is
off.
The camera to move to.
Moves the call's microphone to another device: one of the Devices on Microphone$. The host's controller
swaps the new track into the same stream and carries the mute over, and the driver and the call's recording move to
it, so the agent keeps hearing the user. When the new microphone cannot open, the controller goes back to the one in
use. While the call has no microphone open (it lost the one in use and the default, or the last one went away), the
pick opens one again: the driver and the recording move to it, and it keeps the call's mute. Resolves with the
microphone's state once the switch is over. Nothing changes outside a call or on a host without a controller.
The microphone to move to.
Mute / unmute the local microphone track. Returns the new muted state. While the call has no microphone track (a
switch is putting a new one in, or the call lost its microphone) there is nothing to mute: nothing changes, and it
returns the call's mute, which the next microphone keeps. Outside a call it returns false.
Removes one owner's tools (and its place in the collision order). No-op when it has none.
Removes the handler registered for toolNamePrefix (no-op when none is registered).
Push an updated app-context snapshot mid-session (the continuous-streaming half of client-context
delivery). The host (Explorer) calls this when the user navigates / the active surface's state or
capability manifest changes; the ClientContextChannel turns the delta into a SendContextNote.
No-op semantics when no session is live — the channel simply re-reads on next start.
The latest app-context snapshot (or null to clear).
Drives a client-direct real-time voice session: the browser mints an ephemeral token from the MJ server, then connects DIRECTLY to the realtime provider. Audio frames never transit the MJ server (low latency); only tool calls and final transcripts are relayed back to MJ over GraphQL.
This service is PROVIDER-AGNOSTIC policy/orchestration. All provider wire concerns (transport, event translation, the response state machine, narration-kind tagging, playback tracking) live in a BaseRealtimeClient driver resolved through the MJ ClassFactory by the server-reported
Providerkey (e.g.'openai'→OpenAIRealtimeClient). Future providers (Gemini Live, …) snap in by registering a new driver — this service does not change.The Realtime Co-Agent (server-side) fronts the conversation's current agent — the server bakes the companion instructions + tool set into
SessionConfigJson, which the client driver applies verbatim.Lifecycle: StartRealtimeSession → live duplex → EndRealtimeSession. A start is two halves — MINT (the
StartRealtimeClientSessionmutation) and RUN (everything above) — and a host that must mint through its own server surface enters at the second half via StartRealtimeSessionFromResult; there is one implementation of the run half either way.