ProtectedContextThe host context, available from Initialize until Dispose.
null outside that window — guard with ?. in any code that can run early/late.
ReadonlyEvents$Everything this channel streams out, as typed events: state changes, discrete user actions, frames pushed, opened/completed. Hosts subscribe (the embeddable element re-emits these as DOM events); the model is told about a change through a coalesced note instead — see RecordChange. Completes at Dispose.
ProtectedGoalPoll cadence + deadline for pollGoalResult (the async browser_AchieveGoal result poll).
Protected so tests can shrink them; production polls every 2.5s for up to 5 minutes (a goal loop
past 5 min keeps running server-side, the model just gets a "still running" note).
ProtectedGoalReadonlyOutput$ProtectedPerceptionTuning for the perception coalescer (debounce window, max note size). Override to change it.
ProtectedSessionProtectedSessionMax time ResolveAgentSessionId waits for the session id to bind before giving up, and the poll interval it re-checks on. Protected so tests can shrink the wait; production keeps the defaults (the real mint race is sub-second, 8s is generous headroom).
ProtectedVisualThe shared frame bridge that carries frames to the model's inbound video track. Deliberately NOT named
videoBridge: channels that predate the pump (the Remote Browser, app channels) declare their own
private videoBridge, and a second private of the same name in the base class would stop them compiling.
StaticStrictWhether state snapshots are validated against the descriptor's noun schemas as notes are built —
a development aid that surfaces a channel whose state drifted from its own declaration. On by
default only when a Node-style environment reports a non-production NODE_ENV (tests, tooling);
a browser production build never pays for it. Hosts may set it explicitly.
The runtime capture this channel fronts ('camera' or 'screen'), or null. Such a channel is the capture's policy:
the runtime starts the capture only while the channel is in the session and the session's policy lets the agent see
pixels through it, and the user's "agent can see" choice for the channel decides whether the capture's frames reach
the model. Default null.
The channel definition name — MUST match the MJ: AI Agent Channels row's Name
(e.g. 'Whiteboard'). Used as the persistence key for SerializeState saves
and as the channel tab's stable key.
ProtectedCurrentThe id of the latest state change (0 before any).
How much of this channel the model may perceive without asking: the lowest of the channel's own ceiling
(GetDescriptor().MaxExposure), the server policy and the user's choice. A channel nobody has applied a
policy to simply has its ceiling, which is what every channel had before exposure policy existed.
'none': no state notes and no frames.'state': structured state notes; no frames.'pixels': state notes and frames.It governs what flows to the model UNPROMPTED (perception notes, the contents of opened / completed
notes, frames). It does not REDACT what a verb's result contains: a verb is all-or-nothing. A verb whose
result would show the model something above this level declares ReturnsChannelData on the verb, and
when this level is below it the verb is refused whole (exposure_restricted) and listed to the model as
unavailable; see RefuseVerbForExposure. A verb that does not declare it returns what the channel
says it returns, so a channel whose verbs reveal sensitive data must either declare it or lower its own
MaxExposure.
Why Exposure is below the channel's ceiling; empty when it is not.
The channel's stable primary-instance id. A single-instance channel has exactly one; a
RealtimeChannelDescriptor.MultiInstance channel owns further instance ids itself and
routes by the instanceId argument of ApplyVerb. Appears in every structured note
([channel:<Key>#<instance>] …) and every RealtimeChannelEvent.
Whether this channel shows the agent's video: it sinks outbound video (GetSunkTracks). The runtime marks it as used when the video arrives, and a host presents its surface as the agent (the call overlay puts it in the agent's place in the call rather than in the focus layout).
Where this channel's surface shows when a call starts, and where the user may move it: from the channel's registry
row (UIConfig.Placement and UIConfig.AllowedPlacements). A channel without a row, or whose row says nothing,
starts on its tab and can go anywhere.
Whether the user may share this channel's surface on its own: a host that offers it lists the surface, named by
TabTitle with TabIcon, under "This panel" in its Share menu while the surface is on screen, and the
share then shows only that surface. Default false, so a channel opts in with this getter. A surface that shows the
agent its own call (the camera, the screen share, the agent's video) never should.
OPTIONAL accent color for the channel's tab (a CSS color string, e.g. an hsl() /
token). When a plugin supplies one, the overlay paints the tab's dot + active underline
with it; when omitted (the default null), the overlay derives a stable, deterministic
color from the ChannelName so every channel still reads as a distinct, colored
surface. A channel only overrides this to enforce a specific brand accent.
Font Awesome icon class for the channel's tab (e.g. 'fa-solid fa-chalkboard'). Defaults to a generic puzzle-piece.
Label for the channel's tab on the overlay's surface panel (e.g. 'Whiteboard'). Defaults to ChannelName.
The shared name prefix of every tool this channel exposes (e.g. 'Whiteboard_').
The host registers ONE local-execution route per plugin: tool calls whose name starts
with this prefix go to ApplyAgentTool instead of the server relay.
Default '' — a v2 channel with NO native tools (it is reached only through the ContextTool
proxy). The host registers no route for an empty prefix: an empty prefix would match every
tool name and hijack the server relay.
Whether this channel wants to be offered the artifacts a delegated run just produced (the agent asked Skip or Sage to build something, and it came back as artifacts). A channel that hosts artifacts says yes for the ones it can show — and for an artifact it ALREADY shows, so a newer version replaces it in place.
Asked of every channel in the session, mounted or merely advertised. For an advertised channel it is asked before the channel has been initialized, so it must not depend on Context; the channel's resolved configuration is passed in for exactly that reason. Saying yes for an advertised channel mounts it, so only answer yes when you will do something with the artifacts.
Default: false — a channel that does not host artifacts is never bothered.
Executes ONE browser_* tool call: maps it to a server action and awaits the
ExecuteRemoteBrowserAction mutation, then returns a concise JSON result the model reads.
On success it also feeds the new page URL to the agent as a [browser] context note. Never
throws — bad arguments, no live session, and server failures all become a failed-result
payload so the model can narrate the problem.
Applies exposure policy: the server's decision and the user's own choice. The runtime calls this when it mounts the channel and again whenever the user changes their choice.
Withdrawing exposure takes effect immediately: a pending perception note is dropped, a pending frame is dropped, and the channel's video source stops being forwarded. Restoring it re-baselines (the model's picture of the channel is stale, so the next note is a full snapshot) and sends a fresh frame. When the call is live the model is told about the change, with the reason, so it never assumes it can still see what it can't.
Policy — the server-decided exposure; User — the user's choice (omit for none);
Reasons — why the policy lowered it, from the server's limits.
Sets SurfacePlacement. The runtime calls it with the registry row's placement when it builds the channel, before the channel mounts, as it applies exposure policy with ApplyExposure.
Runs one verb of this channel — the contract-v2 entry point, reached through the ContextTool
proxy (and, via ApplyAgentTool, by native tool calls).
The runtime has ALREADY resolved the verb against the descriptor, enforced InvokableBy, and
validated args against the verb's parameter schema before calling this, so an override can
trust its inputs' shape. It must work with no surface bound, and should not throw — return a
failure result so the model can recover.
Default: adapts the legacy ApplyAgentTool (the verb's native tool name, args as JSON), so every v1 channel is reachable through the proxy with no change. A v2 channel overrides this.
The verb name (as declared in the descriptor).
The validated parameters.
Who is acting: the agent (a model call) or the user (their own interaction).
OptionalinstanceId: stringWhich instance of a multi-instance channel; undefined for the primary instance.
Wires the dynamically-created surface: hands it a snapshot fetcher closing over the
channel context (session id + provider live there), so the surface stays transport-
agnostic. Set BEFORE the surface's first change detection (the pane binds synchronously),
so its ngOnInit poll has the fetcher.
Also asks the server to start a live screencast. When the backend supports ScreenStreaming the
server starts PUSHING frames and reports Streaming: true; we flip the surface to canvas mode (its
poll is then skipped) and OnScreencastFrame paints each pushed frame. When the backend lacks
the capability (Streaming: false) the surface keeps the snapshot poll already running — graceful
fallback, no further action.
HUMAN TAKEOVER is enabled BY DEFAULT (Collaborative): the surface is flipped Interactive = true and
its HumanInput output subscribed, forwarding each pointer/keyboard event to the server via
RELAY_HUMAN_INPUT_MUTATION. Takeover only takes effect on the canvas (screencast) path; the
<img> poll fallback stays view-only. The server gates the actual routing on the backend's
HumanTakeover capability, so this is safe to enable unconditionally.
The channel finished and is handing something back (a submitted form, a chosen option, a
finished game). Emits on Output$ and Events$ and tells the model with a
completed note. The output is validated against the descriptor's Output schema when
StrictStateChecks is on; a mismatch is logged, never swallowed into silence or thrown.
What the channel hands back.
OptionalinstanceId: stringThe instance that completed, for a multi-instance channel (default: the primary instance).
ProtectedConfirmAn AGENT action succeeded and changed the channel's picture: push exactly one confirmation frame now and tell the model not to narrate its own change. Call only on SUCCESS — confirming a failed action would assert an edit landed that did not. A no-op until EnableVisualPerception ran.
Tears the plugin down at session end: release the surface binding, unsubscribe
state-engine subscriptions, then drop the context. Subclasses overriding this MUST
call super.Dispose(). Any final state save has already been flushed by the host
(the debounced RealtimeChannelContext.RequestSave pipeline) before disposal.
ProtectedEmitEmits a typed event on Events$ (NOT to the model — use RecordChange for state the model should perceive, or send an explicit note for a discrete event it must hear about).
The event name (declare it in the descriptor's Events).
Optionalpayload: JSONObjectThe event payload.
OptionalchangeId: numberThe change id the event relates to, when it relates to one.
OptionalinstanceId: stringThe instance the event is about, for a multi-instance channel (default: the primary instance).
ProtectedEnableOpts this channel into VISUAL perception: its surface can be shown to a video-capable model as frames. Call it from OnInitialize with the channel's frame source.
The channel then needs only to say when its picture changed — NotifyVisualChange after a
user edit, ConfirmVisualChange after a successful agent edit — and the base class does the
rest: change-driven (no idle heartbeat), paced to the negotiated cadence, a trailing settle so the
model ends on the final state of a burst, dedupe of identical frames, and a single confirmation
frame (with a "do not narrate" note) after an agent edit. Every frame is tagged with the change id
of the state it was captured at (frame_pushed events), so state and pixels can be cross-checked.
Also makes GetSourcedTracks report the inbound video track, which is what gets it negotiated. Idempotent: calling it again replaces the provider and options.
Renders the channel's current picture as a base64 JPEG (or null).
Optionaloptions: VisualPerceptionOptionsProtectedEnsureReturns the shared frame bridge, creating it if needed and (only when the channel opted into the
legacy poller) starting its fixed-rate poll once the video track is up. null when no context is bound.
The descriptor verb a native tool call runs, or undefined when the tool is not one of the channel's verbs
(the runtime uses it to apply RefuseVerbForExposure on the native-tool route as well as the ContextTool one).
The native tool name the model called.
ProtectedFlushSends any pending coalesced perception note now (e.g. before a surface is torn down).
The remote browser's self-description: a real, server-hosted web browser the user can watch, with
every browser_* tool as a verb (so a tool a subclass adds shows up here too). The model is shown
its pixels (page frames), so the exposure ceiling is pixels.
Produces the latest browser screenshot as a base64-encoded frame for the video bridge. Under active screencast streaming, returns null to avoid redundant server snapshot round-trips and duplicate frame sourcing into the video bridge.
First-run intro shown the first time the user opens the Browser tab (once per user).
Sourced tracks: Remote Browser can source inbound video to the model when the model supports it.
A snapshot of the channel's things (the descriptor's nouns), keyed by noun name. Perception notes are computed as deltas between successive snapshots, so this must be a pure read.
Default: the parsed SerializeState when that is a JSON object, else {} — which makes a
v1 channel's state of record its (single, opaque) snapshot with no code change.
Media tracks this client channel can SINK — samples flowing from the model OUT.
Default [].
The session requests them when it connects, so the driver establishes the ones the model supports. A channel that sinks outbound video shows the agent's video: it follows RealtimeChannelContext.AgentVideo$, and counts as used once the video arrives, so the host shows its surface.
The Angular component the overlay creates dynamically as this channel's tab pane, or null
for a server-only channel that renders no MJ surface (its surface, if any, lives on the
external platform — e.g. a bridge-contributed native whiteboard or meeting-controls channel).
When this returns null, the overlay renders NO tab for the channel and never calls
BindSurface/UnbindSurface — but the channel's tools (GetToolDefinitions /
ApplyAgentTool) and perception (RealtimeChannelContext.SendContextNote) still run.
A created surface instance is handed straight back via BindSurface; the host treats it as
opaque.
Default: null (server-only). A channel with a rendered surface overrides this to return its
component type.
The channel's CLIENT-EXECUTED tool declarations, aggregated by the session service
into the clientTools set declared to the realtime model at session mint. The server
only DECLARES these — execution stays in the browser via ApplyAgentTool.
Default [] — a v2 channel whose verbs are reached through the proxy only.
Whether this channel has a rendered MJ surface (GetSurfaceComponent returns non-null).
The overlay uses this to decide whether to register a surface tab; server-only channels are
false. Override only if surface availability must be decided WITHOUT constructing the type
(the default calls GetSurfaceComponent once).
Binds the host context and invokes the OnInitialize hook. Called exactly once per session, right after ClassFactory instantiation and before any tool call or surface bind.
ProtectedNotifyThe channel's picture changed because of a USER action (or a scene replacement): push a frame, respecting the cadence. Resolves when the decision has been carried out; never rejects. A no-op until EnableVisualPerception ran and the session has an inbound video track.
Feeds one PUSHED tab-audio chunk to the client-side audio player. Called by the session service when a
RemoteBrowserAudioChunk arrives on the push-status stream for THIS session. No-op when the channel
isn't audio-streaming or the player has been torn down.
The pushed audio chunk (base64 webm-opus + codec/seq metadata).
A delegated run produced artifacts this channel said it wants (AcceptsDelegationArtifacts). The channel is mounted and initialized by now. Never throw into the runtime: handle your own failures and report them (the runtime logs anything that escapes, but cannot recover the work).
Default: nothing.
ProtectedOnStarts the video bridge streaming browser frames to the model when inbound video is supported.
ProtectedOnSubclass hook for Open: apply the (already validated) seed inputs. Return a failure
result to refuse the open; return nothing for success. Default: nothing to do.
Forwards one PUSHED screencast frame to the bound surface's canvas. Called by the session service
when a RemoteBrowserScreencastFrame arrives on the push-status stream for THIS session. No-op when
the channel isn't streaming or has no bound surface (e.g. its tab has not been shown yet).
Under streaming the surface's snapshot poll is stopped, so this is the ONLY thing that sees the page while frames are being pushed — which is why the frame now carries the URL (#3496). Without it, a user navigating during a screencast changed the picture and nothing else: the agent kept describing the page it last opened, and the pixels proving otherwise were right there on screen.
Pushes to the video bridge are CHANGE-DRIVEN with a TRAILING-EDGE SETTLE DEBOUNCE:
The frame image as raw base64 JPEG (no data: prefix).
OptionalcurrentUrl: string | nullThe browser's URL when the frame was captured; absent from older servers.
Subclass hook invoked once the realtime session is connected and live (the client driver
is created, connected, and media tracks negotiated), or when the channel is opened in a live
session. Channels that establish media bridges (e.g. video streaming) can start them here when
Context.Client is available.
Default: a channel with visual perception adds the inbound video track it sources when the call
has none (it was opened mid-call) and brings its frame bridge up; any other channel does nothing.
An override that keeps visual perception calls super.OnSessionStarted().
Called by the host when the bound surface moves: to the stage (it fills the call), to its tab, or out of sight because the user hid it. A channel can adapt its own chrome, such as a "Move to stage" button that has nothing to do once the surface is on the stage. Called right after BindSurface with the current placement, then on every move, and not after UnbindSurface. Default: no-op.
Pauses the surface's snapshot poll while it is out of sight; the screencast and the audio keep running.
Opens the channel — mounts it for use and seeds it with inputs. Called by the runtime when the
agent (or user) opens an on-demand channel, after the channel has been initialized.
Validates inputs against the descriptor's Inputs schema (a violation is a structured
'invalid_params' failure, never a half-opened channel), runs the OnOpen hook, then tells
the model the channel is open and what it currently holds (opened note with a state snapshot) —
and takes that snapshot as the baseline later changes are described against.
Optionalinputs: JSONObjectWhat the agent asked to seed the channel with.
ProtectedRecordRecords that the channel's state changed. Call this on EVERY mutation, whoever caused it.
It (1) assigns the change a monotonic id, (2) emits a state_changed event on Events$
immediately (observers see every change), and (3) — unless Perceive is false — schedules a
COALESCED structured note to the model: a burst of N changes inside the debounce window produces
ONE [channel:<Key>#<instance>] state_changed {"changeId":…,"changes":N,"delta":{…}} note carrying
what differs from what the model was last told (the first note is a full snapshot). Notes that
would be too large degrade to a list of changed paths.
Optionaloptions: { Author?: RealtimeChannelActor | "system"; Perceive?: boolean }Author — who made the change (carried on the event); Perceive — whether the
model should be told (default true; a channel that already feeds the model its own perception
— like the Whiteboard's scene deltas — passes false).
The change's id.
Whether the agent may be given this verb's result at the channel's current exposure. A verb that declares
ReturnsChannelData is refused whole when exposure is below that level, because its result would hand the agent
what the user or a zero-data-retention policy held back from it. Policy, not redaction: the result is never filtered.
Applies to the AGENT (the dispatcher and the native-tool route call it); a user acting through the surface is not restricted.
The verb the agent is calling.
The sentence to give the agent, or null when the call may proceed.
The focus pill's "exit" affordance, routed by the overlay to the channel that holds
focus. Implementations should leave focus mode through their OWN surface (so surface
toggles stay in sync), ultimately emitting Context.SetFocusMode(false). The overlay
defensively clears its layout flag as well, so a no-op default is safe.
ProtectedResolveResolves the live RealtimeChannelContext.AgentSessionID, briefly WAITING for it when it
isn't bound yet rather than giving up instantly. AgentSessionID is a live getter over the
session service's current id: it reads null in the window BEFORE the session mints (the
realtime model can fire a tool call the very first beat it connects, before mintSession
resolves) and again AFTER teardown. Server-backed tool paths (e.g. the Remote Browser channel's
browser_* tools) call this instead of reading Context?.AgentSessionID synchronously, so a tool
invoked a beat early WAITS for the session to come live — defense-in-depth against the
"session id missing" race — instead of returning a hard failure to the model.
Returns the id as soon as it's non-null (the common path resolves immediately, no delay), or
null if it's still unbound after SessionIdWaitTimeoutMs — or the channel was
Disposed in the meantime (Context goes null, so we stop waiting on a torn-down session).
ProtectedResolveMaps a verb to the native tool name it is reachable as (inverse of ResolveVerbForTool).
ProtectedResolveMaps a native tool name back to its verb: the descriptor's NativeToolName match first, then the
name minus ToolNamePrefix. Protected so a channel with an unusual naming scheme can
override it.
Restores a PRIOR session's saved channel state (the payload a previous session persisted via SerializeState / RealtimeChannelContext.RequestSave). Invoked by the session host AFTER Initialize and BEFORE any surface binding, when a prior session's saved state exists for this channel.
Returns true when the state was applied; false when the channel ignored it —
either because it keeps no persistent state (this default) or because the payload was
malformed/incompatible. Implementations MUST be tolerant: never throw on bad input,
just return false and start fresh.
Serializes the channel's current state of record (the payload persisted on the
session's channel row), or null when the channel keeps no persistent state.
Default: null.
Called by the host when the surface component is being destroyed (the channel left the session / the overlay was torn down). Drop the instance reference and unsubscribe any output subscriptions — after this, ApplyAgentTool runs in its no-surface mode. Default: no-op.
Checks the current GetState snapshot against the descriptor's noun schemas and returns every violation (empty when valid). Useful in a channel's own tests; the runtime also runs it on every perception note when StrictStateChecks is on.
The REMOTE BROWSER as a pluggable interactive channel — a BaseRealtimeChannelClient resolved from the
MJ: AI Agent Channelsregistry row whoseClientPluginClassis'RealtimeRemoteBrowserChannel'. One instance per session (ClassFactory-created at session start).Topology (CLIENT-DIRECT, like the whiteboard — the browser just happens to live on the server):
browser_*client-executed tool set (REMOTE_BROWSER_TOOL_DEFINITIONS). ApplyAgentTool maps a tool call → a normalized RemoteBrowserAction (MapToolToAction) and awaits theExecuteRemoteBrowserActionGraphQL mutation (via RealtimeChannelContext.ExecuteServerAction) to drive the server browser, returning a concise JSON result for the model. Never throws — argument and transport failures map to a{ success: false, error }payload.[browser]context note carries the new page URL so the agent perceives where the page went; the surface independently polls a live screenshot.RemoteBrowserSnapshot. HUMAN TAKEOVER is enabled by default (Collaborative): the user can click/type into the live canvas and those events relay to the server browser via RELAY_HUMAN_INPUT_MUTATION (server-gated on the backend'sHumanTakeovercapability).The channel keeps NO client-side state of record — the browser's state lives entirely on the server — so SerializeState / RestoreState use the base no-op behavior.