ProtectedContextThe host context, available from Initialize until Dispose.
null outside that window — guard with ?. in any code that can run early/late.
ProtectedGoalPoll cadence + deadline for pollGoalResult (the async browser_AchieveGoal result poll).
Protected so tests can shrink them; production polls every 2.5s for up to 5 minutes (a goal loop
past 5 min keeps running server-side, the model just gets a "still running" note).
ProtectedGoalProtectedSessionProtectedSessionMax time ResolveAgentSessionId waits for the session id to bind before giving up, and the poll interval it re-checks on. Protected so tests can shrink the wait; production keeps the defaults (the real mint race is sub-second, 8s is generous headroom).
The channel definition name — MUST match the MJ: AI Agent Channels row's Name
(e.g. 'Whiteboard'). Used as the persistence key for SerializeState saves
and as the channel tab's stable key.
OPTIONAL accent color for the channel's tab (a CSS color string, e.g. an hsl() /
token). When a plugin supplies one, the overlay paints the tab's dot + active underline
with it; when omitted (the default null), the overlay derives a stable, deterministic
color from the ChannelName so every channel still reads as a distinct, colored
surface. A channel only overrides this to enforce a specific brand accent.
Font Awesome icon class for the channel's tab (e.g. 'fa-solid fa-chalkboard').
Label for the channel's tab on the overlay's surface panel (e.g. 'Whiteboard').
The shared name prefix of every tool this channel exposes (e.g. 'Whiteboard_').
The host registers ONE local-execution route per plugin: tool calls whose name starts
with this prefix go to ApplyAgentTool instead of the server relay.
Executes ONE browser_* tool call: maps it to a server action and awaits the
ExecuteRemoteBrowserAction mutation, then returns a concise JSON result the model reads.
On success it also feeds the new page URL to the agent as a [browser] context note. Never
throws — bad arguments, no live session, and server failures all become a failed-result
payload so the model can narrate the problem.
Wires the dynamically-created surface: hands it a snapshot fetcher closing over the
channel context (session id + provider live there), so the surface stays transport-
agnostic. Set BEFORE the surface's first change detection (the pane binds synchronously),
so its ngOnInit poll has the fetcher.
Also asks the server to start a live screencast. When the backend supports ScreenStreaming the
server starts PUSHING frames and reports Streaming: true; we flip the surface to canvas mode (its
poll is then skipped) and OnScreencastFrame paints each pushed frame. When the backend lacks
the capability (Streaming: false) the surface keeps the snapshot poll already running — graceful
fallback, no further action.
HUMAN TAKEOVER is enabled BY DEFAULT (Collaborative): the surface is flipped Interactive = true and
its HumanInput output subscribed, forwarding each pointer/keyboard event to the server via
RELAY_HUMAN_INPUT_MUTATION. Takeover only takes effect on the canvas (screencast) path; the
<img> poll fallback stays view-only. The server gates the actual routing on the backend's
HumanTakeover capability, so this is safe to enable unconditionally.
Tears the plugin down at session end: release the surface binding, unsubscribe
state-engine subscriptions, then drop the context. Subclasses overriding this MUST
call super.Dispose(). Any final state save has already been flushed by the host
(the debounced RealtimeChannelContext.RequestSave pipeline) before disposal.
First-run intro shown the first time the user opens the Browser tab (once per user).
The Angular component the overlay creates dynamically as this channel's tab pane, or null
for a server-only channel that renders no MJ surface (its surface, if any, lives on the
external platform — e.g. a bridge-contributed native whiteboard or meeting-controls channel).
When this returns null, the overlay renders NO tab for the channel and never calls
BindSurface/UnbindSurface — but the channel's tools (GetToolDefinitions /
ApplyAgentTool) and perception (RealtimeChannelContext.SendContextNote) still run.
A created surface instance is handed straight back via BindSurface; the host treats it as
opaque.
Default: null (server-only). A channel with a rendered surface overrides this to return its
component type.
The channel's CLIENT-EXECUTED tool declarations, aggregated by the session service
into the clientTools set declared to the realtime model at session mint. The server
only DECLARES these — execution stays in the browser via ApplyAgentTool.
Whether this channel has a rendered MJ surface (GetSurfaceComponent returns non-null).
The overlay uses this to decide whether to register a surface tab; server-only channels are
false. Override only if surface availability must be decided WITHOUT constructing the type
(the default calls GetSurfaceComponent once).
Binds the host context and invokes the OnInitialize hook. Called exactly once per session, right after ClassFactory instantiation and before any tool call or surface bind.
Feeds one PUSHED tab-audio chunk to the client-side audio player. Called by the session service when a
RemoteBrowserAudioChunk arrives on the push-status stream for THIS session. No-op when the channel
isn't audio-streaming or the player has been torn down.
The pushed audio chunk (base64 webm-opus + codec/seq metadata).
ProtectedOnSubclass hook invoked from Initialize once Context is bound — wire
state-engine subscriptions (e.g. state change → Context.RequestSave(...)) here.
Default: no-op.
Forwards one PUSHED screencast frame to the bound surface's canvas. Called by the session service
when a RemoteBrowserScreencastFrame arrives on the push-status stream for THIS session. No-op when
the channel isn't streaming or has no bound surface (e.g. the tab pane is collapsed).
Under streaming the surface's snapshot poll is stopped, so this is the ONLY thing that sees the page while frames are being pushed — which is why the frame now carries the URL (#3496). Without it, a user navigating during a screencast changed the picture and nothing else: the agent kept describing the page it last opened, and the pixels proving otherwise were right there on screen.
The frame image as raw base64 JPEG (no data: prefix).
OptionalcurrentUrl: string | nullThe browser's URL when the frame was captured; absent from older servers.
The focus pill's "exit" affordance, routed by the overlay to the channel that holds
focus. Implementations should leave focus mode through their OWN surface (so surface
toggles stay in sync), ultimately emitting Context.SetFocusMode(false). The overlay
defensively clears its layout flag as well, so a no-op default is safe.
ProtectedResolveResolves the live RealtimeChannelContext.AgentSessionID, briefly WAITING for it when it
isn't bound yet rather than giving up instantly. AgentSessionID is a live getter over the
session service's current id: it reads null in the window BEFORE the session mints (the
realtime model can fire a tool call the very first beat it connects, before mintSession
resolves) and again AFTER teardown. Server-backed tool paths (e.g. the Remote Browser channel's
browser_* tools) call this instead of reading Context?.AgentSessionID synchronously, so a tool
invoked a beat early WAITS for the session to come live — defense-in-depth against the
"session id missing" race — instead of returning a hard failure to the model.
Returns the id as soon as it's non-null (the common path resolves immediately, no delay), or
null if it's still unbound after SessionIdWaitTimeoutMs — or the channel was
Disposed in the meantime (Context goes null, so we stop waiting on a torn-down session).
Restores a PRIOR session's saved channel state (the payload a previous session persisted via SerializeState / RealtimeChannelContext.RequestSave). Invoked by the session host AFTER Initialize and BEFORE any surface binding, when a prior session's saved state exists for this channel.
Returns true when the state was applied; false when the channel ignored it —
either because it keeps no persistent state (this default) or because the payload was
malformed/incompatible. Implementations MUST be tolerant: never throw on bad input,
just return false and start fresh.
Serializes the channel's current state of record (the payload persisted on the
session's channel row), or null when the channel keeps no persistent state.
Default: null.
Called by the host when the surface component is being destroyed (tab panel collapsed / overlay torn down). Drop the instance reference and unsubscribe any output subscriptions — after this, ApplyAgentTool runs in its no-surface mode. Default: no-op.
The REMOTE BROWSER as a pluggable interactive channel — a BaseRealtimeChannelClient resolved from the
MJ: AI Agent Channelsregistry row whoseClientPluginClassis'RealtimeRemoteBrowserChannel'. One instance per session (ClassFactory-created at session start).Topology (CLIENT-DIRECT, like the whiteboard — the browser just happens to live on the server):
browser_*client-executed tool set (REMOTE_BROWSER_TOOL_DEFINITIONS). ApplyAgentTool maps a tool call → a normalized RemoteBrowserAction (MapToolToAction) and awaits theExecuteRemoteBrowserActionGraphQL mutation (via RealtimeChannelContext.ExecuteServerAction) to drive the server browser, returning a concise JSON result for the model. Never throws — argument and transport failures map to a{ success: false, error }payload.[browser]context note carries the new page URL so the agent perceives where the page went; the surface independently polls a live screenshot.RemoteBrowserSnapshot. HUMAN TAKEOVER is enabled by default (Collaborative): the user can click/type into the live canvas and those events relay to the server browser via RELAY_HUMAN_INPUT_MUTATION (server-gated on the backend'sHumanTakeovercapability).The channel keeps NO client-side state of record — the browser's state lives entirely on the server — so SerializeState / RestoreState use the base no-op behavior.