AbstractThe plugin's Angular surface component type. The host only ever
sees the default (object) — the typed parameter exists so concrete plugins get a
fully typed BindSurface without casts.
The plugin's Angular surface component type. The host only ever
sees the default (object) — the typed parameter exists so concrete plugins get a
fully typed BindSurface without casts.
ProtectedContextThe host context, available from Initialize until Dispose.
null outside that window — guard with ?. in any code that can run early/late.
ReadonlyEvents$Everything this channel streams out, as typed events: state changes, discrete user actions, frames pushed, opened/completed. Hosts subscribe (the embeddable element re-emits these as DOM events); the model is told about a change through a coalesced note instead — see RecordChange. Completes at Dispose.
ReadonlyOutput$ProtectedPerceptionTuning for the perception coalescer (debounce window, max note size). Override to change it.
ProtectedSessionProtectedSessionMax time ResolveAgentSessionId waits for the session id to bind before giving up, and the poll interval it re-checks on. Protected so tests can shrink the wait; production keeps the defaults (the real mint race is sub-second, 8s is generous headroom).
ProtectedVisualThe shared frame bridge that carries frames to the model's inbound video track. Deliberately NOT named
videoBridge: channels that predate the pump (the Remote Browser, app channels) declare their own
private videoBridge, and a second private of the same name in the base class would stop them compiling.
StaticStrictWhether state snapshots are validated against the descriptor's noun schemas as notes are built —
a development aid that surfaces a channel whose state drifted from its own declaration. On by
default only when a Node-style environment reports a non-production NODE_ENV (tests, tooling);
a browser production build never pays for it. Hosts may set it explicitly.
The runtime capture this channel fronts ('camera' or 'screen'), or null. Such a channel is the capture's policy:
the runtime starts the capture only while the channel is in the session and the session's policy lets the agent see
pixels through it, and the user's "agent can see" choice for the channel decides whether the capture's frames reach
the model. Default null.
AbstractChannelThe channel definition name — MUST match the MJ: AI Agent Channels row's Name
(e.g. 'Whiteboard'). Used as the persistence key for SerializeState saves
and as the channel tab's stable key.
ProtectedCurrentThe id of the latest state change (0 before any).
How much of this channel the model may perceive without asking: the lowest of the channel's own ceiling
(GetDescriptor().MaxExposure), the server policy and the user's choice. A channel nobody has applied a
policy to simply has its ceiling, which is what every channel had before exposure policy existed.
'none': no state notes and no frames.'state': structured state notes; no frames.'pixels': state notes and frames.It governs what flows to the model UNPROMPTED (perception notes, the contents of opened / completed
notes, frames). It does not REDACT what a verb's result contains: a verb is all-or-nothing. A verb whose
result would show the model something above this level declares ReturnsChannelData on the verb, and
when this level is below it the verb is refused whole (exposure_restricted) and listed to the model as
unavailable; see RefuseVerbForExposure. A verb that does not declare it returns what the channel
says it returns, so a channel whose verbs reveal sensitive data must either declare it or lower its own
MaxExposure.
Why Exposure is below the channel's ceiling; empty when it is not.
The channel's stable primary-instance id. A single-instance channel has exactly one; a
RealtimeChannelDescriptor.MultiInstance channel owns further instance ids itself and
routes by the instanceId argument of ApplyVerb. Appears in every structured note
([channel:<Key>#<instance>] …) and every RealtimeChannelEvent.
Whether this channel shows the agent's video: it sinks outbound video (GetSunkTracks). The runtime marks it as used when the video arrives, and a host presents its surface as the agent (the call overlay puts it in the agent's place in the call rather than in the focus layout).
Where this channel's surface shows when a call starts, and where the user may move it: from the channel's registry
row (UIConfig.Placement and UIConfig.AllowedPlacements). A channel without a row, or whose row says nothing,
starts on its tab and can go anywhere.
Whether the user may share this channel's surface on its own: a host that offers it lists the surface, named by
TabTitle with TabIcon, under "This panel" in its Share menu while the surface is on screen, and the
share then shows only that surface. Default false, so a channel opts in with this getter. A surface that shows the
agent its own call (the camera, the screen share, the agent's video) never should.
OPTIONAL accent color for the channel's tab (a CSS color string, e.g. an hsl() /
token). When a plugin supplies one, the overlay paints the tab's dot + active underline
with it; when omitted (the default null), the overlay derives a stable, deterministic
color from the ChannelName so every channel still reads as a distinct, colored
surface. A channel only overrides this to enforce a specific brand accent.
Font Awesome icon class for the channel's tab (e.g. 'fa-solid fa-chalkboard'). Defaults to a generic puzzle-piece.
Label for the channel's tab on the overlay's surface panel (e.g. 'Whiteboard'). Defaults to ChannelName.
The shared name prefix of every tool this channel exposes (e.g. 'Whiteboard_').
The host registers ONE local-execution route per plugin: tool calls whose name starts
with this prefix go to ApplyAgentTool instead of the server relay.
Default '' — a v2 channel with NO native tools (it is reached only through the ContextTool
proxy). The host registers no route for an empty prefix: an empty prefix would match every
tool name and hijack the server relay.
Whether this channel wants to be offered the artifacts a delegated run just produced (the agent asked Skip or Sage to build something, and it came back as artifacts). A channel that hosts artifacts says yes for the ones it can show — and for an artifact it ALREADY shows, so a newer version replaces it in place.
Asked of every channel in the session, mounted or merely advertised. For an advertised channel it is asked before the channel has been initialized, so it must not depend on Context; the channel's resolved configuration is passed in for exactly that reason. Saying yes for an advertised channel mounts it, so only answer yes when you will do something with the artifacts.
Default: false — a channel that does not host artifacts is never bothered.
Executes ONE agent tool call locally (the ACTION direction) and returns the result
JSON string fed back to the model as the tool_response. Called for every tool whose
name starts with ToolNamePrefix. Must work both WITH a bound surface (apply +
UI garnish) and WITHOUT one (apply to the state engine directly — the surface may not
exist yet, e.g. it has not been shown). Should not throw: return a
{ success: false, error } payload so the model can narrate the failure (the host
additionally wraps anything thrown).
This is the LEGACY entry point. Channels written against the v2 contract override ApplyVerb instead and inherit this method: the default maps the tool name back to its verb, calls ApplyVerb as the agent, and serializes the structured result. Channels written against v1 override THIS and inherit ApplyVerb, which adapts it. A channel must override at least one of the two — overriding neither is reported as a failure result, never a recursion.
Applies exposure policy: the server's decision and the user's own choice. The runtime calls this when it mounts the channel and again whenever the user changes their choice.
Withdrawing exposure takes effect immediately: a pending perception note is dropped, a pending frame is dropped, and the channel's video source stops being forwarded. Restoring it re-baselines (the model's picture of the channel is stale, so the next note is a full snapshot) and sends a fresh frame. When the call is live the model is told about the change, with the reason, so it never assumes it can still see what it can't.
Policy — the server-decided exposure; User — the user's choice (omit for none);
Reasons — why the policy lowered it, from the server's limits.
Sets SurfacePlacement. The runtime calls it with the registry row's placement when it builds the channel, before the channel mounts, as it applies exposure policy with ApplyExposure.
Runs one verb of this channel — the contract-v2 entry point, reached through the ContextTool
proxy (and, via ApplyAgentTool, by native tool calls).
The runtime has ALREADY resolved the verb against the descriptor, enforced InvokableBy, and
validated args against the verb's parameter schema before calling this, so an override can
trust its inputs' shape. It must work with no surface bound, and should not throw — return a
failure result so the model can recover.
Default: adapts the legacy ApplyAgentTool (the verb's native tool name, args as JSON), so every v1 channel is reachable through the proxy with no change. A v2 channel overrides this.
The verb name (as declared in the descriptor).
The validated parameters.
Who is acting: the agent (a model call) or the user (their own interaction).
OptionalinstanceId: stringWhich instance of a multi-instance channel; undefined for the primary instance.
Called by the host right after it created the surface component (and BEFORE the
component's first change detection, so inputs set here are visible in its ngOnInit).
The plugin — which knows its own component type — sets inputs (state engine, agent
name, …) and subscribes outputs here, wiring perception/garnish flows back through
Context. May be called again with a NEW instance after an
UnbindSurface (the host recreated the surface).
The channel finished and is handing something back (a submitted form, a chosen option, a
finished game). Emits on Output$ and Events$ and tells the model with a
completed note. The output is validated against the descriptor's Output schema when
StrictStateChecks is on; a mismatch is logged, never swallowed into silence or thrown.
What the channel hands back.
The instance that completed, for a multi-instance channel (default: the primary instance).
ProtectedConfirmAn AGENT action succeeded and changed the channel's picture: push exactly one confirmation frame now and tell the model not to narrate its own change. Call only on SUCCESS — confirming a failed action would assert an edit landed that did not. A no-op until EnableVisualPerception ran.
Tears the plugin down at session end: release the surface binding, unsubscribe
state-engine subscriptions, then drop the context. Subclasses overriding this MUST
call super.Dispose(). Any final state save has already been flushed by the host
(the debounced RealtimeChannelContext.RequestSave pipeline) before disposal.
ProtectedEmitEmits a typed event on Events$ (NOT to the model — use RecordChange for state the model should perceive, or send an explicit note for a discrete event it must hear about).
The event name (declare it in the descriptor's Events).
The event payload.
OptionalchangeId: numberThe change id the event relates to, when it relates to one.
The instance the event is about, for a multi-instance channel (default: the primary instance).
ProtectedEnableOpts this channel into VISUAL perception: its surface can be shown to a video-capable model as frames. Call it from OnInitialize with the channel's frame source.
The channel then needs only to say when its picture changed — NotifyVisualChange after a
user edit, ConfirmVisualChange after a successful agent edit — and the base class does the
rest: change-driven (no idle heartbeat), paced to the negotiated cadence, a trailing settle so the
model ends on the final state of a burst, dedupe of identical frames, and a single confirmation
frame (with a "do not narrate" note) after an agent edit. Every frame is tagged with the change id
of the state it was captured at (frame_pushed events), so state and pixels can be cross-checked.
Also makes GetSourcedTracks report the inbound video track, which is what gets it negotiated. Idempotent: calling it again replaces the provider and options.
Renders the channel's current picture as a base64 JPEG (or null).
ProtectedEnsureReturns the shared frame bridge, creating it if needed and (only when the channel opted into the
legacy poller) starting its fixed-rate poll once the video track is up. null when no context is bound.
The descriptor verb a native tool call runs, or undefined when the tool is not one of the channel's verbs
(the runtime uses it to apply RefuseVerbForExposure on the native-tool route as well as the ContextTool one).
The native tool name the model called.
ProtectedFlushSends any pending coalesced perception note now (e.g. before a surface is torn down).
The channel's SELF-DESCRIPTION — what it is, what it holds, what it accepts, how it behaves in a session. This is the contract an agent reads to operate a channel it has never seen, and what the runtime reads to scope, advertise and validate. Pure data; called often, so keep it cheap.
Default: a descriptor SYNTHESIZED from the v1 members (tools → verbs, intro → instructions),
stamped with the legacy contract version — so a channel that implements nothing new is still
describable and works exactly as before. A v2 channel overrides this with an authored descriptor.
(Build verbs from GetToolDefinitions with BuildToolBackedVerbs when they are the
native tools, so a subclass that adds a tool does not leave the descriptor lying.)
The channel's FIRST-RUN INTRO content, or null when the channel offers no onboarding.
The overlay shows this once per channel per user — the first time the user opens this
channel's surface tab — and remembers "seen" via the user's settings (NOT localStorage),
so it never re-appears on later sessions or other devices.
Default: null (no intro). The base Voice/text channel has no plugin at all, so it never
shows an intro; an interactive channel with a surface worth explaining (whiteboard, remote
browser, …) overrides this to return its ChannelOnboardingDetails. A plugin that
doesn't override it simply shows nothing — onboarding is strictly opt-in.
Media tracks this client channel can SOURCE — samples flowing into the model.
Default [].
A snapshot of the channel's things (the descriptor's nouns), keyed by noun name. Perception notes are computed as deltas between successive snapshots, so this must be a pure read.
Default: the parsed SerializeState when that is a JSON object, else {} — which makes a
v1 channel's state of record its (single, opaque) snapshot with no code change.
Media tracks this client channel can SINK — samples flowing from the model OUT.
Default [].
The session requests them when it connects, so the driver establishes the ones the model supports. A channel that sinks outbound video shows the agent's video: it follows RealtimeChannelContext.AgentVideo$, and counts as used once the video arrives, so the host shows its surface.
The Angular component the overlay creates dynamically as this channel's tab pane, or null
for a server-only channel that renders no MJ surface (its surface, if any, lives on the
external platform — e.g. a bridge-contributed native whiteboard or meeting-controls channel).
When this returns null, the overlay renders NO tab for the channel and never calls
BindSurface/UnbindSurface — but the channel's tools (GetToolDefinitions /
ApplyAgentTool) and perception (RealtimeChannelContext.SendContextNote) still run.
A created surface instance is handed straight back via BindSurface; the host treats it as
opaque.
Default: null (server-only). A channel with a rendered surface overrides this to return its
component type.
The channel's CLIENT-EXECUTED tool declarations, aggregated by the session service
into the clientTools set declared to the realtime model at session mint. The server
only DECLARES these — execution stays in the browser via ApplyAgentTool.
Default [] — a v2 channel whose verbs are reached through the proxy only.
Whether this channel has a rendered MJ surface (GetSurfaceComponent returns non-null).
The overlay uses this to decide whether to register a surface tab; server-only channels are
false. Override only if surface availability must be decided WITHOUT constructing the type
(the default calls GetSurfaceComponent once).
Binds the host context and invokes the OnInitialize hook. Called exactly once per session, right after ClassFactory instantiation and before any tool call or surface bind.
ProtectedNotifyThe channel's picture changed because of a USER action (or a scene replacement): push a frame, respecting the cadence. Resolves when the decision has been carried out; never rejects. A no-op until EnableVisualPerception ran and the session has an inbound video track.
A delegated run produced artifacts this channel said it wants (AcceptsDelegationArtifacts). The channel is mounted and initialized by now. Never throw into the runtime: handle your own failures and report them (the runtime logs anything that escapes, but cannot recover the work).
Default: nothing.
ProtectedOnSubclass hook invoked from Initialize once Context is bound — wire
state-engine subscriptions (e.g. state change → Context.RequestSave(...)) here.
Default: no-op.
ProtectedOnSubclass hook for Open: apply the (already validated) seed inputs. Return a failure
result to refuse the open; return nothing for success. Default: nothing to do.
Subclass hook invoked once the realtime session is connected and live (the client driver
is created, connected, and media tracks negotiated), or when the channel is opened in a live
session. Channels that establish media bridges (e.g. video streaming) can start them here when
Context.Client is available.
Default: a channel with visual perception adds the inbound video track it sources when the call
has none (it was opened mid-call) and brings its frame bridge up; any other channel does nothing.
An override that keeps visual perception calls super.OnSessionStarted().
Called by the host when the bound surface moves: to the stage (it fills the call), to its tab, or out of sight because the user hid it. A channel can adapt its own chrome, such as a "Move to stage" button that has nothing to do once the surface is on the stage. Called right after BindSurface with the current placement, then on every move, and not after UnbindSurface. Default: no-op.
Called by the host when the bound surface comes into sight or goes out of it (the panel collapsed or hidden, another tab active, the call minimized). The surface stays bound and keeps its state; a channel can pause work nobody sees, such as a screenshot poll, and resume it when the surface is shown again. The host calls it right after BindSurface with the surface's current visibility, then on every change, and not after UnbindSurface. Default: no-op, so a surface keeps running out of sight.
Opens the channel — mounts it for use and seeds it with inputs. Called by the runtime when the
agent (or user) opens an on-demand channel, after the channel has been initialized.
Validates inputs against the descriptor's Inputs schema (a violation is a structured
'invalid_params' failure, never a half-opened channel), runs the OnOpen hook, then tells
the model the channel is open and what it currently holds (opened note with a state snapshot) —
and takes that snapshot as the baseline later changes are described against.
What the agent asked to seed the channel with.
ProtectedRecordRecords that the channel's state changed. Call this on EVERY mutation, whoever caused it.
It (1) assigns the change a monotonic id, (2) emits a state_changed event on Events$
immediately (observers see every change), and (3) — unless Perceive is false — schedules a
COALESCED structured note to the model: a burst of N changes inside the debounce window produces
ONE [channel:<Key>#<instance>] state_changed {"changeId":…,"changes":N,"delta":{…}} note carrying
what differs from what the model was last told (the first note is a full snapshot). Notes that
would be too large degrade to a list of changed paths.
Author — who made the change (carried on the event); Perceive — whether the
model should be told (default true; a channel that already feeds the model its own perception
— like the Whiteboard's scene deltas — passes false).
The change's id.
Whether the agent may be given this verb's result at the channel's current exposure. A verb that declares
ReturnsChannelData is refused whole when exposure is below that level, because its result would hand the agent
what the user or a zero-data-retention policy held back from it. Policy, not redaction: the result is never filtered.
Applies to the AGENT (the dispatcher and the native-tool route call it); a user acting through the surface is not restricted.
The verb the agent is calling.
The sentence to give the agent, or null when the call may proceed.
The focus pill's "exit" affordance, routed by the overlay to the channel that holds
focus. Implementations should leave focus mode through their OWN surface (so surface
toggles stay in sync), ultimately emitting Context.SetFocusMode(false). The overlay
defensively clears its layout flag as well, so a no-op default is safe.
ProtectedResolveResolves the live RealtimeChannelContext.AgentSessionID, briefly WAITING for it when it
isn't bound yet rather than giving up instantly. AgentSessionID is a live getter over the
session service's current id: it reads null in the window BEFORE the session mints (the
realtime model can fire a tool call the very first beat it connects, before mintSession
resolves) and again AFTER teardown. Server-backed tool paths (e.g. the Remote Browser channel's
browser_* tools) call this instead of reading Context?.AgentSessionID synchronously, so a tool
invoked a beat early WAITS for the session to come live — defense-in-depth against the
"session id missing" race — instead of returning a hard failure to the model.
Returns the id as soon as it's non-null (the common path resolves immediately, no delay), or
null if it's still unbound after SessionIdWaitTimeoutMs — or the channel was
Disposed in the meantime (Context goes null, so we stop waiting on a torn-down session).
ProtectedResolveMaps a verb to the native tool name it is reachable as (inverse of ResolveVerbForTool).
ProtectedResolveMaps a native tool name back to its verb: the descriptor's NativeToolName match first, then the
name minus ToolNamePrefix. Protected so a channel with an unusual naming scheme can
override it.
Restores a PRIOR session's saved channel state (the payload a previous session persisted via SerializeState / RealtimeChannelContext.RequestSave). Invoked by the session host AFTER Initialize and BEFORE any surface binding, when a prior session's saved state exists for this channel.
Returns true when the state was applied; false when the channel ignored it —
either because it keeps no persistent state (this default) or because the payload was
malformed/incompatible. Implementations MUST be tolerant: never throw on bad input,
just return false and start fresh.
Serializes the channel's current state of record (the payload persisted on the
session's channel row), or null when the channel keeps no persistent state.
Default: null.
Called by the host when the surface component is being destroyed (the channel left the session / the overlay was torn down). Drop the instance reference and unsubscribe any output subscriptions — after this, ApplyAgentTool runs in its no-surface mode. Default: no-op.
Checks the current GetState snapshot against the descriptor's noun schemas and returns every violation (empty when valid). Useful in a channel's own tests; the runtime also runs it on every perception note when StrictStateChecks is on.
Base class for CLIENT-SIDE interactive-channel plugins (per
plans/ai-agent-sessions.md→ "Interactive Channels" / "Pluggable Channel Interfaces").An interactive channel is a bidirectional surface the session's single realtime agent both PERCEIVES and ACTS UPON (whiteboard, shared doc, map, …). A concrete plugin contributes everything the channel needs, so the session service / call overlay carry ZERO channel-specific wiring:
{@link ToolNamePrefix}*calls to — the ACTION direction;nullfrom GetSurfaceComponent (HasSurface isfalse) and the overlay simply skips its tab while still wiring its tools + perception;Registration & resolution (mirrors the realtime model drivers)
Concrete plugins are
@RegisterClass(BaseRealtimeChannelClient, '<ClientPluginClass>')and are resolved at session start from theMJ: AI Agent Channelsregistry: each ACTIVE row'sClientPluginClassis the ClassFactory key (exactly howBaseRealtimeClientdrivers resolve by provider key). Ship aLoad<YourChannel>()no-op alongside the class and call it from a static code path to defeat tree-shaking.Lifecycle — ONE INSTANCE PER SESSION (not a singleton)
ClassFactory.CreateInstance→ Initialize(ctx) → zero or more BindSurface/UnbindSurface cycles (the Angular overlay creates the surface the first time it is shown and keeps it until the channel leaves the session; another host may recreate it), each telling the plugin when its surface goes in and out of sight (OnSurfaceVisibilityChange) and where it is placed (OnSurfacePlacementChange) → Dispose at teardown. ApplyAgentTool MUST work with NO surface bound (apply to the state engine directly; skip the UI garnish) — tool calls can arrive before the surface is first shown.