Member Junction
    Preparing search index...

    Class RemoteBrowserActionResolver

    Resolver for the Remote Browser native realtime channel. A single RemoteBrowserEngine instance (the process-wide singleton) backs every request; ownership is enforced per call against the session's UserID.

    Hierarchy (View Summary)

    Index

    Constructors

    constructor

    Accessors

    MJCoreSchema

    Methods

    AfterCreate AfterDelete AfterUpdate applyRestoreContext ArrayFilterEncryptedFieldsForAPI ArrayMapFieldNamesToCodeNames BeforeCreate BeforeDelete BeforeUpdate CheckAPIKeyScopeAuthorization CheckUserReadPermissions createAuditLogRecord CreateRecord createRecordAccessAuditLogRecord DeleteRecord EmitCloudEvent ExecuteRemoteBrowserAction ExecuteRemoteBrowserGoal FilterEncryptedFieldsForAPI findBy GetRemoteBrowserGoalResult GetRemoteBrowserSelection getRowLevelSecurityWhereClause GetUserFromEmail GetUserFromPayload InterpretRemoteBrowserPage ListenForEntityMessages LoadExternalRecordByKey loadOwnedSession MapFieldNamesToCodeNames packageSPParam PublishCacheInvalidation RelayRemoteBrowserHumanInput RemoteBrowserSnapshot requireUserAndProvider resolveProviderName ReverseMapInputFieldNames RunDynamicViewGeneric RunViewByIDGeneric RunViewByNameGeneric RunViewGenericInternal RunViewsGeneric RunViewsGenericInternal safeFirstArrayElement StartRemoteBrowserAudioStream StartRemoteBrowserScreencast StopRemoteBrowserAudioStream StopRemoteBrowserScreencast TestAndSetClientOldValuesToDBValues UpdateRecord

    Constructors

    Accessors

    Methods

    • Applies an inbound RestoreContext___ blob to a server-side BaseEntity. Mirrors the OldValues___ pattern — the client-side BaseEntity's _restoreContext doesn't traverse the network, so the server must reconstruct it from the mutation input before calling Save().

      Returns true when context was applied; false when no context was on the input.

      Parameters

      • entityObject: BaseEntity
      • input: { RestoreContext___?: { Reason?: string; SourceChangeID?: string } }

      Returns boolean

    • Checks API key scope authorization. Only performs check if request was authenticated via API key (apiKeyHash present in userPayload). For OAuth/JWT auth, this is a no-op.

      Parameters

      • scopePath: string

        The scope path (e.g., 'entity:read', 'agent:execute')

      • resource: string

        The resource name (e.g., entity name, agent name)

      • userPayload: UserPayload

        The user payload from context

      Returns Promise<void>

      AuthorizationError if API key lacks required scope

    • Execute ONE browser action relayed from the client-direct realtime session, returning the outcome + resulting URL.

      Flow:

      1. Ownership gate — the session's UserID must equal the caller's (throws otherwise).
      2. Lazily start (or reuse) the session's browser via RemoteBrowserEngine.StartSessionForAgentSession, resolving the backend from the agent's TypeConfiguration ({ remoteBrowser: { provider } }), else the single Active provider.
      3. Build a strongly-typed RemoteBrowserAction from kind + the supplied fields.
      4. Execute it against the live session and return the result.

      Parameters

      • agentSessionID: string

        The AIAgentSession id the browser is bound to.

      • kind: string

        The action kind ('navigate' | 'click' | 'type' | 'key' | 'scroll' | 'back' | 'forward' | 'wait').

      • __namedParameters: AppContext
      • Optionalurl: string
      • Optionalselector: string
      • Optionalx: number
      • Optionaly: number
      • Optionaltext: string
      • Optionalkey: string
      • OptionaldeltaX: number
      • OptionaldeltaY: number
      • Optionalms: number

      Returns Promise<RemoteBrowserActionResult>

      The action result (success + resulting URL + detail).

    • Execute a high-level GOAL against the session's browser — the agent sets an intent ("log in and open the latest invoice") and the resolved control strategy (computer-use loop or backend native AI) plans

      • executes it autonomously, instead of relaying granular actions. Ownership-gated; lazily starts the browser. Returns the terminal outcome (this is a request/response mutation — for live progress narration in a server-bridged realtime session the broker calls RemoteBrowserEngine.AchieveGoal in-process with an OnProgress callback).

      Parameters

      • agentSessionID: string

        The AIAgentSession id the browser is bound to.

      • goal: string

        The natural-language goal.

      • __namedParameters: AppContext
      • OptionalstartUrl: string
      • OptionalmaxSteps: number
      • OptionalpreferredStrategy: string

      Returns Promise<RemoteBrowserGoalResultType>

      The goal outcome (success, strategy, status, step count, url, detail).

    • Filters encrypted field values before sending to the API client.

      For each encrypted field in the entity:

      • If AllowDecryptInAPI is true: value passes through unchanged (already decrypted by data provider)
      • If AllowDecryptInAPI is false and SendEncryptedValue is true: re-encrypt and send ciphertext
      • If AllowDecryptInAPI is false and SendEncryptedValue is false: replace with sentinel value

      Parameters

      • entityName: string

        Name of the entity

      • dataObject: Record<string, unknown>

        The data object containing field values

      • contextUser: UserInfo

        User context for encryption operations

      • Optionalprovider: IMetadataProvider

      Returns Promise<Record<string, unknown>>

      The filtered data object

    • Returns the remote page's CURRENT text selection — the copy-out half of human clipboard support. The viewer captures a local copy / Cmd+C, calls this to read what the human selected on the live page, and writes the result to the LOCAL clipboard (sidestepping the isolated remote clipboard, the mirror of the 'text' paste-in path). Ownership-gated.

      Gracefully best-effort, never throws past the ownership gate (mirrors RemoteBrowserSnapshot):

      • No live browser for the session → { Text: '' }.
      • Backend lacks HumanTakeover (RemoteBrowserCapabilityNotSupportedError) → { Text: '' }.
      • Nothing selected / any other read failure → { Text: '' } (logged).

      Parameters

      • agentSessionID: string

        The AIAgentSession id the browser is bound to.

      • __namedParameters: AppContext

      Returns Promise<RemoteBrowserSelection>

      The selection text, or { Text: '' } when none is readable.

    • VISION-QUERY path (separate from the navigate/click RemoteBrowserActionResolver.ExecuteRemoteBrowserAction path — this OBSERVES the page, it never drives it). Captures the session's current screenshot and runs a fast vision AI Prompt over it so a non-omnimodal voice agent (audio/text-only) can effectively SEE the page: a concise text description ("kinda see") and, when query names a target, the pixel centroid(s) the agent can then feed to browser_Click(x, y).

      Flow:

      1. Ownership gate — the session's UserID must equal the caller's (throws otherwise).
      2. Get the LIVE session and CaptureScreenshot(). No live session / no screenshot → { Description: null, Elements: [], Detail: 'no live browser' }.
      3. Run the Remote Browser Visual Interpreter prompt with the screenshot as an image_url content block plus an instruction derived from query (empty/"describe" → describe; a named target → locate it).
      4. Parse the strict-JSON result tolerantly and map it to RemoteBrowserInterpretation.

      Best-effort + tolerant by contract: any vision/prompt/parse failure is logged and returned as { Description: null, Elements: [], Detail: <message> }. This mutation NEVER throws past the ownership gate.

      Parameters

      • agentSessionID: string

        The AIAgentSession id the browser is bound to.

      • __namedParameters: AppContext
      • Optionalquery: string

        Optional request — empty/"describe" for a page description, else a visual target to localize.

      Returns Promise<RemoteBrowserInterpretation>

      The interpretation (description + localized elements + optional detail note).

    • Loads a single external-data-source-backed entity record by primary key and returns it in GraphQL field-name (CodeName) shape, or null if not found.

      External entities (Entity.ExternalDataSourceID set) have no MJ base view or sproc — their data is proxied live from a remote system — so the generated single-record resolver cannot run SELECT * FROM <baseView>. Instead it loads through a BaseEntity object, whose InnerLoad the data provider dispatches to the external read router's LoadExternalRecord (a composite-key aware, quoted, parameter-bound single-record lookup), applying the same RLS gate and field post-processing (decryption / datetime normalization) as the MJ-DB path. The caller is responsible for the CheckUserReadPermissions gate beforehand.

      Type Parameters

      • T

      Parameters

      Returns Promise<T>

    • Maps field names to their GraphQL-safe CodeNames and handles encryption for API responses.

      For encrypted fields coming from raw SQL queries (not entity objects):

      • AllowDecryptInAPI=true: Decrypt the value before sending to client
      • AllowDecryptInAPI=false + SendEncryptedValue=true: Keep encrypted ciphertext
      • AllowDecryptInAPI=false + SendEncryptedValue=false: Replace with sentinel

      Parameters

      • entityName: string

        The entity name

      • dataObject: any

        The data object with field values

      • OptionalcontextUser: UserInfo

        Optional user context for decryption (required for encrypted fields)

      • Optionalprovider: IMetadataProvider

      Returns Promise<any>

      The processed data object

    • Relays ONE human-takeover input (pointer move/click or key press) from the user watching the live screencast into the session's server-hosted browser — the "grab the wheel" path. Ownership-gated. The surface captures the event on the live-view canvas, maps display→viewport coordinates, and calls this; the resolver builds a strongly-typed RemoteBrowserHumanInput and routes it over CDP via IRemoteBrowserSession.RouteHumanInput.

      Gracefully best-effort, never throws past the ownership gate:

      • No live browser for the session → false.
      • Unknown/incomplete input (buildHumanInput returns null) → false.
      • Backend lacks HumanTakeover (RemoteBrowserCapabilityNotSupportedError) → false (the live view stays view-only for that backend).
      • Any other failure → logged + false.

      NOTE: this prototype does NOT reject by control mode here — the engine's floor arbiter governs the agent⇄human floor, so we just relay. Finer floor / AgentOnly gating is a follow-up.

      Parameters

      • agentSessionID: string

        The AIAgentSession id the browser is bound to.

      • kind: string

        The input kind ('pointer-move' | 'pointer-click' | 'pointer-down' | 'pointer-up' | 'key' | 'text' | 'scroll').

      • __namedParameters: AppContext
      • Optionalx: number
      • Optionaly: number
      • Optionalbutton: string
      • Optionalkey: string
      • Optionaltext: string

        The pasted text (the 'text' paste-in kind only) — inserted into the page's focused element.

      • OptionaldeltaX: number
      • OptionaldeltaY: number
      • Optionalmodifiers: string

        Optional comma-separated modifier keys held during the input ('Shift', 'Control', 'Alt', 'Meta') — carries Shift-click selection and Ctrl/Cmd+key chords faithfully.

      Returns Promise<boolean>

      true when the input was routed, else false.

    • Return the current viewport screenshot + URL for the session's live browser — the client's live view. Ownership-gated. When the session holds no live browser (never started, or already torn down), the result's fields are null rather than an error.

      Parameters

      • agentSessionID: string

        The AIAgentSession id.

      • __namedParameters: AppContext

      Returns Promise<RemoteBrowserSnapshot>

      The current screenshot + URL, or an empty snapshot when no live browser exists.

    • Reverse-maps GraphQL-safe field names back to entity CodeNames in a mutation input object. For example, _mj__integration_SyncStatus is mapped back to __mj_integration_SyncStatus. Also reverse-maps keys inside the OldValues___ array if present. This is the inverse of MapFieldNamesToCodeNames and must be called before passing GraphQL input to entity SetMany() or field lookups.

      Parameters

      • input: Record<string, unknown>

      Returns Record<string, unknown>

    • Optimized RunViewGenericInternal implementation with:

      • Field filtering at source (Fix #7)
      • Improved error handling (Fix #9)

      Parameters

      • provider: DatabaseProviderBase
      • viewInfo: MJUserViewEntityExtended
      • extraFilter: string
      • orderBy: string
      • userSearchString: string
      • excludeUserViewRunID: string
      • overrideExcludeFilter: string
      • saveViewResults: boolean
      • fields: string[]
      • ignoreMaxRows: boolean
      • excludeDataFromAllPriorViewRuns: boolean
      • forceAuditLog: boolean
      • auditLogDescription: string
      • resultType: string
      • userPayload: UserPayload
      • maxRows: number
      • startRow: number
      • Optionalaggregates: AggregateExpression[]
      • OptionalafterKey: CompositeKey
      • OptionalbypassCache: boolean

      Returns Promise<RunViewResult<any>>

    • Starts streaming the session browser's TAB AUDIO and PUSHES each encoded chunk to the calling user's push-status topic — so a co-agent demoing a video/audio site is HEARD, not just seen. Ownership-gated. Idempotent: a re-call for an already-streaming session is a no-op that reports Streaming: true.

      Capability gating (v1 = by backend implementation): IRemoteBrowserSession.StartAudioStream throws RemoteBrowserCapabilityNotSupportedError on a backend with no audio-capture mechanism — caught here and reported as Streaming: false (the client simply plays no audio). Any other failure is logged and likewise reported as Streaming: false.

      Parameters

      • agentSessionID: string

        The AIAgentSession id the browser is bound to.

      • __namedParameters: AppContext
      • pubSub: PubSubEngine

      Returns Promise<RemoteBrowserAudioStreamResult>

      { Streaming: true } when audio chunks are now being pushed, else { Streaming: false }.

    • Starts a live CDP screencast on the session's browser and PUSHES each encoded frame to the calling user's push-status topic — replacing the client's 700ms snapshot poll with low-latency pushed frames. Ownership-gated. Idempotent: a re-call for an already-streaming session is a no-op that reports Streaming: true.

      Capability gating: IRemoteBrowserSession.StartScreencast throws RemoteBrowserCapabilityNotSupportedError on a backend without ScreenStreaming — caught here and reported as Streaming: false, leaving the client on its polling fallback. Any other failure is logged and likewise reported as Streaming: false (the poll keeps the view alive).

      Parameters

      • agentSessionID: string

        The AIAgentSession id the browser is bound to.

      • __namedParameters: AppContext
      • pubSub: PubSubEngine

      Returns Promise<RemoteBrowserScreencastResult>

      { Streaming: true } when frames are now being pushed, else { Streaming: false }.

    • This routine compares the OldValues property in the input object to the values in the DB that we just loaded. If there are differences, we need to check to see if the client is trying to update any of those fields (e.g. overlap). If there is overlap, we throw an error. If there is no overlap, we can proceed with the update even if the DB Values and the ClientOldValues are not 100% the same, so long as there is no overlap in the specific FIELDS that are different.

      ASSUMES: input object has an OldValues___ property that is an array of Key/Value pairs that represent the old values of the record that the client is trying to update.

      Parameters

      Returns Promise<void>

    Settings

    Member Visibility

    On This Page

    Constructors
    Accessors
    Methods
    AfterCreateAfterDeleteAfterUpdateapplyRestoreContextArrayFilterEncryptedFieldsForAPIArrayMapFieldNamesToCodeNamesBeforeCreateBeforeDeleteBeforeUpdateCheckAPIKeyScopeAuthorizationCheckUserReadPermissionscreateAuditLogRecordCreateRecordcreateRecordAccessAuditLogRecordDeleteRecordEmitCloudEventExecuteRemoteBrowserActionExecuteRemoteBrowserGoalFilterEncryptedFieldsForAPIfindByGetRemoteBrowserGoalResultGetRemoteBrowserSelectiongetRowLevelSecurityWhereClauseGetUserFromEmailGetUserFromPayloadInterpretRemoteBrowserPageListenForEntityMessagesLoadExternalRecordByKeyloadOwnedSessionMapFieldNamesToCodeNamespackageSPParamPublishCacheInvalidationRelayRemoteBrowserHumanInputRemoteBrowserSnapshotrequireUserAndProviderresolveProviderNameReverseMapInputFieldNamesRunDynamicViewGenericRunViewByIDGenericRunViewByNameGenericRunViewGenericInternalRunViewsGenericRunViewsGenericInternalsafeFirstArrayElementStartRemoteBrowserAudioStreamStartRemoteBrowserScreencastStopRemoteBrowserAudioStreamStopRemoteBrowserScreencastTestAndSetClientOldValuesToDBValuesUpdateRecord