Member Junction
    Preparing search index...

    Process-wide watchdog that guarantees no AIAgentRun is left stuck in Status='Running' after the owning process dies (restart, crash, OOM) or its terminal-state write fails.

    Three cooperating mechanisms, all anchored to the database clock so they remain correct across multiple MJAPI instances behind a load balancer (no instance trusts its own clock):

    1. Heartbeat — while a run is in flight, the owning process stamps LastHeartbeatAt every AgentRunWatchdogConfig.heartbeatIntervalMs via SP_HEARTBEAT.
    2. Sweep — SweepOrphanedRuns force-fails any Running run whose heartbeat has gone stale, via SP_SWEEP (one atomic, set-based UPDATE … WHERE Status='Running' AND <stale> inside the proc — so concurrent sweeps from multiple instances are harmless and a run another instance is still heart-beating can't be caught mid-sweep). Run once on boot (closes restart-orphans) and on a timer (closes mid-life orphans).
    3. Graceful shutdown — on SIGTERM/SIGINT (via ShutdownRegistry) this process marks the runs it owns Cancelled via SP_CANCEL, closing the deploy case instantly.
    4. Stop relay — the database row is also the control channel in the OTHER direction. A Stop button (or anything else) marks a Running row Cancelled; every AgentRunWatchdogConfig.cancellationPollIntervalMs this process reads back which of ITS runs were cancelled and aborts the AbortController the owning agent registered with Track. BaseAgent already checks that signal throughout its loop and forwards it into prompts and actions, so the in-flight step ends and the run finalizes as Cancelled. Because the row is the channel, the replica that owns the run is the one that reacts, with no cross-process messaging.

    Every proc filters Status='Running' only — Paused and AwaitingFeedback are legitimately-not-progressing states and are never touched. The watchdog accesses the DB only through these stored procedures (writes) and the base view (reads) — never the base table — so it works as the regular runtime DB user, not just the elevated CodeGen user. Proc calls are built through the provider's Dialect.ProcedureCallSyntax, so they run on SQL Server and PostgreSQL unchanged.

    Hierarchy (View Summary)

    Implements

    Index

    Constructors

    Properties

    ShutdownName: "AgentRunWatchdog" = 'AgentRunWatchdog'

    Optional human-readable identifier surfaced in shutdown logs and used for de-duplication when Register is called repeatedly with the same instance but no name. Falls back to the constructor name.

    ExternalCancelAbortReason: "Cancelled externally" = 'Cancelled externally'

    The abort reason the stop relay raises for a row cancelled with any other reason.

    UserCancelAbortReason: "Cancelled by user request" = 'Cancelled by user request'

    The abort reason the stop relay raises when a run's row was marked Cancelled with CancellationReason = 'User Request' (the Stop button). BaseAgent maps it back to the same reason on the run it finalizes, so the row the UI wrote and the row the agent writes agree.

    Accessors

    • get GlobalKey(): string

      Returns string

    Methods

    • The Global Object Store is a place to store global objects that need to be shared across the application. Depending on the execution environment, this could be the window object in a browser, or the global object in a node environment, or something else in other contexts. The key here is that in some cases static variables are not truly shared because it is possible that a given class might have copies of its code in multiple paths in a deployed application. This approach ensures that no matter how many code copies might exist, there is only one instance of the object in question by using the Global Object Store.

      Returns GlobalObjectStore

    • Abort a run this process owns, right now, without waiting for the poll. The in-process counterpart of the row-driven relay: a caller that already knows the run was cancelled (a resolver on the same replica, a test) can fire the signal directly.

      Parameters

      • runID: string
      • reason: string = AgentRunWatchdog.UserCancelAbortReason

      Returns boolean

      true when a controller was registered for the run and had not already fired.

    • IShutdownable: on graceful shutdown, mark every run this process still owns as Cancelled (via SP_CANCEL) so it doesn't linger as a phantom Running row until the next sweep. Idempotent and never throws — a failed cancel just falls through to the staleness sweep later.

      Returns Promise<void>

    • Begin guarding a run. The owning agent calls this immediately after the run row is saved (so it has a stable ID). Idempotent per run ID. Capturing the provider/user lets the central heartbeat + shutdown statements run without each run re-supplying them.

      Parameters

      • runID: string
      • provider: DatabaseProviderBase
      • contextUser: UserInfo
      • OptionalabortController: AbortController

        The run's own abort controller. When supplied, the stop relay aborts it as soon as the run's row is seen marked Cancelled by someone else (the Stop button), with UserCancelAbortReason when the row says User Request. Without it the run is still heart-beaten and swept, but a stop cannot reach it.

      Returns void

    • Stop guarding a run. Optional — the periodic prune drops terminal runs automatically — but callers may invoke it for prompt cleanup right after writing a terminal state.

      Parameters

      • runID: string

      Returns void

    • Returns the singleton instance of the class. If the instance does not exist, it is created and stored in the Global Object Store. If className is provided it will be used as part of the key in the Global Object Store, otherwise the actual class name will be used. NOTE: the class name used by default is the lowest level of the object hierarchy, so if you have a class that extends another class, the lowest level class name will be used.

      Type Parameters

      Parameters

      • this: new () => T
      • OptionalclassName: string

      Returns T

    • Force-fail any Running agent run whose liveness heartbeat has gone stale (or that never beat and was started long ago), via the dedicated SP_SWEEP proc. The proc does the whole thing as one atomic, set-based UPDATE and returns the count, so it's safe to run on every boot and on a timer, on every instance: the predicate is staleness-based (never "fail all Running") and re-evaluated at write time, so it cannot touch a run a healthy instance is still heart-beating. Returns the number of rows failed.

      Parameters

      Returns Promise<number>