Protected Optional_ProtectedcontentProtectedCrawlProtectedCrawlProtectedMaxProtectedRootProtectedsourcePer-source RunBudget tracker, keyed by normalized source ID. Items
processed in each batch are tallied against the budget of the source
they belong to; when any budget exhausts, the engine's OnAfterBatch
gate returns continue:false and the run pauses gracefully. Next
invocation will re-crawl, change-detection will skip the already-
processed pages, and the remaining ones get processed.
ProtectedURLProtectedvisitedProtectedProviderReturns the active provider — explicit override if set, otherwise the global default.
ProtectedapplyReset crawl-related instance fields back to the class defaults. Called at the start of each content source so prior-source overrides don't leak forward. URLPattern and RootURL default to undefined — derived later if unset.
ProtectedapplyImplemented abstract method from the AutotagBase class. that runs the entire autotagging process. This method is the entry point for the autotagging process. It initializes the connection, retrieves the content sources corresponding to the content source type, sets the content items that we want to process, extracts and processes the text, and sets the results in the database.
OptionalonProgress: AutotagProgressCallbackOptionalcontentSourceIDs: string[]Optionalprovider: IMetadataProviderProtecteddelayProtectedextractPure helper: extract clean body text from raw HTML. No IO. Exposed as a protected method so subclasses and unit tests can exercise it without monkey-patching axios.
Fetch a URL once, extract clean text, and compute a stable checksum over that text. Returns both so callers don't have to fetch twice for "is this changed?" + "what's the content?".
The checksum is computed over the EXTRACTED body text, NOT the raw HTML, because raw HTML routinely contains incidental changes (server timestamps, CSRF tokens, build hashes, ad rotators) that would falsely report a page as "changed" on every crawl. Hashing the extracted text is what users actually mean by "did the content change?"
ProtectedgetBackwards-compatible array form. Drains the streaming variant.
ProtectedgetProtectedgetProtectedgetBackwards-compatible Set form. Drains the streaming variant; URLs end up
in this.visitedURLs as a side effect of streamLowerLevelLinks.
ProtectedgetProtectedgetBackwards-compatible void form. Drains the streaming variant (links go
into visitedURLs as a side effect of streamTopLevelLinks).
ProtectedinstallInstall the engine's OnAfterBatch hook so each batch's items are
counted against the budget of the source they belong to. Returns
continue:false from the gate when any source's budget exhausts,
which the engine then translates into a graceful pause.
ProtectedisSimple check to see if the URL is at the highest level domain.
ProtectednormalizeNormalize a URL for use as a dedup key in visitedURLs. Conservative
normalization that catches the common variations without risking the merge
of two semantically distinct pages:
ProtectedoverlayApply per-source ContentSourceParam rows to this crawler instance. Values
stored in the DB are strings, so we coerce per-key to the right runtime type
instead of bulk-assigning (which previously stuffed strings into number /
boolean fields and relied on JS coercion at use sites).
Unknown keys are silently ignored — same gate as the prior if (key in this)
check, just made explicit.
Given a URL, extracts text from a webpage. Kept for external callers
that just want the text — internal change-detection now uses
fetchAndExtract to avoid redundant fetches.
ProtectedprocessProcess one URL through the change-detection pipeline. Returns the
MJContentItem if the page is new or changed (caller should hand it off
to the LLM stage), or null if the page is unchanged.
One axios.get per URL: the same response body provides both the
change-detection hash and the page text. Compare with byChecksum
scoped to the current ContentSource so identical boilerplate (404 pages,
shared error templates) from a different source can't silently mask
legitimate pages here.
Given a content source, retrieve all content items associated with the content sources. The content items are then processed to determine if they have been modified since the last time they were processed or if they are new content items.
Backwards-compatible array form. Internally drains the streaming variant so there is exactly one implementation of the change-detection logic.
ProtectedSetBackwards-compatible batch form: process an explicit list of URLs and
return all new/changed content items as an array. New code should prefer
streamContentItemsToProcess which pipelines into the LLM batcher.
ProtectedsetupBuild a per-source RunBudget map from each source's ConfigurationObject. Sources with no budget knobs set still get a RunBudget entry (with all limits = null) so the OnAfterBatch hook can update item counts uniformly.
Per-source overrides via ContentSourceParam rows (e.g., MaxItemsPerRun stored as a param) take precedence over the ConfigurationObject value.
ProtectedstreamStreaming variant: yields each newly-discovered URL as the crawler finds it,
so downstream consumers (the content-item streamer that feeds the LLM
batcher) can start working before discovery completes. This is the
canonical implementation; getAllLinksFromContentSource below is a
backwards-compatible array-collecting wrapper.
Streaming variant: yields each new/changed content item as soon as it passes change detection. Lets the crawl and LLM phases overlap so total wall-clock time is roughly max(crawl, classify) instead of crawl + classify.
The canonical implementation lives here; the array-returning
SetContentItemsToProcess is a thin collector wrapper around this.
ProtectedstreamStreaming variant of getLowerLevelLinks. Yields each newly-discovered URL the moment it's added to the visited set, then recurses depth-first into children. This is the canonical implementation — the LLM batcher gets fed in real time during crawl instead of having to wait for the entire recursive discovery to complete.
getLowerLevelLinks below is a thin backwards-compatible wrapper that
drains the stream into a Set.
ProtectedstreamStreaming variant of getTopLevelLinks — yields each URL it adds to the visited set so the LLM batcher gets fed in real time.
Protectedurl
Optional provider override; falls back to Metadata.Provider when not set.