OptionalCrawlWhen true, also adds sibling-path URLs found on the seed page (single-pass, no recursion). Off by default to avoid accidental fan-out across paths the operator didn't intend.
OptionalCrawlWhen true (default), the recursive depth-aware crawler runs. Setting false disables it (single-page behavior, retrieved-as-discovered-from-the-seed-URL only).
OptionalMaxRecursion ceiling for in-domain links. 0 = just the start URL; 2 (the default) =
root + section pages + their child content pages. Higher values combine multiplicatively
with the per-page 1-second crawl delay.
OptionalRootURL prefix used for the in-domain check. When unset, derived as the parent directory of the seed URL. Override to expand or constrain the crawl boundary (e.g., set to the bare origin to crawl the whole site).
OptionalURLRegex string. Only URLs matching this pattern are added to the visited set. Use to scope
to e.g. ^https://example\.com/blog/.*. Unset = match everything.
Per-source crawl/discovery settings specific to AutotagWebsite. All optional with runtime defaults; an empty object is valid and produces the standard behavior (MaxDepth=2, recursive crawl on, sibling-domain fan-out off, no URL filter).