Member Junction
    Preparing search index...

    Class BaseIntegrationConnectorAbstract

    Hierarchy (View Summary)

    Index

    Constructors

    constructor

    Accessors

    DiscoveryIsAuthoritative FetchChangesTimeoutMs IntegrationName MaxConcurrencyHint MonotonicWatermark RateLimitPolicy SupportsBatchWrite SupportsCreate SupportsDelete SupportsGet SupportsListing SupportsSearch SupportsUpdate SupportsUpsert

    Methods

    BatchCreateRecords BatchDeleteRecords BatchUpdateRecords BuildCreatedResult CreateRecord DeleteRecord DiscoverFields DiscoverFieldsViaFetch DiscoverFieldsViaStream DiscoverObjects DiscoverySampleRecordStream ExtractRetryAfterMs FetchChanges GetActionGeneratorConfig GetDefaultConfiguration GetDefaultFieldMappings GetIntegrationObjects GetRecord IntrospectSchema ListRecords PostProcessRecord SearchRecords StableOrderingKey TestConnection UpdateRecord Upsert

    Constructors

    Accessors

    • get DiscoveryIsAuthoritative(): boolean

      §7 — does this connector's discovery (DiscoverObjects/DiscoverFields → IntrospectSchema) return the AUTHORITATIVE, COMPLETE gamut of objects/fields the credentials expose? Default false (safe): a connector must explicitly affirm this. Override to true ONLY when DiscoverObjects hits a real list/describe endpoint that returns EVERYTHING accessible — so an object/field absent from a refresh genuinely means the source dropped it (and may be deactivated). Leave false for stubbed discovery (returns nothing → static metadata is all we have), a cache-driven IntrospectSchema, or any partial/scoped enumeration — there, absence proves nothing and MUST NOT deactivate.

      Returns boolean

    • get FetchChangesTimeoutMs(): number

      Per-connector override for the FetchChanges timeout, in milliseconds. null → use DEFAULT_OPERATION_TIMEOUTS.FetchChangesMs (30s).

      Raise this when a single page is legitimately slow: a connector that fans out one request per parent (ORCID's per-iD /record, any second-layer object) does N requests inside ONE FetchChanges call, so its page time scales with BatchSize — and with how much concurrency the engine's adaptive controller currently allows. Under the fixed 30s, a page that comfortably fit when parallel no longer fit once the controller had cut concurrency, which forced connector authors to shrink BatchSize for the sequential worst case and waste the parallel headroom the rest of the time.

      Note the timeout does NOT itself cut concurrency: ClassifyError gives it NETWORK_TIMEOUT, and only RATE_LIMIT_EXCEEDED feeds the adaptive limiter. A timing-out page simply never ramps the rate UP (the ramp needs a clean fetch), and the object ends incomplete — reported as FETCH_ABORTED_INCOMPLETE. Earlier revisions of this comment described a self-reinforcing timeout→concurrency-cut spiral; that is not what the code does.

      Precedence (highest first): CompanyIntegration.Configuration.fetchTimeoutMs → this property → DEFAULT_OPERATION_TIMEOUTS.FetchChangesMs. Deployments therefore keep the last word without a code change, while a connector that KNOWS it is slow ships a sane default.

      Returns number

    • get MonotonicWatermark(): boolean

      Whether the watermark this connector returns (FetchBatchResult.NewWatermarkValue) is a RELIABLE, monotonically-increasing global maximum — i.e. the connector fetches in watermark order so the last batch's value IS the true high-water mark, and an updated record always re-surfaces at a NEW (higher) watermark. When true, the engine uses that watermark to NARROW the next incremental (instead of advancing a full sync to wall-clock "now", and instead of the keyset clear-and-re-scan), so incrementals fetch only what's new.

      Default false — the safe, backwards-compatible choice: a connector whose source returns records out of watermark order (e.g. HubSpot's creation-ordered list API, where the last batch can carry old modstamps) MUST stay false so the engine keeps advancing to "now" and never saves a stale watermark. Override to true ONLY when the source guarantees monotonic ordering.

      Returns boolean

    Methods

    • Builds a CRUDResult for a record CREATE, failing LOUDLY when the external system returned no usable record ID. A 2xx response with an empty/undefined ID means the create did not durably produce a record we can track — returning Success:true there silently loses the record and causes duplicate creates on the next sync (the HubSpot-association class of bug, fixed in next commit 9f718a7e). This makes that failure explicit at the connector boundary.

      Parameters

      • externalID: string
      • statusCode: number
      • objectName: string

      Returns CRUDResult

    • Discovery via the connector's READ PATH (FetchChanges), TIME-BOUNDED — the way to gather a statistically-significant sample when a single DiscoverFields sample is too small to PROVE a key. "Discovery is the sync read path with the save removed."

      Loops FetchChanges as a read-only FULL fetch (WatermarkValue=null, nothing persisted), threading pagination/keyset cursors across batches, and streams every record through the data-informed field + provable-PK inference. It stops at the discovery TIME BUDGET (default 5 min), or a record cap, or source exhaustion — whichever comes first — so the provable-PK decision is made on as much real data as the budget allows. It NEVER fabricates a key: if even this larger sample yields no provable single/composite PK, the field set comes back PK-less and the object is honestly not added.

      Falls back to the single-sample DiscoverFields if the read path can't run for this object (e.g. a connector whose FetchChanges needs an already-persisted IO row that doesn't exist yet).

      Parameters

      • companyIntegration: MJCompanyIntegrationEntity
      • objectName: string
      • contextUser: UserInfo
      • opts: {
            BatchSize?: number;
            MaxRecords?: number;
            OnFallback?: (err: unknown) => void;
            TimeBudgetMs?: number;
        } = {}
        • OptionalBatchSize?: number
        • OptionalMaxRecords?: number
        • OptionalOnFallback?: (err: unknown) => void

          Called when streaming fails and this method degrades to single-sample DiscoverFields. The degradation is silent otherwise, and it is not a small one: the fallback returns the catalog's own description, which carries no observed widths — so the caller believes it sampled, and the object keeps whatever width the catalog guessed.

        • OptionalTimeBudgetMs?: number

      Returns Promise<ExternalFieldSchema[]>

    • Stage-2 field discovery for sources WITHOUT a describe/introspection endpoint (file feeds, undocumented JSON list endpoints): stream the source's actual records — READ-ONLY, no save, no ack — and derive the full field set + data-informed PK/uniqueness/nullability from the gathered statistics. The connector supplies whatever read-only fetch yields the records; this helper turns that stream into ExternalFieldSchema[].

      Why data-informed: streaming the real values lets pickPrimaryKeyFromStats pick the PK from evidence (uniqueness/non-null statistics) COMBINED with the naming convention, rather than a name guess alone. The PK is a SOFT key, so the pick is best-available, not strict-significance: a confident unique+non-null column wins outright; otherwise a near-unique / convention-named column is taken as a soft key (a PK-less object would stall CodeGen). The scan is time-bounded — it stops on exhaustion OR opts.Discovery.TimeBudgetMs; more rows simply mean stronger claims.

      Provable-only encoding into the standard flags:

      • IsPrimaryKey — set ONLY on the single statistics-first pick. Multiple equally-ranked unique columns leave PK unset here (ambiguous → the pipeline's SoftPKClassifier LLM tiebreaker decides, fed these same stats). Zero unique columns → no PK is fabricated.
      • IsUniqueKey — set when the column was all-distinct over the scan AND uniqueness was provable (the distinct-cap wasn't hit).
      • AllowsNull — asserted true ONLY when a null/absent value was actually observed; otherwise left undefined (permissive default). Never fabricates NOT NULL — critical under a time-capped partial scan where unseen rows could still be null.

      The PK emitted here is SOFT (it rides additionalSchemaInfo via the persist + DDL path; it is NEVER a hard DB key), so a wrong inference can never reject a valid row — the engine dedupes via the record-map. IsReadOnly defaults to true (stream discovery targets read feeds); a writable source overrides via opts.ReadOnly.

      Parameters

      • records:
            | AsyncIterable<Record<string, unknown>, any, any>
            | Iterable<Record<string, unknown>, any, any>

        A read-only sync/async iterable of source records (the caller's fetch yields them).

      • opts: { Discovery?: StreamDiscoveryOptions; Pk?: PkPickOptions; ReadOnly?: boolean } = {}

      Returns Promise<ExternalFieldSchema[]>

    • The record source DiscoverFieldsViaFetch streams for field/PK inference. Default: loop FetchChanges (full fetch), yielding each record's fields until maxRecords. A protocol subclass (e.g. REST) overrides this to sample a template-var CHILD with the correct record-constrained, recursive stream. Yields plain field maps; the caller stops it at maxRecords.

      Parameters

      • companyIntegration: MJCompanyIntegrationEntity
      • objectName: string
      • contextUser: UserInfo
      • batchSize: number
      • maxRecords: number
      • OptionaldeadlineMs: number

        Epoch ms this sample must not outlive, forwarded to the connector as FetchContext.DeadlineMs. Optional so existing overrides of this method keep compiling and keep their current behaviour.

      • OptionalwatchKey: string

        Watchdog key for this sample, when the caller registered one. Diagnostics only.

      Returns AsyncGenerator<Record<string, unknown>>

    • Parse a Retry-After / rate-limit signal out of a failed response or thrown error into milliseconds so the engine can back off precisely. Return undefined when the error is not a throttle (or carries no hint).

      The default reads the standard Retry-After header (RFC 9110 §10.2.3 — the header a 429 and a 503 carry), in both its delay-seconds and HTTP-date forms, from wherever the HTTP client put it. That is not a heuristic and not vendor-specific: there is one correct reading of it, and every HTTP connector benefits from having it read.

      This used to return undefined unconditionally, and no connector in this repo overrode it — so the engine's limiter never learned a delay any vendor had actually stated, and discovery's throttle check (which asked this and only this) concluded "not a throttle" for every 429 MJ ever received.

      Override when the vendor signals its delay somewhere non-standard — in the response body, or in prose (PheedLoop's "Expected available in N second"). Deliberately not parsed here: guessing a duration out of message text risks inventing one, and a wrong Retry-After is worse than none, since it freezes the token bucket for a made-up interval.

      Parameters

      • error: unknown

      Returns number

    • Name of a stable, monotonic ordering key (PK/identity) usable for KEYSET/seek resume on watermark-less objects (plan.md §7 — resume from last-seen key, robust to mid-stream insert/delete). null → keyset resume unavailable for this object.

      Parameters

      • _objectName: string

      Returns string

    Settings

    Member Visibility

    On This Page

    Constructors
    Accessors
    Methods