Class BaseRESTIntegrationConnectorAbstract

Appends default query parameters from the IntegrationObject metadata. DefaultQueryParams is stored as a JSON object like {"key": "value"}. Automatically skips params whose key (case-insensitive) already appears in the URL to avoid duplicates with pagination params.

Runs TransformRecord, then RE-ADDS any source key the transform dropped (present in raw, absent from the output) unless declared in ExcludedSourceKeys. This makes the base-fetch path structurally full-record (§0 forward-compat contract): a TransformRecord override may reshape/coerce but cannot SILENTLY drop a source key from ExternalRecord.Fields, so the framework's custom-column capture sees everything the source returned. The default identity transform returns raw unchanged → fast-path no-op (zero overhead, zero behavior change for connectors that don't override TransformRecord).

Authenticate with the external system and return an auth context. Called once per FetchChanges invocation; the returned context is passed to BuildHeaders and MakeHTTPRequest for every request.

Build HTTP headers for an API request, including auth headers. Called before every HTTP request.

Build the operation request body per BodyShape:

• 'flat' → body = attributes verbatim
• 'wrapped' → body = { [BodyKey]: attributes }
• 'literal' → connector should have overridden the operation; fall back to flat as safety net
• null → default to flat (most common shape)

Appends pagination parameters to a URL based on the object's PaginationType. Override in subclasses to use vendor-specific parameter names.

DISCOVERY-ONLY: how many rows of a KEYLESS parent to read before classifying its key.

Only reached when a parent declares no primary key — with a declared one nothing is buffered and the chain stays lazy. The default is the value-statistic classifier's own significance floor: below it the classifier abstains, above it more rows change no verdict, and every row here is a fetch that recurses up the whole dependency chain.

Overridable the same way every other discovery bound is — per-connection Configuration (discoveryParentKeySampleRows), then MJ_INTEGRATION_DISCOVERY_PARENT_KEY_SAMPLE_ROWS, then this getter. Bounded above by the per-table sample target, so lowering that lowers this too and the two can never disagree in the direction that matters.

DISCOVERY-ONLY: last-resort SAFETY bound on the recursion depth for multi-level template chains (a parent that is itself a template-var child), so a malformed metadata cycle cannot loop unboundedly. This is only the fallback when the consumer sets no discoverySampleMaxDepth in Configuration and no env override is present — the consumer decides the real value. A recursion cap MUST have some bound, so a conservative default remains here; override the getter to change the fallback.

REST override (§sample-discover): a SINGLE-template-var CHILD is sampled with the recursive, record-constrained StreamRecordsForDiscovery. Flat objects fall back to the generic FetchChanges loop; MULTI-var (composition) children are deferred — StreamRecordsForDiscovery adjourns them (declared-only until first sync) rather than fire malformed URLs.

Source keys a connector DELIBERATELY removes in its TransformRecord override (vendor noise, a flattened parent blob, a nested child collection emitted as its own object). These are the only sanctioned drops — excluded from the applyTransformPreservingKeys re-add. Default: none. Override per-object to declare intentional removals.

Best-effort error message extraction from a vendor response. Override for vendor-specific shapes.

Extract the new record's external ID from a create response per IDLocation.

Extract pagination state from the vendor-specific response.

Get the base URL for API requests (e.g., "https://api.example.com/v1"). Combined with the object's APIPath to form the full request URL.

Gets IntegrationObjectField records from the engine's cache for a given object ID. Returns only active fields sorted by Sequence.

MEMOISED. This sits on per-record paths — RawToExternalRecord / TransformRecord resolve an object's fields for every record transformed, and several callers then run a .find() for the primary key over the freshly-sorted result — so the filter and sort were being repeated per record over a list that only changes when the engine reloads its metadata.

Invalidation is the ARRAY IDENTITY of the engine's field cache: it is replaced wholesale on load/refresh, so a new array yields a new memo automatically, including after a schema refresh. A fresh copy is returned per call, preserving the previous contract for callers that sort or splice the result.

Gets an IntegrationObject from the engine's cache by integration ID and object name. Throws if not found.

Execute an HTTP request. The concrete connector owns the transport layer (fetch, axios, got, etc.).

Extract the data array from the vendor-specific response envelope.

Substitute the ExternalID into a URL path template at runtime. Path templates may use {ID}, {id}, or {ExternalID} as the placeholder; for symmetry with FetchChanges template-var handling, any unsubstituted {var} in the template is left in place (caller's responsibility to ensure consistency).

For IDLocation='body' or 'header' the ID is not substituted into the path (caller must handle separately); we just return the raw path.

Per-call cap on how many PARENTS a parent-iterated object processes before yielding a resumable batch (the engine loops on HasMore). Kept conservative so a single FetchChanges call fits the engine's op-timeout even at a slow initial adaptive rate; the AIMD bucket ramps up over batches. Override in a concrete connector if a vendor tolerates larger per-call parent runs.

Optional per-record CUSTOMIZATION hook called between NormalizeResponse (vendor envelope-stripping) and ToExternalRecord (composite-PK assembly).

Distinct from NormalizeResponse — that strips the vendor's response envelope to expose individual records. This hook is for vendor-specific record-level customization the standard pipeline shouldn't carry: nested-field flattening, empty-string→null coercion for date columns, computed fields, removing vendor metadata blobs (e.g. Salesforce 'attributes'), etc.

Default is identity (no transform). Override only when a concrete connector needs vendor-specific shape changes.