§7 — EntityDefinitions enumerates the COMPLETE credentialed gamut (standard + custom +
solution-installed tables / attributes), not a filtered subset, so a comprehensive refresh
may safely + reversibly deactivate objects no longer in the response.
Verbatim from the upstream identity handoff. Drives the three-way name invariant. Returned as a string literal (not the INTEGRATION_NAME const ref) so the T1 ThreeWayName static parser can extract it from source.
Highest SAFE per-layer concurrency the source tolerates (plan.md §7 peak parallelization) — the
ceiling the engine's adaptive controller ramps toward. null → use configured syncConcurrency.
§7 — the watermark this connector advances (delta-link / modifiedon high-water) IS monotonic:
change-tracking yields a forward-only delta token, and the modifiedon fallback fetches in
ascending modifiedon order so the last batch carries the true maximum. Lets the engine narrow
the next incremental instead of advancing to wall-clock now.
§7 — Dataverse enforces a per-environment Web API service-protection limit (≈ 6000 requests / 300s sliding window, plus an execution-time budget). Conservative sustained rate keeps the connector under the request-count limit; the engine's AIMD bucket adapts on 429s via ExtractRetryAfterMs. (≈6000/300 ≈ 20 req/s; we stay below to leave headroom.)
Whether this connector supports batched target writes (plan.md §7 aggressive batching).
Whether this connector supports creating new records in the external system.
Whether this connector supports deleting records from the external system.
Whether this connector supports reading/fetching records. Always true.
Whether this connector supports paginated listing of records.
Whether this connector supports searching/querying records with filters.
Whether this connector supports updating existing records in the external system.
Whether this connector supports idempotent upserts (create-or-update keyed
by a unique business property). Connectors override this AND Upsert to enable it.
ProtectedAppendAppends default query parameters from the IntegrationObject metadata. DefaultQueryParams is stored as a JSON object like {"key": "value"}. Automatically skips params whose key (case-insensitive) already appears in the URL to avoid duplicates with pagination params.
ProtectedapplyOverrides the base re-add so the DYNAMIC OData annotation keys (<col>@odata.bind, FormattedValue,
lookuplogicalname, etc.) TransformRecord strips are NOT restored. The base loop re-adds any
raw key absent from the transform output unless it is in the static ExcludedSourceKeys
set — which can't enumerate the per-record annotation suffixes. Here we re-add a dropped key only
when it is neither statically excluded NOR an annotation key (by predicate). This keeps full-record
pass-through for genuine columns while making the annotation removal stick (auditable, by rule).
ProtectedAuthenticateMints/caches a Microsoft Entra ID access token via client-credentials. The scope is the
environment's .default (<EnvironmentUrl>/.default) so the token's audience matches the
Dataverse resource. Token is cached by OAuth2TokenManager until near expiry.
Batch-create. Default loops single-record CreateRecord, so the engine may always call the batch form.
Batch-delete. Default loops single-record DeleteRecord.
Batch-update. Default loops single-record UpdateRecord.
ProtectedBuildBuilds a CRUDResult for a record CREATE, failing LOUDLY when the external system returned no usable record ID. A 2xx response with an empty/undefined ID means the create did not durably produce a record we can track — returning Success:true there silently loses the record and causes duplicate creates on the next sync (the HubSpot-association class of bug, fixed in next commit 9f718a7e). This makes that failure explicit at the connector boundary.
ProtectedBuildStandard OData/Dataverse headers — bearer token + JSON + OData version + return-representation.
ProtectedBuildBuild the operation request body per BodyShape:
ProtectedBuildHonors the OData @odata.nextLink cursor (used verbatim) and otherwise injects the page-size
request as a $top-free first page — Dataverse caps page size via the Prefer: odata.maxpagesize
header (added in MakeFirstPageHeaders), NOT a query param, and does NOT support $skip.
Optionalcursor: stringOptional_effectivePageSize: numberGeneric create: reads CreateAPIPath/Method/BodyShape/BodyKey/IDLocation from the IO row.
Generic delete: reads DeleteAPIPath/DeleteMethod/DeleteIDLocation from the IO row.
Case-2 runtime discovery for a single object's columns: describes the table's Attributes
collection and maps each Property → ExternalFieldSchema, surfacing Type/MaxLength,
the GUID PK (IsPrimaryId / PrimaryIdAttribute → IsPrimaryKey), and scalar Lookup
columns → FK (with ForeignKeyTarget = the referenced table's logical name).
Discovery via the connector's READ PATH (FetchChanges), TIME-BOUNDED — the way to gather a statistically-significant sample when a single DiscoverFields sample is too small to PROVE a key. "Discovery is the sync read path with the save removed."
Loops FetchChanges as a read-only FULL fetch (WatermarkValue=null, nothing persisted), threading pagination/keyset cursors across batches, and streams every record through the data-informed field + provable-PK inference. It stops at the discovery TIME BUDGET (default 5 min), or a record cap, or source exhaustion — whichever comes first — so the provable-PK decision is made on as much real data as the budget allows. It NEVER fabricates a key: if even this larger sample yields no provable single/composite PK, the field set comes back PK-less and the object is honestly not added.
Falls back to the single-sample DiscoverFields if the read path can't run for this object (e.g. a connector whose FetchChanges needs an already-persisted IO row that doesn't exist yet).
Optionalopts: { BatchSize?: number; MaxRecords?: number; TimeBudgetMs?: number }ProtectedDiscoverStage-2 field discovery for sources WITHOUT a describe/introspection endpoint (file feeds,
undocumented JSON list endpoints): stream the source's actual records — READ-ONLY, no save,
no ack — and derive the full field set + data-informed PK/uniqueness/nullability from the
gathered statistics. The connector supplies whatever read-only fetch yields the records; this
helper turns that stream into ExternalFieldSchema[].
Why data-informed: streaming the real values lets pickPrimaryKeyFromStats pick the PK from
evidence (uniqueness/non-null statistics) COMBINED with the naming convention, rather than a
name guess alone. The PK is a SOFT key, so the pick is best-available, not strict-significance:
a confident unique+non-null column wins outright; otherwise a near-unique / convention-named
column is taken as a soft key (a PK-less object would stall CodeGen). The scan is time-bounded —
it stops on exhaustion OR opts.Discovery.TimeBudgetMs; more rows simply mean stronger claims.
Provable-only encoding into the standard flags:
IsPrimaryKey — set ONLY on the single statistics-first pick. Multiple equally-ranked unique
columns leave PK unset here (ambiguous → the pipeline's SoftPKClassifier LLM tiebreaker
decides, fed these same stats). Zero unique columns → no PK is fabricated.IsUniqueKey — set when the column was all-distinct over the scan AND uniqueness was provable
(the distinct-cap wasn't hit).AllowsNull — asserted true ONLY when a null/absent value was actually observed; otherwise
left undefined (permissive default). Never fabricates NOT NULL — critical under a time-capped
partial scan where unseen rows could still be null.The PK emitted here is SOFT (it rides additionalSchemaInfo via the persist + DDL path; it is
NEVER a hard DB key), so a wrong inference can never reject a valid row — the engine dedupes via
the record-map. IsReadOnly defaults to true (stream discovery targets read feeds); a writable
source overrides via opts.ReadOnly.
A read-only sync/async iterable of source records (the caller's fetch yields them).
Optionalopts: { Discovery?: StreamDiscoveryOptions; Pk?: PkPickOptions; ReadOnly?: boolean }OptionalDiscovery?: StreamDiscoveryOptionsTime budget / sample caps for the scan (see StreamDiscoveryOptions).
OptionalPk?: PkPickOptionsSignificance threshold + naming-rank tiebreaker (see PkPickOptions).
OptionalReadOnly?: booleanWhether discovered fields are read-only. Default true.
Case-2 (auth-gated) runtime discovery: enumerates EVERY table the credential can see via the
EntityDefinitions describe endpoint (standard + custom + solution-installed) — NOT a baked
catalog. The documented standard catalog seeded in the Declared metadata file is the floor;
this returns what the live environment actually exposes (the ceiling), which is strictly a
superset for custom-bearing tenants.
ProtectedDiscoveryDISCOVERY-ONLY: last-resort SAFETY bound on the recursion depth for multi-level template chains (a
parent that is itself a template-var child), so a malformed metadata cycle cannot loop unboundedly.
This is only the fallback when the consumer sets no discoverySampleMaxDepth in Configuration and
no env override is present — the consumer decides the real value. A recursion cap MUST have some
bound, so a conservative default remains here; override the getter to change the fallback.
ProtectedDiscoveryREST override (§sample-discover): a SINGLE-template-var CHILD is sampled with the recursive, record-constrained StreamRecordsForDiscovery. Flat objects fall back to the generic FetchChanges loop; MULTI-var (composition) children are deferred — StreamRecordsForDiscovery adjourns them (declared-only until first sync) rather than fire malformed URLs.
ProtectedExcludedThe well-known static OData control annotation keys TransformRecord drops. The base
applyTransformPreservingKeys uses this to avoid re-adding them. Dataverse ALSO emits dynamic,
per-lookup annotation suffixes (<col>@odata.bind, <col>@OData.Community.Display.V1.FormattedValue,
<col>@Microsoft.Dynamics.CRM.lookuplogicalname) whose exact names vary by record and cannot be
enumerated statically — those are handled by the applyTransformPreservingKeys override
below (predicate exclusion), so they never re-appear. This static list covers the always-present
envelope keys for completeness / auditability.
ProtectedExtractBest-effort error message extraction from a vendor response. Override for vendor-specific shapes.
ProtectedExtractDataverse returns the created record's URI in the OData-EntityId response header
(e.g. https://org.crm.dynamics.com/api/data/v9.2/accounts(00000000-...)); the GUID is the
trailing (...) segment. Falls back to the Location header, then the body PK, so a tenant
configured to return-representation still resolves. An empty ID makes the create FAIL LOUDLY
via the base's BuildCreatedResult (never a silent duplicate-create).
ProtectedExtractCursor pagination via @odata.nextLink. The nextLink is an ABSOLUTE continuation URL that
encodes the exact next request (including the opaque $skiptoken); it is followed VERBATIM and
never hand-built. A present nextLink means another page exists.
Parse Dataverse's throttle signal. The service returns HTTP 429 with a Retry-After header
(seconds) on a service-protection limit; honor it precisely.
Routes to the change-tracking delta path when the table declares it AND a watermark exists;
otherwise delegates to the base metadata-driven fetch (which applies the modifiedon watermark
fallback + standard @odata.nextLink pagination). The base path already handles
TransformRecord (annotation stripping) + full-record pass-through, so the delta path mirrors it.
Returns the ActionGeneratorConfig for this connector, combining the integration name, category, icon, and objects into a ready-to-use configuration for ActionMetadataGenerator.Generate().
Override in subclasses to customize the config (e.g., icon, category). Returns null by default if GetIntegrationObjects() returns empty.
ProtectedGetPer-connection base the engine joins the IO APIPath onto. The frozen Dataverse IO APIPaths are
ABSOLUTE-from-root (they already include /api/data/v9.2/...), so this returns the org ROOT
(<EnvironmentUrl>) and the resolved request URL is <EnvironmentUrl>/api/data/v9.2/<entityset>.
The versioned Web API base (<EnvironmentUrl>/api/data/v9.2) is ApiBaseUrl, used by the
connector's OWN calls (WhoAmI / EntityDefinitions / delta). Never a hardcoded org URL.
ProtectedGetGets IntegrationObjectField records from the engine's cache for a given object ID. Returns only active fields sorted by Sequence.
ProtectedGetGets an IntegrationObject from the engine's cache by integration ID and object name. Throws if not found.
Returns a proposed default configuration for quick setup. Override in subclasses to provide connector-specific defaults including schema name, objects to sync, and field mappings. Returns null by default (no quick setup available).
Returns suggested default field mappings for an external object to MJ entity. Override in subclasses to provide intelligent defaults.
Name of the external object
Name of the target MJ entity
Array of default field mappings (empty by default)
Returns the integration objects and their fields that this connector supports, for use by the ActionMetadataGenerator. This is static metadata that does NOT require a live connection — it describes the connector's known object model.
Override in subclasses to provide connector-specific objects/fields. Returns an empty array by default (no action generation available).
Generic get-one: hits APIPath/{ID} via GET. Override if API uses non-standard get shape.
Builds SourceSchemaInfo from IntegrationEngineBase cached metadata. Provides richer metadata than the base class implementation, including FK relationships derived from RelatedIntegrationObjectID.
Lists records with cursor-based pagination.
Override in subclasses that support paginated listing.
Check SupportsListing before calling.
ProtectedMakeHTTP transport with retry/backoff on 429/503/504 (Dataverse service-protection limits). The
thrown error on a retryable status carries the response headers so ExtractRetryAfterMs
can read Retry-After for the engine's adaptive bucket.
Optionalbody: unknownProtectedNormalizeUnwraps the OData collection envelope's value array. A single-record GET (/<set>(id)) is
returned as a one-element array. Honors a metadata-declared ResponseDataKey if set, defaulting
to the OData value convention.
Type-driven post-processing hook (plan.md §10): a connector may normalize/enforce a record's
values to the resolved column formats AFTER transform/normalize and BEFORE write. Default
returns the record unchanged. (Named for this system — NOT MCP, not take.) The engine ALSO
applies target-type constraint enforcement; this is the connector-side complement.
Searches for records matching the given filters.
Override in subclasses that support search/query operations.
Check SupportsSearch before calling.
§7 — keyset/seek resume key for watermark-less objects: the table's GUID primary key
(<logicalName>id), read from the IO's Configuration. Stable + monotonic-enough for
resume-from-last-seen; null when no PK is known.
ProtectedSubstituteSubstitutes the target ID into a Dataverse single-record path. Supports the standard GUID form
/<entityset>(id) AND the alternate-key upsert form /<entityset>(<altkey>='<value>') — when
the ExternalID already contains an = it is an alternate-key expression and is inserted as-is
(only the value is escaped); otherwise it is treated as a GUID. Handles both ({id}) template
placeholders and a bare (id) suffix.
ProtectedTemplatePer-call cap on how many PARENTS a parent-iterated object processes before yielding a resumable batch (the engine loops on HasMore). Kept conservative so a single FetchChanges call fits the engine's op-timeout even at a slow initial adaptive rate; the AIMD bucket ramps up over batches. Override in a concrete connector if a vendor tolerates larger per-call parent runs.
Validates the service principal can reach the environment by issuing a fresh token and a
lightweight WhoAmI call (an unbound function returning the calling user/org GUIDs).
ProtectedTransformRemoves the OData control annotations (@odata.etag, @odata.context, @odata.id,
<lookup>@odata.bind, <lookup>@OData.Community.Display.V1.FormattedValue, etc.) Dataverse
sprinkles onto every record — they are response transport metadata, not columns. Every other
source key is preserved (full-record pass-through); the removed keys are declared in
ExcludedSourceKeys so the base re-add doesn't restore them and change-detection ignores them.
Generic update: reads UpdateAPIPath/Method/BodyShape/BodyKey/IDLocation from the IO row.
Upserts a record — a single idempotent create-or-update keyed by a unique
business property (e.g. email), eliminating the search-then-create race window.
Override in subclasses whose external system exposes a keyed upsert primitive.
Check SupportsUpsert before calling.
Connector for Microsoft Dynamics 365 / Dataverse (Common Data Service) via the OData v4 Web API.
<EnvironmentUrl>/api/data/v9.2, per-connection (every customer environment differs — nothing is hardcoded).DiscoverObjects/DiscoverFieldsparse the credentialedEntityDefinitionsdescribe endpoint at runtime (case-2 auth-gated discovery) so BOTH standard and custom/solution-installed tables surface. The documented standard catalog lives in the Declared metadata file, not in this class.@odata.nextLinkcursor, followed verbatim.OData-EntityIdresponse header; update PATCH/<entityset>(id); delete DELETE/<entityset>(id); alternate-key upsert path form).Prefer: odata.track-changes→@odata.deltaLink) when a table declares it;modifiedon-polling fallback otherwise.