Class DuplicateRecordDetector

Write the LLM verdict + run id onto a single match row's generated columns. The recommendation / confidence / reasoning are taken from THIS candidate's own verdict (judged independently), so a false-positive candidate reads NotDuplicate even when another candidate in the same set is a confident Merge. Falls back to the set-level summary only when the reasoner returned no per-candidate verdict for this record. The proposed survivor + field map stay set-level (they describe the merge of the set's true duplicates).

Carry the set-level verdict + resolved survivor field map onto the result so the auto-merge step (AutoMergeAboveAbsolute) can consult the recommendation and apply the literal {FieldName, Value} overrides via RecordMergeRequest.FieldMap. The UI still reads the persisted per-row MJDuplicateRunDetailMatchEntity.LLMProposedFieldMap (the raw choices) and lets the reviewer override before a manual merge.

Assemble the reasoning input for a matched set: source description, candidate descriptions, and the differing-field deltas loaded for the whole set.

Build a map of recordID → JSON metadata string from loaded BaseEntity records. Extracts the entity's name field and a few key display fields for rich UI rendering.

Create a new DuplicateRun record.

Create match records for a single run detail, saving in parallel batches.

When a per-set DuplicateReasoningOutput is supplied, each match row gets THIS candidate's own verdict (recommendation / confidence / reasoning), while the set-level proposed survivor / field map / run id ride along on every row.

Create DuplicateRunDetail records for a batch of record IDs.

Execute one guarded auto-merge for an eligible candidate, applying the LLM's resolved field map when present, and stamp the match record on success.

Execute a vector query — uses hybrid search with RRF when the provider supports it.

Remove matches whose target record no longer exists in the source entity.

Vector indexes (e.g. Pinecone) retain embeddings keyed by each record's composite key via a deterministic vector ID. When records are deleted — or re-seeded with new primary keys — their old vectors linger as "ghosts". A nearest-neighbour query then happily returns those ghosts, often the record's OWN former vector (identical name, ~0.98 score), which to a user looks exactly like a record matching itself. The per-record self-match filter can't catch these because the ghost carries the OLD id.

We verify every distinct match key for the batch against the live table in a single batched query and drop any that don't resolve. Single-column primary keys only — the same assumption the rest of this engine makes.

Mutates queryResults in place.

Try to find an existing DuplicateRun for a given ListID. Returns null if none found.

Generate human-readable template text for each record using the entity document template.

Loads the template from TemplateEngineServer and renders it via Nunjucks, matching the same approach used by the vectorization pipeline.

Read the maxConcurrentRequests from the VectorDatabase entity's Configuration column, falling back to DEFAULT_QUERY_CONCURRENCY if not set.

Initializes embedding model, vector DB, and index name providers. Called once per detection run rather than per-record. Uses AIEngine and KnowledgeHubMetadataEngine caches to avoid redundant database queries.

Decide whether a single candidate is eligible for automatic merge.

Back-compat path (EnableLLMReasoning = false): eligible iff the vector score meets the absolute threshold — byte-for-byte the original behavior. AutomationLevel is ignored.

Reasoning path (EnableLLMReasoning = true): AutomationLevel governs.

• ReviewAll / LLMGated → never auto-merge (everything goes to human review).
• AutoMergeAboveAbsolute → at/above the absolute threshold AND the set's LLM recommendation is 'Merge'.

Gate predicate: reasoning enabled + non-empty set + top score clears the threshold. A null ReasoningThreshold means "no gate" — reasoning runs for any non-empty set.

True when two composite keys identify the same record. Compares the normalized (trimmed, lower-cased) URL-segment representation, so it is robust to UUID-casing differences across platforms (SQL Server upper- vs PostgreSQL lower-case) and to a bare-PK vs concatenated key shape.

Populate a candidate's composite key from a vector match's RecordID, tolerant of the two formats different vector providers emit:

• Concatenated ("ID|", or "F1|v1||F2|v2" for composite PKs) — e.g. providers that stored the full key. Parsed via CompositeKey.LoadFromConcatenatedString.
• Bare primary-key value ("") — what the in-process SimpleVectorServiceProvider returns, since it stores MJ: Entity Record Documents.RecordID as the raw PK value. For this form we rebuild the key using the source record's PK field name(s).

Getting this wrong leaves KeyValuePairs empty (the bare value has no field delimiter, so LoadFromConcatenatedString no-ops), which silently breaks self-match filtering, candidate field-delta loading (the reasoner sees an empty candidate), and MatchRecordID persistence.

Return the set of primary key values (normalized for case-insensitive UUID comparison) that actually exist in the entity, for the given candidate IDs. Batches the IN-list to stay within reasonable query sizes.

Load record IDs directly from the entity, optionally filtered. Uses Fields: ['ID'] and ResultType: 'simple' for efficiency.

Load record IDs from a list's detail records.

Load record IDs by running a saved view.

Load the IDs of records to check, using the appropriate strategy based on the request. Returns an array of primary key value strings.

Load full entity objects for a batch of composite keys.

Load records from an entity that are members of the specified list. Kept for backward compatibility.

Load the template entity from TemplateEngineServer for the given entity document.

Locate the matching run-detail row, create the per-candidate match records (carrying the set's LLM verdict when present), mark the detail complete, and stamp the created match-record IDs back onto the result (kept in lockstep with Duplicates for auto-merge).

Persist match results and update run detail records.

When LLM reasoning is enabled on the entity document AND a matched set clears the reasoning gate (top vector score >= ReasoningThreshold), reasoning runs ONCE for the whole set; the verdict is written onto every match row's LLM* columns and carried on the result for the auto-merge step. When reasoning is disabled the loop is unchanged.

Automatically merge records that meet the absolute match threshold.

Query the vector DB for duplicates of each record, with concurrency control.

Creates one async task per record, then executes them via RunWithConcurrency with the specified concurrency limit. Each task embeds and queries a single record against the vector index.

Supports hybrid search (vector + keyword) with RRF fusion when the vector DB provider supports it (checked via SupportsHybridSearch).

Post-query, results are:

1. Parsed from raw vector matches into typed PotentialDuplicate objects
2. Filtered to remove self-matches (same record as source)
3. Filtered by the potential match threshold (from options or entity document)

Extract a display label from a candidate's vector metadata snapshot, if present.

Case-insensitive, trimmed equality of two record-id (URL-segment) strings.

Resolve an existing DuplicateRun or create a new one. Supports both list-based and list-free operation.

Resolve the reasoner's per-field survivor choices ({FieldName, SourceRecordID}) into literal {FieldName, Value} entries for RecordMergeRequest.FieldMap, by reading the chosen record's value out of the matched-set deltas. Choices whose field or record can't be located in the deltas are skipped (the survivor's existing value then stands), so a partial/garbled choice can never inject an undefined override into the merge.

Resolve the reasoning provider for this entity document's ReasoningMode via the ClassFactory. Returns null when ClassFactory falls back to the abstract base (no implementation registered for the mode) so the caller can skip gracefully.

Resolve a record's display label: prefer the name loaded from the record (via the delta builder), then a secondary fallback (e.g. a candidate's vector-metadata name), then the raw record-id key as a last resort. Guards against the loaded label itself being the record key (the engine's own fallback) so we still try the secondary source.

Run LLM reasoning for one source record's matched set, ONCE, only when:

1. the entity document has EnableLLMReasoning = true, AND
2. the set is non-empty, AND
3. the set's top MatchProbability >= the (non-null) ReasoningThreshold.

Returns undefined in every other case — including when reasoning is disabled — so the surrounding persist/merge path stays byte-for-byte identical to the vector-only behavior. A failed reasoning call returns an output with Success = false (never throws), so one bad set never aborts the run.

Drop inverse duplicates in place: if A→B was already persisted this run, skip B→A.

Validates that an entity document exists and is usable. Uses the KnowledgeHubMetadataEngine cache for instant lookups without database queries.

Null out a hallucinated survivor: the reasoner can return a SurvivorRecordID that isn't any record in the set (a made-up GUID, or a candidate that was filtered out). Persisting it would carry a phantom id to the UI and auto-merge. Validate against the set's record ids (source + candidates) and drop it if it doesn't match.

Run vectorization for the entity document's records.