Class DuplicateRecordDetector

Protected getter/setter for the AIEmbeddingRunner instance used by duplicate detection.

Write the LLM verdict + run id onto a single match row's generated columns. The recommendation / confidence / reasoning are taken from THIS candidate's own verdict (judged independently), so a false-positive candidate reads NotDuplicate even when another candidate in the same set is a confident Merge. Falls back to the set-level summary only when the reasoner returned no per-candidate verdict for this record. The proposed survivor + field map stay set-level (they describe the merge of the set's true duplicates). The run id is the set's, unless this candidate's verdict came from a run of its own.

Carry the set-level verdict + resolved survivor field map onto the result, and each candidate's own verdict onto its PotentialDuplicate, so the auto-merge step (AutoMergeAboveAbsolute) can consult both recommendations and apply the literal {FieldName, Value} overrides via RecordMergeRequest.FieldMap. The UI still reads the persisted per-row MJDuplicateRunDetailMatchEntity.LLMProposedFieldMap (the raw choices) and lets the reviewer override before a manual merge.

The entry-time decision's input, bounded: at most DUPLICATE_ENTRY_CHECK_MAX_DECISION_FIELDS differing fields, the ones the person entered first, and every value and label cut to DUPLICATE_ENTRY_CHECK_MAX_FIELD_TEXT_LENGTH. A candidate's stored values are cut too: a long Notes field on an existing record would otherwise reach the model in full.

Assemble the reasoning input for a matched set: source description, candidate descriptions, and the differing-field deltas loaded for the whole set.

Build a map of recordID → JSON metadata string from loaded BaseEntity records. Extracts the entity's name field and a few key display fields for rich UI rendering.

Build an unsaved entity object holding the entered values, so the entity document's template renders it as it renders a saved record. Values for fields the entity does not have are ignored, and text longer than DUPLICATE_ENTRY_CHECK_MAX_FIELD_TEXT_LENGTH is cut, so the template, the embedding and the decision state never see more of a field than that.

How many neighbors to query for so that TopK survive the record filter: TopK times RECORD_FILTER_TOPK_FACTOR when a filter is set, TopK otherwise.

Create a new DuplicateRun record.

Create match records for a single run detail, saving in parallel batches.

When a per-set DuplicateReasoningOutput is supplied, each match row gets THIS candidate's own verdict (recommendation / confidence / reasoning), while the set-level proposed survivor / field map / run id ride along on every row.

Create DuplicateRunDetail records for a batch of record IDs.

The fields whose values make up a record's display name: every IsNameField field in Sequence order, or the entity's single NameField when none is flagged.

Execute one guarded auto-merge for an eligible candidate, applying the LLM's resolved field map when present, and stamp the match record on success.

Execute a vector query — uses hybrid search with RRF when the provider supports it.

Remove matches whose target record no longer exists in the source entity.

Vector indexes (e.g. Pinecone) retain embeddings keyed by each record's composite key via a deterministic vector ID. When records are deleted — or re-seeded with new primary keys — their old vectors linger as "ghosts". A nearest-neighbour query then happily returns those ghosts, often the record's OWN former vector (identical name, ~0.98 score), which to a user looks exactly like a record matching itself. The per-record self-match filter can't catch these because the ghost carries the OLD id.

We verify every distinct match key for the batch against the live table in a single batched query and drop any that don't resolve. Keys are compared in compact URL-segment form, so this works for a single-column key of any name and for composite keys.

With the entity document's record filter, a match whose record fails the filter is dropped the same way: its vector may predate the filter, or the record may have changed since sync.

Mutates queryResults in place.

Try to find an existing DuplicateRun for a given ListID. Returns null if none found.

The entity document that turns the entry-time check on for an entity: an Active document with EnableLLMReasoning on whose ReasoningMode is 'Decision' or 'DecisionThenPrompt'. The check makes a decision-model call on every pause in typing, so switching reasoning off on the document turns the check off too, as it turns off reasoning in batch runs (IsReasoningGateOpen). When several qualify, the oldest wins (earliest __mj_CreatedAt, then lowest ID), so adding a document never silently changes which one an entity uses.

Generate the template text for each record: the text that is embedded and queried with.

It must be the text vector sync embedded for the record, or the query vector describes a different document than the stored one and genuine duplicates fall below the threshold. So the template is rendered exactly as sync renders it: from the row RunView returns for the record (LoadTemplateRows), with every Entity param's related rows (a person's Phones, Emails...), through the same EntityDocumentTemplateDataBuilder sync uses.

Read the maxConcurrentRequests from the VectorDatabase entity's Configuration column, falling back to DEFAULT_QUERY_CONCURRENCY if not set.

The entity document's record filter (Configuration.recordFilter.extraFilter): the records that take part in duplicate detection. A batch run checks only records that pass it, and every run offers only candidates that pass it. CheckSingleRecord and the entry-time check still check the record they are given, since the caller asked about that record. Vector sync applies the same filter to the records it vectorizes.

Initializes embedding model, vector DB, and index name providers. Called once per detection run rather than per-record. Uses AIEngine and KnowledgeHubMetadataEngine caches to avoid redundant database queries.

Decide whether a single candidate is eligible for automatic merge.

Back-compat path (EnableLLMReasoning = false): eligible iff the vector score meets the absolute threshold — byte-for-byte the original behavior. AutomationLevel is ignored.

Reasoning path (EnableLLMReasoning = true): AutomationLevel governs.

• ReviewAll / LLMGated → never auto-merge (everything goes to human review).
• AutoMergeAboveAbsolute → at/above the absolute threshold AND the set's LLM recommendation is 'Merge' AND this candidate's own verdict is 'Merge'. A set-level 'Merge' only says that SOME candidate is a duplicate, so it never merges a candidate the reasoner judged otherwise, or returned no verdict for.

Gate predicate: reasoning enabled + non-empty set + top score clears the threshold. A null ReasoningThreshold means "no gate" — reasoning runs for any non-empty set.

True when two composite keys identify the same record. Compares the normalized (trimmed, lower-cased) URL-segment representation, so it is robust to UUID-casing differences across platforms (SQL Server upper- vs PostgreSQL lower-case) and to a bare-PK vs concatenated key shape.

Narrow a list's or view's record ids to those that pass the entity document's record filter, so a record the filter excludes is never checked whatever the run's source.

Populate a candidate's composite key from a vector match's RecordID, tolerant of the two formats different vector providers emit:

• Concatenated ("ID|", or "F1|v1||F2|v2" for composite PKs) — e.g. providers that stored the full key. Parsed via CompositeKey.LoadFromConcatenatedString.
• Bare primary-key value ("") — what the in-process SimpleVectorServiceProvider returns, since it stores MJ: Entity Record Documents.RecordID as the raw PK value. For this form we rebuild the key using the source record's PK field name(s).

Getting this wrong leaves KeyValuePairs empty (the bare value has no field delimiter, so LoadFromConcatenatedString no-ops), which silently breaks self-match filtering, candidate field-delta loading (the reasoner sees an empty candidate), and MatchRecordID persistence.

Return the set of record keys (compact URL segments, normalized for case-insensitive UUID comparison) that actually exist in the entity, for the given candidate ids — themselves compact segments. A single-column key uses one IN (...) per chunk; a composite key needs one (F1=.. AND F2=..) term per record. Batched to stay within reasonable query sizes.

The display names of the candidates the context user can read, from one RunView over the candidate keys run as that user, so entity permissions and row-level security apply. A candidate missing from the map must not be shown. When the RunView fails the map is empty: a candidate is shown only when the user is known to be able to read it.

Load record IDs directly from the entity, optionally filtered. Selects only the primary key column(s) with ResultType: 'simple' for efficiency.

Load record IDs from a list's detail records.

Load record IDs by running a saved view.

Resolve the ids to check. Every id is a compact CompositeKey URL segment — the bare value for a single-column key (whatever the column is called), F1|v1||F2|v2 for a composite key — which is the form MJ: List Details.RecordID already holds; CompositeKey.FromURLSegment rebuilds the real key from it for any entity.

Every loader reads with IgnoreMaxRows. Without it RunView falls back to the entity's UserViewMaxRows (1,000 by default) and returns only the first page of ids. The run then sets TotalItemCount from that page, so it checks a fraction of the records and still reports itself complete: "1000 of 1000" on a 61,671-record entity.

Load full entity objects for a batch of composite keys.

Load records from an entity that are members of the specified list. Kept for backward compatibility.

Load the template entity from TemplateEngineServer for the given entity document.

Each record's row as vector sync reads it — RunView with ResultType: 'simple' and every column — in the order of records, so the template renders the values sync rendered. BaseEntity.GetAll() differs: it turns date fields into Date objects, for one.

An unsaved record (the entry-time check), or one whose row is no longer there, renders from its own values.

Locate the matching run-detail row, create the per-candidate match records (carrying the set's LLM verdict when present), mark the detail complete, and stamp the created match-record IDs back onto the result (kept in lockstep with Duplicates for auto-merge).

Persist match results and update run detail records.

When LLM reasoning is enabled on the entity document AND a matched set clears the reasoning gate (top vector score >= ReasoningThreshold), reasoning runs ONCE for the whole set; the verdict is written onto every match row's LLM* columns and carried on the result for the auto-merge step. When reasoning is disabled the loop is unchanged.

Automatically merge records that meet the absolute match threshold.

Render one record with the entity document's template, embed it, and query the vector index for its candidates. Shared by CheckSingleRecord (a saved record) and CheckRecordValues (an unsaved one).

Query the vector DB for duplicates of each record, with concurrency control.

Creates one async task per record, then executes them via RunWithConcurrency with the specified concurrency limit. Each task embeds and queries a single record against the vector index.

Supports hybrid search (vector + keyword) with RRF fusion when the vector DB provider supports it (checked via SupportsHybridSearch).

Post-query, results are:

1. Parsed from raw vector matches into typed PotentialDuplicate objects
2. Filtered to remove self-matches (same record as source)
3. Filtered by the potential match threshold (from options or entity document)

Extract a display label from a candidate's vector metadata snapshot, if present.

Case-insensitive, trimmed equality of two record-id (URL-segment) strings.

The decision provider the entry-time check asks: whatever the class factory holds for the 'Decision' mode, which is DecisionReasoningProvider unless an app registers a subclass. Null when that registration is not a decision provider.

Resolve an existing DuplicateRun or create a new one. Supports both list-based and list-free operation.

Resolve the reasoner's per-field survivor choices ({FieldName, SourceRecordID}) into literal {FieldName, Value} entries for RecordMergeRequest.FieldMap, by reading the chosen record's value out of the matched-set deltas. Choices whose field or record can't be located in the deltas are skipped (the survivor's existing value then stands), so a partial/garbled choice can never inject an undefined override into the merge.

Resolve the reasoning provider for this entity document's ReasoningMode via the ClassFactory. Returns null when ClassFactory falls back to the abstract base (no implementation registered for the mode) so the caller can skip gracefully.

Resolve a record's display label: prefer the name loaded from the record (via the delta builder), then a secondary fallback (e.g. a candidate's vector-metadata name), then the raw record-id key as a last resort. Guards against the loaded label itself being the record key (the engine's own fallback) so we still try the secondary source.

True when a result carries at least one candidate that ProcessAutoMerges could act on. Used to decide what the run must carry across batches: results with no eligible candidate are already persisted and are never read again, so holding them only grows the heap.

Deliberately reuses IsAutoMergeEligible, so a subclass that narrows eligibility narrows what is retained too, and the two can never disagree.

Run LLM reasoning for one source record's matched set, ONCE, only when:

1. the entity document has EnableLLMReasoning = true, AND
2. the set is non-empty, AND
3. the set's top MatchProbability >= the (non-null) ReasoningThreshold.

Returns undefined in every other case — including when reasoning is disabled — so the surrounding persist/merge path stays byte-for-byte identical to the vector-only behavior. A failed reasoning call returns an output with Success = false (never throws), so one bad set never aborts the run.

Drop inverse duplicates in place: if A→B was already persisted this run, skip B→A.

Keep at most TopK candidates per result, in the order the query ranked them. Mutates in place.

Validates that an entity document exists and is usable. Uses the KnowledgeHubMetadataEngine cache for instant lookups without database queries.

Null out a hallucinated survivor: the reasoner can return a SurvivorRecordID that isn't any record in the set (a made-up GUID, or a candidate that was filtered out). Persisting it would carry a phantom id to the UI and auto-merge. Validate against the set's record ids (source + candidates) and drop it if it doesn't match.

Run vectorization for the entity document's records.