Entity Search Guide (SearchEntity / SearchEntities)

A focused guide to MJ’s per-entity ranked search API. For the broader picture of which search API to use when, start with SEARCH_OVERVIEW_GUIDE.md.

What this API does

“Give me the N records of entity X most relevant to this free-text query.”

SearchEntity blends two signals against the records of one entity:

1. Lexical — substring / prefix matching on the entity’s name field and any field marked IncludeInUserSearchAPI. A RunView with LIKE filters.
2. Semantic — vector cosine against embeddings stored on MJ: Entity Record Documents.VectorJSON. Embeddings are produced from a Nunjucks-rendered template tied to the entity via an EntityDocument of type Search.

The two ranked lists fuse via canonical weighted Reciprocal Rank Fusion (ComputeRRF), permission filtering drops records the caller can’t read, and the top-K slice is returned. All modes (lexical-only, semantic-only, hybrid) are first-class via the mode option.

API surface

Declared on IMetadataProvider, implemented on every concrete provider:

SearchEntity(params: SearchEntityParams): Promise<EntitySearchResult[]>
SearchEntities(params: SearchEntityParams[]): Promise<EntitySearchResult[][]>

type SearchEntityParams = {
    entityName: string;
    searchText: string;
    options?: SearchEntitiesOptions;
};

type SearchEntitiesOptions = {
    mode?: 'lexical' | 'semantic' | 'hybrid';   // default 'hybrid'
    rrfK?: number;                              // default 60
    weights?: { lexical?: number; semantic?: number };  // default 1.0 each
    topK?: number;                              // default 10
    minScore?: number;                          // default 0
    entityDocumentId?: string;                  // override which EntityDocument
    contextUser?: UserInfo;
};

type EntitySearchResult = {
    entityRecordDocumentId: string | null;
    recordId: string;
    score: number;
    matchType: 'lexical' | 'semantic' | 'hybrid';
    components: { lexical?: number; semantic?: number };
};

Singular vs. plural

• SearchEntity({ ... }) — one entity, returns EntitySearchResult[].
• SearchEntities([{...}, {...}, ...]) — many entities. Returns EntitySearchResult[][] aligned by input order: result[i] is the ranked list for params[i].

Use the plural form whenever you want results from more than one entity for the same request. On the client (GraphQLDataProvider), the plural form is one GraphQL round-trip regardless of how many entities you list. On the server (GenericDatabaseProvider), the plural form fans out via Promise.all, so per-entity passes run concurrently.

Even the singular form on the client uses the plural transport under the hood — a single-element batch — so there’s no network penalty for calling the singular variant.

Quick examples

Agent prompt seeding (the original motivation)

import { Metadata } from '@memberjunction/core';

const md = new Metadata();
const candidates = await md.SearchEntity({
    entityName: 'MJ: Entities',
    searchText: userPrompt,
    options: { topK: 10, contextUser },
});

// Bake candidates into the system prompt as the "likely relevant entities" hint.
const promptHint = candidates
    .map(c => `- ${c.recordId} (score ${c.score.toFixed(3)})`)
    .join('\n');

Cross-entity “find anything relevant to X”

const groups = await md.SearchEntities([
    { entityName: 'Invoices',  searchText: query, options: { topK: 5, contextUser } },
    { entityName: 'Customers', searchText: query, options: { topK: 5, contextUser } },
    { entityName: 'Notes',     searchText: query, options: { topK: 5, contextUser } },
]);

const [invoices, customers, notes] = groups;

From an Action (agent-friendly)

await runAction({
    ActionName: 'Search Entity',
    Params: [
        { Name: 'EntityName', Value: 'Accounts' },
        { Name: 'SearchText', Value: 'enterprise contracts in NA region' },
        { Name: 'TopK', Value: 20 },
        { Name: 'Mode', Value: 'hybrid' },
        { Name: 'SemanticWeight', Value: 1.5 },  // bias toward semantic relevance
    ],
});

Mode tuning

Goal	Recommended options
Exact name lookups dominate (e.g. user typed “Invoices”)	mode: 'hybrid', default weights — lexical wins ties via score
Semantic intent dominates (vague descriptive queries)	mode: 'hybrid', weights: { semantic: 1.5 }
Skip embedding round-trip / EntityDocument not configured	mode: 'lexical'
Pure embedding similarity, no name bias	mode: 'semantic'

Provider implementation (how it works under the hood)

Polymorphism, not registration. No startup wiring required.

Provider	What it does
ProviderBase (MJCore)	Holds the concrete orchestration: resolve EntityDocument → parallel lexical + semantic passes → weighted RRF blend → permission post-filter → topK / minScore slice.
	The semantic pass is protected abstract — each concrete provider supplies its own.
GenericDatabaseProvider (server)	Implements searchEntitiesSemanticPass: AIEngine.EmbedTextLocal → SimpleVectorServiceProvider.QueryIndex. Direct calls, no indirection.
GraphQLDataProvider (client)	Overrides both SearchEntity and SearchEntities outright with a one-round-trip GraphQL proxy to the SearchEntities resolver. The abstract semantic pass is a no-op (unreachable).
SearchEntitiesResolver (MJServer)	Receives the batch, looks up the user via UserCache, calls md.SearchEntities, returns groups aligned by input order.

Result: same one-line API call on the client and the server; the right work happens at the right tier.

Configuration

Semantic and hybrid modes require an Active EntityDocument of type Search registered for the target entity. The MJ install seeds a standard set of Search EntityDocuments (see /metadata/entity-documents/), so several core catalogs are searchable out of the box, all on the in-process Simple Vector Service Provider + gte-small (Local) embedding stack (no API key, no cost):

Seeded Search EntityDocument	Target entity
MJ Entities Search	MJ: Entities
AI Agents Search	MJ: AI Agents
Actions Search	MJ: Actions
AI Prompts Search	MJ: AI Prompts
AI Models Search	MJ: AI Models

Opting another entity in is purely additive metadata — no code.

Enabling on your own entity (3 steps)

1. Register an EntityDocument of type Search for your entity. Add an entry to /metadata/entity-documents/.entity-documents.json (or push via mj sync from a custom metadata folder):

   {
     "fields": {
       "Name": "Accounts Search",
       "EntityID": "@lookup:MJ: Entities.Name=Accounts",
       "TypeID": "@lookup:MJ: Entity Document Types.Name=Search",
       "Status": "Active",
       "VectorDatabaseID": "@lookup:MJ: Vector Databases.Name=Simple Vector Service Provider",
       "VectorIndexID": "@lookup:MJ: Vector Indexes.Name=Default - SVS + gte-small (Local)",
       "AIModelID": "@lookup:MJ: AI Models.Name=gte-small (Local)",
       "TemplateText": "@file:templates/accounts-search.njk",
       "PotentialMatchThreshold": 0.7,
       "AbsoluteMatchThreshold": 0.95
     }
   }

   VectorIndexID is required. VectorizeEntity throws a configuration error if the EntityDocument has no Vector Index. Use the seeded Default - SVS + gte-small (Local) index unless you’ve created your own.

   TemplateText (not TemplateID) is the convenient path: on mj sync push, the MJ: Entity Documents server entity auto-creates the backing Template + Template Content records from the @file: content and stamps TemplateID for you. Omit primaryKey/sync for new records — mj-sync stamps them on first push.

2. Write the template at templates/accounts-search.njk. A reasonable default:

   {{ Name }}
   {{ Description }}
   {{ Industry }} - {{ Region }}

   Templates can reference any field on the record — including denormalized view fields (e.g. {{ Vendor }}, {{ Category }}). Keep them focused on the text a user would actually phrase — descriptive fields, business-language tags, not internal IDs.

3. Run the vector sync. The seeded Entity Vector Sync - Daily scheduled job (cron 0 0 4 * * *, RunImmediatelyIfNeverRun: true) drives the Vectorize Entity action over every Active Search EntityDocument, generates embeddings for each record, and writes them to EntityRecordDocument.VectorJSON. No manual step required — though you can force a run from the Scheduling dashboard. If no Active Search EntityDocuments exist, the job is a clean no-op (ResultCode: NO_DOCUMENTS), not a failure.

That’s it. Provider.SearchEntity({ entityName: 'Accounts', searchText: query }) now returns semantic results.

Swapping to a remote vector DB

For high-scale corpora, set VectorDatabaseID on the EntityDocument to a Pinecone / Qdrant / pgvector provider row instead of Simple Vector Service Provider. The orchestration in ProviderBase.SearchEntity is provider-agnostic — the semantic pass in GenericDatabaseProvider calls whichever VectorDBBase driver the EntityDocument names.

Weighted RRF — when and how to tune

The hybrid mode passes [lexicalRanked, semanticRanked] through ComputeRRF(lists, k, weights). Defaults:

options.weights = { lexical: 1.0, semantic: 1.0 }
options.rrfK = 60   // paper standard

Increase a weight to bias toward that signal. A weight of 0 suppresses the list entirely (equivalent to mode: 'lexical' or mode: 'semantic').

The weights option is the same shape used everywhere RRF is invoked in MJ — SearchEngine cross-scope fusion, dupe detection, etc. — so tuning intuition transfers.

Permissions

Results are post-filtered by row-level read permission on the target entity. The implementation over-fetches (typically topK * 2) before filtering so the final result count usually matches the requested topK. If your user can’t read N of the top results, the next-best matches fill in.

Permission evaluation uses the existing RunView pipeline — no separate permission codepath to maintain.

Common pitfalls

• No EntityDocument configured → semantic mode returns empty; hybrid degrades to lexical-only. Log: SearchEntity: no active 'Search' EntityDocument for entity "...".
• EntityDocument exists but vectors haven’t synced yet → semantic mode returns empty silently. Run the vector sync job or wait for the next poll.
• Embedding model misconfigured → AIEngine.EmbedTextLocal returns null; semantic pass returns empty. Check the configured local embedding model.
• searchText is empty / whitespace → returns [] immediately, no work done.
• Caller forgets contextUser on the server → permission filter has no user to evaluate against; behavior depends on the entity’s permission rules. Always pass contextUser server-side.

Out of scope (today)

• Field-level enrichment of the embedding template — only the EntityDocument’s Nunjucks template feeds the embedding.
• Cross-process / distributed in-memory vector cache (the SimpleVectorServiceProvider cache is per-process).
• An admin UI for “enable search on this entity” — metadata + mj sync is the v1 surface.

See plans/entity-search-via-entity-document.md for the original design discussion and the deferred ideas.