@memberjunction/search-engine

Reusable server-side search engine for MemberJunction applications. Provides hybrid search blending vector similarity, full-text search, entity LIKE-search, external storage search, and taxonomy tag-weighted retrieval using Reciprocal Rank Fusion (RRF).

🔍 Search Architecture

The engine coordinates multiple search providers through a unified interface (BaseSearchProvider), runs them concurrently, and fuses candidates with rank-based or score-based fusion:

🏷️ TagSearchProvider

Purpose & Concept

In enterprise search, searching for a concept such as "Cheddar" or "Machine Learning" should find the products, articles, companies, or tickets associated with that tag — rather than returning the administrative tag definition record itself ("Cheddar Cheddar - MJ: Tags").

TagSearchProvider implements tag-weighted record retrieval:

1. Excludes Administrative Taxonomy Records: Internal taxonomy entities (MJ: Tags, MJ: Tagged Items, MJ: Tag Synonyms, MJ: Tag Scopes, MJ: Tag Co-Occurrences) have AllowUserSearchAPI = 0.
2. Matches Query to Taxonomy Graph: Matches query terms against active tags and synonyms loaded into memory via TagEngineBase.
3. Retrieves Associated Records: Queries MJ: Tagged Items (and MJ: Content Item Tags) for records associated with the matched tags.
4. Weights by Continuous Relevance: Multiplies the tag match confidence ($0.0 - 1.0$) by the tagged item's continuous weight (TaggedItem.Weight, $0.0 - 1.0$).
5. Deduplicates Across Tags: When a record is tagged with multiple matching tags, takes the highest score, aggregates all matched tags into the result, and formats a descriptive snippet.

Tag Matching Confidence Matrix

TagSearchProvider uses a multi-tier matching strategy against TagEngineBase.Instance:

Continuous Weight Scaling

Tags in MemberJunction are not binary on/off flags. Autotagging pipelines and users assign relevance weights:

• Weight = 1.0: Central topic or manually applied tag.
• Weight = 0.8: Highly relevant topic.
• Weight = 0.5: Moderately relevant topic.
• Weight = 0.1: Tangential reference.

The final score for a tag-retrieved record is: $$\text{Score} = \text{round}(\text{TagConfidence} \times \text{TaggedItem.Weight}, 2)$$

Example:

• Query: "Cheddar" (Exact match $\implies \text{Confidence} = 1.00$).
• Product "Vermont Farmhouse Sharp" has Weight = 0.95.
• Final Score = $0.95$.
• Snippet: Tagged with "Sharp Cheddar" (95% relevance).

Configuration in Metadata

The provider is registered in metadata/search-providers/.search-providers.json and in [__mj].[SearchProvider]:

{
  "ID": "E89F43A1-7023-4158-9A7B-4B6CD7E19F12",
  "Name": "Tags",
  "DriverClass": "TagSearchProvider",
  "DisplayName": "Tags",
  "Description": "Retrieves records linked via the taxonomy knowledge graph and Tagged Items, weighted by tag match confidence and relevance weight.",
  "Priority": 3,
  "SupportsPreview": true
}
Copy

🔀 Fusion and Enrichment

1. Reciprocal Rank Fusion (RRF): Results from TagSearchProvider participate in SearchFusion.Fuse(). When an item matches across both direct entity LIKE-search and tag-weighted retrieval, RRF boosts the record's final rank, and SearchFusion.Deduplicate() combines score breakdowns (ScoreBreakdown.Tag + ScoreBreakdown.Entity) and merges tags.
2. Security & Permission Push-Down: SearchEngine automatically validates entity read permissions and executes verifyOwnershipAndRowFilters on tag results via RunView, guaranteeing that users only see records they are permitted to view. For content items and chunks, the protected VerifyOriginRecords gate then follows the result to the record it was derived from (through MJ: Entity Record Documents) and keeps it only when that origin record is readable under the origin entity's own permissions and row filters. See SEARCH_SCOPES_AND_RAG_GUIDE.md § 4.
3. Automatic Record Name Enrichment: SearchEnricher.Enrich() resolves canonical record names using GetEntityRecordNames for all tag results and attaches entity icons.

🧪 Testing

Run package unit tests:

cd packages/SearchEngine
npm run test
Copy