@memberjunction/web-search-engine

Metadata-driven web search for MemberJunction: a set of external search vendors behind one interface, with administrator-controlled priority and automatic failover.

The administrator picks the vendor. The agent never does.

import { WebSearchEngine } from '@memberjunction/web-search-engine';

await WebSearchEngine.Instance.Config(false, contextUser);
const result = await WebSearchEngine.Instance.Search({ Query: 'association management trends' }, contextUser);

if (result.Success) {
  for (const hit of result.Hits) console.log(hit.Title, hit.URL);
  console.log('served by', result.ProviderUsed);
}
Copy

Why this exists

Google discontinues the Custom Search JSON API on 2027-01-01 and has already closed it to new customers. Microsoft retired the Bing Search APIs on 2025-08-11. Both replaced a commodity SERP API with an answer-shaped grounding product locked to their own agent stack. Selling raw ranked web results to developers is a business both companies have chosen to exit.

The lesson is not "pick a different vendor". It is that any single web-search dependency has an expiry date, so the architecture has to make the next shutdown a configuration change.

Before this package, MJ had four independent web-search Actions — Google, Tavily, Perplexity, DuckDuckGo — with three incompatible output contracts (Items, Results, Content). Nothing could transparently fall back onto a different shape, so "Perplexity as a backup" was never actually a backup. Swapping vendors meant editing every agent that named one.

Now it is a row in a table.

How it differs from @memberjunction/search-engine

They sound alike and are not interchangeable. search-engine searches content MJ owns; this package searches the open web.

The decisive detail is the result contract: SearchResultItem requires ID, EntityName and RecordID — the primary key of a source record — and the pipeline ends in SearchEnricher, which resolves entity icons and record display names by entity lookup. A web hit has none of those. Fitting one in would mean lying in the type and dragging it through enrichment designed to decorate entity records.

What the two do share is the plugin pattern, and this package reuses all of it: a provider table carrying DriverClass / Priority / Status / CredentialID, ClassFactory resolution, and the Initialize() → CheckAvailability() → IsAvailable() lifecycle.

Blending the two — web results and internal records in one ranked list — is reachable through the existing RRF fusion, but it needs a result type that does not require EntityName. That is a deliberate future step, not a side effect of this package.

Three ways in

Pick by who is calling.

All three land in the same engine and obey the same provider configuration.

Who uses this

No agent or skill binds a search vendor directly. Every agent and skill that does web lookups — including the Web Research skill — binds the provider-neutral Web Search Action, and leaves Provider unset so the engine can fail over.

Google Custom Search and Perplexity Search remain registered for direct or manual execution, but carry no agent or skill bindings. Anything new binds Web Search, never a vendor.

Verify at any time:

grep -rn "Google Custom Search\|Perplexity Search" metadata/agents/ metadata/ai-skills/
# no output = no vendor bindings remain
Copy

Selection: how a provider is chosen

Without an explicit provider

1. Filter to providers that are Active, report themselves available, and can do what was asked
2. Order by Priority ascending — lower wins
3. Try each in turn until one succeeds

Failover is conditional, and the condition is the part that matters:

Zero hits is a success. A narrow query legitimately matches nothing. Treating that as failure makes the engine retry a query that will keep returning nothing, at four vendors' expense.

With an explicit provider

Provider pins the search to one vendor, by Name or DriverClass, case-insensitively. There is no failover — the call fails, with a code that says exactly why:

Those are four different problems with four different fixes, which is why they are four codes and not one. And the reason a pinned provider never silently falls back: a substitution is invisible. Results keep arriving, from a vendor nobody chose, until something eventually surfaces it — usually an invoice.

Attempts — read this one

Every provider tried is recorded in result.Attempts, including on the success path.

If the primary provider rate-limits every call and the secondary quietly serves everything, the system looks completely healthy. Nothing else makes that visible while the spend moves to a vendor nobody selected.

const result = await WebSearchEngine.Instance.Search({ Query: 'q' }, user);
// result.ProviderUsed === 'Tavily'
// result.Attempts === [
//   { ProviderName: 'Brave',  Succeeded: false, FailureKind: 'transient', ErrorMessage: '429 …' },
//   { ProviderName: 'Tavily', Succeeded: true,  HitCount: 10 },
// ]
Copy

Configuring providers

Rows live in __mj.WebSearchProvider, seeded from metadata/web-search-providers/.

Capabilities are not columns. Whether a driver can synthesize an answer or honour a domain filter is a property of the implementation, declared on the class — exactly as search-engine declares SourceType. The table holds configuration: on/off, order, credentials, limits.

To change the primary vendor, edit one row's Priority and run mj sync push. No code change, no agent change, no redeploy.

Bundled drivers

Why DuckDuckGo has no HTML fallback

An earlier revision fell back to fetching and regex-parsing DuckDuckGo's HTML results page, which answered far more queries than the Instant Answer API does. It was removed deliberately.

Running regular expressions over remote HTML is running them over attacker-influenceable input, and CodeQL flagged the parser with two high-severity findings: polynomial backtracking in the result patterns (the [^"]*…[^"]* shape) and incomplete sanitization in its tag-stripping.

Hardening those patterns would have narrowed the surface; deleting the parser removes it. For a fallback inside a last-resort provider this engine already tells operators not to rely on, the capability was not worth the exposure — and "the regex looks safe now" is exactly the kind of claim that is true when written and wrong later.

If you need real web coverage, configure a provider with an API. That is what the rest of this package is for.

Credentials

Each driver reads its key from the linked Credential record first, then falls back to an environment variable:

The fallback exists so a host upgrading from the per-vendor Actions does not have to migrate its secrets into the Credential store on the same day it gains the engine.

A driver with no key self-disables rather than throwing. An unconfigured vendor is a normal state in a product shipped to many hosts, not an error — the engine simply routes around it.

⚠️ Caching rights

AllowResultCaching defaults to 0, and that default is deliberate.

Caching rights differ sharply between vendors: some sell result storage as a distinct plan tier, others forbid persistent caching outright, and at least one major grounding product's terms prohibit caching, syndicating or analysing results at all. Violating any of it is silent — nothing fails, you are simply in breach.

Nothing in this engine caches today. The column exists so that the first caching layer reads a per-provider gate instead of inventing one, and so the contractual answer lives next to the credential it applies to. Confirm your plan's terms before setting it to 1.

Writing a driver

import { RegisterClass } from '@memberjunction/global';
import { BaseWebSearchProvider, WebSearchCapabilities } from '@memberjunction/web-search-engine';

@RegisterClass(BaseWebSearchProvider, 'AcmeWebSearchProvider')
export class AcmeWebSearchProvider extends BaseWebSearchProvider {
  public readonly Capabilities: WebSearchCapabilities = {
    Answer: false, DomainFilter: true, Freshness: true, Region: true, MaxResultsCap: 25,
  };

  private apiKey: string | undefined;

  public async CheckAvailability(): Promise<void> {
    this.apiKey = this.GetSecret('apiKey', 'ACME_API_KEY');
    this.apiKey ? this.MarkAvailable() : this.MarkUnavailable('No Acme API key.');
  }

  public async ExecuteSearch(params: WebSearchParams): Promise<WebSearchProviderResponse> {
    try {
      const response = await HttpGet<AcmeResponse>(ENDPOINT, { /* … */ });
      return { Success: true, Hits: (response.Data?.results ?? []).map(toHit) };
    } catch (e) {
      return classifyHttpFailure(e, 'Acme');   // sets FailureKind for you
    }
  }
}
Copy

Four rules, each of which has a failure mode if ignored:

1. Declare capabilities honestly. Claiming DomainFilter: true and then ignoring the filter returns confident results for a different question than the one asked.
2. Classify failures correctly. classifyHttpFailure handles the common cases; the distinction that matters is 400/422 (permanent) versus everything else (transient).
3. Self-disable, don't throw, when the driver is simply not configured on this host.
4. Export a Load…() anchor and call it from the package index. Bundlers drop a module whose exports are unused, and a dropped module never runs its decorator — so the driver silently vanishes from ClassFactory and the engine reports "no registered driver" for a DriverClass that plainly exists in the source.

Result codes

Search() never throws for a search-level failure — branch on Success, as with RunView. Exceptions escaping a driver are caught, recorded as a transient attempt, and failed over.

Setup after the migration

The migration, its CodeGen tail and the Remote Operation's server half are all committed, so a local database only needs the schema applied and the metadata seeded:

pnpm run mj:migrate                  # creates __mj.WebSearchProvider
pnpm mj sync push --dir metadata     # seeds the provider rows, the Action, the scopes
pnpm install && pnpm run build
Copy

Migrate before pushing metadata. metadata/web-search-providers/ writes into __mj.WebSearchProvider, so the table has to exist first. (A different ordering constraint — metadata before mj codegen, because remote operations generate from rows rather than schema — applies when regenerating, not when applying what is already committed.)

Verify before committing:

node .github/scripts/check-migration-entityfield-sequence.mjs   # apply-time MAX(Sequence)+1, never a literal
npm run check:codegen-tail                                      # generated entity/resolvers/form are committed
Copy

The PostgreSQL counterpart is not authored per-PR — conversion is build-engineer work at release time.

Testing

cd packages/WebSearchEngine && pnpm test
Copy

The selection tests are the ones that matter. Each pins a property with a way of passing while broken: conditional failover, an explicit provider never being substituted, and Attempts being recorded on the success path.

Related

• guides/SEARCH_OVERVIEW_GUIDE.md — which search API to reach for
• @memberjunction/search-engine — internal search over content MJ owns
• packages/Actions/CLAUDE.md — why the Action is a thin wrapper