Member Junction
    Preparing search index...

    Interface OrganicKeyDetectionConfig

    Configuration for organic-key cluster detection.

    Organic keys (per MJ PR #2193) are cross-entity navigation rules based on shared business values rather than declared FKs (e.g. matching contact records across CRM systems by email). This pass clusters columns whose descriptions + sample values indicate they share a value space, and emits proposed clusters for human review.

    Off by default — opt-in via enabled: true. Requires an AI provider (the existing ai config) for both embeddings and per-cluster refinement.

    NOTE: This pass deliberately does NOT iterate. Clustering happens once, the LLM refiner runs once per cluster, and the pass terminates. No convergence loop, no multi-pass expansion. If a future version reintroduces iteration, maxIterations would join this config section.

    interface OrganicKeyDetectionConfig {
        autoCreateRelatedViewOnForm?: boolean;
        clusteringSensitivity?: "strict" | "balanced" | "permissive";
        embedding?: {
            batchSize?: number;
            cachePath?: string;
            dimensions?: number;
            endpoint?: string;
            model?: string;
            provider?:
                | "gemini"
                | "openai"
                | "mistral"
                | "azure"
                | "bedrock"
                | "ollama"
                | "local";
        };
        enabled: boolean;
        maxRefinementRetries?: number;
        mergeThreshold?: number;
        minClusterSize?: number;
        minDistinctTables?: number;
        onRefinementError?: "fail"
        | "skip";
        outputFile?: string;
        refinementConcurrency?: number;
        sampleValueCount?: number;
        tokenBudget?: number;
    }
    Index

    Properties

    autoCreateRelatedViewOnForm? clusteringSensitivity? embedding? enabled maxRefinementRetries? mergeThreshold? minClusterSize? minDistinctTables? onRefinementError? outputFile? refinementConcurrency? sampleValueCount? tokenBudget?

    Properties

    autoCreateRelatedViewOnForm?: boolean

    Set AutoCreateRelatedViewOnForm on every emitted organic key. Default FALSE.

    Was a hardcoded true: each emitted key auto-created one related-record grid per spoke — around 22 per key on a real run — each grid a value-join on a column the detector had never verified. A machine-proposed key should be opted into.

    clusteringSensitivity?: "strict" | "balanced" | "permissive"

    How aggressive should clustering be? The detector auto-calibrates the merge distance from the actual embedding geometry — this knob just picks how loose to be when deciding what counts as "close enough" to merge:

    • 'strict': few, tight clusters. Only obvious matches. Lower recall, higher precision.
    • 'balanced': default. Catches most real organic keys without too much noise.
    • 'permissive': more clusters, including borderline cases. Higher recall, more LLM review work.

    Internally maps to a percentile of the pairwise distance distribution (strict ~ p1, balanced ~ p5, permissive ~ p15).

    embedding?: {
        batchSize?: number;
        cachePath?: string;
        dimensions?: number;
        endpoint?: string;
        model?: string;
        provider?:
            | "gemini"
            | "openai"
            | "mistral"
            | "azure"
            | "bedrock"
            | "ollama"
            | "local";
    }

    Embedding provider override. Maps to a MemberJunction BaseEmbeddings driver (resolved via the ClassFactory). The API key is taken from the top-level ai config.

    When absent this defaults to the embedding driver of the SAME vendor as the configured LLM provider, because that is the one credential the operator has definitely supplied. It used to default to the literal openai, which meant the shipped default configuration handed a Gemini key to OpenAIEmbedding. A vendor with no embedding driver (anthropic, groq, openrouter, …) is a named error asking for an explicit choice; local needs no key at all.

    Type Declaration

    • OptionalbatchSize?: number
    • OptionalcachePath?: string

      Disk path for the embedding cache. Default: <outputDir>/embedding-cache.json.

    • Optionaldimensions?: number
    • Optionalendpoint?: string
    • Optionalmodel?: string
    • Optionalprovider?: "gemini" | "openai" | "mistral" | "azure" | "bedrock" | "ollama" | "local"
    enabled: boolean

    Enable the organic-key detection pass (default: false).

    maxRefinementRetries?: number

    Max retries per LLM call on transient failure. Default: 2.

    mergeThreshold?: number

    Advanced / debugging override — explicit cosine-distance threshold. Set ONLY to bypass auto-calibration (e.g. ablation studies, reproducing a specific run). Normal users should set clusteringSensitivity instead.

    minClusterSize?: number

    Minimum members for a cluster to be reported. Default: 2.

    minDistinctTables?: number

    Minimum distinct tables a cluster must span. Default: 2.

    onRefinementError?: "fail" | "skip"

    When a single cluster's refinement errors, fail the whole pass or just skip that cluster. Default: 'skip'.

    outputFile?: string

    Path (relative to outputDir) for the detected-organic-keys JSON output. Default: detected-organic-keys.json.

    refinementConcurrency?: number

    Concurrency for per-cluster LLM refinement calls. Default: 4.

    sampleValueCount?: number

    Number of sample values per column to include in the embedding input and refiner prompt. Default: 5.

    tokenBudget?: number

    Token budget for the normalization pass. Default: 0 = unlimited.

    Enforced by NOT SCHEDULING further tables once the running total exceeds it; calls already in flight are allowed to finish, so the recorded total may overshoot by up to concurrency tables' worth. The result reports BudgetExhausted and how many tables were skipped, because normalizing 300 of 500 tables and reporting success is its own defect.

    This was declared and read nowhere, so the pass had no token, cost or call-count cap at all.

    Settings

    Member Visibility

    On This Page

    Properties