A library for synchronizing MemberJunction database metadata with local file system representations. This library is integrated into the MemberJunction CLI (mj) and is accessed through mj sync commands. It enables developers and non-technical users to manage MJ metadata using their preferred editors and version control systems while maintaining the database as the source of truth.
MetadataSync is included with the MemberJunction CLI. Install the CLI globally:
npm install -g @memberjunction/cli
Then use the sync commands:
mj sync --help
MemberJunction is a powerful metadata-driven system where configuration, business logic, AI prompts, templates, and more are stored as metadata in the database. This approach provides tremendous flexibility and runtime configurability, but it can create friction in modern development workflows.
The Metadata Sync tool bridges the gap between database-stored metadata and file-based workflows by:
flowchart LR
subgraph Files["Local File System"]
JSON["JSON Metadata Files"]
EXT["External Files\n(.md, .html, .sql)"]
CFG[".mj-sync.json\nConfiguration"]
end
subgraph Engine["MetadataSync Engine"]
PULL["PullService"]
PUSH["PushService"]
VAL["ValidationService"]
SYNC["SyncEngine\n(Reference Resolution)"]
end
subgraph DB["MemberJunction Database"]
ENT["Entity Records"]
REL["Related Records"]
META["Entity Metadata"]
end
DB -->|"mj sync pull"| PULL
PULL --> Files
Files -->|"mj sync push"| VAL
VAL --> PUSH
PUSH --> SYNC
SYNC --> DB
style Files fill:#2d6a9f,stroke:#1a4971,color:#fff
style Engine fill:#7c5295,stroke:#563a6b,color:#fff
style DB fill:#2d8659,stroke:#1a5c3a,color:#fff
For Developers:
For Non-Technical Users:
For Organizations:
This tool preserves the power of MJ's metadata-driven architecture while adding the convenience of file-based workflows. The database remains the source of truth for runtime operations, while files become the medium for creation, editing, and deployment.
@file:filename.ext to link external files from JSON@parent: and @root: to reference parent/root entity fields-v) for detailed output--include and --exclude options--include and --exclude options--include or --exclude to filter which entity directories are processedai-*, *-test, etc.--include and --exclude togetherflowchart TD
subgraph Services["Service Layer"]
IS["InitService"]
PS["PullService"]
PUS["PushService"]
SS["StatusService"]
WS["WatchService"]
VS["ValidationService"]
FRS["FileResetService"]
FS["FormattingService"]
end
subgraph Core["Core Engine"]
SE["SyncEngine\n(Reference Resolution)"]
CM["ConfigManager"]
JP["JsonPreprocessor\n(@include handling)"]
RDA["RecordDependencyAnalyzer\n(Topological Sort)"]
end
subgraph IO["I/O Layer"]
FE["FieldExternalizer"]
RP["RecordProcessor"]
FWB["FileWriteBatch"]
JWH["JsonWriteHelper"]
FBM["FileBackupManager"]
end
subgraph Data["Data & Auditing"]
TM["TransactionManager"]
SL["SQLLogger"]
DA["DeletionAuditor"]
DRS["DatabaseReferenceScanner"]
end
Services --> Core
Services --> IO
PUS --> Data
Core --> IO
style Services fill:#2d6a9f,stroke:#1a4971,color:#fff
style Core fill:#7c5295,stroke:#563a6b,color:#fff
style IO fill:#2d8659,stroke:#1a5c3a,color:#fff
style Data fill:#b8762f,stroke:#8a5722,color:#fff
The tool works with any MemberJunction entity -- both core system entities and user-created entities. Each entity type can have its own directory structure, file naming conventions, and related entity configurations.
This tool should NOT be used to modify metadata that is reflected from the underlying database catalog. Examples include:
These properties are designed to flow from the database catalog up into MJ metadata, not the other way around. Attempting to modify these via file sync could create inconsistencies between the metadata and actual database schema.
The tool is intended for managing business-level metadata such as:
For more information about how CodeGen reflects system-level data from the database into the MJ metadata layer, see the CodeGen documentation.
The push command preloads existing records and caches files/lookups so the
per-record sync work avoids redundant DB round-trips and disk I/O.
RunViews)Before processing records, the push scans every JSON file in the target
directories (in parallel, with a concurrency cap), collects primary keys
from records and nested relatedEntities, groups them by entity, and
issues a single filtered RunView per entity. This replaces the
prior per-record existence check with one batched read per entity at
push start.
SyncMetadataEngine extends BaseEngine)SyncMetadataEngine plugs into MJ's BaseEntity event bus via BaseEngine.
Saves and deletes performed during the push update the in-memory cache in
place, so subsequent lookups inside the same run see the new state without
a re-fetch. The engine overrides canUseImmediateMutation to allow this
in-place update for its own preload configs — the filter is built from
the preloaded PKs, so updates and deletes stay safe, and newly-created
records are deliberately added to the cache so later lookups can find them
within the same push.
@include output are cached on
first read so the validation and sync passes don't reparse. The cache
entry for a file is invalidated whenever the push writes that file back
to disk.@lookup (and @parent / @root) values are
memoized per (entity, lookup-fields) key. The cache is indexed by
entity so deleting a record invalidates only the affected entries
instead of clearing the whole cache. Keys are URI-encoded to prevent
collisions when user data contains = / & / |.Real-world reductions vary with the size and shape of your metadata tree; expect the biggest wins on syncs that read many records of the same entity (which is the common case).
The tool uses a hierarchical directory structure with cascading defaults:
.mj-sync.json files define entities and base defaults.mj-folder.json files define folder-specific defaults (optional)filePattern configured in .mj-sync.json.md, .html, etc.) are referenced from the JSON filesflowchart TD
subgraph Root["metadata/"]
RS[".mj-sync.json\n(Global Config)"]
end
subgraph AI["ai-prompts/"]
AS[".mj-sync.json\nentity: AI Prompts"]
subgraph CS["customer-service/"]
CF[".mj-folder.json\n(Folder Defaults)"]
GJ[".greeting.json\n(Record)"]
GM["greeting.prompt.md\n(External Content)"]
end
subgraph AN["analytics/"]
AF[".mj-folder.json"]
DJ[".daily-report.json"]
DM["daily-report.prompt.md"]
end
end
subgraph TE["templates/"]
TS[".mj-sync.json\nentity: Templates"]
end
Root --> AI
Root --> TE
AS --> CS
AS --> AN
style Root fill:#64748b,stroke:#475569,color:#fff
style AI fill:#2d6a9f,stroke:#1a4971,color:#fff
style CS fill:#7c5295,stroke:#563a6b,color:#fff
style AN fill:#7c5295,stroke:#563a6b,color:#fff
style TE fill:#2d6a9f,stroke:#1a4971,color:#fff
metadata/
+-- .mj-sync.json # Global sync configuration
+-- ai-prompts/
| +-- .mj-sync.json # Defines entity: "AI Prompts"
| +-- customer-service/
| | +-- .mj-folder.json # Folder metadata (CategoryID, etc.)
| | +-- .greeting.json # AI Prompt record with embedded models
| | +-- greeting.prompt.md # Prompt content (referenced)
| | +-- greeting.notes.md # Notes field (referenced)
| +-- analytics/
| +-- .mj-folder.json # Folder metadata (CategoryID, etc.)
| +-- .daily-report.json # AI Prompt record
| +-- daily-report.prompt.md # Prompt content (referenced)
+-- templates/ # Reusable JSON templates
| +-- standard-prompt-settings.json
| +-- standard-ai-models.json
+-- template-entities/
+-- .mj-sync.json # Defines entity: "Templates"
+-- email/
| +-- .mj-folder.json
| +-- .welcome.json
| +-- welcome.template.html
+-- reports/
+-- .mj-folder.json
+-- .invoice.json
+-- invoice.template.html
Each JSON file contains one record:
{
"fields": { ... },
"relatedEntities": { ... }
}
JSON files can contain arrays of records:
[
{
"fields": { ... },
"relatedEntities": { ... }
},
{
"fields": { ... },
"relatedEntities": { ... }
}
]
This is useful for grouping related records in a single file, reducing file clutter for entities with many small records, and maintaining logical groupings while using @file: references for large content.
{
"fields": {
"Name": "Customer Greeting",
"Description": "Friendly customer service greeting",
"TypeID": "@lookup:AI Prompt Types.Name=Chat",
"CategoryID": "@lookup:AI Prompt Categories.Name=Customer Service",
"Temperature": 0.7,
"MaxTokens": 1000,
"Prompt": "@file:greeting.prompt.md",
"Notes": "@file:../shared/notes/greeting-notes.md",
"SystemPrompt": "@url:https://raw.githubusercontent.com/company/prompts/main/system/customer-service.md"
},
"primaryKey": {
"ID": "550e8400-e29b-41d4-a716-446655440000"
},
"sync": {
"lastModified": "2024-01-15T10:30:00Z",
"checksum": "sha256:abcd1234..."
}
}
{
"fields": {
"Name": "Customer Service Chat",
"Description": "Main customer service prompt",
"TypeID": "@lookup:AI Prompt Types.Name=Chat",
"TemplateText": "@file:customer-service.md",
"Status": "Active"
},
"relatedEntities": {
"MJ: AI Prompt Models": [
{
"fields": {
"PromptID": "@parent:ID",
"ModelID": "@lookup:AI Models.Name=GPT 4.1",
"VendorID": "@lookup:MJ: AI Vendors.Name=OpenAI",
"Priority": 1,
"Status": "Active"
},
"primaryKey": {
"ID": "BFA2433E-F36B-1410-8DB0-00021F8B792E"
},
"sync": {
"lastModified": "2025-06-07T17:18:31.687Z",
"checksum": "a642ebea748cb1f99467af2a7e6f4ffd3649761be27453b988af973bed57f070"
}
},
{
"fields": {
"PromptID": "@parent:ID",
"ModelID": "@lookup:AI Models.Name=Claude 4 Sonnet",
"Priority": 2,
"Status": "Active"
}
}
]
},
"primaryKey": {
"ID": "C2A1433E-F36B-1410-8DB0-00021F8B792E"
},
"sync": {
"lastModified": "2025-06-07T17:18:31.698Z",
"checksum": "7cbd241cbf0d67c068c1434e572a78c87bb31751cbfe7734bfd32f8cea17a2c9"
}
}
{
"primaryKey": {
"UserID": "550e8400-e29b-41d4-a716-446655440000",
"RoleID": "660f9400-f39c-51e5-b827-557766551111"
},
"fields": {
"GrantedAt": "2024-01-15T10:30:00Z",
"GrantedBy": "@lookup:Users.Email=admin@company.com",
"ExpiresAt": "2025-01-15T10:30:00Z",
"Notes": "@file:user-role-notes.md"
},
"sync": {
"lastModified": "2024-01-15T10:30:00Z",
"checksum": "sha256:abcd1234..."
}
}
| Key | Purpose |
|---|---|
fields |
Entity field values |
relatedEntities |
Embedded related entity records |
primaryKey |
Record identifier |
sync |
Sync metadata (lastModified, checksum) |
__mj_sync_notes |
System-managed resolution tracking |
deleteRecord |
Deletion directive |
Any key that is not one of the reserved keys above is preserved but ignored during sync operations. By convention, use an underscore prefix (_) for comment keys:
{
"_comments": [
"This file configures encryption settings for the Test Tables entity"
],
"fields": {
"Name": "Test Tables",
"BaseView": "vwTestTables"
},
"primaryKey": {
"ID": "0fde4c2c-26b1-45e9-b504-5d4a6f4201cf"
}
}
The tool supports special reference types that can be used in any field that accepts text content. These references are processed during push/pull operations to handle external content, lookups, and environment-specific values.
flowchart TD
subgraph References["@ Reference Types"]
FILE["@file:\nExternal file content"]
URL["@url:\nRemote URL content"]
LOOKUP["@lookup:\nEntity ID resolution"]
PARENT["@parent:\nParent record field"]
ROOT["@root:\nRoot record field"]
ENV["@env:\nEnvironment variable"]
INCLUDE["@include\nJSON composition"]
end
subgraph External["External Resources"]
style External fill:#2d8659,stroke:#1a5c3a,color:#fff
EF["Local Files"]
UR["Remote URLs"]
end
subgraph Database["Database Lookups"]
style Database fill:#2d6a9f,stroke:#1a4971,color:#fff
EL["Entity Records"]
end
subgraph Context["Record Context"]
style Context fill:#b8762f,stroke:#8a5722,color:#fff
PR["Parent Entity"]
RR["Root Entity"]
end
subgraph Runtime["Runtime Values"]
style Runtime fill:#7c5295,stroke:#563a6b,color:#fff
EV["Environment Vars"]
end
FILE --> EF
URL --> UR
LOOKUP --> EL
PARENT --> PR
ROOT --> RR
ENV --> EV
INCLUDE --> EF
style References fill:#64748b,stroke:#475569,color:#fff
When a field value starts with @file:, the tool will:
@include directives within themExamples:
@file:greeting.prompt.md -- File in same directory as JSON@file:./shared/common-prompt.md -- Relative path@file:../templates/standard-header.md -- Parent directory reference@file:spec.json -- JSON file with @include directives (processed automatically)When a field value starts with @url:, the tool will:
Examples:
@url:https://example.com/prompts/greeting.md -- Remote content@url:https://raw.githubusercontent.com/company/prompts/main/customer.md -- GitHub raw contentEnable entity relationships using human-readable values:
@lookup:EntityName.FieldName=Value@lookup:EntityName.Field1=Value1&Field2=Value2@lookup:EntityName.FieldName=Value?create@lookup:EntityName.FieldName=Value?create&Field2=Value2@lookup:EntityName.FieldName=Value?allowDefer@lookup:EntityName.FieldName=Value?create&allowDeferExamples:
@lookup:AI Prompt Types.Name=Chat -- Single field lookup, fails if not found@lookup:Users.Email=john@example.com&Department=Sales -- Multi-field lookup for precise matching@lookup:AI Prompt Categories.Name=Examples?create -- Creates if missing@lookup:AI Prompt Categories.Name=Examples?create&Description=Example prompts -- Creates with description@lookup:Dashboards.Name=Data Explorer?allowDefer -- Defers lookup if not found, retries at end of pushWhen you need to match records based on multiple criteria, use the multi-field syntax:
{
"CategoryID": "@lookup:AI Prompt Categories.Name=Actions&Status=Active",
"ManagerID": "@lookup:Users.Email=manager@company.com&Department=Engineering&Status=Active"
}
The ?allowDefer flag enables handling of circular dependencies between entities during push operations. Use this when Entity A references Entity B and Entity B references Entity A -- or any situation where a lookup target might not exist yet during initial processing.
How it works:
The flag is permission-based, not imperative. The lookup is always attempted first, and only deferred if it fails:
flowchart TD
A["@lookup:Entity.Field=Value?allowDefer"] --> B{Try lookup now}
B -->|Found| C[Return ID immediately]
B -->|Not found| D{Has ?allowDefer?}
D -->|Yes| E[Skip this field, continue processing]
D -->|No| F[Push fails - transaction rolled back]
E --> G[Save record without deferred field]
G --> H[Queue record for re-processing]
H --> I[Phase 2.5: Re-process entire record]
I -->|Success| J[Update record with resolved field]
I -->|Failure| F
style C fill:#2d8659,stroke:#1a5c3a,color:#fff
style F fill:#b8762f,stroke:#8a5722,color:#fff
style J fill:#2d8659,stroke:#1a5c3a,color:#fff
style I fill:#7c5295,stroke:#563a6b,color:#fff
When to use ?allowDefer:
Processing phases:
?allowDefer fails (record not found), the field is skipped but the record still savesExample: Application / Dashboard circular reference
The Applications entity can have DefaultNavItems (a JSON field) that contains nested references to Dashboards, while Dashboards have an ApplicationID that references Applications.
Since Applications are processed before Dashboards (alphabetical order), the Dashboard lookup in DefaultNavItems needs ?allowDefer:
// .data-explorer-application.json
{
"fields": {
"Name": "Data Explorer",
"DefaultNavItems": [
{
"Label": "Explorer",
"ResourceType": "Dashboard",
"RecordID": "@lookup:Dashboards.Name=Data Explorer?allowDefer"
}
]
}
}
// .data-explorer-dashboard.json
// Note: No ?allowDefer needed - Applications are processed first
{
"fields": {
"Name": "Data Explorer",
"ApplicationID": "@lookup:Applications.Name=Data Explorer"
}
}
Combining flags:
You can combine ?allowDefer with ?create:
"CategoryID": "@lookup:Categories.Name=New Category?create&allowDefer"
This means: "Look up the category, create if missing, and if the lookup still fails for some reason, defer it."
Important notes:
Reference fields from the immediate parent entity in embedded collections:
@parent:ID -- Get the parent's ID field@parent:Name -- Get the parent's Name fieldReference fields from the root entity in nested structures:
@root:ID -- Get the root entity's ID@root:CategoryID -- Get the root's CategoryIDReference fields synchronously in-memory from the owner record for children nested inside collections, embeds, or extensions:
@owner:ShipToPersonID -- Copies the owner's ShipToPersonID value into a nested child or subtype extension field without database queries.Read a value from an environment variable at push time:
@env:VARIABLE_NAME — push fails when the variable is not set.@env:VARIABLE_NAME?optional — when the variable is not set, push leaves the field out: an update keeps the database value, a create gets the column default, and push gives a warning. Not allowed in primaryKey.defaults (.mj-sync.json or .mj-folder.json), an unset ?optional variable skips that one default, with a warning. It does not fall back to a default from a higher folder or from the entity .mj-sync.json.?optional variable inside a JSON object field leaves out the whole field.Pull keeps the reference. When a file holds @env:VAR, mj sync pull writes the reference back, not the database value, so the resolved value is not written to the file. If VAR is set and its value differs from the database, pull warns (it never prints either value). This applies to:
"Token": "@env:VAR"), also with --multi-file;@file:x.json, pull updates x.json and keeps the @file: reference (not when x.json uses @include or a nested @file:; pull then warns and leaves x.json as it is);@lookup: criteria or create fields that use @env: (@lookup:Entity.Field=@env:VAR). Pull keeps the lookup. When the record that the database value points to is missing or does not match the criteria, pull warns (it does not check when a criteria variable is not set);collections, embeds and extension items;"primaryKey": { "ID": "@env:VAR" }). Pull finds the existing record by the variable's value. When a key variable of a top-level record is not set, pull stops before it writes any file. When the key variable of a relatedEntities child, a collections child or an embed is not set, pull also stops, with an error that names the file and the variable.When the database value of a JSON field that holds @env: is not JSON, pull keeps the file value and warns.
Pull leaves out a read-only field (one that push ignores) whose database value holds the value of an @env: reference in the record file, such as a key path that the database builds from an @env: key. A UUID value matches without case; another value matches when it has 6 or more characters. When the existing file already holds such a field, pull removes it from the file, with merge too. With -v, pull names the field and the variable, never the value.
With --multi-file, pull also finds an existing record in the folder's other record files, so it keeps the @env: references of that record in the multi-file. As in a single-file pull, a record file that cannot be read, or an unset key variable in any of them, stops pull before it writes.
?optional does not work inside @lookup:, because ? starts the lookup flags. Validation and push report it as an error.
Push does not print values that come from @env:. The --verbose diff and the --change-detail report show <hidden> for the old value and <from @env:VAR> for the new one. Save errors replace each value of 6 or more characters with <from @env:VAR>; a shorter value can print. On create, push keeps an @env: primaryKey reference in the file when the database saved the variable's value as the key. Push also resolves an @env: primaryKey of a collections child (to find the existing child, or to set the key of a new one) and of a record to delete; an unset key variable stops push with an error that names the variable. When a variable that a record's fields or primaryKey uses is not set (and the reference is not ?optional), push stops with an error that names the record file and the line where the record starts, the entity, the field and the variable, for example /abs/path/.tags.json:12: MJ: Tags primaryKey.ID uses @env:MJ_KEY_ID, which is not set. With SQL logging on, push leaves the SET line of an env-sourced field out of the log and writes -- <Entity>.<Field>: not recorded, value comes from @env:VAR in its place. A replayed update then keeps the target's value. A replayed create gets the column default, or fails if the column is NOT NULL with no default. This works for SQL Server logs only; other providers keep the value in the log, and push gives the names-only warning. A primary key from @env: stays in the log; push warns with names only. When a value of 6 or more characters is still in the log, push warns with the variable name.
mj sync watch resolves @env: as push does, for the parts of a record file that watch handles. See Watch.
Query: … with its parameters, logged by the provider) can print the SQL text with values. MetadataSync's own output masks these values (values of 6 or more characters), but the MJCore and provider logs do not.mj sync watch writes record-change data to its SQL log. On an update, this data can hold the OLD database value of an env-sourced field. The SQL-log warnings cannot detect old values. Check a watch log before you share it.@env: reference that is only inside a file brought in by @include, or by a nested @file: inside a side file, is not kept. Pull writes the database value there.externalizeFields side file (Field.sub) that you write by hand with @env: references inside is not kept. Pull writes the database value there. Pull itself never moves a sub-object that holds @env: into a side file.@file: JSON side file that pull rewrites, and a file that externalizeFields writes, are not part of the pull backup. A failed pull (for example, an unset key variable on a child) does not roll them back. A side file holds @env: references, never their values. An externalized file holds database values. It does not hold the value of an @env: reference that is in the record file, but it can hold one when the reference is only in a nested @file: or @include file (see above).@env: references.@file: JSON side file to keep its @env: leaves, the other reference leaves in that file (such as @lookup: or @file:) get the database value.mj sync status does not resolve an @env: primaryKey. It counts such a record as deleted. It prints no value.@env: primaryKey, push does not find the records that refer to it only in the database, so it does not list them. If such a record exists, the delete fails with a foreign-key error, or, for an entity with cascade deletes, the database removes or changes those records. MetadataSync masks values of 6 or more characters in the error text.collections child that has no primaryKey in the file is not matched on pull, so its @env: fields get the database value. Give each child its key. Push writes keys back only for top-level records.?optional on an @env: value inside a lookup's create fields (?create&Code=@env:C?optional) is not an error. When C is not set, push leaves the whole foreign-key field out of the save, with a warning.pushWhen)An entity folder's .mj-sync.json can gate the whole folder on an environment variable:
{ "entity": "MJ: AI Agents", "pushWhen": "@env:DEMO_AGENTS_ENABLED" }
true or 1 (any case, surrounding spaces ignored).Skipped folder <name>: <VAR> is not set (or is not true), and continues. --format=json lists them in data.skippedDirectories.pushWhen.@env:VAR. pushWhen in the root .mj-sync.json is a validation error.pushWhen applies to the folders push runs over: the entity folders directly under the root, or the --dir folder. A pushWhen in a nested entity folder (for example actions/runtime) has no effect unless --dir names that folder.mj sync push does not validate a folder it will skip. mj sync validate validates every folder, and warns about a pushWhen in a nested folder.mj sync watch reads pushWhen too. See Watch.collections, embeds, extension)MetadataSync natively supports three composition axes directly within RecordData, persisted in a single Save() call on the root entity graph:
collections: Declared RelatedRecordCollection sets (e.g. Lines, AgendaItems). Supports upsert mode (default: match by PK, leaves unmentioned items intact) and authoritative mode (configured in .mj-sync.json: computes deletion set, enforced with a 20% max implied delete safety rail unless --allow-bulk-delete is passed).embeds: 1:1 peer record embeds keyed by foreign key field name (e.g. ShipToAddressID). Peer records are ensured and assigned before parent entity save.extension: 1:1 IsA subtype extensions (e.g. Event Order Lines extending Order Lines). Leaf fields only (parent fields in leaf extensions are rejected during validation). Resolved automatically via prospective subtype resolution (EnsureISAChild()).The tool automatically detects primary key fields from entity metadata:
{"ID": "value"} or {"CustomKeyName": "value"}@lookup, @parent, and other reference typesYou can use @lookup references in primary key fields to avoid hardcoding GUIDs. This is especially useful when decorating existing records:
{
"fields": {
"Encrypt": true,
"AllowDecryptInAPI": false
},
"primaryKey": {
"ID": "@lookup:Entity Fields.EntityID=@lookup:Entities.Name=Test Tables&Name=ServerOnlyEncrypted"
}
}
In this example:
@lookup:Entities.Name=Test Tables resolves to the Entity IDEntityID and NameNote: Primary key lookups must resolve immediately -- the ?allowDefer flag is not supported in primary key fields since the primary key is needed to determine if a record exists.
When a field value is an array or object, the tool automatically:
@lookup:, @file:, @parent:, @root: references inside the objectThis is useful for JSON-typed fields like Configuration, Settings, Metadata, etc.
{
"fields": {
"Name": "Agent Memory Manager Job",
"CronExpression": "0 */15 * * * *",
"Configuration": {
"AgentID": "@lookup:AI Agents.Name=Memory Manager",
"InitialMessage": "Analyze recent conversations",
"Settings": {
"MaxNotes": 5,
"Strategy": "Relevant",
"TargetAgentID": "@lookup:AI Agents.Name=Sage"
}
}
}
}
When pushed to the database, the Configuration field is resolved (lookups become GUIDs) and stringified as JSON for storage.
The tool supports deleting records from the database using a special deleteRecord directive in JSON files:
{
"fields": {
"Name": "Obsolete Prompt",
"Description": "This prompt is no longer needed"
},
"primaryKey": {
"ID": "550e8400-e29b-41d4-a716-446655440000"
},
"deleteRecord": {
"delete": true
}
}
After successfully deleting the record, the tool updates the JSON file with a deletedAt timestamp. Important notes:
primaryKey to identify which record to deletedeletedAt is set, the deletion will not be attempted again--dry-run to preview what would be deleteddeleteRecord is present, normal create/update operations are skipped{@include} References in FilesEnable content composition within non-JSON files (like .md, .html, .txt) using JSDoc-style include syntax:
{@include path/to/file.ext}@file: references# My Prompt Template
## System Instructions
{@include ./shared/system-instructions.md}
## Context
{@include ../common/context-header.md}
## Task
Please analyze the following...
Enable modular JSON composition by including external JSON files directly into your metadata files:
Object Context -- Property Spreading (Default)
{
"name": "Parent Record",
"@include": "child.json",
"description": "Additional fields"
}
Multiple Includes with Dot Notation (Eliminates VS Code Warnings)
{
"name": "Parent Record",
"@include.data": "shared/data-fields.json",
"description": "Middle field",
"@include.config": "shared/config-fields.json",
"status": "Active"
}
Use dot notation (@include.anything) to include multiple files at different positions in your object. The part after the dot is ignored by the processor but makes each key unique, eliminating VS Code's duplicate key warnings.
Array Context -- Element Insertion
[
{"name": "First item"},
"@include:child.json",
{"name": "Last item"}
]
Explicit Mode Control
{
"@include": {
"file": "child.json",
"mode": "spread"
}
}
"child.json", "./child.json", "../shared/base.json", "subfolder/config.json"@include directives are processed first (recursively)@file references are recursively resolved (including nested ones in JSON)@ references (@lookup, etc.)This ensures that included content can contain other special references that will be properly resolved.
mj sync push does not support @template:. Push saves the literal text (for example @template:templates/settings.json), not the content of the file. mj sync validate reports a field value that starts with @template: as an error. Use @file: instead.
The tool implements a cascading inheritance system for field defaults, similar to CSS or OOP inheritance:
.mj-sync.json) -- Base defaults for all records.mj-folder.json) -- Every .mj-folder.json from the sync root down to the folder of the record file. Any of them, the root's too, wins over the entity's .mj-sync.json defaults.mj-folder.json wins over a higher onePush and mj sync watch apply these defaults. See Folder Defaults for the details.
flowchart TD
ELD[".mj-sync.json\nTemperature: 0.7\nMaxTokens: 1500"] --> FD1[".mj-folder.json\nTemperature: 0.8\n(overrides entity)"]
FD1 --> FD2[".mj-folder.json\nTemperature: 0.6\n(overrides parent)"]
FD2 --> REC["urgent.json\nTemperature: 0.9\n(overrides all)"]
ELD -..->|inherited| FD1
FD1 -..->|inherited| FD2
FD2 -..->|inherited| REC
FINAL["Final Values:\nTemperature: 0.9 (record)\nMaxTokens: 1500 (entity)"]
REC --> FINAL
style ELD fill:#2d6a9f,stroke:#1a4971,color:#fff
style FD1 fill:#7c5295,stroke:#563a6b,color:#fff
style FD2 fill:#7c5295,stroke:#563a6b,color:#fff
style REC fill:#2d8659,stroke:#1a5c3a,color:#fff
style FINAL fill:#b8762f,stroke:#8a5722,color:#fff
ai-prompts/.mj-sync.json -> Temperature: 0.7, MaxTokens: 1500
+-- customer-service/.mj-folder.json -> Temperature: 0.8 (overrides)
| +-- greeting.json -> Uses Temperature: 0.8, MaxTokens: 1500
| +-- escalation/.mj-folder.json -> Temperature: 0.6 (overrides again)
| +-- urgent.json -> Temperature: 0.9 (record override)
__mj_sync_notesWhen you use @lookup or @parent references in your metadata files, MetadataSync can track how these references were resolved during push operations. This information is written to a __mj_sync_notes key in each record, providing transparency into the resolution process.
Note: This feature is disabled by default to keep metadata files clean. Enable it when you need to debug lookup resolutions or understand how references are being resolved.
Add emitSyncNotes to your .mj-sync.json configuration:
Root-level configuration (applies to all entity directories):
{
"version": "1.0",
"emitSyncNotes": true,
"directoryOrder": ["..."]
}
Entity-level override (in an entity directory's .mj-sync.json):
{
"entity": "AI Prompts",
"emitSyncNotes": true
}
The inheritance works as follows:
emitSyncNotes takes precedence if explicitly set.mj-sync.jsonfalse if not set anywhere{
"fields": {
"Name": "ServerOnlyEncrypted",
"Encrypt": true,
"EncryptionKeyID": "@lookup:MJ: Encryption Keys.Name=Test Encryption Key"
},
"primaryKey": {
"ID": "@lookup:Entity Fields.EntityID=@lookup:Entities.Name=Test Tables&Name=ServerOnlyEncrypted"
},
"sync": {
"lastModified": "2025-12-25T16:14:32.605Z",
"checksum": "7e989e08396f6cffb8b2d70958018b21..."
},
"__mj_sync_notes": [
{
"type": "lookup",
"field": "primaryKey.ID",
"expression": "@lookup:Entity Fields.EntityID=@lookup:Entities.Name=Test Tables&Name=ServerOnlyEncrypted",
"resolved": "F501E294-5F5F-44C6-AD06-5C9754A13D29",
"nested": [
{
"expression": "@lookup:Entities.Name=Test Tables",
"resolved": "0fde4c2c-26b1-45e9-b504-5d4a6f4201cf"
}
]
},
{
"type": "lookup",
"field": "fields.EncryptionKeyID",
"expression": "@lookup:MJ: Encryption Keys.Name=Test Encryption Key",
"resolved": "85B814C8-A01B-4AE3-A252-DC9D54C914C7"
}
]
}
| Field | Description |
|---|---|
type |
Resolution type: "lookup" for @lookup references, "parent" for @parent references |
field |
Field path where the resolution occurred (e.g., "primaryKey.ID", "fields.CategoryID") |
expression |
The original reference expression before resolution |
resolved |
The resolved value (typically a GUID) |
nested |
(Optional) Array of nested resolutions for expressions containing nested @lookup references |
The __mj_sync_notes key uses a double underscore prefix (__) to clearly indicate it is system-managed. Do not manually edit this section -- it is regenerated on each push when emitSyncNotes is enabled.
mj sync push writes through BaseEntity.Save(), so an entity's custom validation, Save()
override and lifecycle hooks run only if the class that carries them is registered in the CLI
process. MJ core's server subclasses always are (the CLI imports the lite manifest). An Open App's
classes, and a host's own generated entities, are loaded by the mj prerun hook through
@memberjunction/dynamic-packages from mj.config.cjs (dynamicPackages.server[],
codeGeneration.packages) and from an mj-app.json beside the config — before the push opens the
database provider.
When no subclass is registered for an entity being pushed, the push warns once per entity
("No entity subclass is registered for 'X' … records will be written with the generic BaseEntity")
on both the directory-push path and the lookup auto-create path; the warning lands in the result
envelope's warnings and never changes the exit code. --no-app-packages (or
MJ_DYNAMIC_PACKAGES=none) skips the app packages deliberately — restoring a dump, bulk ingestion
without side effects. Details and troubleshooting:
guides/DYNAMIC_PACKAGE_LOADING_GUIDE.md.
All MetadataSync functionality is accessed through the MemberJunction CLI (mj) under the sync namespace:
# Initialize a directory for metadata sync
mj sync init
# Validate metadata files
mj sync validate
mj sync validate --dir="./metadata"
mj sync validate --verbose
mj sync validate --format=json
mj sync validate --save-report
# Pull metadata from database to files
mj sync pull --entity="AI Prompts"
mj sync pull --entity="AI Prompts" --filter="CategoryID='customer-service-id'"
mj sync pull --entity="AI Prompts" --multi-file="all-prompts"
# Push local changes to database
mj sync push
mj sync push --dir="ai-prompts"
mj sync push -v
mj sync push --dry-run
mj sync push --isolated-transactions --parallel-batch-size=20 # parallel, NOT rolled back on failure
# Directory filtering
mj sync push --exclude="actions"
mj sync push --exclude="actions,templates"
mj sync push --exclude="*-test,*-old"
mj sync push --include="prompts,agent-types"
mj sync push --include="ai-*"
# Status and watch
mj sync status
mj sync watch
# CI/CD mode
mj sync push --ci
# Skip validation
mj sync push --no-validate
# Reset file checksums
mj sync file-reset
# Incremental push (skip unchanged files and records)
mj sync push --incremental
mj sync push --dir ./metadata --incremental --verbose
# Incremental pull (only records updated since last pull)
mj sync pull --incremental
# Pull records updated after a specific timestamp
mj sync pull --entity "MJ: AI Prompts" --since "2026-04-07T10:00:00Z"
--dir can name the metadata root or one entity folder. For an entity folder, push walks up through parent folders that have a .mj-sync.json and reads its root settings from the first one that has no entity; if it finds none, it uses the folder's own file. --include and --exclude are not applied to a single entity folder. With --incremental, an entity-folder push shares the root's state file.
--incremental)Skips unchanged files and records using stored checksums, significantly speeding up repeated pushes when only a few files have changed.
--incremental establishes baseline checksums (runs at normal speed)sync.checksum already present in metadata files)~/.mj/sync-state/ (machine-local, not committed to version control)mj sync push --dir ./metadata --incremental --verbose--incremental compares file content only. After you set a variable that ?optional left out, push once without --incremental.--incremental)Only pulls records updated since the last successful pull, avoiding a full re-pull of every record.
__mj_UpdatedAt column (maintained by database triggers on all tracked entities)__mj_DeletedAt) and removes their local files (with --multi-file, their entries in the file)~/.mj/sync-state/--since)Explicit alternative to --incremental -- pulls only records updated after a given ISO timestamp.
mj sync pull --entity "MJ: AI Prompts" --since "2026-04-07T10:00:00Z"
mj sync watch)mj sync watch pushes a record file each time it changes.
directoryOrder, the root ignoreDirectories, and pushWhen. It reads pushWhen once when it starts and does not watch a skipped folder. When every folder is skipped, it stops with No entity directories to watch: pushWhen skips every folder. It has no --include or --exclude flag.filePattern (default *.json, dot files included), without node_modules and .mj-*.json files. It also watches .md, .txt, .html, .liquid and .sql files, which a record can bring in with @file:.fields and primaryKey of a file that holds one record. It does not handle relatedEntities, collections, embeds or extension. A file that holds an array of records fails. A relative @file: in a record field resolves from the folder of the record file (push resolves it from the entity folder).@env:. It applies folder defaults, @env: references, ?optional and an @env: primary key as push does. As in push, a name in fields or in defaults that matches an entity field without regard to case is set with the entity's spelling. Another name in fields is set as written, so a virtual property such as TemplateText works. Another default name applies only when the entity has a property with that exact name. With -v, it shows <hidden> / <from @env:VAR> in place of env values. Its SQL log leaves env-sourced fields out of the SQL statements, with the same names-only warnings as push. The record-change data in its log is not masked and can hold old database values (see Known limits).push.autoCreateMissingRecords.@env: key fields from the file. A literal key gets a new value from the entity (a new UUID), and watch writes that value over the file key. Push uses the file key.push.alwaysPush.The following improvements apply automatically with no flags required:
Lazy embedding model loading -- The AIEngine no longer loads the embedding model (~50 MB Xenova/all-mpnet-base-v2) at startup. The model loads on the first semantic search call (FindSimilarAgents, etc.). CLI commands that never use semantic search (sync, codegen) skip the ~8 s model load entirely. API server behavior is unchanged -- embeddings generate on the first search request.
Indexed batch context lookups -- Push operations resolve @lookup references using an indexed data structure instead of a linear scan, reducing lookup cost from O(N) to O(1).
Batched pull queries -- Pull operations pre-fetch related entities with a single IN query per type, replacing the previous N+1 query pattern (one query per parent record per related type).
The tool uses the existing mj.config.cjs for database configuration and a hierarchical structure of .mj-sync.json and .mj-folder.json files for sync behavior.
{
"version": "1.0.0",
"directoryOrder": [
"prompts",
"agent-types"
],
"ignoreDirectories": [
"output",
"examples"
],
"push": {
"validateBeforePush": true,
"requireConfirmation": true,
"autoCreateMissingRecords": false,
"alwaysPush": false,
"isolatedTransactions": false
},
"sqlLogging": {
"enabled": true,
"outputDirectory": "./sql_logging",
"formatAsMigration": false,
"filterPatterns": ["*EntityFieldValue*"],
"filterType": "exclude"
},
"userRoleValidation": {
"enabled": true,
"allowedRoles": ["Administrator", "Developer"],
"allowUsersWithoutRoles": false
},
"watch": {
"debounceMs": 1000,
"ignorePatterns": ["*.tmp", "*.bak"]
},
"emitSyncNotes": false
}
{
"entity": "AI Prompts",
"filePattern": "*.json",
"defaults": {
"TypeID": "@lookup:AI Prompt Types.Name=Chat",
"Temperature": 0.7,
"MaxTokens": 1500,
"Status": "Active"
},
"pull": {
"filePattern": "*.json",
"updateExistingRecords": true,
"createNewFileIfNotFound": true,
"mergeStrategy": "merge",
"filter": "Status = 'Active'",
"externalizeFields": [
{
"field": "Prompt",
"pattern": "@file:{Name}.prompt.md"
}
],
"relatedEntities": {
"MJ: AI Prompt Models": {
"entity": "MJ: AI Prompt Models",
"foreignKey": "PromptID",
"filter": "Status = 'Active'"
}
}
}
}
{
"defaults": {
"CategoryID": "@lookup:AI Prompt Categories.Name=Customer Service",
"Temperature": 0.8,
"Tags": ["customer-service", "support"]
}
}
Push and mj sync watch apply .mj-folder.json defaults from the sync root down to the folder of the record file. A deeper folder wins over a higher one, and a field in the record wins over every default. A .mj-folder.json default (the root's too) wins over a default in the entity's .mj-sync.json. .mj-folder.json holds only defaults; the folder's entity comes from .mj-sync.json. Push applies them to records nested in a file (relatedEntities) too; watch does not handle nested records. A default applies when the nested record's entity has a field with that name. Default and record field names match the entity's fields without regard to case. A default whose name is not an entity field applies only when the entity has a property with that exact name, such as the virtual property TemplateText. Validation reports another spelling of such a name as an unknown field.
@file: in a default resolves from the folder of the record file, not from the folder of the .mj-folder.json.@env:VAR?optional variable in a default skips that one default, with a warning. It does not fall back to a default from a higher folder or from the entity .mj-sync.json.When set to true, the push command automatically creates records when a primaryKey is specified but the record does not exist in the database. Useful when migrating data between environments or restoring records from backups.
{
"push": {
"autoCreateMissingRecords": true
}
}
When set to true, forces ALL records to be saved to the database regardless of their dirty state. This bypasses the normal dirty checking mechanism.
Use cases:
{
"push": {
"alwaysPush": true
}
}
Note: This flag should be used judiciously as it causes database writes for all records. Enable temporarily when needed, then disable for normal operations.
Whether each JSON-root graph gets its own connection and transaction. Defaults to false, and can be set per entity directory as well as at the root.
false: all-or-nothing. Every create, update and delete runs in one database transaction, one graph at a time. If anything fails, nothing is saved and the metadata files are restored.true: sibling graphs run in parallel (--parallel-batch-size) on their own connections, and each create and update is committed as soon as it is saved. A failure does not roll those back.An entity's own .mj-sync.json overrides the root, and the CLI flags --isolated-transactions and --no-isolated-transactions override both, in either direction. See Atomic and Isolated Pushes.
// root .mj-sync.json — the default for this tree
{ "push": { "isolatedTransactions": false } }
// orders/.mj-sync.json — this entity opts in
{ "entity": "MJ_BizApps_Orders: Orders", "push": { "isolatedTransactions": true } }
Choose it for throughput on entities whose partial writes are acceptable. Nested transaction scopes inside a save work either way: on the shared connection they become savepoints.
Directory order is configured in the root-level .mj-sync.json file only (not inherited by subdirectories):
{
"version": "1.0.0",
"directoryOrder": [
"prompts",
"agent-types"
]
}
directoryOrder are processed first, in the specified orderIgnore directories are configured in .mj-sync.json files and are cumulative through the directory hierarchy:
{
"version": "1.0.0",
"ignoreDirectories": [
"output",
"examples",
"templates"
]
}
Child directories inherit parent ignore patterns and can add their own.
SQL logging captures all database operations during push commands for creating migration files, debugging, and deployment:
{
"sqlLogging": {
"enabled": true,
"outputDirectory": "./sql_logging",
"formatAsMigration": true,
"filterPatterns": ["*AIPrompt*", "/^EXEC sp_/i"],
"filterType": "exclude"
}
}
| Option | Type | Default | Description |
|---|---|---|---|
enabled |
boolean | false | Enable SQL logging during push operations |
outputDirectory |
string | "./sql_logging" | Directory for SQL log files |
formatAsMigration |
boolean | false | Format as Flyway migration files |
filterPatterns |
string[] | undefined | Patterns to filter SQL statements |
filterType |
"exclude" or "include" | "exclude" | How to apply filter patterns |
Filter Pattern Types:
/ and optionally end with flags (e.g., /spCreate.*Run/i)* as a wildcard (e.g., *AIPrompt*)Validates UserID fields against specific roles in the MemberJunction system:
{
"userRoleValidation": {
"enabled": true,
"allowedRoles": [
"Administrator",
"Developer",
"Content Manager"
],
"allowUsersWithoutRoles": false
}
}
| Option | Type | Default | Description |
|---|---|---|---|
enabled |
boolean | false | Enable user role validation |
allowedRoles |
string[] | [] | Role names that are allowed |
allowUsersWithoutRoles |
boolean | false | Allow users without any assigned roles |
The pull command supports smart update capabilities with extensive configuration options:
{
"entity": "AI Prompts",
"filePattern": "*.json",
"pull": {
"filePattern": "*.json",
"createNewFileIfNotFound": true,
"newFileName": ".all-new.json",
"appendRecordsToExistingFile": true,
"updateExistingRecords": true,
"preserveFields": ["customField", "localNotes"],
"mergeStrategy": "merge",
"backupBeforeUpdate": true,
"filter": "Status = 'Active'",
"externalizeFields": [
{
"field": "TemplateText",
"pattern": "@file:{Name}.template.md"
}
],
"excludeFields": ["InternalID", "TempField"],
"lookupFields": {
"CategoryID": {
"entity": "AI Prompt Categories",
"field": "Name"
}
},
"relatedEntities": {
"MJ: AI Prompt Models": {
"entity": "MJ: AI Prompt Models",
"foreignKey": "PromptID",
"filter": "Status = 'Active'"
}
},
"ignoreNullFields": false,
"ignoreVirtualFields": false
},
"push": {
"skipGeoCoding": true
}
}
push.skipGeoCoding maps to EntitySaveOptions.SkipGeoCoding for this entity only. Use it on display-only geo entities (People/Organizations whose coords are virtual PrimaryAddressLatitude). Do not use a global CLI --skip-geocode as the only control. Addresses with native lat/lng already set do not call the provider even without this flag.
A push is atomic by default, so every save runs inside the push transaction. Durable After* entity actions without a queue submitter wait until the push transaction ends before they run. In a directory using isolated transactions each graph runs on its own CreateIndependentInstance(), and those actions run once that instance's own save commits. See Atomic and Isolated Pushes.
| Option | Type | Default | Description |
|---|---|---|---|
filePattern |
string | Entity filePattern | Pattern for finding existing files |
createNewFileIfNotFound |
boolean | true | Create files for records not found locally |
newFileName |
string | -- | Filename for new records when appending |
appendRecordsToExistingFile |
boolean | false | Append new records to a single file |
updateExistingRecords |
boolean | true | Update existing records in local files |
preserveFields |
string[] | [] | Fields that retain local values during updates |
mergeStrategy |
string | "merge" | How to merge: "merge", "overwrite", or "skip" |
backupBeforeUpdate |
boolean | false | Back up every file a pull rewrites. The backups restore the files if the pull fails and are removed once it succeeds |
backupDirectory |
string | ".backups" | Directory for backup files, relative to the folder of the files being backed up. It must stay inside that folder |
filter |
string | -- | SQL WHERE clause for filtering records |
externalizeFields |
array/object | -- | Fields to save as external files |
excludeFields |
string[] | [] | Fields to omit from pulled data |
lookupFields |
object | -- | Foreign keys to convert to @lookup references |
relatedEntities |
object | -- | Related entities to pull as embedded collections |
ignoreNullFields |
boolean | false | Exclude fields with null values |
ignoreVirtualFields |
boolean | false | Exclude virtual fields (view-only fields) |
merge (default): Combines fields from database and local file, with database values taking precedence for existing fields (except @env: references)overwrite: Completely replaces local record with database version (except preserved fields and @env: references)skip: Leaves existing records unchanged, only adds new recordsmj sync pull flags override the folder's pull settings for one run:
--merge-strategy=merge|overwrite|skip replaces pull.mergeStrategy.--preserve-fields and --exclude-fields add to pull.preserveFields and pull.excludeFields. Repeat the flag or separate names with commas.--update-existing / --no-update-existing sets pull.updateExistingRecords, and --create-new / --no-create-new sets pull.createNewFileIfNotFound. Without the flag, the folder setting applies (default true).--preserve-fields and --exclude-fields apply to the folder's own records, not to related (child) records.
Pull stops before it writes any file when a record file in the folder cannot be read (for example, it is not valid JSON). Otherwise pull would not find the records in that file and would write them again as new files.
--multi-file writes the query result to one file. With --since/--incremental it updates matched entries in place, adds new ones and keeps the rest. It uses --exclude-fields, and --merge-strategy only to decide if an externalized field keeps its existing @file: path. It does not use --preserve-fields, --update-existing or --create-new. An empty or whitespace-only file counts as no records. Pull stops, and does not write, when the file is not valid JSON. With --since/--incremental, pull removes the entries of records that were soft-deleted since the timestamp from the file; it does not delete the file.
With mergeStrategy: "skip", existing records are left as they are and count as skipped; new records are still written.
| excludeFields | preserveFields | |
|---|---|---|
| Purpose | Completely omit fields from files | Protect local values from overwrite |
| Use Case | Remove system/internal fields | Keep custom local modifications |
| Effect | Fields never appear in JSON | Fields exist but retain local values |
| Example | Internal IDs, timestamps | Custom file paths, local notes |
When a preserved field contains a @file: reference, the tool updates the content at the existing file path rather than creating a new file with a generated name.
The externalizeFields configuration supports dynamic file naming with placeholders:
"externalizeFields": [
{
"field": "TemplateText",
"pattern": "@file:{Name}.template.md"
},
{
"field": "SQLQuery",
"pattern": "@file:queries/{CategoryName}/{Name}.sql"
}
]
Supported placeholders:
{Name} -- The entity's name field value{ID} -- The entity's primary key{FieldName} -- The field being externalized{AnyFieldName} -- Any field from the entity recordAll values are sanitized for filesystem compatibility (lowercase, spaces to hyphens, special characters removed).
The ignoreVirtualFields option controls whether virtual fields (computed view-only fields) are included in pulled data.
With ignoreVirtualFields: false (default):
{
"fields": {
"Name": "Test Action",
"CategoryID": "@lookup:Action Categories.Name=System",
"Category": "System",
"UserID": "123",
"User": "John Smith"
}
}
With ignoreVirtualFields: true:
{
"fields": {
"Name": "Test Action",
"CategoryID": "@lookup:Action Categories.Name=System",
"UserID": "123"
}
}
The tool supports managing related entities as embedded collections within parent JSON files:
{
"fields": {
"Name": "Parent Entity"
},
"relatedEntities": {
"Child Entity": [
{
"fields": {
"ParentID": "@parent:ID",
"Name": "Child 1"
},
"relatedEntities": {
"Grandchild Entity": [
{
"fields": {
"ChildID": "@parent:ID",
"RootID": "@root:ID",
"Name": "Grandchild 1"
}
}
]
}
}
]
}
}
Benefits:
The tool supports automatic recursive patterns for self-referencing entities, eliminating the need to manually define each nesting level for hierarchical data structures.
{
"pull": {
"relatedEntities": {
"AI Agents": {
"entity": "AI Agents",
"foreignKey": "ParentID",
"recursive": true,
"maxDepth": 10
}
}
}
}
| Option | Type | Default | Description |
|---|---|---|---|
recursive |
boolean | false | Enable automatic recursive fetching |
maxDepth |
number | 10 | Maximum recursion depth |
When recursive: true is set:
lookupFields, externalizeFields, etc.A push is all-or-nothing by default. One database transaction covers every create, update and delete, and graphs run one at a time inside it.
An entity directory can opt out with push.isolatedTransactions: true, which gives each of its JSON-root graphs its own connection and transaction so siblings run in parallel. That buys throughput and costs atomicity: those creates and updates commit as they are saved and are not rolled back with the push.
| Shared (default) | Isolated (isolatedTransactions: true) |
|
|---|---|---|
| Creates and updates | Host connection, inside the push transaction | One connection per JSON-root graph |
| Graphs at once | 1 | --parallel-batch-size (default 10) |
| A failure anywhere | Rolls back everything that directory wrote, with every other shared directory; metadata files are restored | Rolls back deletes and deferred records only. Creates and updates that already ran stay committed |
| After a failure the push says | rolled back successfully. Nothing from this push was saved. |
Which files and records are still in the database |
The setting resolves per entity directory: the CLI flag wins, then that directory's .mj-sync.json, then the root's, then shared.
# Default: all-or-nothing
mj sync push
# One run with isolation forced on or off, whatever the files say
mj sync push --isolated-transactions
mj sync push --no-isolated-transactions
mj sync push --isolated-transactions --parallel-batch-size=20
A mixed push is not all-or-nothing as a whole. It is all-or-nothing for the shared directories, and best effort for the isolated ones. A failure names the files and records that stayed.
Order isolated directories first in directoryOrder when they touch rows the shared directories also write. The push transaction holds its locks until the end, so an isolated directory running after a shared one can block against the push's own transaction.
flowchart TD
B[Begin push transaction] --> P1[Phase 1: creates and updates]
P1 --> P2[Phase 2: deletes, reverse dependency order]
P2 --> P25[Phase 2.5: deferred records]
P25 --> C{Commit}
C -->|OK| P3[Phase 3: write files with deletions, save incremental state]
C -->|Rejected| RB
P1 -->|Any record error| RB[Roll back, restore files, report what is left]
P2 -->|Any delete error| RB
P25 -->|Any deferred error| RB
style RB fill:#b8762f,stroke:#8a5722,color:#fff
style P3 fill:#2d8659,stroke:#1a5c3a,color:#fff
autoCreateMissingRecords=false. A dry run keeps counting errors instead, since it writes nothing.--incremental) is saved only after the commit, so a failed push never marks its files as pushed.Records are grouped into dependency levels:
Records are also grouped into JSON-root graphs: an Action and its nested Action Params share one graph. Every DB read and write for a graph uses the graph's provider: GetEntityObject, Save, Load, RunView, lookups and RecordGeoCode. That is an ORM invariant (BaseEntity.BindProvider). Mixing the host connection with a graph instance in one tree is a deadlock, because the child FK waits on an uncommitted parent on another pooled connection.
In a shared directory every graph's provider is the host, and graphs run one at a time. The pool refuses to hand the host to a second graph while another graph holds it.
In an isolated directory each graph gets one CreateIndependentInstance().
TransactionDepth is already 0. In the second case Save() has settled, so a fresh instance at the next level is safe.--parallel-batch-size, plus any still-open leftover-depth graphs.BeginTransaction or a nested entity action, is committed at drain on success and rolled back on failure.CreateIndependentInstance fails later, the file is aborted; a graph never falls back to the host.CreateIndependentInstance calls: one per graph per level when Save() settles, so L×N instead of N. If a large isolated push is slow, look here first.The MetadataSync tool includes a comprehensive validation system that checks metadata files for correctness before pushing to the database.
flowchart TD
START["mj sync validate"] --> LOAD["Load .mj-sync.json"]
LOAD --> DIRS["Discover Entity Directories"]
DIRS --> LOOP["For Each Directory"]
LOOP --> ENT["Validate Entity Names"]
LOOP --> FLD["Validate Field Names & Types"]
LOOP --> REF["Validate References\n(@file, @lookup, @env)"]
LOOP --> DEP["Analyze Dependencies"]
ENT --> COLLECT["Collect Errors & Warnings"]
FLD --> COLLECT
REF --> COLLECT
DEP --> COLLECT
COLLECT --> RESULT{Valid?}
RESULT -->|Yes| PASS["Validation Passed"]
RESULT -->|No| FAIL["Validation Failed\n(Errors Listed)"]
style START fill:#2d6a9f,stroke:#1a4971,color:#fff
style PASS fill:#2d8659,stroke:#1a5c3a,color:#fff
style FAIL fill:#b8762f,stroke:#8a5722,color:#fff
style COLLECT fill:#7c5295,stroke:#563a6b,color:#fff
By default, validation runs automatically before push operations:
# These commands validate first, then proceed if valid
mj sync push
mj sync pull --entity="AI Prompts"
mj sync validate
mj sync validate --dir="./metadata"
mj sync validate --verbose
mj sync validate --format=json
mj sync validate --save-report
# Skip validation checks (use with caution)
mj sync push --no-validate
mj sync pull --entity="AI Prompts" --no-validate
Files Validated:
filePattern (default *.json) the way push finds them. Files in sub-folders are included when the pattern matches them (for example **/*.json). .mj-*.json files and files under node_modules are left out. A file that is not valid JSON is an error.ignoreDirectories) are not validated.mj sync push --ci can stop on errors that earlier versions did not find: unknown fields and unknown top-level keys, a __mj_ field or the primary key in fields, @template: values, ?optional in @lookup:, missing @file: targets and conflicting record entries. Fix the files, or push with --no-validate.Entity Validation:
Field Validation:
__mj_ field or the primary key in fields is an error. Another read-only field (such as a joined display field that pull writes) is a warning, because push ignores it. For a virtual field, the warning says to set pull.ignoreVirtualFields: true; for another read-only field, it says to add the field to pull.excludeFields. Field names match the entity's fields without regard to case. A name that is not an entity field must match an entity property exactly (for example TemplateText); validation reports another spelling as an unknown field.Record Conflicts:
primaryKey. This is allowed when they do not disagree.primaryKey and the field, and gives the full path and start line (path:line) of each entry.primaryKey holds an @parent:, @owner: or @root: reference (also inside a @lookup:), or an object value, are not compared, because the same text can name different records.Reference Validation:
@file: references point to existing files. In record fields, the path resolves from the entity folder, as push does?optional inside @lookup: is an error@lookup: references find matching records@template: values are errors, because push does not support them@parent: and @root: have proper contextDependency Order Validation:
Validation Report
Files: 4
Entities: 29
Errors: 2
Warnings: 5
Errors
1. Field "Status" does not exist on entity "Templates"
Entity: Templates
Field: Status
File: ./metadata/templates/.my-template.json
2. File not found: ./shared/footer.html
Entity: Templates
Field: FooterHTML
File: ./metadata/templates/.my-template.json
{
"isValid": false,
"summary": {
"totalFiles": 4,
"totalEntities": 29,
"totalErrors": 2,
"totalWarnings": 5
},
"errors": [
{
"type": "field",
"entity": "Templates",
"field": "Status",
"file": "./metadata/templates/.my-template.json",
"message": "Field \"Status\" does not exist on entity \"Templates\"",
"suggestion": "Check spelling of 'Status'."
}
]
}
| Error | Cause | Solution |
|---|---|---|
Field "X" does not exist |
Typo or wrong entity | Check entity definition in generated files |
Entity "X" not found |
Wrong entity name | Use exact entity name from database |
File not found |
Bad @file: reference | Check file path is relative and exists |
Lookup not found |
No matching record | Verify lookup value or use ?create |
Circular dependency |
A -> B -> A references | Restructure to avoid cycles |
Required field missing |
Missing required field | Add field with appropriate value |
Conflicting values for <entity> record … |
Two entries for one record set a field to different values | Set the same value in both entries, or set the field in one entry only |
@template: is not supported by mj sync push |
A field value starts with @template: |
Use @file: instead |
The MetadataSync package exports several service classes for programmatic use:
| Service | Purpose |
|---|---|
InitService |
Initialize directory structure for metadata sync |
PullService |
Pull metadata from database to local files |
PushService |
Push local file changes to database |
StatusService |
Compare local files with database state |
WatchService |
Watch for file changes and auto-sync |
ValidationService |
Validate metadata files for correctness |
FileResetService |
Reset file checksums and primary keys |
FormattingService |
Format validation results for display |
import { ValidationService, FormattingService } from '@memberjunction/metadata-sync';
// Create validator instance
const validator = new ValidationService({
verbose: false,
outputFormat: 'human',
maxNestingDepth: 10,
checkBestPractices: true
});
// Validate a directory
const result = await validator.validateDirectory('/path/to/metadata');
// Check results
if (result.isValid) {
console.log('Validation passed!');
} else {
console.log(`Found ${result.errors.length} errors`);
// Format results for display
const formatter = new FormattingService();
const humanOutput = formatter.formatValidationResult(result, true);
console.log(humanOutput);
// Or get JSON output
const jsonOutput = formatter.formatValidationResultAsJson(result);
// Or get markdown report
const markdownReport = formatter.formatValidationResultAsMarkdown(result);
}
import { SyncEngine, PushService } from '@memberjunction/metadata-sync';
import { initializeProvider, getSystemUser } from '@memberjunction/metadata-sync';
// Initialize the data provider
await initializeProvider();
const systemUser = await getSystemUser();
// Create services
const syncEngine = new SyncEngine(systemUser);
const pushService = new PushService(syncEngine, systemUser);
// Push with callbacks
const result = await pushService.push(
{ dir: './metadata', verbose: true },
{
onProgress: (msg) => console.log(msg),
onSuccess: (msg) => console.log(msg),
onError: (msg) => console.error(msg)
}
);
console.log(`Created: ${result.created}, Updated: ${result.updated}`);
# Example GitHub Actions workflow
- name: Validate Metadata
run: |
npm install @memberjunction/cli
npx mj sync validate --dir=./metadata --format=json > validation-results.json
- name: Check Validation Results
run: |
if [ $(jq '.isValid' validation-results.json) = "false" ]; then
echo "Metadata validation failed!"
jq '.errors' validation-results.json
exit 1
fi
- name: Push Metadata to Production
run: |
npx mj sync push --ci --dir=./metadata
Before creating entity JSON files, follow this checklist:
packages/MJCoreEntities/src/generated/entity_subclasses.ts and search for the entity class? in TypeScript definitions.mj-sync.json).mj-sync.json with proper glob patterns# Open in your IDE:
packages/MJCoreEntities/src/generated/entity_subclasses.ts
# Search for your entity class (Ctrl+F):
class TemplateEntity
mkdir templates
cd templates
# Create entity config (dot-prefixed configuration file)
echo '{
"entity": "Templates",
"filePattern": "*.json"
}' > .mj-sync.json
{
"fields": {
"Name": "My First Template",
"Description": "A test template",
"UserID": "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E"
}
}
# Dry run to check for errors
mj sync push --dir="templates" --dry-run
# If successful, do actual push
mj sync push --dir="templates"
Always Required:
ID -- Primary key (GUID, auto-generated if not provided)Name -- Human-readable nameUserID -- Creator/owner (use System User: ECAFCCEC-6A37-EF11-86D4-000D3A4E707E)Be Careful With:
Status fields -- Some entities have them, others do not2024-01-15T10:30:00Z| Error Message | Cause | Solution |
|---|---|---|
No entity directories found |
Missing .mj-sync.json or wrong filePattern | Check .mj-sync.json exists and uses "*.json" |
Field 'X' does not exist on entity 'Y' |
Using non-existent field | Check BaseEntity class in entity_subclasses.ts |
User ID cannot be null |
Missing required UserID | Add "UserID": "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E" |
Processing 0 records |
Files do not match filePattern | Check files match pattern in .mj-sync.json |
| Failed validation | Wrong data type or format | Check BaseEntity class for field types |
Shows high-level progress:
Processing AI Prompts in demo/ai-prompts
-> Processing 2 related MJ: AI Prompt Models records
Created: 1
Updated: 2
Shows detailed field-level operations with hierarchical indentation:
Processing AI Prompts in demo/ai-prompts
Setting Name: "Example Greeting Prompt 3" -> "Example Greeting Prompt 3"
Setting Description: "A simple example prompt..." -> "A simple example prompt..."
-> Processing 2 related MJ: AI Prompt Models records
Setting PromptID: "@parent:ID" -> "C2A1433E-F36B-1410-8DB0-00021F8B792E"
Setting ModelID: "@lookup:AI Models.Name=GPT 4.1" -> "123-456-789"
Setting Priority: 1 -> 1
Created MJ: AI Prompt Models record
npm install -g @memberjunction/climj sync pull --entity="AI Prompts" to get latest prompts with their modelsmj sync push --dry-runmj sync push --ci on merge- name: Push Metadata to Production
run: |
npm install -g @memberjunction/cli
mj sync push --ci --entity="AI Prompts"
This package depends on:
Key third-party dependencies:
chokidar -- File watching for watch modefast-glob -- Fast file pattern matchingcosmiconfig -- Configuration file discoveryzod -- Runtime validationchalk -- Terminal output formattingIf you were previously using the standalone mj-sync command:
Update your installation: Install the MJ CLI instead of standalone MetadataSync
npm install -g @memberjunction/cli
Update your scripts: Replace mj-sync with mj sync in all scripts
# Old command (standalone package)
mj-sync push --dir="metadata"
# New command
mj sync push --dir="metadata"
Configuration unchanged: All .mj-sync.json configuration files work exactly the same
See the MemberJunction Contributing Guide for development setup and guidelines.
The push path was substantially reworked to eliminate per-record DB
round-trips on metadata trees with thousands of records. End-to-end
measurements on a representative ~36,500-record metadata/ tree (mostly
idempotent, including a metadata/integrations/ dir with 23,789 records):
| Scenario | Pre-optimization | Post-optimization | Speedup |
|---|---|---|---|
| Full sync (incl. integrations) | ~6m 49s | ~1m 4s | ~6.5× |
| Partial sync (excluding integrations) | ~1m 37s | ~30.5s | ~3.2× |
The changes sit entirely inside the push flow; behaviour, configuration,
and on-disk formats are unchanged.
Upfront preloading via SyncMetadataEngine (extends BaseEngine).
Before processing any file the engine scans every JSON file under each
configured entity directory once, collects the set of entity names
referenced (including nested relatedEntities), and issues a single
unfiltered RunView per entity through BaseEngine.Load. The
resulting BaseEntity instances live on dynamic per-entity property
slots that the sync path consults instead of round-tripping the DB
per record.
Why unfiltered: metadata entities (Actions, Prompts, Agents,
Templates, …) are bounded by design. Loading all rows once is faster
than computing a giant WHERE … IN (…) clause, and it lets
@lookup: resolution hit the cache even for records that are not
directly mentioned in local files. An oversize warning fires if any
single entity comes back with more than 100,000 rows so operators
notice when something non-metadata-scale slips into a sync workflow.
O(1) PK index on the preload cache. Each preloaded slot is also
indexed into a per-entity Map<serializedPK, BaseEntity>. The
sync's loadEntity path uses this for O(1) hash lookups instead of
the previous Array.find(... serializePrimaryKey(GetAll())) scan.
This was the single biggest fix. The naïve Array.find was O(N×K)
where N = cached rows and K = records being processed. For
MJ: Integration Object Fields (~100K+ DB rows, ~23,789 records
processed) that was ~1.2B comparisons → ~38 minutes. After the Map
index the integrations dir alone dropped from 38 minutes to seconds.
The index has a self-healing array-scan fallback for entries
introduced by BaseEngine's event-driven slot mutations.
Resolved-lookup cache + cached file reads. Resolved @lookup:
keys are memoized in a per-engine Map<lookupKey, ID>, indexed by
entity so the cache invalidates only the affected entries on delete
instead of clearing wholesale. The parsed + @include-preprocessed
contents of every file are also cached, so validation and sync
passes don't re-read or re-parse the same file twice. File-cache
entries are invalidated at every write site (immediate and deferred
writes) so later passes within the same push see fresh contents.
Skip preload for unresolved PK references. Records whose
primaryKey still carries an unresolved @lookup: / @parent: /
@root: / @file: / @env: / @template: reference are skipped
at the preload step — SyncEngine's per-record path resolves them
later. Without this guard the preload would inline literal
@lookup:… strings into a WHERE ID = '…' filter and SQL Server
would (correctly) reject the cast to uniqueidentifier.
Provider plumbing. The push flow no longer reaches for
Metadata.Provider directly during cache and lookup writes —
SyncEngine.getProvider() is the single entry point. Single-process
today; ready for multi-provider scenarios without further surgery.
Fixed-width / space-padded character types (nchar/char on SQL Server;
char/character/bpchar on PostgreSQL) used to surface their storage
padding through BaseEntity.Get, causing dirty-check to compare e.g.
"Input " against "Input" and false-positive every record as
dirty. Once preload populated the in-memory comparison this manifested
as thousands of spurious "updates" per sync (~4,279 on MJ: Action Params alone).
The fix is in @memberjunction/core:
EntityFieldInfo.FixedWidthColumn getter, delegating to a new
IsFixedWidthStringSQLType predicate in @memberjunction/sql-dialect
so the list of fixed-width type names stays in one place per dialect.EntityField.Value setter and BaseEntity.Get raw fast-path now
rtrim string values when FixedWidthColumn is true, memoizing back
into _raw so the trim runs once per field per record.This is independent of MetadataSync but was exposed by the preload work and is required for the "Unchanged" counts to be accurate.
SyncMetadataEngine.drainWarnings() exposes non-fatal warnings
collected during preload (malformed relatedEntities shapes, oversized
entities). PushService surfaces them through callbacks.onWarn so
they appear in the CLI's standard warning channel alongside everything
else.