Skip to content
MemberJunction

@memberjunction/doc-utils

A TypeScript library for dynamically retrieving, parsing, and caching MemberJunction library documentation from the official documentation website. Designed to support AI models, code generation tools, and other documentation-driven features that need structured access to MJ library metadata.

Overview

Section titled “Overview”

The @memberjunction/doc-utils package provides a singleton DocumentationEngine that loads library and library item metadata from the MemberJunction entity system, then fetches the corresponding HTML documentation pages from the official MJ documentation site (https://memberjunction.github.io/MJ/). Parsed content is cached in memory for efficient repeated access.

This package is particularly useful for:

  • Feeding structured library documentation to AI models for context-aware code generation
  • Building developer tools that reference MJ API documentation
  • Providing searchable documentation indexes for MJ packages and their exported items

Architecture

Section titled “Architecture”
graph TB
    subgraph DocUtils["DocUtils Package"]
        DE["DocumentationEngine<br/>(Singleton)"]
        LEE["MJLibraryEntityExtended"]
        LIEE["MJLibraryItemEntityExtended"]
    end

    subgraph MJCore["@memberjunction/core"]
        BE["BaseEngine"]
        BEnt["BaseEntity"]
    end

    subgraph CoreEntities["@memberjunction/core-entities"]
        LE["LibraryEntity"]
        LIE["LibraryItemEntity"]
    end

    subgraph External["External Services"]
        DB[("MJ Database<br/>Libraries &<br/>Library Items")]
        DOCS["MJ Docs Site<br/>memberjunction.github.io"]
    end

    DE -->|extends| BE
    LEE -->|extends| LE
    LIEE -->|extends| LIE
    LE -->|extends| BEnt
    LIE -->|extends| BEnt

    DE -->|loads metadata| DB
    DE -->|fetches HTML| DOCS
    DE -->|manages| LEE
    DE -->|manages| LIEE
    LEE -->|contains| LIEE

    style DE fill:#2d6a9f,stroke:#1a4971,color:#fff
    style LEE fill:#2d8659,stroke:#1a5c3a,color:#fff
    style LIEE fill:#2d8659,stroke:#1a5c3a,color:#fff
    style BE fill:#7c5295,stroke:#563a6b,color:#fff
    style BEnt fill:#7c5295,stroke:#563a6b,color:#fff
    style LE fill:#7c5295,stroke:#563a6b,color:#fff
    style LIE fill:#7c5295,stroke:#563a6b,color:#fff
    style DB fill:#b8762f,stroke:#8a5722,color:#fff
    style DOCS fill:#b8762f,stroke:#8a5722,color:#fff

Data Flow

Section titled “Data Flow”

The engine follows a two-phase loading process: first it loads entity metadata from the database, then it fetches and parses HTML documentation for each library item.

sequenceDiagram
    participant App as Application
    participant DE as DocumentationEngine
    participant DB as MJ Database
    participant Site as Docs Website

    App->>DE: Config(forceRefresh, contextUser)
    DE->>DB: Load Libraries (entity metadata)
    DB-->>DE: MJLibraryEntityExtended[]
    DE->>DB: Load Library Items (entity metadata)
    DB-->>DE: MJLibraryItemEntityExtended[]

    Note over DE: AdditionalLoading phase

    loop For each Library
        loop For each Item in Library
            DE->>DE: Build URL from library name + item type
            DE->>Site: GET /{type}/{sanitized_name}.{ItemName}.html
            Site-->>DE: HTML response
            DE->>DE: Parse HTML via JSDOM (extract div.col-content)
            DE->>DE: Cache HTMLContent and URL on item
        end
    end

    DE-->>App: Config complete, data cached
    App->>DE: Access Libraries / LibraryItems
    DE-->>App: Cached entity data with HTML content

Class Hierarchy

Section titled “Class Hierarchy”
classDiagram
    class BaseEngine {
        <<abstract>>
        +Load(configs, provider, forceRefresh, contextUser)
        #AdditionalLoading(contextUser)*
    }

    class DocumentationEngine {
        +Instance$ DocumentationEngine
        +Libraries MJLibraryEntityExtended[]
        +LibraryItems MJLibraryItemEntityExtended[]
        +Config(forceRefresh, contextUser, provider)
        #AdditionalLoading(contextUser)
        #GetContent(url, rootSelector) string
        #fetchDocumentation(url) string
        #parseDocumentation(html, rootSelector) string
    }

    class BaseEntity {
        <<abstract>>
        +Get(fieldName)
        +Set(fieldName, value)
        +Save()
    }

    class LibraryEntity {
        +ID string
        +Name string
        +Description string?
        +Status "Active" | "Disabled" | "Pending"
        +TypeDefinitions string?
        +SampleCode string?
    }

    class MJLibraryEntityExtended {
        +Items MJLibraryItemEntityExtended[]
    }

    class LibraryItemEntity {
        +ID string
        +Name string
        +LibraryID string
        +Type "Class"|"Function"|"Interface"|"Module"|"Type"|"Variable"
        +Library string
    }

    class MJLibraryItemEntityExtended {
        +URL string
        +HTMLContent string
        +TypeURLSegment string
    }

    BaseEngine <|-- DocumentationEngine
    BaseEntity <|-- LibraryEntity
    BaseEntity <|-- LibraryItemEntity
    LibraryEntity <|-- MJLibraryEntityExtended
    LibraryItemEntity <|-- MJLibraryItemEntityExtended
    DocumentationEngine o-- MJLibraryEntityExtended
    DocumentationEngine o-- MJLibraryItemEntityExtended
    MJLibraryEntityExtended o-- MJLibraryItemEntityExtended

    style DocumentationEngine fill:#2d6a9f,stroke:#1a4971,color:#fff
    style MJLibraryEntityExtended fill:#2d8659,stroke:#1a5c3a,color:#fff
    style MJLibraryItemEntityExtended fill:#2d8659,stroke:#1a5c3a,color:#fff
    style BaseEngine fill:#7c5295,stroke:#563a6b,color:#fff
    style BaseEntity fill:#7c5295,stroke:#563a6b,color:#fff
    style LibraryEntity fill:#7c5295,stroke:#563a6b,color:#fff
    style LibraryItemEntity fill:#7c5295,stroke:#563a6b,color:#fff

Installation

Section titled “Installation”
Terminal window
npm install @memberjunction/doc-utils

Dependencies

Section titled “Dependencies”

MemberJunction packages:

PackagePurpose
@memberjunction/coreBase engine class, metadata provider, user context
@memberjunction/core-entitiesLibraryEntity and LibraryItemEntity base classes
@memberjunction/global@RegisterClass decorator for entity registration

External packages:

PackagePurpose
axiosHTTP requests to the documentation website
jsdomHTML parsing and DOM manipulation for content extraction

Usage

Section titled “Usage”

Initializing the Engine

Section titled “Initializing the Engine”

The DocumentationEngine uses the singleton pattern. Call Config() once to load metadata and documentation content before accessing data.

import { DocumentationEngine } from '@memberjunction/doc-utils';
import { UserInfo } from '@memberjunction/core';


// Get the singleton instance
const docEngine = DocumentationEngine.Instance;


// Server-side: pass contextUser for proper data isolation
const contextUser: UserInfo = /* from your authentication context */;
await docEngine.Config(false, contextUser);


// Client-side: contextUser is optional
await docEngine.Config();

Accessing Libraries and Items

Section titled “Accessing Libraries and Items”

After configuration, libraries and their items are available through cached properties.

const docEngine = DocumentationEngine.Instance;


// Get all libraries
const libraries = docEngine.Libraries;
console.log(`Loaded ${libraries.length} libraries`);


// Find a specific library
const coreLib = libraries.find(lib => lib.Name === '@memberjunction/core');
if (coreLib) {
    console.log(`${coreLib.Name} (${coreLib.Status})`);
    console.log(`Items: ${coreLib.Items.length}`);


    // Iterate through items in the library
    for (const item of coreLib.Items) {
        console.log(`  ${item.Type}: ${item.Name}`);
        console.log(`  URL: ${item.URL}`);
        console.log(`  Content length: ${item.HTMLContent.length} chars`);
    }
}

Accessing All Library Items

Section titled “Accessing All Library Items”

The flat LibraryItems array provides access to all items across all libraries.

const docEngine = DocumentationEngine.Instance;


// Get all library items regardless of library
const allItems = docEngine.LibraryItems;


// Filter by type
const allClasses = allItems.filter(item => item.Type === 'Class');
const allInterfaces = allItems.filter(item => item.Type === 'Interface');


// Find a specific item by name and library
const baseEntity = allItems.find(
    item => item.Name === 'BaseEntity' && item.Library === '@memberjunction/core'
);


if (baseEntity) {
    console.log(`Documentation URL: ${baseEntity.URL}`);
    console.log(`HTML Content: ${baseEntity.HTMLContent}`);
}

Force Refreshing Documentation

Section titled “Force Refreshing Documentation”

Pass true to Config() to reload all metadata from the database and re-fetch documentation content.

const docEngine = DocumentationEngine.Instance;


// Force a complete reload
await docEngine.Config(true);

API Reference

Section titled “API Reference”

DocumentationEngine

Section titled “DocumentationEngine”

The primary class providing access to MJ library documentation. Extends BaseEngine and uses the singleton pattern.

MemberTypeDescription
Instance (static)DocumentationEngineReturns the singleton instance
LibrariesMJLibraryEntityExtended[]All loaded libraries with their items
LibraryItemsMJLibraryItemEntityExtended[]Flat list of all library items across all libraries
Config(forceRefresh?, contextUser?, provider?)Promise<void>Initializes the engine: loads metadata, fetches and parses documentation

MJLibraryEntityExtended

Section titled “MJLibraryEntityExtended”

Extended entity class for libraries. Registered via @RegisterClass(BaseEntity, "Libraries").

MemberTypeDescription
IDstringUnique identifier
NamestringLibrary package name (e.g., @memberjunction/core)
Descriptionstring | nullHuman-readable description
Status'Active' | 'Disabled' | 'Pending'Library availability status
TypeDefinitionsstring | nullType/function definitions for reference by humans and AI
SampleCodestring | nullUsage examples for the library
ItemsMJLibraryItemEntityExtended[]All items belonging to this library

MJLibraryItemEntityExtended

Section titled “MJLibraryItemEntityExtended”

Extended entity class for individual library items (classes, interfaces, functions, etc.). Registered via @RegisterClass(BaseEntity, "Library Items").

MemberTypeDescription
IDstringUnique identifier
NamestringItem name (e.g., BaseEntity, RunView)
LibraryIDstringForeign key to the parent library
LibrarystringDenormalized library name from the view
Type'Class' | 'Function' | 'Interface' | 'Module' | 'Type' | 'Variable'The kind of exported item
URLstringGenerated documentation URL for this item
HTMLContentstringParsed HTML content from the documentation page
TypeURLSegmentstring (getter)URL path segment derived from Type (e.g., classes, interfaces)

URL Generation

Section titled “URL Generation”

Documentation URLs are constructed automatically using the pattern:

https://memberjunction.github.io/MJ/{typeSegment}/{sanitizedLibraryName}.{ItemName}.html

Where:

  • {typeSegment} is the pluralized, lowercase item type (classes, interfaces, functions, modules, types, variables)
  • {sanitizedLibraryName} is the library name with @, ., /, and \ characters replaced by _
  • {ItemName} is the exact name of the exported item

For example, BaseEntity (a Class) in @memberjunction/core resolves to:

https://memberjunction.github.io/MJ/classes/_memberjunction_core.BaseEntity.html

URL Routing by Item Type

Section titled “URL Routing by Item Type”
graph LR
    Item["Library Item"]

    Item -->|Class| C["/classes/"]
    Item -->|Interface| I["/interfaces/"]
    Item -->|Function| F["/functions/"]
    Item -->|Module| M["/modules/"]
    Item -->|Type| T["/types/"]
    Item -->|Variable| V["/variables/"]

    style Item fill:#2d6a9f,stroke:#1a4971,color:#fff
    style C fill:#2d8659,stroke:#1a5c3a,color:#fff
    style I fill:#2d8659,stroke:#1a5c3a,color:#fff
    style F fill:#2d8659,stroke:#1a5c3a,color:#fff
    style M fill:#b8762f,stroke:#8a5722,color:#fff
    style T fill:#b8762f,stroke:#8a5722,color:#fff
    style V fill:#b8762f,stroke:#8a5722,color:#fff

Integration with MemberJunction

Section titled “Integration with MemberJunction”

This package integrates with the broader MemberJunction ecosystem through several mechanisms:

  • Entity System: Uses @RegisterClass to register MJLibraryEntityExtended and MJLibraryItemEntityExtended as entity subclasses, ensuring the MJ class factory returns the extended types when loading Libraries and Library Items.
  • BaseEngine Pattern: Extends BaseEngine with the Config() / AdditionalLoading() lifecycle, allowing consistent initialization and caching across all MJ engine classes.
  • User Context: Supports contextUser for proper data isolation and security when running on the server side.
  • Metadata Provider: Accepts an optional IMetadataProvider for environments with custom metadata access patterns.

Build

Section titled “Build”
Terminal window
# Build the package
cd packages/DocUtils
npm run build


# Development mode with file watching
npm start

Source Files

Section titled “Source Files”
FileDescription
src/index.tsPublic API surface — re-exports from Engine.ts
src/Engine.tsCore implementation: DocumentationEngine, MJLibraryEntityExtended, MJLibraryItemEntityExtended

Notes

Section titled “Notes”
  • Documentation is fetched from https://memberjunction.github.io/MJ/ using axios HTTP GET requests.
  • HTML content is parsed with jsdom, extracting the inner HTML of the div.col-content element from each page.
  • All fetched content is cached in memory after the initial Config() call. Subsequent calls to Config() without forceRefresh=true return immediately.
  • Library names containing special characters (@, ., /, \) are sanitized by replacing them with underscores for URL compatibility.
  • If a documentation page cannot be fetched (network error, 404, etc.), the HTMLContent for that item is set to an error or placeholder string rather than throwing an exception.