@memberjunction/ai-llamacpp

MemberJunction AI provider for llama.cpp, enabling MJ agents and prompts to run against a local llama-server process. Because llama-server exposes an OpenAI-compatible /v1/chat/completions API, this package is a thin subclass of @memberjunction/ai-openai that simply points the client at your local endpoint.

Architecture

graph TD
    A["LlamaCppLLM
(Provider)"] -->|extends| B["OpenAILLM
(@memberjunction/ai-openai)"]
    B -->|extends| C["BaseLLM
(@memberjunction/ai)"]
    A -->|HTTP| D["llama-server
(localhost:8080/v1)"]
    D -->|runs| E["GGUF models
(Qwen, Llama, DeepSeek, ...)"]
    C -->|registered via| F["@RegisterClass"]

    style A fill:#7c5295,stroke:#563a6b,color:#fff
    style B fill:#2d6a9f,stroke:#1a4971,color:#fff
    style C fill:#2d6a9f,stroke:#1a4971,color:#fff
    style D fill:#2d8659,stroke:#1a5c3a,color:#fff
    style E fill:#b8762f,stroke:#8a5722,color:#fff
    style F fill:#b8762f,stroke:#8a5722,color:#fff
Copy

Features

• Fully local inference — no cloud dependencies, works offline / in airplane mode
• OpenAI-compatible — inherits chat, streaming, JSON mode, and parameter handling from OpenAILLM
• Any GGUF model — run whatever you loaded into llama-server (Qwen3, Llama 3.x, DeepSeek-R1, Phi, Gemma, etc.)
• No API key required by default — a placeholder is supplied; pass a real key if you started llama-server --api-key <key>
• Configurable endpoint — defaults to http://localhost:8080/v1 but accepts any host/port via constructor args or env vars

Environment variables

The endpoint and API key can be overridden without changing code. Precedence is constructor argument → env var → hardcoded default.

Examples:

# Point every LlamaCppLLM instance at a different port
export LLAMACPP_PORT=9090

# Point at a box elsewhere on the LAN
export LLAMACPP_HOST=10.0.0.5
export LLAMACPP_PORT=8080

# Full URL override (e.g. a reverse-proxied endpoint)
export LLAMACPP_BASE_URL=https://llama.internal.example.com/v1

# Authenticated llama-server
export LLAMACPP_API_KEY=my-secret
Copy

Installation

npm install @memberjunction/ai-llamacpp
Copy

Setting up llama-server

This package is a client. Before it can do anything you need a running llama-server process with a model loaded. The four steps below get you from zero to a working endpoint.

1. Install llama.cpp

Pick the path that matches your platform:

macOS (Apple Silicon or Intel)

brew install llama.cpp
Copy

This gives you llama-server, llama-cli, and friends on your PATH, with Metal GPU support built in.

Linux

# Homebrew on Linux (simplest, CPU-only by default)
brew install llama.cpp

# Or prebuilt releases (pick the one matching your CUDA / ROCm / Vulkan stack):
#   https://github.com/ggml-org/llama.cpp/releases
Copy

Windows Download a prebuilt zip from the releases page (pick win-cuda for NVIDIA, win-vulkan for AMD/Intel, or the plain win-avx2 CPU build).

Build from source (any platform, latest features, custom GPU backend)

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# Pick ONE backend:
cmake -B build -DGGML_CUDA=ON        # NVIDIA
cmake -B build -DGGML_METAL=ON       # Apple Silicon (default on macOS)
cmake -B build -DGGML_VULKAN=ON      # AMD / Intel GPUs, cross-platform
cmake -B build -DGGML_HIPBLAS=ON     # AMD ROCm
cmake -B build                        # CPU only
cmake --build build --config Release -j
# Binaries land in ./build/bin/
Copy

Verify the install:

llama-server --version
Copy

2. Get a GGUF model

llama.cpp only loads models in the GGUF format. Hugging Face has thousands. Some sensible starting points:

Quantization cheat sheet. The suffix on the filename (e.g. Q4_K_M) is the quantization. Rules of thumb:

• Q4_K_M — best size/quality tradeoff for most users. Start here.
• Q5_K_M / Q6_K — higher quality, ~25–50% more VRAM.
• Q8_0 — near-lossless, about half the size of full precision.
• IQ3_XXS / IQ2_M — squeeze a bigger model into less VRAM at real quality cost.

Download with curl or the huggingface-cli:

# Raw download
curl -L -o qwen2.5-coder-7b-q4.gguf \
  https://huggingface.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf

# Or with the HF CLI
huggingface-cli download bartowski/Qwen2.5-Coder-7B-Instruct-GGUF \
  Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf --local-dir ./models
Copy

3. Start the server

Minimal command:

llama-server -m ./models/qwen2.5-coder-7b-q4.gguf --port 8080
Copy

Realistic command with the flags that actually matter for agent workloads:

llama-server \
  -m ./models/qwen2.5-coder-7b-q4.gguf \
  --host 0.0.0.0 --port 8080 \
  -c 32768 \
  -ngl 99 \
  -np 4 \
  --flash-attn
Copy

4. Verify it's running

# Health check
curl http://localhost:8080/health

# Sanity chat request
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Say hi in one word."}]
  }'
Copy

llama-server also exposes a web UI at http://localhost:8080 for interactive testing.

Running as a long-lived service

For anything beyond a dev session, don't run llama-server in a terminal window. Options:

• systemd (Linux) — write a unit file pointing at llama-server with the flags above and Restart=on-failure.
• launchd (macOS) — same idea with a .plist.
• Docker — ghcr.io/ggml-org/llama.cpp:server (CUDA variant: :server-cuda). Mount your models directory and map port 8080.
• tmux / screen — fine for a dev box.

Common gotchas

• "Context length exceeded" errors — raise -c or shrink your prompts. The default is whatever's baked into the model, often much smaller than you want.
• Slow / CPU-only inference despite having a GPU — -ngl was 0 (the default) or the llama.cpp build doesn't have your GPU backend compiled in. Check startup logs for CUDA / Metal / Vulkan detection.
• Model won't load / "failed to open" — wrong path, or the file is a .safetensors / .bin instead of .gguf. Only GGUF loads.
• 429 / queued requests under load — raise -np (parallel slots). Each slot needs its own KV cache, so VRAM usage scales with it.
• First request is very slow — the model is being loaded into VRAM. Subsequent requests are fast. Keep the process running.

Usage

import { LlamaCppLLM } from '@memberjunction/ai-llamacpp';

// API key is ignored unless llama-server was started with --api-key
const llm = new LlamaCppLLM();

const result = await llm.ChatCompletion({
    model: 'qwen2.5-coder-32b', // the name reported by llama-server (any value works for most builds)
    messages: [
        { role: 'system', content: 'You are a helpful assistant.' },
        { role: 'user', content: 'Write a TypeScript function that reverses a linked list.' }
    ],
    temperature: 0.2,
});

if (result.success) {
    console.log(result.data.choices[0].message.content);
}
Copy

Connecting to a remote or non-default endpoint

// Custom host/port (e.g. another box on your LAN, or a container)
const llm = new LlamaCppLLM('', 'http://192.168.1.42:9090/v1');
Copy

Using an authenticated endpoint

If you started llama-server --api-key my-secret:

const llm = new LlamaCppLLM('my-secret');
Copy

Streaming

Streaming is inherited from OpenAILLM:

await llm.StreamingChatCompletion({
    model: 'qwen2.5-coder-32b',
    messages: [{ role: 'user', content: 'Explain quicksort.' }],
}, {
    OnContent: (chunk) => process.stdout.write(chunk),
    OnComplete: (final) => console.log('\n\nDone:', final.data.usage),
});
Copy

Configuration via MJ Metadata

Register llama.cpp as a vendor and add a model record pointing at it:

How It Works

LlamaCppLLM is a ~15-line subclass of OpenAILLM that:

1. Defaults baseURL to http://localhost:8080/v1.
2. Substitutes a placeholder API key when none is provided, since the OpenAI SDK requires a non-empty string but llama-server runs unauthenticated by default.

All chat, streaming, tool-call, and parameter-handling logic is inherited — there's no duplicated code. This mirrors how xAILLM and OpenRouterLLM wrap their respective OpenAI-compatible endpoints.

Class Registration

Registered as LlamaCppLLM via @RegisterClass(BaseLLM, 'LlamaCppLLM'), which is how the MJ ClassFactory discovers it at runtime.

llama.cpp vs. Ollama / LM Studio

MJ ships providers for all three. Quick comparison:

• llama.cpp (this package) — direct access to llama-server. Best performance ceiling, fine-grained sampler control, latest features. You manage GGUF files yourself.
• Ollama (@memberjunction/ai-ollama) — friendlier model management (ollama pull ...), automatic load/unload, better default concurrency. Uses llama.cpp internally.
• LM Studio (@memberjunction/ai-lmstudio) — GUI model management on macOS/Windows, also uses llama.cpp internally.

Use this package when you want to talk to llama-server directly — no daemon, no extra process, no model manager in between.

Dependencies

• @memberjunction/ai — core AI abstractions
• @memberjunction/ai-openai — parent class providing OpenAI-compatible behaviour
• @memberjunction/global — class registration system