ctxshard — Deterministic SQLite Memory & Pre-Turn Context Compiler for Local LLMs (4B–14B)

ctxshard is a zero-dependency, physically sharded multi-tenant memory engine, state graph, and stdio MCP server built on Node.js native node:sqlite (FTS5 + WAL). Instead of running vector databases or hoping a 4B–9B small language model remembers to call search_memory, ctxshard deterministically compiles matching entities, aliases, relational assertions, directives, and live resource state from isolated <tenant>.sqlite files into a <800-token prompt frame in ~1ms.

Works out of the box with llama.cpp (llama-server), Ollama, LM Studio, vLLM, Claude Desktop, Cursor, OpenCode, Goose, and custom local agent loops.

Why Small Local Models (4B–9B) Break on Standard AI Memory

Most agent memory frameworks (Letta, Mem0, Zep / Graphiti, vector RAG) assume frontier reasoning budgets or external embedding infrastructure. When paired with local SLMs (Qwen3.5-4B, Qwen2.5-7B, Llama-3.2-3B/8B, Gemma-3-4B, Phi-4, Ministral-8B), four failure modes appear immediately:

1. Skipped Memory Tool Calls: 4B–9B models frequently answer directly without calling recall() or search_nodes() first, hallucinating missing context.

2. Vector RAG Pollution in 8k–16k Windows: Cosine-similarity chunks inject noisy, low-precision text into narrow local KV caches while burning extra RAM/VRAM on embedding models (sqlite-vec, Chroma, sentence-transformers).

3. Cross-Tenant Context Bleed: Storing personal, employer, and client projects inside one shared vector index or JSONL file risks leaking facts across workspaces via fuzzy similarity.

4. Restart Amnesia & node-gyp Friction: Live agent state maps vanish on process restart, and C++ SQLite addons (better-sqlite3) frequently fail to compile across Node upgrades.

ctxshard fixes all four in deterministic CPU code using Node's built-in node:sqlite—zero runtime npm dependencies, zero embeddings, and hard file-per-tenant isolation.

Related MCP server: nuzo-memory

How ctxshard Compares to Existing Agent Memory Tools

Architecture

User Prompt: "Why is ACME slow right now?"
      │
      ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Pre-Turn Context Compiler (compileContext — ~1.0ms CPU, 0 LLM tokens)   │
│  1. Boundary-safe entity + alias linker ("ACME" -> client:acme-core)    │
│  2. SQLite FTS5 keyword fallback across entities_fts & assertions_fts   │
│  3. 1-hop graph walk (parent, children, linked live resource telemetry) │
│  4. Priority packer with hard token ceiling (default <= 800 tokens)     │
└───────────────┬─────────────────────────────────────────┬───────────────┘
                │                                         │
                ▼                                         ▼
┌───────────────────────────────┐         ┌───────────────────────────────┐
│ principal.sqlite (Global)     │         │ shards/acme.sqlite (Isolated) │
│ • Principal identity & role   │         │ • entities + entities_fts     │
│ • Tenant shard registry       │         │ • assertions + assertions_fts │
│ • Append-only audit_log       │         │ • resources (live state+meta) │
└───────────────────────────────┘         │ • directives (standing KV)    │
                                          └───────────────────────────────┘
                │
                ▼
Injected System Frame (~245 tokens) ──► Local 4B–9B Model (llama.cpp / Ollama)

flowchart LR
    Prompt["User Prompt"] --> Compiler["Deterministic Context Compiler\n(~1ms CPU · Hard Token Budget)"]
    Principal[("principal.sqlite\nIdentity + Audit Log")] --> Compiler
    ShardA[("shards/acme.sqlite\nEntities · FTS5 · State")] --> Compiler
    ShardB[("shards/personal.sqlite\nPhysically Sealed")] -. "Air-gapped when\nacme is active" .- Compiler
    Compiler --> Frame["Compact Context Frame\n(<= 800 tokens)"]
    Frame --> SLM["Local 4B–14B LLM\n(llama.cpp / Ollama / vLLM)"]

Quick Start

1. Drop-In stdio MCP Server (Claude Desktop, Cursor, OpenCode, Goose)

Add ctxshard to your .mcp.json or claude_desktop_config.json:

{
  "mcpServers": {
    "ctxshard": {
      "command": "npx",
      "args": ["-y", "ctxshard", "mcp"]
    }
  }
}

Optional flags: --dir /custom/path/to/data and --tenant my-workspace.

2. Pre-Turn Context Compilation in TypeScript (llama.cpp / Ollama / OpenAI-Compatible)

Install into any Node.js >=22.5.0 project:

npm install ctxshard

Compile deterministic tenant memory before calling your local model—so even a 4B model that never calls a memory tool still answers with exact facts and live state:

import { CtxShardEngine } from 'ctxshard';

const memory = new CtxShardEngine({
  defaultTenant: 'acme',
  principal: {
    name: 'Jordan Lee',
    role: 'Staff Infrastructure Engineer',
    directives: ['Never cross-pollinate client data', 'Cite live resource status first'],
  },
});

// 1. Teach durable entities + plain-sentence assertions (scoped to shards/acme.sqlite)
memory.upsertEntity({
  id: 'client:acme',
  kind: 'client',
  name: 'Acme Logistics',
  aliases: ['ACME'],
  about: 'Primary freight routing platform',
  resourceId: 'site:api.acme.internal',
  fact: 'Runs PostgreSQL 17 on us-west-2; maintenance window is Tuesdays 02:00 UTC.',
});

// 2. Persist live resource state (telemetry polls merge metrics without wiping learned tags)
memory.upsertResources([
  {
    id: 'site:api.acme.internal',
    kind: 'site',
    name: 'api.acme.internal',
    status: 'warn',
    metrics: { tier: 'enterprise', latencyMs: 940, sslDays: 62 },
  },
]);

// 3. Compile a token-bounded frame in ~1ms before calling llama-server / Ollama
const userPrompt = 'Why is ACME degraded right now, and when can we fail over Postgres?';
const frame = memory.compileContext(userPrompt, { maxTokens: 600 });

const res = await fetch('http://127.0.0.1:8080/v1/chat/completions', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    model: 'qwen-local',
    messages: [
      { role: 'system', content: `Use this verified workspace context:\n\n${frame.text}` },
      { role: 'user', content: userPrompt },
    ],
  }),
});

3. Human CLI Inspector (Zero Black-Box Embeddings)

Every shard is plain relational SQLite. Inspect, teach, export, or physically purge any tenant from your terminal:

# Seed an entity + fact from CLI
npx ctxshard entity add client "Acme Logistics" --tenant acme --about "Freight platform" --fact "Bills quarterly"

# Preview the exact compiled prompt frame and token count
npx ctxshard compile "When does Acme Logistics bill?" --tenant acme --max-tokens 600

# Atomic single-file backup or complete physical deletion (GDPR / client offboarding)
npx ctxshard tenant export acme ./backups/acme-2026.sqlite
npx ctxshard tenant purge acme

Benchmarks (npm run bench)

Measured locally on Node.js (node:sqlite WAL + PRAGMA data_version write-invalidated cache) across 1,000 seeded entities + 1,000 assertions + 1,000 live resources:

Run npm run bench on your own hardware to reproduce.

MCP Tools & Resources Exposed

When running ctxshard mcp, the following small-model-safe tools and resources are exposed:

• compile_context — Deterministically compile a token-budgeted context frame for a prompt.

• tenant_switch / tenant_list — Switch active <tenant>.sqlite shard or inspect shard counts.

• entity_upsert / entity_lookup — Upsert typed graph nodes (person, company, brand, client, project, service, place, other) or search via SQLite FTS5.

• assertion_add — Attach deduplicated plain-sentence facts with provenance (human | agent | ingest).

• resource_upsert / resource_list — Persist tracked infrastructure/business resources (client, site, service, machine, agent, lead, ticket) across restarts while merging live telemetry metrics cleanly.

• directive_set / directive_recall — Store and search per-tenant standing rules and preferences.

• Resource ctxshard://context/active — Live text snapshot of the active tenant shard.

Pairing with local-llm-relay

Building full autonomous workflows on 4B–9B local models? Pair ctxshard (long-term sharded memory & pre-turn context compilation) with local-llm-relay (out-of-context SQLite dataset handles ds-* for 200+ row tool outputs, persistent multi-turn checklists, and max_tokens batch-shrinking recovery).

FAQ

How do I give an Ollama or llama.cpp 4B model long-term memory without a vector database?

Install ctxshard, record entities/assertions via entity_upsert (or the stdio MCP server), and call engine.compileContext(userMessage, { maxTokens: 600 }) before sending your chat completion request. Relevant entities, aliases, facts, and live resource statuses are matched in <2ms via word-boundary linking and SQLite FTS5—requiring zero vector embeddings.

How does multi-tenant isolation work for agencies, consultants, or work-vs-personal setups?

Each tenant ID maps to a separate SQLite database file on disk (~/.local/share/ctxshard/shards/<tenant_id>.sqlite). Queries against tenant client-a never open or scan client-b.sqlite or personal.sqlite. Syncing a single workspace to a work laptop or permanently wiping an offboarded client is a single file copy (tenant export) or file delete (tenant purge).

Why use SQLite FTS5 + an entity graph instead of vector embeddings for SLMs?

Exact identifiers—hostnames, error codes, account slugs, ports, SKU names, and policy constraints—degrade under dense vector compression and return fuzzy near-misses that confuse small 4B–9B models. Relational entity/alias matching + FTS5 prefix search is exact, deterministic, auditable in plain SQL, and uses 0 MB of GPU memory.

License

MIT © Cynthia Schomp