Open Context Engine

Open Context Engine (OCE) is a local-first, high-performance retrieval engine that indexes your entire codebase, embeds it into vectors, and serves it up to LLMs via semantic search, keyword search, and the Model Context Protocol (MCP). No cloud required. No code leaves your machine.

Whether you're building a coding assistant, automating PR reviews, or just want to ask natural-language questions about your own code—OCE gives your AI ground truth.

📘 Docs, benchmarks & guides: opencontextengine.com — MCP setup for Claude Code & Cursor, the agentic-workflow guide, and measured retrieval benchmarks.

✨ What Makes It Special

Related MCP server: codesteer-atlas

🚀 Quick Start

1. Install

npm install -g @dhaxor/open-context     # Node 22+

2. Pick a model — free options included

oce setup

oce setup checks what you already have (a running Ollama, keys in your environment) and lists every way to get a working model, free ones first:

oce setup -p google records your choice; after that oce trace just works.

Hybrid search out of the box — no key, no account. The first index embeds with local MiniLM (in-process, on-device, nothing sent anywhere), so semantic + keyword search works immediately. Want the strongest code retrieval? A Voyage key upgrades the embedding model — oce setup --embeddings voyage — and the index rebuilds with it. Local does the job fine if you'd rather not.

3. Open Trace

oce trace      # the workspace, in your browser
oce            # the same thing, in your terminal

Both index the current directory on start and keep one conversation going — see Trace below for what makes it different.

4. Or search from the command line

oce index                                        # full index (incremental afterwards: --incremental)
oce search "how does the auth middleware work?"

🔭 Trace

Trace is the agent workspace. Every coding agent shows you what it did; Trace also shows you what it knew — because OCE owns the retrieval, not just the chat loop.

• Evidence rail. What is in the model's context window right now: every retrieved chunk ranked by its fusion score, the graph edge that pulled it in ("called by search.ts"), and a budget meter split into retrieval / files / history / system. Pin a chunk and it survives compaction; evict one and it is genuinely removed from the next request.

• Retrieval readout instead of a spinner — retrieved 5 · top 0.94 · graph +2 · 251ms.

• The spine. A timeline of checkpoints, one per turn, on the same hash chain as the audit log. Click one to rewind: the conversation and the files the agent changed roll back — and a file you edited by hand since is left alone.

• Approvals with a blast radius. Every edit and shell command waits for y / always / n, showing the diff, a risk score, and how many indexed callers the file has.

• Sub-agents you can watch. Delegated research streams into its own collapsible block instead of going silent for minutes.

• Parallel sessions. In a git repository, each new session gets its own worktree and branch, so agents never edit the same checkout. Review a session's diff and land it (commit + --no-ff merge) when it's done — uncommitted work is never discarded when you close one.

• The composer. @path pins a file into context (fuzzy-matched against the index), !command runs a shell command through the same approval flow, and /command for everything else — the same in the browser and terminal.

oce trace binds to 127.0.0.1 only and mints a fresh access token each launch. Useful flags: --port, --no-open, --headless (API only), --no-parallel.

In the terminal

Bare oce opens the same workspace full-screen in your terminal: the same evidence rail (from 100 columns wide, or Ctrl+R), the same checkpoint gutter, the same approvals. Esc or Ctrl+C interrupts a run; Ctrl+C twice or /exit quits. Pipes, --print, and dumb terminals fall back to the classic line-by-line REPL, which you can also ask for with oce --classic.

As a desktop app

desktop/ wraps Trace in an Electron shell that adds desktop notifications (only when an approval is waiting, a turn finishes in the background, or something fails), an application menu, and trace:// deep links. It is built from a source checkout for now — see desktop/README.md.

The classic REPL

oce --classic streams styled output, shows each tool call with a live spinner and timing, and — before any file edit or shell command — prints a diff/command preview and asks for approval (y / always / n). Ctrl+C interrupts the current run without quitting the session.

Approval modes (as in the other leading coding CLIs):

• default (suggest) — every edit and shell command asks first

• --auto-edit — file edits apply automatically; shell still asks

• --full-auto — nothing asks (for containers / CI)

Switch mid-session with /mode. Interactive sessions have the edit and shell tools available behind approvals; the non-interactive --print path keeps them opt-in via --allow-edits / --allow-shell.

Slash commands: /help /reset /compact /plan /diff /usage /tools /mode /sessions /resume /exit.

Sessions persist automatically to .open-context/sessions/. oce --continue resumes the last conversation in the workspace; oce --resume <id> restores a specific one (list them with /sessions).

Harness features:

• Plan tracking — the agent maintains a live step checklist for multi-step work, rendered as you watch.

• Sub-agent delegation — broad explorations run in a scoped read-only child agent, so their large intermediate output never floods the main context.

• Summarizing compaction — when history gets long, older turns are condensed into a context note by the model rather than dropped, so decisions and file paths survive; force it with /compact.

• Environment awareness — platform, git branch/status, date, and index stats are injected into the system prompt.

• Cost + parallelism — every turn ends with a stats line (steps, tool calls, token usage, wall time); read-only tool calls in a turn run in parallel, while any turn with an edit or shell call stays strictly ordered.

• --route sends each query to a cost-appropriate model tier; --memory remembers codebase insights across sessions; --audit logs runs to the tamper-evident audit log.

Non-interactive use for scripts and CI:

oce --print "where is the auth middleware defined?"           # one answer, then exit
oce --print "list the exported symbols in src/core" --json     # {answer, stats, toolCalls}

The programmatic API mirrors all of this: defaultAgentTools({ context }) stays read-only unless you pass includeEdits/shell/plan/delegate, and ContextAgent accepts hooks (a PermissionManager composes in here), compaction, and environmentProvider.

5. Connect to Claude / Cursor via MCP

oce mcp --workspace ./my-project

# Or serve a shared HTTP endpoint (e.g. one index for the whole team):
oce mcp --workspace ./my-project --http --port 8940 --auth-token "$TOKEN"

The MCP server indexes the workspace on startup and then watches for changes, so the index stays live without a manual re-index. Pass --no-watch to disable the watcher. (The MCP handshake is established before indexing begins, so the first index of a large repo won't block your client from connecting.)

In HTTP mode the server speaks MCP's Streamable HTTP transport, binds to loopback by default, answers GET /health, and — when --auth-token (or OCE_MCP_AUTH_TOKEN) is set — requires Authorization: Bearer <token> on every request.

Then add to your MCP config:

{
  "mcpServers": {
    "open-context": {
      "command": "oce",
      "args": ["mcp", "--workspace", "/path/to/project"]
    }
  }
}

6. Or use the VS Code extension

The extension/ folder ships a full VS Code experience on top of the same engine: a sidebar chat grounded in your index (streaming, markdown, code-block copy/insert/apply), agent edits with per-file diff / undo / redo and a per-turn "undo all", quick search (Ctrl+Shift+K), auto-indexing on save, and an index health panel for debugging retrieval. Each platform package carries one Node-API SQLite binding, which runs on every VS Code from 1.103 and on remote hosts (SSH, WSL, Codespaces) — see extension/PUBLISHING.md for the support matrix, or build your own VSIX:

cd extension && npm ci --ignore-scripts
npm run package -- linux-x64   # → ../artifacts/open-context-linux-x64-<version>.vsix

📖 Usage Guide

CLI Reference

oce <command> [options]

Commands:
  index <workspace>      Index a codebase
  search <query>         Search the indexed codebase
  watch                  Index, then keep the index live as files change
  mcp                    Start MCP server (stdio, or --http for a shared endpoint)
  agent                  Start interactive agent (auto-indexes on startup)
  eval                   Score retrieval quality against a labeled query set
  push-index <dest>      Publish the index as a team artifact (Team)
  pull-index <src>       Install a team index, re-embed only local changes (Team)
  bench                  Benchmark parse/chunk throughput on this workspace
  policy                 Show the effective policy (user + workspace + org lock)
  audit                  Inspect / verify the tamper-evident audit log
  activate <key>         Activate a Team/Enterprise license
  license / deactivate   Show or remove the active license

Options:
  -w, --workspace <path>   Project root (default: cwd)
  -p, --provider <name>    Embedding provider: voyage | openai | ollama | local
  -m, --model <model>      Embedding model name
  --api-key <key>          API key (falls back to env vars)
  --store-path <path>      Custom store directory (default: .open-context/)
  --max-file-size <bytes>  Skip files larger than this
  --chunk-size <lines>     Lines per chunk (default: 80)
  --chunk-overlap <lines>  Overlap between chunks (default: 15)

Programmatic API

import { OpenContext, ContextAgent, defaultAgentTools } from "@dhaxor/open-context";

// 1. Create and index
const ctx = await OpenContext.create({
  workspaceRoot: "./my-project",
  embedding: {
    provider: "voyage",
    model: "voyage-code-3",
    dimension: 1024,
    apiKey: process.env.VOYAGE_API_KEY,
    batchSize: 32,
  },
});

await ctx.indexWorkspace();

// 2. Search
const results = await ctx.search("authentication middleware flow");
console.log(results);

// 3. Agent with tools
const agent = new ContextAgent({
  provider: "openai",
  model: "gpt-4o",
  apiKey: process.env.OPENAI_API_KEY,
  tools: defaultAgentTools({ context: ctx, includeEdits: true }),
  // Optional harness features:
  maxParallelTools: 4,                       // read-only tool calls fan out concurrently
  hooks: {
    preToolCall: (tc) => tc.name === "remove-file"
      ? { behavior: "deny", reason: "deletions are reviewed manually" }
      : { behavior: "allow" },
    postToolCall: (_tc, result) => result.replaceAll(process.env.HOME!, "~"),
  },
});

const answer = await agent.run("How does the auth middleware work?");
console.log(answer);
console.log(agent.getLastRunStats());        // steps, tool calls, token usage, duration
const saved = agent.exportSession();          // persist; importSession(saved) restores

// 4. Cleanup
ctx.close();

MCP Server Integration

import { runMCPServer } from "@dhaxor/open-context";

await runMCPServer({
  workspaceRoot: "./my-project",
  embedding: { provider: "voyage", model: "voyage-code-3", dimension: 1024, batchSize: 32 },
});

Exposed MCP tools:

• codebase-retrieval — Natural language search over code

• list-files — Browse indexed files with filters

• read-file — Read file contents with optional line ranges

• find-symbol-definition — Exact-name lookup of where a function/class/type is defined

• find-symbol-references — Every indexed usage site of an identifier

• index-status — Chunk count, file count, search mode, embedding model

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                        Open Context Engine                   │
├─────────────────────────────────────────────────────────────┤
│  CLI / API / MCP / Agent                                     │
├─────────────────────────────────────────────────────────────┤
│  HybridRetriever                                             │
│  ├── Vector Search (sqlite-vec)                              │
│  ├── BM25 Search (SQLite FTS5, porter-stemmed)               │
│  ├── Reciprocal Rank Fusion + symbol/editor/recency boosts   │
│  ├── Optional Re-ranking (Voyage / Cohere)                   │
│  └── Graph Expansion (callers/callees via code graph)        │
├─────────────────────────────────────────────────────────────┤
│  AstChunker + CodeChunker                                    │
│  ├── Tree-sitter parsers (TS, JS, Py, Go, Rust, Java, C#)   │
│  └── Boundary-aware fallback chunking                        │
├─────────────────────────────────────────────────────────────┤
│  Embedding Providers                                         │
│  ├── Voyage Code-3 (recommended)                             │
│  ├── OpenAI text-embedding-3-*                               │
│  └── Ollama (local, nomic-embed-text)                        │
├─────────────────────────────────────────────────────────────┤
│  SqliteStore                                                 │
│  ├── chunks (metadata)                                       │
│  ├── chunks_vec (vector index — skipped in keyword-only)     │
│  ├── chunks_fts (full-text index)                            │
│  ├── graph_edges (import/call/definition graph)              │
│  └── files (index state)                                     │
└─────────────────────────────────────────────────────────────┘

⚙️ Configuration

Embedding Models

The local provider needs the optional @huggingface/transformers package; models download once into ~/.open-context/models (override with OCE_MODEL_DIR) and run offline afterwards.

Agent LLMs

openai, anthropic, google (Gemini, streaming + tools), ollama (fully local — oce -p ollama --llm-model qwen2.5-coder:7b needs no API key), and custom (any OpenAI-compatible endpoint via --llm-base-url). With -p ollama for the LLM, --embedding-provider local for embeddings, and reranker: { provider: "local" }, the entire stack runs offline — index, search, rerank, and agent.

Config file

Persistent settings live in ~/.open-context/config.json (user) and <workspace>/.open-context/config.json (project — commit it). Flags and env vars always win. API keys are deliberately rejected in config files; use env vars.

{
  "embedding": { "provider": "local", "model": "jina-embeddings-v2-base-code" },
  "llm": { "provider": "ollama", "model": "qwen2.5-coder:7b" },
  "search": { "topK": 20, "minScore": 0.15 },
  "chunkSize": 80
}

Search Configuration

{
  search: {
    topK: 15,              // Final results returned
    candidateK: 60,        // Candidates before fusion/rerank
    maxOutputLength: 80000,// Max chars in formatted output
    minScore: 0.0,         // Score floor
    bm25Weight: 1.0,       // Keyword search weight
    vectorWeight: 1.0,     // Semantic search weight
    rerank: true,          // Enable reranking if configured
    expandSymbols: true,   // Pull in referenced definitions
  }
}

Re-ranking

{
  reranker: {
    provider: "voyage",    // or "cohere"
    model: "rerank-2",     // optional
    apiKey: "your-key",
  }
}

👥 Team Index Sync (Team)

Index once — in CI or on a lead's machine — and let the whole team pull the result instead of each paying to embed the same monorepo:

# CI / lead machine: refresh the index and publish it (any file path or
# HTTP(S) PUT endpoint — S3/GCS presigned URLs work as-is)
oce push-index https://artifacts.example.com/oce/main.db.gz --token "$TOKEN"

# Each teammate: install it, then only their local diff gets embedded
oce pull-index https://artifacts.example.com/oce/main.db.gz --token "$TOKEN"
# → Installed team index: 12,481 chunks, 1,904 files, built ... at commit ab12cd34
# → Reconciled: 3 file(s) re-embedded locally, 1,901 reused from the artifact

The artifact is the store itself (vectors + FTS + code graph + file hashes) gzipped, so reconciliation is a plain incremental index: only files whose hashes differ from the artifact re-embed. No OCE cloud service is involved — artifacts go to storage you control.

On top of that, a machine-wide embedding cache (~/.open-context/embed-cache.db, on by default in the CLI and extension) keys vectors by content hash: identical code never embeds twice, across repos, branches, and store rebuilds. Disable with --no-embed-cache or openContext.embedding.cache.enabled.

🛡️ Policy Controls

Commit a policy file and every OCE surface (CLI, extension, library) enforces it — a --allow-shell flag can't override it, and the merge is always most-restrictive-wins:

// <workspace>/.open-context/policy.json (or ~/.open-context/policy.json for user-wide)
{
  "agent": {
    "shell": { "enabled": false },                  // or pin: { "allowlist": ["git", "npm"] }
    "edits": { "enabled": true },
    "webSearch": { "enabled": false }
  },
  "embedding": { "localOnly": true },               // only ollama/local providers may run
  "ignore": ["secrets/**", "*.pem"],                // never indexed, on top of .contextignore
  "audit": { "required": true }                     // force audit logging on
}

oce policy prints the effective merge and where each rule came from. Orgs on a Team+ plan can additionally ship an Ed25519-signed policy.lock (issued with scripts/license-tool.mjs sign-policy) that developers cannot loosen or delete their way out of.

🧾 Audit Log

oce agent --audit (or the openContext.agent.audit.enabled setting) appends every run, tool call, and MCP invocation to .open-context/audit/audit.jsonl. Each record is SHA-256 hash-chained to the previous one, so edits, deletions, and reordering are detectable:

oce audit                 # recent events
oce audit --type tool-call --since 2026-07-01
oce audit --verify        # ✓ chain intact — or exactly where it was tampered

Audit logging is an Enterprise feature; a workspace/org policy with "audit": { "required": true } also switches it on.

🛠️ Development

git clone https://github.com/Dhaxor/open-context-engine.git
cd open-context-engine
npm install
npm run build
npm test

Project Structure

src/
├── core/
│   ├── context.ts          # Main OpenContext class
│   ├── ast-chunker.ts      # Tree-sitter chunking
│   ├── chunker.ts          # Fallback line chunking
│   ├── embedder.ts         # Embedding providers
│   ├── retriever.ts        # Hybrid search + RRF + boosts
│   ├── reranker.ts         # Re-ranking providers
│   ├── sqlite-store.ts     # SQLite persistence (vec + FTS5 + graph)
│   ├── code-graph.ts       # Import/call/definition edge queries
│   ├── ast-graph-extractor.ts # Single-DFS AST edge extraction
│   ├── graph-expander.ts   # Graph-aware result expansion
│   ├── chunk-pool.ts       # Worker-thread parse/chunk pool (auto on big repos)
│   ├── policy.ts           # Workspace/org policy loading + enforcement
│   ├── audit.ts            # Tamper-evident (hash-chained) audit log
│   ├── license.ts          # Offline Ed25519 license gate
│   ├── search.ts           # Output formatting
│   ├── file-filter.ts      # Gitignore-aware file collection
│   └── file-watcher.ts     # Watch for changes
├── agent/
│   ├── agent.ts            # ContextAgent + tools
│   ├── providers.ts        # LLM callers (OpenAI, Anthropic)
│   ├── model-router.ts     # Cost-appropriate tier per query
│   ├── session-memory.ts   # Cross-session insight memory
│   ├── edit-tools.ts       # File editing tools
│   └── extra-tools.ts      # Shell + web search
├── eval/
│   └── runner.ts           # Retrieval-quality scoring (oce eval)
├── mcp/
│   └── server.ts           # MCP server implementation
└── cli/
    └── index.ts            # oce CLI entrypoint
extension/                  # VS Code extension (chat, edit review, health)

🧪 How Search Works

1. Chunking: Your code is parsed by Tree-sitter into semantic units (functions, classes, methods). Unsupported files fall back to boundary-aware line chunking.

2. Embedding: Each chunk is embedded using your chosen provider. Voyage Code-3 is optimized for code and gives the best retrieval quality.

3. Indexing: Vectors go into sqlite-vec, text into FTS5, metadata into SQLite tables. All local.

4. Query: Your natural language query is embedded and searched simultaneously via vector similarity and BM25.

5. Fusion: Results are fused with Reciprocal Rank Fusion, then boosted by symbol-name matches, your editor context (active file, open tabs), and git recency.

6. Re-ranking (optional): Top candidates are re-scored by a dedicated re-ranker for higher precision.

7. Graph Expansion: Top results are expanded along the code graph — callers, callees, and referenced definitions ride along so the LLM sees the full picture.

If the sqlite-vec extension can't load on your platform, steps 2 and 4's vector half are skipped automatically and the engine runs keyword-only (BM25) — slower to find conceptual matches, but everything keeps working and getStatus() / the CLI / the extension all surface the mode.

📏 Measuring Retrieval Quality

oce eval scores the engine against a labeled query set — so ranking changes are measured, not guessed:

# Score the engine against the bundled gold set for this repo
oce eval --cases eval/oce.eval.json

# Save a baseline, change a ranking knob, then compare
oce eval --cases eval/oce.eval.json --out baseline.json
# ...tweak weights / expansion / reranker...
oce eval --cases eval/oce.eval.json --baseline baseline.json

# Measure what symbol/graph expansion contributes
oce eval --cases eval/oce.eval.json --no-expand --baseline baseline.json

Reports recall@k, MRR, nDCG@k, hit-rate, and per-query latency, plus per-case deltas (improved / regressed) against a saved baseline. Metrics are file-granular: a case passes when the gold file ranks in the top-k unique files returned.

Current numbers, measured fresh on this repo's committed 44-case gold set (eval/oce.eval.json) and the external 27-case Flask set (eval/external/flask.eval.json) — baselines committed beside them:

Two honest readings, and both matter:

• Hybrid earns its keep where queries paraphrase intent — +6 points of recall on this repo's set, and it's the mode the agent actually runs in.

• Adaptive fusion closed the definition-query gap. Equal-weight RRF used to demote confident BM25 hits below vector-preferred docs/tests (MiniLM hybrid scored 0.852 on Flask, losing to keyword-only's 0.926). Queries are now classified by shape — identifier spellings, quoted code and file paths lean BM25; paraphrases lean semantic (src/core/query-profile.ts) — and the local-MiniLM hybrid reaches parity with keyword-only (0.926) on the Flask set while keeping hybrid's edge on paraphrase queries. The voyage columns predate adaptive fusion (query-time change only; the committed baselines remain the equal-weight reference). Re-run any of it yourself: oce eval --no-adaptive-fusion A/Bs the old behavior.

The June baseline this section used to quote (hybrid 0.977) no longer reproduces on today's engine (0.943) — drift the harness now catches on every run instead of silently shipping.

Reports record which search mode produced them (hybrid vs keyword-only), and baseline comparisons warn loudly on a mode mismatch.

Cross-repo validation: the bundled set only proves retrieval on this repository. eval/external/ holds gold sets for codebases OCE has never seen — starting with Flask 3.1.2 (Python, 27 cases, recall@10 0.926 keyword-only) — with committed baselines and a workflow for adding more. A ranking change that wins on eval/oce.eval.json should be re-run there before it counts as progress.

A case file is a JSON array (or { cases: [...] }):

{ "id": "rrf-fusion", "query": "where is reciprocal rank fusion implemented", "expectedPaths": ["src/core/retriever.ts"] }

Keep your eval sets out of the index by listing their directory in .contextignore — a gold set maps queries to answers, so indexing it contaminates the rankings it measures.

📦 Requirements

• Node.js 22 or newer (22 and 24 LTS install without a compiler; Node 20 is end-of-life and the SQLite binding no longer ships prebuilt binaries for it)

• A chat model — oce setup lists free options (Ollama locally, or a free-tier key from Google, Groq, Cerebras, or OpenRouter)

• Embeddings are optional: without them, search runs keyword-only (BM25). For semantic ranking use Ollama (nomic-embed-text, free and local), the optional @huggingface/transformers package (--provider local), or a Voyage / OpenAI key

• Platforms without a sqlite-vec build (e.g. win32-arm64, Alpine) also run keyword-only

📄 License

MIT © Gain