Context Budget Engine

Your turn

Connect it to the AI assistant you already use. No API key needed.

git clone https://github.com/sivram04/context-budget-engine && cd context-budget-engine
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e .
claude mcp add cbe -- cbe-mcp

Then ask Claude Code something about your codebase and tell it to use read_smart. Run savings at any time to see how many tokens it did not have to read.

Related MCP server: project-graph-mcp

The principles

Read the part, not the file. Measure, don't assume. A change ships only if it is cheaper and solves as many tasks. The loop edits policies, never the judge. Report the noise, not just the best run.

What it does for you

What it will not do

• It never edits your code. It only decides what the assistant reads.

• The loop cannot grade itself. It may only write policy files: never the tasks, the judge, the gate or the agent.

• It never claims a win the gate did not measure. Rejected rounds and noisy runs are published below, next to the good ones.

• It never sends your code anywhere. The MCP tools run locally; only the optional summarize strategy calls a model, and only if you configure one.

How it works

            ┌──────── cbe improve (self-improvement loop) ────────┐
            │  run → measure → diagnose → propose policy → gate   │
            └───────────────┬─────────────────────────────────────┘
                            │ edits only the policy (policies/vN.yaml)
 test agent / Claude Code   ▼
        └──► read_smart(path, query) ──► policy picks a strategy ──► reduced view
                                          full · skeleton · extract · grep_window
                                          log_tail · head · summarize (cheap model)
 benchmark + judge (read-only) ──► results/*.jsonl ──► cbe report

The gate (src/cbe/improve.py): a candidate must be cheaper on the 27 training tasks by a minimum margin with no meaningful loss of success, and must not lose success on the 13 held-out tasks. Proposals come from an LLM that reads the failures and token hot spots, and from a model-free local search so the loop still works with weak models.

Results

All raw per-task results are in results/. Agent model: Gemini 3.1 Flash-Lite.

1. Baseline vs engine, 3 runs each, all 40 tasks

Paired by task, v1 is cheaper on 27/40 tasks (median ratio 0.75). The −40% is far outside the ~6% run-to-run spread; the success difference is within noise. On bug fixes alone, the first run used 49,476 → 24,490 tokens per task (−50%).

2. Claude Code, with vs without the MCP tools

Same model (Sonnet), same repo copy, isolated config; only the reading tools differ (scripts/claude_code_check.py).

Fewer input tokens on 8 of 10 questions. Caveats: one run per arm; the files are short; and the list-price cost came out about equal ($0.031 vs $0.033 per question) because the MCP tool calls caused extra cache writes, which are priced higher than cache reads.

3. The self-improvement loop, honestly

First attempt (1 run per evaluation): 6 rounds, all rejected. Three times the optimizer proposed a policy identical to v1, and those re-runs of the same policy scored 88.9%, 96.3% and 92.6%. One run cannot tell a real gain from noise, so the gate was rebuilt.

Noise-aware loop (3 runs per evaluation, ≥5% cheaper, ≤2 pts success drop):

One small verified improvement, and five candidates refused, four of which would have traded success for tokens. All of them came from the model-free search: the Flash-Lite optimizer only re-proposed existing policies. Whole study (noise check + loop): $3.49.

Usage

MCP tools

Claude Desktop / Cursor: add to claude_desktop_config.json / .cursor/mcp.json:

{ "mcpServers": { "cbe": { "command": "/full/path/to/.venv/bin/cbe-mcp" } } }

Commands

Any OpenAI-compatible endpoint works for the benchmark and the loop (Gemini, Groq, local Ollama). Copy .env.example to .env:

CBE_PROVIDER=gemini GEMINI_API_KEY=... CBE_MAX_USD=3.0

Benchmark

40 tasks on a fixed snapshot of a real codebase (bench_repos/gateway, an LLM gateway with 87 tests):

• 20 code-understanding questions with exact answers, e.g. "What is the default cosine-similarity threshold for a semantic cache hit?"

• 20 bug fixes: one bug is planted per task from a plain-language report; it passes only if the repo's own tests pass afterwards.

scripts/verify_tasks.py proves every planted bug breaks its tests and every expected answer exists in the source. The 27 train / 13 held-out split is a stable hash of the task id.

Project layout

src/cbe/strategies.py   ways to read less (AST-based for Python, generic for other languages)
src/cbe/policy.py       policy schema + strict validation (what the loop may change)
src/cbe/engine.py       picks a strategy per read, accounts tokens, keyless fallback
src/cbe/agent.py        small JSON-protocol coding agent used for measurement
src/cbe/bench/          tasks, workspaces, judge (read-only), runner
src/cbe/improve.py      diagnose / propose / noise-aware gate loop
src/cbe/mcp_server.py   MCP tools for real assistants
policies/               v0-full (baseline), v1 (hand-written)
results/                raw results, loop logs and accepted policies (v2)

Contributing

pip install -e ".[dev]"
pytest
ruff check src tests scripts
python scripts/verify_tasks.py

Ideas that would help most: strategies for more languages (tree-sitter), a stronger optimizer model for the loop, and benchmarks on larger codebases.

License

MIT. See LICENSE.