Repo Doctor

A hand-written agent loop drives a custom MCP server over stdio to produce scored, evidence-cited audits of public GitHub repos - benchmarked for run-to-run score stability, not just demoed once.

Repo Doctor is a small, from-scratch agent: no LangChain, no SDK tool-runner, no agent framework. The loop itself is 115 lines (agent/loop.py, the whole package under 1,000) implementing tool use directly - call an LLM, execute whatever tools it asks for, feed the results back, repeat until it emits one JSON-schema-validated structured output. The tools themselves live behind a custom MCP server, the same protocol Claude Desktop speaks, so the identical four tools work whether the agent driving them is this CLI or Claude Desktop's own agent (see docs/claude-desktop.md).

What's inside

GitHub repo URL
      |
      v
repo-doctor CLI -- hand-written agent loop (no framework, no SDK tool-runner)
      |
      +-- Claude claude-opus-4-8  --or--  Ollama llama3.1:8b (tool-use turns)
      |
      v
    stdio  <-- MCP protocol -->  repo-doctor MCP server (FastMCP)
                                    |
                                    +-- fetch_repo_meta      GitHub API: description, topics, license, stars
                                    +-- read_repo_files      README, .gitignore, LICENSE, 2-level file tree
                                    +-- scan_git_history     full history: secrets, emails, large binaries
                                    +-- check_repo_hygiene   CI workflows, tests dir, pinned deps, demo media
      |
      v
submit_report -- client-side tool, JSON-schema-validated structured output
      |
      +-- repo-doctor audit <url>       one markdown report: score /100, evidence per line, proposed fixes
      +-- repo-doctor bench --runs 3    the same audit repeated per repo, for score stability

Plain Python on purpose - anthropic, mcp, requests, jsonschema, and nothing that hides the tool-use loop from you.

Related MCP server: github-security-mcp

Results

repo-doctor bench audits a fixed list of 8 public repos, 3 times each, on claude-opus-4-8, and reports the mean score, the spread (max - min) across runs, mean latency, tokens, estimated cost, and failures, per repo:

23 of 24 audits completed; the one failure (a gpt-2 run) is counted in the table, not discarded. The full 24-audit bench cost $4.69 in API usage.

The spread column is the point: a rubric graded by an LLM does not return the same number twice, and burying that fact would make the tool look more precise than it is. For grounding, one real single-repo audit outside the bench loop: auditing Infojitsu/nn-digit-recognition scored 65 / 100 (identity 15/20, README 17/20, hygiene 3/20, history safety 20/20, reproducibility 10/20) in 3 agent-loop iterations, using 13,659 input / 2,945 output tokens (about $0.14 on claude-opus-4-8) - the same audit repo-doctor audit <url> reproduces on demand.

Quickstart

Prereqs: Python 3.11+, a public GitHub repo to audit, and either an Anthropic API key or Ollama for a free local fallback.

python -m venv .venv && .venv/Scripts/activate    # or source .venv/bin/activate
pip install -e .

export ANTHROPIC_API_KEY=sk-ant-...   # never commit this; rotate any key you've shared or demoed with
repo-doctor audit https://github.com/Infojitsu/nn-digit-recognition

Prefer a file? cp .env.example .env and put the key there instead - the CLI loads .env from the directory you run it in (the file is gitignored, and a real environment variable always wins over a .env entry).

No API key, no cost - run entirely local instead:

ollama pull llama3.1:8b
repo-doctor audit https://github.com/Infojitsu/nn-digit-recognition --provider ollama

Write a report to a file with -o report.md, or run the stability bench with repo-doctor bench --runs 3 -o docs/bench.md. To drive the same MCP server from Claude Desktop instead of this CLI, see docs/claude-desktop.md.

Honest evaluation, by construction

• Every score line in a report cites the exact tool output it came from (the schema's evidence field) - the agent cannot hand out unearned points without pointing at where they came from.

• submit_report is validated against a JSON Schema (Draft 2020-12) before it is accepted; a report that fails validation gets one corrective retry, and a second failure ends the run as a documented failure rather than a silently wrong score.

• History safety is graded from a deterministic scanner over the full git history (regex-matched secrets, personal emails, large binaries), not LLM judgment - the model only reads the (capped) findings, it does not decide what counts as a secret.

• repo-doctor bench always audits the same 8 repos in the same order and counts failures explicitly, rather than only reporting the runs that happened to succeed.

• 85 pytest tests (84 offline + 1 live-marked) cover the tool loop, both providers, all four MCP server tools, the hygiene/history/file scanners, a real-stdio findings round-trip, and a pure-ASCII policy for every tracked text file.

Limitations

• LLM judgment varies run to run - see the spread column above; a score is a snapshot from one run, not a certified measurement.

• The git-history secret scan is regex-based over known key/token shapes, not entropy-based, so it will miss high-entropy secrets that don't match a known pattern.

• Public GitHub repos only - no private repos, no non-GitHub hosts.

• Scores are a rubric, not ground truth: a low score means the repo is missing rubric items, not that the underlying code doesn't work.

• --provider ollama runs fully local and free (llama3.1:8b via Ollama), but local-model tool-calling is noticeably less reliable than Claude's. In our own smoke test, llama3.1:8b made three tool calls and then failed submit_report schema validation twice; the run ended with the loop's honest-failure exit (audit failed: submit_report failed schema validation twice, exit code 1) rather than a report. Treat Ollama as a zero-cost way to try the tool, not a like-for-like substitute for the bench numbers above (which are all claude-opus-4-8).

License

Code is MIT. Repo Doctor reads public GitHub repositories at audit time (via the GitHub API and a local clone into a gitignored cache, .repodoctor_cache/) and never redistributes repo contents; note the clone cache persists on disk under .repodoctor_cache/ until you delete it manually.