# Repo Doctor

**A hand-written agent loop drives a custom MCP server over stdio to produce
scored, evidence-cited audits of public GitHub repos - benchmarked for
run-to-run score stability, not just demoed once.**


Repo Doctor is a small, from-scratch agent: no LangChain, no SDK tool-runner,
no agent framework. The loop itself is 115 lines (`agent/loop.py`, the whole
package under 1,000) implementing tool use directly - call an LLM, execute
whatever tools it asks for, feed the results back, repeat until it emits one
JSON-schema-validated structured output. The tools themselves live behind a custom
[MCP](https://modelcontextprotocol.io) server, the same protocol Claude
Desktop speaks, so the identical four tools work whether the agent driving
them is this CLI or Claude Desktop's own agent (see
[docs/claude-desktop.md](docs/claude-desktop.md)).

## What's inside

```
GitHub repo URL
      |
      v
repo-doctor CLI -- hand-written agent loop (no framework, no SDK tool-runner)
      |
      +-- Claude claude-opus-4-8  --or--  Ollama llama3.1:8b (tool-use turns)
      |
      v
    stdio  <-- MCP protocol -->  repo-doctor MCP server (FastMCP)
                                    |
                                    +-- fetch_repo_meta      GitHub API: description, topics, license, stars
                                    +-- read_repo_files      README, .gitignore, LICENSE, 2-level file tree
                                    +-- scan_git_history     full history: secrets, emails, large binaries
                                    +-- check_repo_hygiene   CI workflows, tests dir, pinned deps, demo media
      |
      v
submit_report -- client-side tool, JSON-schema-validated structured output
      |
      +-- repo-doctor audit <url>       one markdown report: score /100, evidence per line, proposed fixes
      +-- repo-doctor bench --runs 3    the same audit repeated per repo, for score stability
```

Plain Python on purpose - `anthropic`, `mcp`, `requests`, `jsonschema`, and
nothing that hides the tool-use loop from you.

## Results

`repo-doctor bench` audits a fixed list of 8 public repos, 3 times each, on
`claude-opus-4-8`, and reports the mean score, the spread (max - min) across
runs, mean latency, tokens, estimated cost, and failures, per repo:

| Repo | Runs | Mean score | Spread | Mean s | Tokens in/out | Est. cost | Failures |
|---|---|---|---|---|---|---|---|
| Infojitsu/nn-digit-recognition | 3 | 69 | 1 | 32.2 | 41702/7116 | $0.39 | 0 |
| Infojitsu/face-recognition-from-scratch | 3 | 75 | 6 | 36.3 | 66220/7942 | $0.53 | 0 |
| psf/requests | 3 | 72 | 3 | 51.2 | 100569/11752 | $0.80 | 0 |
| pallets/flask | 3 | 69 | 6 | 51.4 | 124802/11412 | $0.91 | 0 |
| tiangolo/fastapi | 3 | 79 | 9 | 49.2 | 97767/7265 | $0.67 | 0 |
| karpathy/micrograd | 3 | 68 | 5 | 28.9 | 41465/6151 | $0.36 | 0 |
| kennethreitz/setup.py | 3 | 55 | 4 | 33.2 | 58267/7262 | $0.47 | 0 |
| openai/gpt-2 | 3 | 53 | 0 | 41.6 | 64776/9551 | $0.56 | 1 |

23 of 24 audits completed; the one failure (a gpt-2 run) is counted in the
table, not discarded. The full 24-audit bench cost $4.69 in API usage.

The spread column is the point: a rubric graded by an LLM does not return the
same number twice, and burying that fact would make the tool look more
precise than it is. For grounding, one real single-repo audit outside the
bench loop: auditing
[Infojitsu/nn-digit-recognition](https://github.com/Infojitsu/nn-digit-recognition)
scored **65 / 100** (identity 15/20, README 17/20, hygiene 3/20, history
safety 20/20, reproducibility 10/20) in 3 agent-loop iterations, using
13,659 input / 2,945 output tokens (about $0.14 on claude-opus-4-8) - the
same audit `repo-doctor audit <url>` reproduces on demand.

## Quickstart

Prereqs: Python 3.11+, a public GitHub repo to audit, and either an
Anthropic API key or [Ollama](https://ollama.com) for a free local fallback.

```bash
python -m venv .venv && .venv/Scripts/activate    # or source .venv/bin/activate
pip install -e .

export ANTHROPIC_API_KEY=sk-ant-...   # never commit this; rotate any key you've shared or demoed with
repo-doctor audit https://github.com/Infojitsu/nn-digit-recognition
```

Prefer a file? `cp .env.example .env` and put the key there instead - the CLI
loads `.env` from the directory you run it in (the file is gitignored, and a
real environment variable always wins over a `.env` entry).

No API key, no cost - run entirely local instead:

```bash
ollama pull llama3.1:8b
repo-doctor audit https://github.com/Infojitsu/nn-digit-recognition --provider ollama
```

Write a report to a file with `-o report.md`, or run the stability bench
with `repo-doctor bench --runs 3 -o docs/bench.md`. To drive the same MCP
server from Claude Desktop instead of this CLI, see
[docs/claude-desktop.md](docs/claude-desktop.md).

## Honest evaluation, by construction

- Every score line in a report cites the exact tool output it came from (the
  schema's `evidence` field) - the agent cannot hand out unearned points
  without pointing at where they came from.
- `submit_report` is validated against a JSON Schema (Draft 2020-12) before
  it is accepted; a report that fails validation gets one corrective retry,
  and a second failure ends the run as a documented failure rather than a
  silently wrong score.
- History safety is graded from a deterministic scanner over the full git
  history (regex-matched secrets, personal emails, large binaries), not LLM
  judgment - the model only reads the (capped) findings, it does not decide
  what counts as a secret.
- `repo-doctor bench` always audits the same 8 repos in the same order and
  counts failures explicitly, rather than only reporting the runs that
  happened to succeed.
- 85 pytest tests (84 offline + 1 live-marked) cover the tool loop, both
  providers, all four MCP server tools, the hygiene/history/file scanners,
  a real-stdio findings round-trip, and a pure-ASCII policy for every
  tracked text file.

## Limitations

- LLM judgment varies run to run - see the spread column above; a score is a
  snapshot from one run, not a certified measurement.
- The git-history secret scan is regex-based over known key/token shapes,
  not entropy-based, so it will miss high-entropy secrets that don't match a
  known pattern.
- Public GitHub repos only - no private repos, no non-GitHub hosts.
- Scores are a rubric, not ground truth: a low score means the repo is
  missing rubric items, not that the underlying code doesn't work.
- `--provider ollama` runs fully local and free (`llama3.1:8b` via Ollama),
  but local-model tool-calling is noticeably less reliable than Claude's.
  In our own smoke test, llama3.1:8b made three tool calls and then failed
  `submit_report` schema validation twice; the run ended with the loop's
  honest-failure exit (`audit failed: submit_report failed schema validation
  twice`, exit code 1) rather than a report. Treat Ollama as a zero-cost way
  to try the tool, not a like-for-like substitute for the bench numbers
  above (which are all `claude-opus-4-8`).

## License

Code is [MIT](LICENSE). Repo Doctor reads public GitHub repositories at
audit time (via the GitHub API and a local clone into a gitignored cache,
`.repodoctor_cache/`) and never redistributes repo contents; note the clone
cache persists on disk under `.repodoctor_cache/` until you delete it
manually.

<!--
GitHub About: LLM agent with a hand-written tool loop and a custom MCP server that produces scored, evidence-cited audits of GitHub repos.
Topics: mcp, llm-agents, tool-use, anthropic, claude, ollama, python, github-api, developer-tools
-->