CodePecker

An MCP server that reviews a piece of code across four dimensions — security, standards, production readiness, sustainability — automatically fixes the issues, verifies the fix by running the code's tests, and reports what it did. Any MCP-capable agent (Claude Code, Codex, Copilot) can call it as a tool; there's also a CLI for local demos.

review → remediate → run tests → repeat (bounded)   →   findings + scorecard + fixed code + diff + citations

Quickstart (30 seconds)

git clone <this repo> && cd CodePecker
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt && pip install -e . --no-deps
cp .env.example .env                              # add a free Groq key — see Setup
codepecker examples/urlshortener_sample/store.py  # review a flawed sample file

No key yet? pytest -q runs the whole suite fully offline. Full details, other providers, and MCP client wiring are below.

Related MCP server: code-review-mcp-server

How it works

For each of the four dimensions, CodePecker gathers findings two ways:

• Deterministic checks (regex/code) for rules that must be caught reliably — hardcoded secrets, eval/unsafe deserialization, bare except, missing tests. No LLM, so they never "forget".

• An LLM judge for the nuanced rules (input validation, logging, timeouts, N+1 queries, …), with guardrails: it may only cite rules from the batch it was given, any evidence it quotes must appear in the code, and severity/dimension come from the rule metadata — hallucinated findings are dropped in code.

Each rule lives in a markdown file in codepecker-skill/rules/ (RAG), tagged deterministic: true|false so it's enforced by exactly one path. A hand-written, bounded agent loop then asks the model to remediate and re-runs the tests — a fix that resolves a finding but breaks the tests is not accepted.

The rule corpus is packaged as an Agent Skill: codepecker-skill/ is a valid skill (a SKILL.md entry point over the same rules/ folder). So the same corpus serves two surfaces from one source of truth — a Claude agent can load it as a skill to suggest fixes at the desk, and the MCP server reads the same rules/ to enforce them (deterministic checks, guardrails, test-verified remediation). One corpus, no drift: the skill suggests, the MCP tool guarantees.

Setup

Requires Python 3.10+.

git clone <this repo> && cd CodePecker
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install -e . --no-deps            # makes `codepecker` + `python -m codepecker.*` work
cp .env.example .env                  # then add your key (below)

Verify the install (fully offline — no key needed):

pytest -q                             # the test suite should pass

Keys — the default needs just one. Text runs on Groq (fast Llama 3.3 70B) and embeddings run locally (no key). Put your Groq key in .env:

GROQ_API_KEY=...        # free key at https://console.groq.com/keys

Provider-agnostic via LiteLLM — switch model or provider with no code change, e.g. CODEPECKER_TEXT_MODEL=openai/gpt-4o (one key does both), or go fully offline with CODEPECKER_TEXT_MODEL=ollama/llama3.1. See .env.example.

Usage

CLI (local demo)

codepecker examples/urlshortener_sample/store.py
# or:  python -m codepecker.cli examples/urlshortener_sample/store.py

Prints the findings, a rule-coverage scorecard (how the code did against all rules per dimension — passes included, so you can attest coverage, not just violations), the remediated code, a unified diff, the rules cited, and a metrics summary. Each run is appended to metrics.jsonl.

If the file imports sibling modules, pass them with --support (repeatable) so the sandbox can run the code's tests instead of failing on the import:

python -m codepecker.cli examples/urlshortener_sample/service.py \
  --support examples/urlshortener_sample/store.py \
  --support examples/urlshortener_sample/shortener.py

Useful flags:

A realistic, multi-file example. examples/auth_sample/ is an auth module split across several files (hardcoded secrets, weak crypto, missing validation) — closer to real code than a single snippet. Review its entry point, bringing the siblings it imports so the sandbox can run the tests:

python -m codepecker.cli examples/auth_sample/auth.py \
  --support examples/auth_sample/crypto.py \
  --support examples/auth_sample/db.py

See examples/GROUND_TRUTH.md for the exact issues this auth sample is seeded with.

Reliable live demo: the loop makes many LLM calls, so a free tier's tokens-per-minute cap can throttle a full run. The loop is resilient — a mid-run rate limit is recorded and the review still completes (with degraded coverage noted) rather than crashing. For a smooth end-to-end demo, use a higher-limit tier or run the text model locally: CODEPECKER_TEXT_MODEL=ollama/llama3.1 (no key, no limits).

MCP server (in a coding agent)

CodePecker speaks MCP over stdio — no ports, no daemon. Every client points at the same command; only the config file and its shape differ:

• command /absolute/path/to/CodePecker/.venv/bin/python

• args ["-m", "codepecker.server"]

• env GROQ_API_KEY (or whichever provider key your model IDs need)

Use the absolute path to the venv's Python so the agent inherits CodePecker's dependencies. Sanity-check that it launches (it waits on stdio; Ctrl-C to exit):

python -m codepecker.server

claude mcp add codepecker \
  --env GROQ_API_KEY=your-key \
  -- /absolute/path/to/CodePecker/.venv/bin/python -m codepecker.server

Add --scope project to share it with your team via a checked-in .mcp.json.

Create .vscode/mcp.json in the workspace — note the top-level servers key and the type field (VS Code's shape differs from the mcpServers one below):

{
  "servers": {
    "codepecker": {
      "type": "stdio",
      "command": "/absolute/path/to/CodePecker/.venv/bin/python",
      "args": ["-m", "codepecker.server"],
      "env": { "GROQ_API_KEY": "your-key" }
    }
  }
}

Open Copilot Chat → switch to Agent mode → codepecker shows up in the tools picker. (To avoid hardcoding the key, use VS Code's "inputs" secret prompt.)

Identical mcpServers block; only the file location differs:

• Cursor — .cursor/mcp.json (project) or ~/.cursor/mcp.json (global)

• Windsurf — ~/.codeium/windsurf/mcp_config.json

• Claude Desktop — claude_desktop_config.json (macOS: ~/Library/Application Support/Claude/)

{
  "mcpServers": {
    "codepecker": {
      "command": "/absolute/path/to/CodePecker/.venv/bin/python",
      "args": ["-m", "codepecker.server"],
      "env": { "GROQ_API_KEY": "your-key" }
    }
  }
}

Add to ~/.codex/config.toml (TOML, not JSON):

[mcp_servers.codepecker]
command = "/absolute/path/to/CodePecker/.venv/bin/python"
args = ["-m", "codepecker.server"]
env = { GROQ_API_KEY = "your-key" }

MCP config conventions move fast. If a client has renamed a key or moved its config file, check that client's own MCP docs — only the command / args / env values above are CodePecker-specific.

The server exposes one tool:

review_and_remediate(code, language="python", tests="", tests_dir="", support_files={})

• code — the source to review.

• tests (optional) — a separate test file; the code should import as solution (from solution import ...). Passing it runs the tests and suppresses the "no tests" finding.

• tests_dir (optional) — path to the code's test directory; confirms tests exist (rule RDY-03) without running them. This is how coverage is attested in detect-only mode (CODEPECKER_REMEDIATE=false), which skips test execution.

• support_files (optional) — {"sibling.py": "<source>", …} for modules the code (or its tests) imports, so they resolve in the sandbox instead of crashing test collection.

Detect-only vs. remediate is controlled by the CODEPECKER_REMEDIATE env var (default true); set it false to report findings without ever calling the model to rewrite code — the same behaviour as the CLI's --no-remediate.

Evaluation

python eval/run_eval.py

Runs CodePecker over the labeled golden set (eval/golden/) and reports precision/recall/F1 per dimension, remediation resolution + test-pass rates, and mean iterations/latency; writes eval/report.json. This is the "how do I know it's good?" evidence and is meant to run in CI. (It drives the full loop over every sample, so use a decent rate-limit tier.)

Design decisions (the short "why")

Project layout

src/codepecker/
  config.py            env-driven model IDs + tuning constants
  types.py             LLM Protocols (DIP/ISP) + the Finding type
  llm_client.py        the only module that talks to a provider (LiteLLM)
  vector_store.py      ChromaDB adapter (RAG index)
  knowledge/loader.py  parse + embed the markdown knowledge banks
  tools/
    deterministic_checks.py   code checks, keyed by rule id
    judge.py                  batched, guardrailed LLM judge
    run_tests.py              sandboxed pytest runner
  agent.py             review_and_remediate() — the bounded loop
  metrics.py           append-only metrics log + summary
  evaluation.py        pure detection/remediation metrics (used by eval/run_eval.py)
  cli.py               local demo runner
  server.py            FastMCP server (stdio)
codepecker-skill/      the corpus as an Agent Skill (one source of truth)
  SKILL.md             agent-facing entry point (the "suggest" surface)
  rules/               the rules: security/ standards/ readiness/ sustainability/
eval/                  golden samples + run_eval.py
tests/                 the test suite

Testing

pytest -q                       # 97 offline tests (local embeddings, faked LLM)
pytest -m "live or not live"    # + the 1 live acceptance test (needs GROQ_API_KEY)

The default suite is fully offline and deterministic; the one live test is opt-in.

Non-goals / next steps

MVP simplifications, called out honestly:

• Sandbox is a subprocess + timeout, not a container — production wants gVisor/a microVM with no network and resource limits.

• Deterministic checks are regex-based — production would use AST analysis.

• Local embeddings (all-MiniLM-L6-v2) trade recall for zero keys — swap in a hosted embedder for higher-quality retrieval at scale.

• Not yet: metadata-routed retrieval for very large rule sets, a metrics dashboard, real GitHub integration, runtime energy profiling, remote HTTP/Cloud Run deploy.