# Promptest MCP

An MCP server that grades your AI prompts against [Promptest](https://promptest.cortexresearch.group)'s validated 5-category rubric, tracks what actually happened when you used them, and gives you guidance grounded in the [CRG-RIS study](https://github.com/cortexresearch/promptest-crg-ris-study) of what really predicts good outcomes — not just what looks good on paper.

## Why this exists

Promptest's rubric reliably scores prompt quality. But a CRG-RIS evaluation of that rubric found something more specific and more useful: **overall score didn't predict cost or scope discipline on a real coding task — but whether the prompt named the bug's root cause and exact fix location did**, by a wide margin (37–41% lower cost, and the only variant that avoided a security-flagged over-broad fix).

This MCP server bakes that finding directly into its guidance, and — because it's an MCP tool that lives inside your actual coding sessions — it can track your own real outcomes over time, not just score prompts in isolation the way the web app does.

## Tools

- **`grade_prompt`** — scores a prompt against the 5-category rubric, returns a grade, category breakdown, and guidance. Returns a `taskId`.
- **`log_outcome`** — record what actually happened: did it work, how many turns did it take, did the fix stay in scope, did you name the root cause/location. Pass the `taskId` from `grade_prompt`.
- **`get_prompting_history`** — your personal history and stats, including the CRG-RIS-validated comparison: average turns when you named the root cause/location vs. when you didn't.
- **`explain_rubric`** — explains the rubric and the CRG-RIS findings without grading anything.

## Install

### 1. Build

```bash
git clone https://github.com/cortexresearch/promptest-mcp.git
cd promptest-mcp
npm install
npm run build
```

### 2. Add to your MCP client config

**Claude Code** (`.claude/settings.json` or via `claude mcp add`):

```json
{
  "mcpServers": {
    "promptest": {
      "command": "node",
      "args": ["/absolute/path/to/promptest-mcp/dist/index.js"],
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-..."
      }
    }
  }
}
```

**Claude Desktop** (`claude_desktop_config.json`): same shape, under `mcpServers`.

### 3. Set your API key

Set exactly one of these in the `env` block above, matching whichever provider you want as the judge:

| Env var | Provider |
|---|---|
| `ANTHROPIC_API_KEY` | Anthropic (Claude) |
| `OPENAI_API_KEY` | OpenAI |
| `GOOGLE_API_KEY` | Google (Gemini) |
| `OPENROUTER_API_KEY` | OpenRouter (any model) |

The server auto-detects which provider to use from whichever key is set. To force a specific provider or model, set `PROMPTEST_PROVIDER` (`anthropic` \| `openai` \| `google` \| `openrouter`) and/or `PROMPTEST_MODEL`.

**Your key never leaves your machine except to call the provider you chose.** This server has no backend, sends nothing to Promptest or Cortex Research Group, and doesn't phone home.

## Where your data lives

History is stored locally at `~/.promptest/history.json` — plain JSON, nothing hidden, delete it any time to start fresh.

## Background

- [Promptest](https://promptest.cortexresearch.group) — the web app this rubric was built for
- [CRG-RIS study](https://github.com/cortexresearch/promptest-crg-ris-study) — the research this MCP's guidance is grounded in, including the honestly-reported finding that the rubric's overall score alone didn't predict real-world cost, and what did

## License

MIT