cold-run

I tested 47 agent skills. I deleted 23 of them. Three were making Claude's code worse.

Then I built the tool that does it, pointed it at the 24 survivors, and it graded three of the first five NEGATIVE.

That is not a bug. It is the most interesting thing I found, and I am publishing it rather than waiting for you to find it: runs/bare-baseline/.

npx cold-run

The finding

A skill that does nothing is the boring failure. The common one is worse:

A skill spends the agent's attention. It buys a specific check, and it pays for that check out of everything else the task needed.

Asked to build a cached lookup, the agent following amplification-check added jitter and single-flight — exactly what the skill asks for. The agent that had never seen the skill added those and an LRU cap, stale-while-revalidate, and negative caching. The skilled version's cache grows without bound.

Asked to add retries inside a Lambda, the skilled agent produced a tidy retry-budget table and a blanket except that retries a definitive 404 as eagerly as a timeout.

You cannot see this by reading a skill. Both of those skills are well written. You only see it by running the task twice and diffing the artifacts.

What it does

Finds your skills, picks an adversarial test task for each one, runs a cold/skilled pair in parallel, and tells you which of your skills change nothing.

cold-run v0.1  —  30 skills in ~/.claude/skills

  REAL      retry-guard                added a breaker + cap; cold declined one
  NONE      be-thorough                cold output was identical
  NONE      check-edge-cases           cold found two more edge cases
  NEGATIVE  refactor-first             skilled dropped a null check cold kept
  ...

  11 change nothing.
   2 make the output worse.
  17 earn their place.

  Deleting the 13 frees ~7,400 tokens of always-loaded
  context, every session.

  full report      -> cold-run-report.md
  every transcript -> .cold-run/

It never deletes anything. It writes the report and every transcript, and the decision is yours.

Cost is real and it asks first. Four model calls per skill. On a library over 12 it tests a spread of five so your first run is cheap; --all does the rest.

npx cold-run                    # auto-finds your skills
npx cold-run path/to/skills     # or point it somewhere
npx cold-run --all --yes        # everything, no prompt

Runs through your local claude CLI if you have one, otherwise ANTHROPIC_API_KEY. Zero dependencies — npx starts instantly.

As an MCP server

Same audit, driven by whatever agent you already talk to — Claude Code, Claude Desktop, Cursor. It uses your agent's own subagents, so there's no second API key and no hidden spend.

claude mcp add cold-run -- npx -y --package=cold-run cold-run-mcp

{
  "mcpServers": {
    "cold-run": {
      "command": "npx",
      "args": ["-y", "--package=cold-run", "cold-run-mcp"]
    }
  }
}

Then just ask: "which of my skills actually do anything?"

The five verdicts

The 24 that survived, and the asterisk on them

The library that produced this tool is in skills/. Every one has a recorded cold/skilled pair behind it — see EVIDENCE.md and runs/ledger.md.

Read this before you install them. That audit ran both agents under my machine's global operating contracts — house rules that force stop-and-ask behaviour and a fixed reporting format. cold-run deliberately strips all of that. Under the bare baseline, three of the first five survivors grade NEGATIVE. Both results are published, in full, with transcripts: runs/bare-baseline/.

Which number applies to you depends on what your own CLAUDE.md already forces. Neither has been run at full scale under the bare baseline. I would rather ship the contradiction than hide it. The 23 that didn't make it, and why none could be repaired, are in runs/dropped.md.

A few of the survivors, so you can judge the bar:

• yagni-audit — "implement a rate limiter." Cold wrote ~700 lines: sync, async, keyed, Redis, Lua, decorators. Skilled wrote 15.

• trust-source-check — cold authenticated services with a shared static env-var token. Skilled required a JWKS-verified OIDC token with audience pinning.

• reverse-path-proof — skilled ran a real Postgres container and found the rollback aborts on a 255-character name, wiping every name in the table. Cold had the same bug.

• pit-of-success — cold shipped immediate hard delete as the default. Skilled made it reversible for 30 days and put purge behind a confirmation echoing the user id.

Install them the normal way, or don't. The tool is the point.

Skills are graded on two gates here, not one

1. Cold-run delta — does it change the artifact? 23 skills failed this. → EVIDENCE.md

2. Routing — does the description actually get the skill loaded? A perfect skill with a description that never fires is dead weight that still costs tokens. Tested as a classifier against 360 written requests; two rounds failed before all 24 passed. → tests/router/

A skill passing one and failing the other is still broken.

vs. skill-doctor

skill-doctor is a linter. It reads your skills and checks frontmatter validity, broken resource links, and whether a trigger description looks weak, then scores 0-100. It never runs anything.

cold-run is the other half. It executes the skill and an agent that cannot see the skill, on the same task, and compares the artifacts.

A skill can score 100/100 on structure and still be a no-op. That is the failure this catches, and it is the common one — 23 of my 47 were structurally perfect.

Run both. They fail on different things.

Prior art

obra/superpowers is the workflow layer — brainstorm, plan, execute, verify. There is no overlap with this: zero workflow skills here, zero domain checks there. Its writing-skills prescribes the same subagent testing this tool automates.

License

MIT.