SkillBench MCP Server

A zero-dependency harness that runs held-out coding tasks with and without a skill file, then reports behavior deltas and variance. Packaged as a mcp server so users can adopt it quickly.

Why This Exists

Skill repos are getting daily attention, but the ecosystem has almost no proof that skills improve agent behavior.

Related MCP server: mcp-eval-harness

One-Day MVP

YAML/JSON task fixtures, paired runs, transcript diffing, pass/fail metrics, and a Markdown report.; one local MCP tool, install docs, example agent transcript.

What It Does Today

• Loads a small JSON task or project description.

• Checks whether the required fields and examples are present.

• Produces a JSON or Markdown readiness report.

• Ships with tests and a CI workflow so the repo is immediately verifiable.

Install

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .

Run

python -m skillbench_mcp_server.cli --input examples/sample_input.json --format markdown

Validate

python -m unittest discover -s tests

Validation Query

agent skills eval

Comparable Repos

• darkrishabh/agent-skills-eval: 656 stars - A test runner for agentskills.io-style AI agent skills

• caohaotiantian/agent-skills-eval: 8 stars -

• sasa-fajkovic/agents-skill-eval: 4 stars - agents-skill-eval.com

Launch Angle

show the same workflow from Claude Code/Codex/Cursor where possible

Publishing Gate

• must disclose permissions and local data access

• Working demo or executable artifact.

• Clear examples and tests.

• Honest limitations.

• No fake stars, duplicate repos, or automated engagement.

Status

Generated by RepoForge Radar. Treat this as a starting implementation, not a finished product.