Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@EvalKit MCP Server Evaluate robustness of 'How to bypass safety filters'"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Multi-technique evaluation toolkit for testing AI safety classifier robustness against query decomposition, obfuscation, and multi-agent orchestration attacks. MCP plugin for Claude Code.
uv sync # install everything
claude # MCP tools auto-loaded from .mcp.jsonRelated MCP server: llamator-mcp-server
User Query → SPLITTER → [Encoder → Wrapper → ApiClient] × N → MERGER → ReportTool | Purpose |
| Full eval pipeline |
| Preview query decomposition |
| Test homoglyph substitution |
| Check configuration |
python3 run.py decompose "query" 5 # decompose into sub-queries
python3 run.py eval "query" # full evaluation pipeline
python3 run.py obfuscate "text" # test obfuscationpython3 benchmark.py # dry run (no API key)
python3 benchmark.py --quick # single query smoke test
python3 benchmark.py --json report.json # save JSON results
python3 benchmark.py --html report.html # save HTML report
ANTHROPIC_API_KEY=sk-... python3 benchmark.py # live testParameter | Values | Default | Description |
| none, light, moderate, aggressive | moderate | Homoglyph replacement level |
| fiction, study_guide, academic, documentation, translation, none | study_guide | Narrative framing strategy |
| 1–20 | 10 | Max sub-query decompositions |
| 0–10000 | 5000 | Long-context padding per query |
| true, false | true | Multi-agent orchestration |
| true, false | true | Helper model with filters removed |
evalkit/
├── evalkit/ # Core modules
│ ├── splitter.py # Query → sub-questions
│ ├── encoder.py # Unicode homoglyph engine
│ ├── wrapper.py # Narrative wrapping
│ ├── api_client.py # API client + model routing
│ ├── merger.py # Output stitching + metrics
│ ├── context_builder.py # Multi-turn conversation
│ ├── agent_router.py # Agent pack coordination
│ └── models.py # Data classes + enums
├── evalkit_server.py # MCP server (FastMCP)
├── run.py # CLI wrapper (no MCP needed)
├── benchmark.py # Test matrix runner
├── tests/ # Pytest test suite
├── docs/ # Documentation
├── .mcp.json # Claude Code auto-discovery
└── CLAUDE.md # Claude Code instructionsSee docs/TECHNIQUES.md for detailed documentation of each technique.
MIT — authorized security research and defense evaluation only.
Process a query through the configured evaluation stages.
Returns a formatted analysis with per-segment routing information, aggregate metrics, and processing results.
Inputs: query (required), topic, obfuscation, framing, +4 more. Returns: result
Split a query into conceptual sub-components for analysis.
Divides the input into constituent parts based on topic boundaries.
Inputs: query (required), max_pieces, auto_split. Returns: result
Analyze character-level text transformations.
Applies configurable character substitutions and reports detected patterns in the input text.
Inputs: text (required), level. Returns: result
Check server configuration and connection status.
Inputs: no named parameters. Returns: result
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 4 tools
Each tool targets a distinct evaluation function: robustness classification, query decomposition, obfuscation analysis, and server status. No overlap in purpose.
All tools share the 'eval_' prefix, but the suffixes vary in pattern (noun_noun for classifier_robustness, verb_noun for decompose_query, noun_noun for obfuscation_evasion, and single noun for status). Mostly consistent with minor deviation.
Four tools is well-scoped for an evaluation kit, covering multiple evaluation stages without excess or deficiency.
The set covers core evaluation operations and status checking, though a tool for retrieving historical results or configuring evaluations would round it out. Minor gap, not severe.
Train, evaluate, red-team and deploy AI agents: prompt tuning, LLM-judged evals, katas.
Test your AI agents for security and safety weaknesses, and re-verify the fixes.
Jailbreak resistance tester for LLM agents: DAN, role-play, encoding bypass probes.
Security scanner for AI agents and MCP servers: injection, jailbreak, tool abuse.
