Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@EvalKit MCP Server Evaluate robustness of 'How to bypass safety filters'"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Multi-technique evaluation toolkit for testing AI safety classifier robustness against query decomposition, obfuscation, and multi-agent orchestration attacks. MCP plugin for Claude Code.
uv sync # install everything
claude # MCP tools auto-loaded from .mcp.jsonRelated MCP server: llamator-mcp-server
User Query → SPLITTER → [Encoder → Wrapper → ApiClient] × N → MERGER → ReportTool | Purpose |
| Full eval pipeline |
| Preview query decomposition |
| Test homoglyph substitution |
| Check configuration |
python3 run.py decompose "query" 5 # decompose into sub-queries
python3 run.py eval "query" # full evaluation pipeline
python3 run.py obfuscate "text" # test obfuscationpython3 benchmark.py # dry run (no API key)
python3 benchmark.py --quick # single query smoke test
python3 benchmark.py --json report.json # save JSON results
python3 benchmark.py --html report.html # save HTML report
ANTHROPIC_API_KEY=sk-... python3 benchmark.py # live testParameter | Values | Default | Description |
| none, light, moderate, aggressive | moderate | Homoglyph replacement level |
| fiction, study_guide, academic, documentation, translation, none | study_guide | Narrative framing strategy |
| 1–20 | 10 | Max sub-query decompositions |
| 0–10000 | 5000 | Long-context padding per query |
| true, false | true | Multi-agent orchestration |
| true, false | true | Helper model with filters removed |
evalkit/
├── evalkit/ # Core modules
│ ├── splitter.py # Query → sub-questions
│ ├── encoder.py # Unicode homoglyph engine
│ ├── wrapper.py # Narrative wrapping
│ ├── api_client.py # API client + model routing
│ ├── merger.py # Output stitching + metrics
│ ├── context_builder.py # Multi-turn conversation
│ ├── agent_router.py # Agent pack coordination
│ └── models.py # Data classes + enums
├── evalkit_server.py # MCP server (FastMCP)
├── run.py # CLI wrapper (no MCP needed)
├── benchmark.py # Test matrix runner
├── tests/ # Pytest test suite
├── docs/ # Documentation
├── .mcp.json # Claude Code auto-discovery
└── CLAUDE.md # Claude Code instructionsSee docs/TECHNIQUES.md for detailed documentation of each technique.
MIT — authorized security research and defense evaluation only.
Process a query through the configured evaluation stages.
Returns a formatted analysis with per-segment routing information, aggregate metrics, and processing results.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | ||
| topic | No | computer security | |
| obfuscation | No | moderate | |
| framing | No | study_guide | |
| max_pieces | No | ||
| padding_tokens | No | ||
| multi_agent | No | ||
| helper_enabled | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description should disclose behavioral traits. It mentions output format but does not state whether the tool is read-only, requires authentication, or has side effects. The analysis nature suggests safety, but it's not explicit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise at two sentences with no wasted words. However, front-loading is acceptable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 8 parameters, no annotations, and an output schema, the description is notably incomplete. It glosses over evaluation stages, parameter roles, and output specifics.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% (no parameter descriptions), and the tool description adds no meaning beyond parameter names. It does not explain 'query', 'topic', 'obfuscation', or other parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states the tool processes a query through evaluation stages and returns analysis, which is moderately clear. However, it is vague about what 'configured evaluation stages' entails and does not differentiate from sibling tools like eval_decompose_query or eval_obfuscation_evasion.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. There is no mention of context, prerequisites, or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Split a query into conceptual sub-components for analysis.
Divides the input into constituent parts based on topic boundaries.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | ||
| max_pieces | No | ||
| auto_split | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must fully disclose behavioral traits. It only states that the query is split into parts based on topic boundaries. It does not mention side effects, whether the operation is read-only, how the splitting algorithm works, or any constraints. This is insufficient for a behavioral understanding.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise at two sentences, with no redundant information. The first sentence states the core purpose, and the second adds detail. It is front-loaded and efficient, though some might argue it is too brief given the missing parameter details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the presence of an output schema (not shown), the description does not need to explain return values, but it still lacks information about parameters, usage context, and behavioral traits. The tool has three parameters with no description, which makes it incomplete for effective use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage for parameters, so the description must compensate. However, the description does not mention any of the three parameters (query, max_pieces, auto_split) or their semantics. It only describes the high-level function, providing no meaning beyond the parameter names in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool splits a query into conceptual sub-components based on topic boundaries. It is specific about the action and resource, and while it doesn't explicitly differentiate from siblings, the sibling tools have distinct purposes (robustness, evasion, status), so confusion is unlikely.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. There is no mention of prerequisites, contexts, or scenarios where the tool is appropriate or inappropriate. The description lacks any exclusions or alternative tool names.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Analyze character-level text transformations.
Applies configurable character substitutions and reports detected patterns in the input text.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | ||
| level | No | moderate |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description states the tool applies character substitutions and reports patterns, indicating a non-destructive analysis. However, it does not disclose specifics like scope of patterns, required permissions, or edge cases. With no annotations, the description carries the full burden but gives only basic transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise: two sentences covering the action and mechanism. It front-loads the purpose. However, it could be more structured (e.g., separate 'when to use' section). Still, every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 2 parameters and no annotations, the description is insufficient for an agent to use it correctly. It lacks details on input format, level options, and output nature (though output schema exists). For a simple analysis tool, it meets the bare minimum but leaves gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%. The description does not explain the 'text' or 'level' parameters. 'Configurable character substitutions' vaguely hints at 'level', but does not describe expected values or effect. The description adds no semantic value beyond what the schema provides, which is minimal.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: analyzing character-level text transformations and reporting detected patterns. The verb 'analyze' paired with the resource 'character-level text transformations' is specific. Sibling tools have distinct purposes (classifier robustness, query decomposition, status), so this tool is well-differentiated.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives. The description implies it's for detecting obfuscation/evasion, but does not provide when-not-to-use or mention sibling tools. An agent would need to infer usage context from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Check server configuration and connection status.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description only states the tool 'check's configuration and status, which is a generic read behavior. No additional behavioral context (e.g., auth needs, side effects) is disclosed beyond the obvious.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence that conveys the purpose without any redundant or extraneous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (no parameters, output schema present), the description adequately covers its purpose. No additional details are necessary for this self-contained status check.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are no parameters, so the description does not need to add meaning beyond the schema. A baseline score of 4 is appropriate as there is no information to provide.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses the specific verb 'check' with resources 'server configuration and connection status', clearly distinguishing it from sibling tools like eval_classifier_robustness which have different purposes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies this tool is for checking status, and sibling names suggest distinct use cases (e.g., robustness, decomposition), but no explicit guidance on when to use or avoid using this tool is given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.2.0eval_classifier_robustnesseval_decompose_queryeval_obfuscation_evasioneval_statusScored across 4 tools
Each tool targets a distinct evaluation function: robustness classification, query decomposition, obfuscation analysis, and server status. No overlap in purpose.
All tools share the 'eval_' prefix, but the suffixes vary in pattern (noun_noun for classifier_robustness, verb_noun for decompose_query, noun_noun for obfuscation_evasion, and single noun for status). Mostly consistent with minor deviation.
Four tools is well-scoped for an evaluation kit, covering multiple evaluation stages without excess or deficiency.
The set covers core evaluation operations and status checking, though a tool for retrieving historical results or configuring evaluations would round it out. Minor gap, not severe.
Prompt injection detection API for AI agents. Scan untrusted text before passing it to an LLM.
The WAF for agents. Pattern-based + heuristic firewall scans prompts, RAG documents, tool argume...
35-probe LLM/agent security red-team scan (injection, jailbreak, MCP abuse) with report.
Pre-execution governance for AI agents. Deterministic PASS/FAIL/REVIEW verdicts, replayable proof.
