@mhdd_24/ai-benchmark-mcp
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@@mhdd_24/ai-benchmark-mcpList all available AI benchmarks"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
@mhdd_24/ai-benchmark-mcp
MCP server for Run standardized model benchmarks.
Same architecture as @mhdd_24/sublime-mcp.
Full documentation: docs/WIKI.md
How it works (30 seconds)
You (chat) → MCP client → ai-benchmark-mcp → AI Benchmark APIs / CLIs / local toolsRelated MCP server: metrillm-mcp
Prerequisites
Requirement | Notes |
Node.js 18+ | ESM TypeScript MCP server |
Credentials / CLIs | See environment variables below |
Install
Option A — npm (after publish)
npm install -g @mhdd_24/ai-benchmark-mcpOption B — npx
npx @mhdd_24/ai-benchmark-mcpOption C — clone and build
git clone https://github.com/Mhdd-24/AI-Benchmark-MCP.git
cd AI-Benchmark-MCP
npm install
npm run build
node dist/index.jsConfigure Cursor
Edit ~/.cursor/mcp.json:
{
"mcpServers": {
"aibench": {
"command": "npx",
"args": ["-y", "@mhdd_24/ai-benchmark-mcp"],
"env": {
"_": "optional"
}
}
}
}Local development:
{
"command": "node",
"args": ["/absolute/path/to/AI-Benchmark-MCP/dist/index.js"]
}Environment variables
Variable | Description |
— | No required env |
Tools
Tool | Description |
| Show run standardized model benchmarks configuration / health. |
| List run standardized model benchmarks entities. |
| Inspect a run standardized model benchmarks artifact. |
| Compare two run standardized model benchmarks items. |
License
ISC
Available Tools
4 toolsaibench_compareC
Compare two run standardized model benchmarks items.
| Name | Required | Description | Default |
|---|---|---|---|
| left | Yes | Left | |
| right | Yes | Right |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It only says 'Compare...' with no mention of whether this is read-only, what output it produces, whether it requires prior benchmark runs, or how it behaves if inputs are invalid. The description does not contradict annotations because there are none, but it leaves the tool's behavior largely opaque.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no filler, which is concise. However, it is underspecified and grammatically unclear, so the brevity is not an asset: it lacks enough structure to convey the intended meaning precisely.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the sibling tools and the absence of an output schema, the description should clarify what the comparison returns, how left and right identify benchmark items, and what conditions must hold for a valid comparison. None of that is present, so the description is not complete enough for an agent to call the tool correctly without additional inference.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 100% description coverage, but the parameter descriptions are minimal ('Left' and 'Right') and add no real semantics. The tool description adds a small amount of meaning by implying left and right are the two items to compare, but it does not explain their expected format, whether they are identifiers, names, or paths. Baseline 3 is appropriate because the schema covers parameter names, even though the descriptions are thin.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states the verb 'Compare' and a resource ('standardized model benchmarks items'), which gives a general sense of the tool. However, the phrase 'two run standardized model benchmarks items' is grammatically awkward and vague, and it doesn't specify what aspect is compared or what the result is. It is distinguishable from siblings only at a coarse level: compare vs. list/status/inspect.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance about when to use this tool versus aibench_status, aibench_list, or aibench_inspect. No mention of alternatives, prerequisites, or exclusions is present. The only implied usage is 'use this when you need to compare two items,' which is too thin to route an agent reliably.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aibench_inspectC
Inspect a run standardized model benchmarks artifact.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Name/id |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It does not disclose whether this is a read-only operation, what side effects exist, what the output looks like, or any constraints. The verb 'Inspect' suggests read-only, but that is not explicit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very short, which is concise, but the phrasing is awkward and unclear. It is not front-loaded with the most useful information because the core action is muddled.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with one parameter and no output schema, the description should at least clarify what 'inspect' returns and how it differs from status/list/compare. It does neither, leaving the agent to guess.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the only parameter 'name' has a description 'Name/id'. The description adds no additional meaning beyond the schema, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Inspect a run standardized model benchmarks artifact' is grammatically awkward and vague. It does not clearly state what 'inspect' returns or what a 'run standardized model benchmarks artifact' is. It does not distinguish itself from siblings like aibench_status or aibench_list.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus aibench_status, aibench_list, or aibench_compare. The description implies inspection of a specific artifact but does not state conditions or alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aibench_listC
List run standardized model benchmarks entities.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max results | |
| query | No | Filter |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the burden of behavioral disclosure. It only says 'List', implying a read-only operation, but does not mention permissions, return format, side effects, or limitations. The behavior beyond the verb is entirely unspecified.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, but it is poorly constructed and ambiguous ('List run standardized model benchmarks entities'). It is not front-loaded with key information and fails to convey a clear purpose efficiently.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity and lack of annotations, the description is incomplete. It does not explain what the tool returns, how it relates to sibling tools, or any constraints like pagination or ordering. The existence of sibling tools increases the need for differentiation, which is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema fully describes both parameters with clear descriptions ('Max results' and 'Filter'), giving 100% schema coverage. The tool description adds no additional meaning beyond what the schema provides, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a verb ('List') and a resource ('standardized model benchmarks entities'), but the phrasing 'List run' is awkward and ambiguous. It does not clearly differentiate from sibling tools like aibench_status or aibench_inspect, which likely also deal with benchmarks.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided about when to use this tool versus the siblings aibench_status, aibench_inspect, or aibench_compare. The description does not mention any alternative tools or conditions for selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
aibench_statusC
Show run standardized model benchmarks configuration / health.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It says 'Show ... configuration / health' which implies a read-only status operation, but it does not disclose what kind of output to expect, whether it queries live state, or any side effects. The slash in 'configuration / health' is ambiguous about what is actually returned.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very short, but the grammar is awkward and the slash construction is unclear. It is concise but not well-structured; a clearer phrasing like 'Show the configuration and health status of standardized model benchmark runs' would be more effective.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter status tool, the description is minimal. It does not explain what 'health' means, what configuration aspects are shown, or how this relates to the sibling tools. Given the sibling set (list, inspect, compare), an agent would struggle to know exactly what status information this returns.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so there is nothing to document. The description's mention of 'configuration / health' gives some sense of what the status report covers, which is sufficient for a no-parameter tool.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Show run standardized model benchmarks configuration / health' indicates the tool reports on configuration/health of benchmark runs, but the phrasing is awkward and ambiguous ('Show run standardized model benchmarks configuration / health' could be read as showing a run, or showing configuration/health). It does not clearly distinguish from siblings like aibench_list or aibench_inspect.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus aibench_list, aibench_inspect, or aibench_compare. The description implies it shows status/health, but does not state when that is the right choice or what alternatives exist.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
4 tool updates
v1.0.0- First observed
aibench_compare - First observed
aibench_inspect - First observed
aibench_list - First observed
aibench_status
TDQS
Scored across 4 tools
The four tools are mostly distinct: status shows configuration/health, list enumerates entities, inspect examines a single artifact, and compare contrasts two items. There is slight potential confusion between 'inspect' and 'compare' since both involve examining artifacts, but their purposes are clear enough.
All tools share the 'aibench_' prefix and use simple verb-based names (status, list, inspect, compare). This is consistent, though the prefix is a bit verbose and 'aibench_status' is more of a noun than a verb_noun pattern.
Four tools is a reasonable, focused set for a benchmark server. It covers the core operations without being bloated, though it is on the smaller side.
The set covers status, listing, inspection, and comparison, which are the main read-only operations for benchmarks. However, there are no tools to run, create, or delete benchmarks, so the surface is incomplete for a full lifecycle.
Related MCP Connectors
Benchmark-first release surface with a read-only MCP endpoint and operator CLI.
Governed data discovery, exact queries, decisions, simulations, and runtime utilities over MCP.
Render, verify, describe, and safely edit Mermaid diagrams through MCP.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
Related MCP Servers
AlicenseNot gradedqualityDmaintenanceEnables running LLM evaluations, experiments, and custom evaluators through a standardized MCP interface.16Apache 2.0- AlicenseNot gradedqualityCmaintenanceEnables benchmarking of local LLM models (performance and quality) and sharing results to a public leaderboard via MCP tools.8 npm8Apache 2.0
- AlicenseBqualityAmaintenanceEnables building and running custom LLM benchmarks with multi-judge evaluation, supporting GUI, MCP client, and CLI usage for ranked, auditable results.3MIT
- AlicenseNot gradedqualityAmaintenanceEnables managing Onde Inference accounts and model catalog operations through MCP tools such as login, app management, model registration, and assignment. Returns structured JSON over stdio for use with any MCP client.Apache 2.0