Skip to main content
Glama
Mhdd-24

@mhdd_24/ai-benchmark-mcp

by Mhdd-24

@mhdd_24/ai-benchmark-mcp

MCP server for Run standardized model benchmarks.

Same architecture as @mhdd_24/sublime-mcp.

Full documentation: docs/WIKI.md


How it works (30 seconds)

You (chat) → MCP client → ai-benchmark-mcp → AI Benchmark APIs / CLIs / local tools

Related MCP server: metrillm-mcp

Prerequisites

Requirement

Notes

Node.js 18+

ESM TypeScript MCP server

Credentials / CLIs

See environment variables below


Install

Option A — npm (after publish)

npm install -g @mhdd_24/ai-benchmark-mcp

Option B — npx

npx @mhdd_24/ai-benchmark-mcp

Option C — clone and build

git clone https://github.com/Mhdd-24/AI-Benchmark-MCP.git
cd AI-Benchmark-MCP
npm install
npm run build
node dist/index.js

Configure Cursor

Edit ~/.cursor/mcp.json:

{
  "mcpServers": {
    "aibench": {
      "command": "npx",
      "args": ["-y", "@mhdd_24/ai-benchmark-mcp"],
      "env": {
        "_": "optional"
      }
    }
  }
}

Local development:

{
  "command": "node",
  "args": ["/absolute/path/to/AI-Benchmark-MCP/dist/index.js"]
}

Environment variables

Variable

Description

No required env


Tools

Tool

Description

aibench_status

Show run standardized model benchmarks configuration / health.

aibench_list

List run standardized model benchmarks entities.

aibench_inspect

Inspect a run standardized model benchmarks artifact.

aibench_compare

Compare two run standardized model benchmarks items.


License

ISC

Available Tools

4 tools
aibench_compareC

Compare two run standardized model benchmarks items.

ParametersJSON Schema
NameRequiredDescriptionDefault
leftYesLeft
rightYesRight

TDQS

C2.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It only says 'Compare...' with no mention of whether this is read-only, what output it produces, whether it requires prior benchmark runs, or how it behaves if inputs are invalid. The description does not contradict annotations because there are none, but it leaves the tool's behavior largely opaque.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence with no filler, which is concise. However, it is underspecified and grammatically unclear, so the brevity is not an asset: it lacks enough structure to convey the intended meaning precisely.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the sibling tools and the absence of an output schema, the description should clarify what the comparison returns, how left and right identify benchmark items, and what conditions must hold for a valid comparison. None of that is present, so the description is not complete enough for an agent to call the tool correctly without additional inference.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 100% description coverage, but the parameter descriptions are minimal ('Left' and 'Right') and add no real semantics. The tool description adds a small amount of meaning by implying left and right are the two items to compare, but it does not explain their expected format, whether they are identifiers, names, or paths. Baseline 3 is appropriate because the schema covers parameter names, even though the descriptions are thin.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states the verb 'Compare' and a resource ('standardized model benchmarks items'), which gives a general sense of the tool. However, the phrase 'two run standardized model benchmarks items' is grammatically awkward and vague, and it doesn't specify what aspect is compared or what the result is. It is distinguishable from siblings only at a coarse level: compare vs. list/status/inspect.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance about when to use this tool versus aibench_status, aibench_list, or aibench_inspect. No mention of alternatives, prerequisites, or exclusions is present. The only implied usage is 'use this when you need to compare two items,' which is too thin to route an agent reliably.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aibench_inspectC

Inspect a run standardized model benchmarks artifact.

ParametersJSON Schema
NameRequiredDescriptionDefault
nameYesName/id

TDQS

C2.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It does not disclose whether this is a read-only operation, what side effects exist, what the output looks like, or any constraints. The verb 'Inspect' suggests read-only, but that is not explicit.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is very short, which is concise, but the phrasing is awkward and unclear. It is not front-loaded with the most useful information because the core action is muddled.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with one parameter and no output schema, the description should at least clarify what 'inspect' returns and how it differs from status/list/compare. It does neither, leaving the agent to guess.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the only parameter 'name' has a description 'Name/id'. The description adds no additional meaning beyond the schema, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose2/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'Inspect a run standardized model benchmarks artifact' is grammatically awkward and vague. It does not clearly state what 'inspect' returns or what a 'run standardized model benchmarks artifact' is. It does not distinguish itself from siblings like aibench_status or aibench_list.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus aibench_status, aibench_list, or aibench_compare. The description implies inspection of a specific artifact but does not state conditions or alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aibench_listC

List run standardized model benchmarks entities.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoMax results
queryNoFilter

TDQS

C2.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must carry the burden of behavioral disclosure. It only says 'List', implying a read-only operation, but does not mention permissions, return format, side effects, or limitations. The behavior beyond the verb is entirely unspecified.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence, but it is poorly constructed and ambiguous ('List run standardized model benchmarks entities'). It is not front-loaded with key information and fails to convey a clear purpose efficiently.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity and lack of annotations, the description is incomplete. It does not explain what the tool returns, how it relates to sibling tools, or any constraints like pagination or ordering. The existence of sibling tools increases the need for differentiation, which is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema fully describes both parameters with clear descriptions ('Max results' and 'Filter'), giving 100% schema coverage. The tool description adds no additional meaning beyond what the schema provides, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a verb ('List') and a resource ('standardized model benchmarks entities'), but the phrasing 'List run' is awkward and ambiguous. It does not clearly differentiate from sibling tools like aibench_status or aibench_inspect, which likely also deal with benchmarks.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided about when to use this tool versus the siblings aibench_status, aibench_inspect, or aibench_compare. The description does not mention any alternative tools or conditions for selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

aibench_statusC

Show run standardized model benchmarks configuration / health.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

C2.7/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It says 'Show ... configuration / health' which implies a read-only status operation, but it does not disclose what kind of output to expect, whether it queries live state, or any side effects. The slash in 'configuration / health' is ambiguous about what is actually returned.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is very short, but the grammar is awkward and the slash construction is unclear. It is concise but not well-structured; a clearer phrasing like 'Show the configuration and health status of standardized model benchmark runs' would be more effective.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter status tool, the description is minimal. It does not explain what 'health' means, what configuration aspects are shown, or how this relates to the sibling tools. Given the sibling set (list, inspect, compare), an agent would struggle to know exactly what status information this returns.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so there is nothing to document. The description's mention of 'configuration / health' gives some sense of what the status report covers, which is sufficient for a no-parameter tool.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'Show run standardized model benchmarks configuration / health' indicates the tool reports on configuration/health of benchmark runs, but the phrasing is awkward and ambiguous ('Show run standardized model benchmarks configuration / health' could be read as showing a run, or showing configuration/health). It does not clearly distinguish from siblings like aibench_list or aibench_inspect.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus aibench_list, aibench_inspect, or aibench_compare. The description implies it shows status/health, but does not state when that is the right choice or what alternatives exist.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 4 tool updatesv1.0.0
    • First observedaibench_compare
    • First observedaibench_inspect
    • First observedaibench_list
    • First observedaibench_status

TDQS

C2.8/5.0

Scored across 4 tools

Disambiguation4/5

The four tools are mostly distinct: status shows configuration/health, list enumerates entities, inspect examines a single artifact, and compare contrasts two items. There is slight potential confusion between 'inspect' and 'compare' since both involve examining artifacts, but their purposes are clear enough.

Naming Consistency4/5

All tools share the 'aibench_' prefix and use simple verb-based names (status, list, inspect, compare). This is consistent, though the prefix is a bit verbose and 'aibench_status' is more of a noun than a verb_noun pattern.

Tool Count4/5

Four tools is a reasonable, focused set for a benchmark server. It covers the core operations without being bloated, though it is on the smaller side.

Completeness3/5

The set covers status, listing, inspection, and comparison, which are the main read-only operations for benchmarks. However, there are no tools to run, create, or delete benchmarks, so the surface is incomplete for a full lifecycle.

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables benchmarking of local LLM models (performance and quality) and sharing results to a public leaderboard via MCP tools.
    8 npm
    8
    Apache 2.0
  • A
    license
    B
    quality
    A
    maintenance
    Enables building and running custom LLM benchmarks with multi-judge evaluation, supporting GUI, MCP client, and CLI usage for ranked, auditable results.
    3
    MIT