This server acts as a reliability coach for AI agents, helping to improve tool descriptions and selection quality through pure local computation—no API keys or external calls required.
Score tool descriptions: Get a 0–100 heuristic score analyzing clarity and LLM-friendliness, with reasons, suggestions, and interpretation to reduce wrong tool calls.
Estimate token cost: Roughly calculate token consumption of tool definitions, showing per-tool breakdowns and advising on progressive loading.
Simulate tool choice: Predict which tool an agent is most likely to pick for a given prompt using keyword heuristics, ideal for pre‑production testing.
Generate agent tests: Automatically create three simple test prompts to verify an agent correctly selects and uses a specific tool.
Run a full reliability report: Combine description scores and token estimates into an actionable summary with overall status (Good / Acceptable / Needs attention) and recommendations.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@mcp-agent-reliability Simulate which tool an agent would pick for 'send email to John'"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Problem: AI agents frequently call the wrong tools. Vague tool descriptions, overloaded context windows, and zero visibility into selection quality cause wasted tokens, failed tasks, and frustrated users.
Solution: mcp-agent-reliability is a lightweight MCP server that acts as a reliability coach for your agents. It helps you:
Score how clear and LLM-friendly your tool descriptions are (0–100)
Estimate how many tokens your tools will consume
Simulate which tool an agent is most likely to pick for a given prompt
Generate simple test prompts to verify correct tool selection
Produce a full reliability report with actionable recommendations
All features are pure computation — no paid API keys, no external calls, zero ongoing cost.
Built for entrepreneurs, founders, and teams who are tired of agents calling the wrong tools and burning money.
Related MCP server: clair
Imagine you give a 10-year-old child a big list of 30 toys and say “go play with the right one”.
If the labels are confusing, the child will pick the wrong toy.
AI agents are the same.
When you connect many MCP servers, the agent sees a long menu of tools.
If the descriptions are vague, it picks the wrong tool → wasted tokens → failed tasks.
This server is the label checker and practice teacher for that menu.
In 2026, as agents become more autonomous and tool counts grow, reliability is no longer optional — it is the difference between a demo and a production system.
Feature | Description |
Tool Description Scoring | Heuristic 0–100 score with reasons and concrete suggestions |
Token Cost Estimation | Rough token count for a list of tools + advice on progressive loading |
Tool Choice Simulation | Keyword-heuristic prediction of which tool an agent would select |
Test Prompt Generation | 3 ready-to-use prompts to verify an agent picks the correct tool |
Reliability Report | Combined score + token summary with overall status and recommendation |
Zero External Cost | Pure local computation, no API keys required |
Stateless-friendly | Compatible with modern MCP updates |
TypeScript + Official SDK | Built on |
flowchart TD
A[MCP Client<br/>Cursor / Claude / etc.] -->|stdio| B[mcp-agent-reliability Server]
B --> C[ListTools Handler]
B --> D[CallTool Handler]
D --> E1[score_tool_description]
D --> E2[estimate_token_cost]
D --> E3[simulate_tool_choice]
D --> E4[generate_agent_tests]
D --> E5[reliability_report]
E1 & E2 & E3 & E4 & E5 --> F[Pure Heuristic Utils<br/>scoring.ts]
F --> G[JSON Response back to Client]Transport: stdio (standard for local MCP servers)
Runtime: Node.js ≥ 18
Core logic: Pure functions in src/utils/scoring.ts (no network, no side effects)
Tools: Five focused tools registered via the official MCP SDK
git clone https://github.com/princeruhulofficial/mcp-agent-reliability.git
cd mcp-agent-reliability
npm install
npm run build
npm startnpx -y mcp-agent-reliabilitynpm run devAdd this to your MCP config (~/.cursor/mcp.json or claude_desktop_config.json):
Local path version:
{
"mcpServers": {
"agent-reliability": {
"command": "node",
"args": ["/ABSOLUTE/PATH/TO/mcp-agent-reliability/dist/index.js"]
}
}
}After npm publish (recommended for others):
{
"mcpServers": {
"agent-reliability": {
"command": "npx",
"args": ["-y", "mcp-agent-reliability"]
}
}
}Replace
/ABSOLUTE/PATH/TO/...with the real full path on your machine.
Restart the client after saving the config.
score_tool_descriptionPurpose: Score how clear, specific, and LLM-friendly a tool description is (0–100). Use this before adding a new tool to an agent to reduce wrong tool calls.
Parameters:
Name | Type | Required | Description |
| string | Yes | The full tool description text to score |
| string | No | Optional name of the tool (e.g. |
Return schema (example):
{
"score": 85,
"reasons": [
"Good length for an LLM to read",
"Language looks specific",
"Mentions inputs or outputs — helpful for the model",
"Overall: strong description — agent should select it reliably"
],
"suggestions": [],
"interpretation": "Excellent — agent should pick this tool reliably"
}Example call:
Tool: score_tool_description
name: create_invoice
description: Create a new invoice for a customer. Requires customer_id and amount. Returns invoice_id.estimate_token_costPurpose: Roughly estimate how many tokens a list of tool definitions will consume in the agent context window. Helps decide whether to enable progressive loading.
Parameters:
Name | Type | Required | Description |
| array | Yes | List of objects with |
Return schema (example):
{
"total_estimated_tokens": 1240,
"tool_count": 5,
"average_per_tool": 248,
"breakdown": [
{ "name": "create_invoice", "tokens": 210 },
{ "name": "send_email", "tokens": 185 }
],
"advice": "Low — should be fine for most agents"
}Advice thresholds:
> 15000 → High — consider progressive disclosure or fewer tools
> 8000 → Moderate — monitor context usage
otherwise → Low — should be fine
simulate_tool_choicePurpose: Given a user prompt and a list of available tools, predict which tool an agent is most likely to pick. Useful for testing tool selection before production.
Parameters:
Name | Type | Required | Description |
| string | Yes | The user message or task the agent will see |
| array | Yes | List of tools ( |
Return schema (example):
{
"predicted_tool": "create_invoice",
"confidence": 78,
"all_scores": [
{ "name": "create_invoice", "score": 6 },
{ "name": "send_email", "score": 2 }
],
"note": "This is a keyword-heuristic simulation, not a real LLM. Use it for quick checks."
}generate_agent_testsPurpose: Generate 3 simple test prompts that you can feed to an agent to verify it correctly selects and uses a given tool.
Parameters:
Name | Type | Required | Description |
| string | Yes | Name of the tool to test |
| string | Yes | Description of the tool |
Return schema (example):
{
"tool": "create_invoice",
"test_prompts": [
"Please use the create_invoice tool to Create a new invoice for a customer...",
"I need to Create a new invoice for a customer. Can you call the right tool?",
"Call create_invoice with a safe example input and show me the result."
],
"how_to_use": "Copy each prompt into your agent chat (with only this tool enabled) and check if it calls the correct tool."
}reliability_reportPurpose: Create a short reliability report for a set of tools. Combines description scores and token estimates into one actionable summary.
Parameters:
Name | Type | Required | Description |
| array | Yes | List of tools ( |
Return schema (example):
{
"overall_status": "Good",
"average_description_score": 78,
"total_estimated_tokens": 1240,
"tool_count": 5,
"tools": [
{
"name": "create_invoice",
"score": 85,
"estimated_tokens": 210,
"top_suggestion": "Looks good"
}
],
"recommendation": "You are in a healthy range. Keep monitoring as you add more tools."
}Overall status logic:
Needs attention if average score < 55 or total tokens > 20 000
Acceptable with room to improve if average score < 70
otherwise Good
The scoring engine is a pure heuristic (no LLM calls). It starts at a neutral 50 and adjusts based on observed MCP failure patterns:
Check | Effect |
Description length < 20 chars | −25 |
Length between 40–300 chars | +15 |
Length ≥ 300 chars | −10 |
Contains vague words ( | −15 |
Language looks specific | +10 |
Mentions inputs / outputs / returns | +10 |
Destructive action without safety note | −10 |
Tool name follows | +5 |
Final score clamped to 0–100 | — |
Interpretation bands:
≥ 80 → Excellent — agent should pick this tool reliably
60–79 → OK — improve with the suggestions
< 60 → Weak — high chance of wrong or missed tool calls
Token estimation uses ≈ 3.5 characters per token (slightly denser than plain text because of schema overhead) plus a fixed 40-token schema boilerplate per tool.
score_tool_description
name: create_invoice
description: Create a new invoice for a customer. Requires customer_id and amount. Returns invoice_id.→ Score around 85, interpretation “Excellent”.
score_tool_description
name: handle_stuff
description: Does things with data.→ Low score, suggestions to be more specific and mention inputs/outputs.
Pass a list of your real tools to reliability_report and get an overall status + per-tool breakdown in one call.
Who | How they use it |
Founders / Entrepreneurs | Quickly check if their agent’s tool set is production-ready before shipping |
Agent builders | Score every new tool description before adding it to the system |
Teams with many MCP servers | Estimate total token overhead and decide on progressive disclosure |
QA / Testing | Generate test prompts and simulate tool choice before real LLM runs |
Cost-conscious operators | Catch token-heavy tool lists early |
Pure computation — no external API, no secrets, no side effects
Fast & free — runs entirely locally
Actionable — every score comes with reasons and concrete suggestions
Focused — only five tools, each solving one clear problem
Honest — the simulator is a heuristic, not a real LLM (clearly stated)
Entrepreneur-friendly — simple language, clear value, zero ongoing cost
All tools are synchronous pure functions
Typical response time: < 5 ms on modern hardware
Memory footprint: negligible (no large models or caches)
Scales linearly with number of tools (usually tens, not thousands)
No network calls
No data leaves your machine
No API keys required or stored
No logging of tool descriptions or prompts beyond the current request
MIT licensed — audit the full source in minutes
Q: Does this replace a real LLM evaluation?
A: No. It is a fast, free, local heuristic for early feedback. Use it before expensive LLM-based evals.
Q: Why not use an LLM to score descriptions?
A: That would require API keys and cost money. This version is deliberately zero-cost. An optional LLM-backed mode is on the roadmap.
Q: Can I use it with remote / hosted MCP?
A: Current version is stdio-only. A hosted version is planned.
Q: Is the token estimate accurate?
A: It is a rough approximation (±20–30% typical). Good enough for “is this too heavy?” decisions.
Q: Will you publish to npm?
A: Yes — once the package is published, the npx one-liner will work for everyone.
Optional LLM-backed scoring (higher accuracy when you want it)
Hosted version with dashboard
Integration with progressive disclosure patterns
npm package publication for one-command install
More sophisticated simulation (optional embedding similarity)
Export reports as Markdown / HTML
Contributions are welcome!
Fork the repository
Create a feature branch (git checkout -b feature/amazing-improvement)
Make your changes and add tests if relevant
Open a Pull Request with a clear description
Please keep the core philosophy: pure, free, fast, and entrepreneur-friendly.
MIT License — see LICENSE for details.
Roughly estimate how many tokens a list of tool definitions will consume in the agent context window. Helps decide whether to enable progressive loading.
| Name | Required | Description | Default |
|---|---|---|---|
| tools | Yes | List of tools with their name and description |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden of behavioral disclosure. It notes the estimate is 'rough,' setting expectations about accuracy, but does not disclose return format, potential side effects, or how the estimate is computed. For a non-destructive estimation tool, this is minimal but not misleading.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences that are front-loaded and free of redundancy. The first sentence states the core function, the second adds the decision context. No filler or unnecessary detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has a simple input schema and no output schema. The description covers purpose and use case but does not specify the return format (e.g., numeric value, range, aggregate) or whether the estimate is per-tool or total. This is a moderate gap, though acceptable for an estimation tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides 100% coverage for the single 'tools' parameter, including nested object structure and descriptions. The tool description adds context about tool definitions and token consumption, but does not add significant detail beyond what the schema already specifies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action and resource: 'estimate how many tokens a list of tool definitions will consume in the agent context window.' It also provides context for its use case, distinguishing it from sibling tools like score_tool_description or reliability_report.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when deciding on progressive loading ('Helps decide whether to enable progressive loading'), but it does not explicitly state when not to use it or mention alternative tools. Guidance is implied rather than explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Generate 3 simple test prompts that you can feed to an agent to verify it correctly selects and uses a given tool.
| Name | Required | Description | Default |
|---|---|---|---|
| tool_name | Yes | Name of the tool to test | |
| description | Yes | Description of the tool |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must bear the burden. It discloses that the tool generates 3 prompts and the intended use, but does not detail any side effects, permissions, or output format. For a simple generation tool, this is minimal but acceptable.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, clear sentence with no superfluous words. Front-loaded with the action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the core function and purpose, but lacks guidance on usage context, output structure, and any behavioral caveats. Given the tool's simplicity, this is adequate but leaves some gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides descriptions for both parameters (tool_name and description), with 100% coverage. The tool description adds no additional parameter information beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Generate 3 simple test prompts' with the purpose of verifying an agent's tool selection and usage. This distinguishes it from sibling tools like score_tool_description or simulate_tool_choice.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the use case (generating test prompts for tool verification) but does not explicitly state when to use it vs alternatives, nor does it mention any exclusions or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Create a short reliability report for a set of tools. Combines description scores and token estimates into one actionable summary.
| Name | Required | Description | Default |
|---|---|---|---|
| tools | Yes | List of tools to evaluate |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full transparency burden. It discloses that the tool creates a short actionable report and combines description scores and token estimates, which hints at internal computation. However, it does not mention whether the operation is read-only, whether it internally invokes other tools, or how the estimates are derived, leaving some behavioral ambiguity.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that front-loads the primary action ('Create a short reliability report') and then states the value proposition. There is no filler or repetition of schema details, making it extremely efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has one parameter and no output schema, but the description adequately conveys purpose and composition. It explains that the report merges description scores and token estimates, and sibling tool names provide additional context. A minor gap is that the output format is unspecified, which would improve completeness, but it is still sufficient for an agent to understand how to invoke the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already provides a description for the only parameter 'tools' ('List of tools to evaluate'), so schema coverage is 100%. The description's phrase 'set of tools' adds no new semantic detail beyond what the schema conveys, so a baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Create' and the resource 'a short reliability report for a set of tools.' It specifies that the tool combines description scores and token estimates, which differentiates it from siblings like score_tool_description (which scores individual descriptions) and estimate_token_cost (which estimates token costs).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than explicit. The description indicates that the tool aggregates scores and token estimates into a summary, suggesting it is used when an overall reliability snapshot is needed, but it does not explicitly state when to use this tool versus the sibling functions or provide any exclusions or alternative guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Score how clear, specific and LLM-friendly a tool description is (0-100). Use this before adding a new tool to an agent to reduce wrong tool calls.
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | Optional name of the tool (e.g. create_invoice) | |
| description | Yes | The full tool description text to score |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full burden. It states the scoring criteria (clarity, specificity, LLM-friendliness) but does not disclose behavior such as whether the tool is read-only, what output format to expect, or any limitations. For an apparently safe utility, this lack of explicit safety disclosure is a notable gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the primary purpose, and every word contributes value. There is no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with two parameters and no output schema, the description adequately explains the core function and a primary use case. Missing return-value details are acceptable given the tool's simplicity, but a bit more context on how to interpret the 0-100 score would have made it fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description text adds no additional parameter-level meaning beyond what the schema already provides for 'name' and 'description'. It does not compensate further, but it doesn't need to at this coverage level.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: scoring how clear, specific, and LLM-friendly a tool description is on a 0-100 scale. It uses a specific verb (score) and resource (tool description), distinguishing it from sibling tools like estimate_token_cost or simulate_tool_choice.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear when-to-use instruction: 'Use this before adding a new tool to an agent to reduce wrong tool calls.' It doesn't explicitly mention alternatives or exclusions, but the context is specific enough to guide the agent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Given a user prompt and a list of available tools, predict which tool an agent is most likely to pick. Useful for testing tool selection before production.
| Name | Required | Description | Default |
|---|---|---|---|
| tools | Yes | List of tools the agent can choose from | |
| prompt | Yes | The user message or task the agent will see |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description must disclose behavioral traits. It only states what the tool does, not whether it is read-only, deterministic, or how predictions are generated. Missing important behavioral context for a simulation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no fluff. The first sentence states the core function, the second gives a use case. Efficient and appropriately sized.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description omits expected output format or any caveats about the prediction (e.g., confidence score, top-n tools). With no output schema, this is a significant gap for a prediction tool, making it incomplete for practical use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description mentions 'user prompt' and 'list of available tools' but adds no extra detail beyond the schema's own parameter descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states it predicts which tool an agent is most likely to pick given a prompt and tool list. The verb 'predict' and resource 'tool choice' are specific, and this distinguishes it from siblings that score descriptions or generate tests.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides a clear use context: 'testing tool selection before production.' However, it does not explicitly mention alternatives or when not to use the tool, so it falls short of a full 'when/when-not' guide.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v1.0.0estimate_token_costgenerate_agent_testsreliability_reportscore_tool_descriptionsimulate_tool_choiceScored across 5 tools
Each tool targets a distinct aspect of agent reliability: scoring descriptions, estimating token costs, simulating tool choice, generating test prompts, and producing a combined report. There is no overlap or ambiguity between them.
Most names follow a clear verb_noun pattern (score_tool_description, estimate_token_cost, simulate_tool_choice, generate_agent_tests), but 'reliability_report' deviates as a noun_noun construction. The pattern is mostly consistent with one minor deviation.
At 5 tools, the server is well-scoped for its purpose of assessing and improving MCP tool reliability. Each tool serves a distinct function without redundancy or bloat.
The set covers the core lifecycle: evaluating descriptions, estimating cost, predicting selection, generating tests, and summarizing results. A minor gap is the lack of direct test execution or runtime monitoring, but the provided surface is reasonably complete for its intended scope.
MCP server for building and testing AI agents with multi-model experimentation and insights.
Analytics for MCP servers. Find out which of your tools agents get wrong. MCPulse shows you which tools AI agents retry, which come back empty, and which they never call at all. Two lines inside your own server. It never sees your arguments or your results. getmcpulse.com
The OpenRouter for tools. One MCP connection gives any AI agent 254 hosted tools, pay per call.
Hosted MCP server for LLM cost estimation, model comparison, and budget-aware routing.
