mcp-agent-reliability

1. Value Proposition

Problem: AI agents frequently call the wrong tools. Vague tool descriptions, overloaded context windows, and zero visibility into selection quality cause wasted tokens, failed tasks, and frustrated users.

Solution: mcp-agent-reliability is a lightweight MCP server that acts as a reliability coach for your agents. It helps you:

• Score how clear and LLM-friendly your tool descriptions are (0–100)

• Estimate how many tokens your tools will consume

• Simulate which tool an agent is most likely to pick for a given prompt

• Generate simple test prompts to verify correct tool selection

• Produce a full reliability report with actionable recommendations

All features are pure computation — no paid API keys, no external calls, zero ongoing cost.

Built for entrepreneurs, founders, and teams who are tired of agents calling the wrong tools and burning money.

Related MCP server: clair

2. Why This Project Exists

Imagine you give a 10-year-old child a big list of 30 toys and say “go play with the right one”.
If the labels are confusing, the child will pick the wrong toy.

AI agents are the same.

When you connect many MCP servers, the agent sees a long menu of tools.
If the descriptions are vague, it picks the wrong tool → wasted tokens → failed tasks.

This server is the label checker and practice teacher for that menu.

In 2026, as agents become more autonomous and tool counts grow, reliability is no longer optional — it is the difference between a demo and a production system.

3. Features

4. Architecture

flowchart TD
    A[MCP Client<br/>Cursor / Claude / etc.] -->|stdio| B[mcp-agent-reliability Server]
    B --> C[ListTools Handler]
    B --> D[CallTool Handler]
    D --> E1[score_tool_description]
    D --> E2[estimate_token_cost]
    D --> E3[simulate_tool_choice]
    D --> E4[generate_agent_tests]
    D --> E5[reliability_report]
    E1 & E2 & E3 & E4 & E5 --> F[Pure Heuristic Utils<br/>scoring.ts]
    F --> G[JSON Response back to Client]

• Transport: stdio (standard for local MCP servers)

• Runtime: Node.js ≥ 18

• Core logic: Pure functions in src/utils/scoring.ts (no network, no side effects)

• Tools: Five focused tools registered via the official MCP SDK

5. Installation

Option A — Local (recommended while developing)

git clone https://github.com/princeruhulofficial/mcp-agent-reliability.git
cd mcp-agent-reliability
npm install
npm run build
npm start

Option B — After publishing to npm

npx -y mcp-agent-reliability

Option C — From source with tsx (dev)

npm run dev

6. MCP Client Configuration

Cursor / Claude Desktop / most MCP clients

Add this to your MCP config (~/.cursor/mcp.json or claude_desktop_config.json):

Local path version:

{
  "mcpServers": {
    "agent-reliability": {
      "command": "node",
      "args": ["/ABSOLUTE/PATH/TO/mcp-agent-reliability/dist/index.js"]
    }
  }
}

After npm publish (recommended for others):

{
  "mcpServers": {
    "agent-reliability": {
      "command": "npx",
      "args": ["-y", "mcp-agent-reliability"]
    }
  }
}

Replace /ABSOLUTE/PATH/TO/... with the real full path on your machine.

Restart the client after saving the config.

7. All 5 Tools

7.1 score_tool_description

Purpose: Score how clear, specific, and LLM-friendly a tool description is (0–100). Use this before adding a new tool to an agent to reduce wrong tool calls.

Parameters:

Return schema (example):

{
  "score": 85,
  "reasons": [
    "Good length for an LLM to read",
    "Language looks specific",
    "Mentions inputs or outputs — helpful for the model",
    "Overall: strong description — agent should select it reliably"
  ],
  "suggestions": [],
  "interpretation": "Excellent — agent should pick this tool reliably"
}

Example call:

Tool: score_tool_description
name: create_invoice
description: Create a new invoice for a customer. Requires customer_id and amount. Returns invoice_id.

7.2 estimate_token_cost

Purpose: Roughly estimate how many tokens a list of tool definitions will consume in the agent context window. Helps decide whether to enable progressive loading.

Parameters:

Return schema (example):

{
  "total_estimated_tokens": 1240,
  "tool_count": 5,
  "average_per_tool": 248,
  "breakdown": [
    { "name": "create_invoice", "tokens": 210 },
    { "name": "send_email", "tokens": 185 }
  ],
  "advice": "Low — should be fine for most agents"
}

Advice thresholds:

• > 15000 → High — consider progressive disclosure or fewer tools

• > 8000 → Moderate — monitor context usage

• otherwise → Low — should be fine

7.3 simulate_tool_choice

Purpose: Given a user prompt and a list of available tools, predict which tool an agent is most likely to pick. Useful for testing tool selection before production.

Parameters:

Return schema (example):

{
  "predicted_tool": "create_invoice",
  "confidence": 78,
  "all_scores": [
    { "name": "create_invoice", "score": 6 },
    { "name": "send_email", "score": 2 }
  ],
  "note": "This is a keyword-heuristic simulation, not a real LLM. Use it for quick checks."
}

7.4 generate_agent_tests

Purpose: Generate 3 simple test prompts that you can feed to an agent to verify it correctly selects and uses a given tool.

Parameters:

Return schema (example):

{
  "tool": "create_invoice",
  "test_prompts": [
    "Please use the create_invoice tool to Create a new invoice for a customer...",
    "I need to Create a new invoice for a customer. Can you call the right tool?",
    "Call create_invoice with a safe example input and show me the result."
  ],
  "how_to_use": "Copy each prompt into your agent chat (with only this tool enabled) and check if it calls the correct tool."
}

7.5 reliability_report

Purpose: Create a short reliability report for a set of tools. Combines description scores and token estimates into one actionable summary.

Parameters:

Return schema (example):

{
  "overall_status": "Good",
  "average_description_score": 78,
  "total_estimated_tokens": 1240,
  "tool_count": 5,
  "tools": [
    {
      "name": "create_invoice",
      "score": 85,
      "estimated_tokens": 210,
      "top_suggestion": "Looks good"
    }
  ],
  "recommendation": "You are in a healthy range. Keep monitoring as you add more tools."
}

Overall status logic:

• Needs attention if average score < 55 or total tokens > 20 000

• Acceptable with room to improve if average score < 70

• otherwise Good

8. Scoring Methodology

The scoring engine is a pure heuristic (no LLM calls). It starts at a neutral 50 and adjusts based on observed MCP failure patterns:

Interpretation bands:

• ≥ 80 → Excellent — agent should pick this tool reliably

• 60–79 → OK — improve with the suggestions

• < 60 → Weak — high chance of wrong or missed tool calls

Token estimation uses ≈ 3.5 characters per token (slightly denser than plain text because of schema overhead) plus a fixed 40-token schema boilerplate per tool.

9. Examples

Score a strong description

score_tool_description
name: create_invoice
description: Create a new invoice for a customer. Requires customer_id and amount. Returns invoice_id.

→ Score around 85, interpretation “Excellent”.

Score a weak description

score_tool_description
name: handle_stuff
description: Does things with data.

→ Low score, suggestions to be more specific and mention inputs/outputs.

Full reliability report

Pass a list of your real tools to reliability_report and get an overall status + per-tool breakdown in one call.

10. Use Cases

11. Design Principles

1. Pure computation — no external API, no secrets, no side effects

2. Fast & free — runs entirely locally

3. Actionable — every score comes with reasons and concrete suggestions

4. Focused — only five tools, each solving one clear problem

5. Honest — the simulator is a heuristic, not a real LLM (clearly stated)

6. Entrepreneur-friendly — simple language, clear value, zero ongoing cost

12. Performance

• All tools are synchronous pure functions

• Typical response time: < 5 ms on modern hardware

• Memory footprint: negligible (no large models or caches)

• Scales linearly with number of tools (usually tens, not thousands)

13. Security & Privacy

• No network calls

• No data leaves your machine

• No API keys required or stored

• No logging of tool descriptions or prompts beyond the current request

• MIT licensed — audit the full source in minutes

14. FAQ

Q: Does this replace a real LLM evaluation?
A: No. It is a fast, free, local heuristic for early feedback. Use it before expensive LLM-based evals.

Q: Why not use an LLM to score descriptions?
A: That would require API keys and cost money. This version is deliberately zero-cost. An optional LLM-backed mode is on the roadmap.

Q: Can I use it with remote / hosted MCP?
A: Current version is stdio-only. A hosted version is planned.

Q: Is the token estimate accurate?
A: It is a rough approximation (±20–30% typical). Good enough for “is this too heavy?” decisions.

Q: Will you publish to npm?
A: Yes — once the package is published, the npx one-liner will work for everyone.

15. Roadmap

• Optional LLM-backed scoring (higher accuracy when you want it)

• Hosted version with dashboard

• Integration with progressive disclosure patterns

• npm package publication for one-command install

• More sophisticated simulation (optional embedding similarity)

• Export reports as Markdown / HTML

16. Contributing

Contributions are welcome!

1. Fork the repository

2. Create a feature branch (git checkout -b feature/amazing-improvement)

3. Make your changes and add tests if relevant

4. Open a Pull Request with a clear description

Please keep the core philosophy: pure, free, fast, and entrepreneur-friendly.

17. License

MIT License — see LICENSE for details.