Skip to main content
Glama
cdywolf

llm-eval-mcp

by cdywolf
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

Related Servers

Alternatives to llm-eval-mcp

No user-submitted related servers found.

    Related Servers

    • judge-audit-mcp

      asif786ka
      A
      license
      A
      quality
      C
      maintenance
      An MCP server that audits LLM-as-judge evaluations, detecting judge drift across runs, measuring bias through controlled probes, and comparing judge agreement with human raters.
      6
      MIT

    • Astro-LLM

      EthanPany
      F
      license
      Not graded
      quality
      C
      maintenance
      An MCP server that enables agents to evaluate LLMs daily through capability benchmarks and value alignment tests, providing tools to list models, get almanac, judge dilemmas, match user values, and score models.
      1
      -

    • groundcheck

      offquestxo
      A
      license
      A
      quality
      B
      maintenance
      An MCP server that lets any AI agent evaluate RAG outputs -- faithfulness scoring, hallucination detection, and retrieval quality metrics -- with zero API keys, using MCP sampling.
      6
      MIT
    Deploy Server
    One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

    TDQS

    A4.5/5.0

    Scored across 3 tools

    Disambiguation5/5

    Each tool has a distinct role: generating adversarial tasks, judging a single answer, and aggregating evaluation stats. There is no overlap in purpose or expected inputs.

    Naming Consistency5/5

    All tool names follow a clear verb_noun pattern with snake_case: generate_adversarial_tasks, run_llm_as_judge, get_eval_stats. The verbs are action-oriented and descriptive.

    Tool Count5/5

    With 3 tools, the server is tightly scoped to the core evaluation workflow: generate tasks, judge responses, and view stats. Each tool is essential and the count is within the ideal 3-15 range.

    Completeness4/5

    The set covers the main lifecycle of an evaluation run: generating adversarial cases, scoring a response against expected behavior, and retrieving aggregate metrics. A minor gap is the lack of task management (list, delete, update) but tasks are ephemeral, so this is acceptable.

    Maintenance

    ActivitySlowing
    ResponsivenessNo issues

    Resources