A
licenseA
qualityA
maintenanceEnables building and running custom LLM benchmarks with multi-judge evaluation, supporting GUI, MCP client, and CLI usage for ranked, auditable results.
8
MIT
sure-but-wrong
by Ayesha458
Related Servers
Alternatives to sure-but-wrong
No user-submitted related servers found.
Related Servers
- FlicenseAqualityBmaintenanceEnables MCP clients to score retrieval quality with recall, precision, nDCG, hit rate and MRR, compare 2-3 configurations with seeded paired-bootstrap confidence intervals, audit agent tool calls against expected traces, and detect prompt regressions against golden sets. It also generates Markdown, JSON or JUnit reports with pass/fail gates, all deterministically and offline without paid API keys.5-
- AlicenseNot gradedqualityCmaintenanceEnables benchmarking of local LLM models (performance and quality) and sharing results to a public leaderboard via MCP tools.14 npm8Apache 2.0
- FlicenseNot gradedqualityBmaintenanceEnables running deterministic multi-model prompt regression and golden dataset benchmark suites, exposing structured results and telemetry through the Model Context Protocol for integration with MCP-compliant clients.8-
- AlicenseNot gradedqualityAmaintenanceEnables MCP clients to list and inspect benchmark tasks, score submitted answers, run evaluations through OpenRouter, and retrieve provenance data for exploit-resistance testing.398 npmMIT
- AlicenseNot gradedqualityAmaintenanceEnables users to list and inspect benchmark tasks, grade answers, run benchmark evaluations, and retrieve provenance through its MCP interface.211 npmMIT
TDQS
A3.9/5.0
Scored across 8 tools
Disambiguation5/5
Each tool maps to a distinct phase of the benchmark lifecycle: list_questions (inspect bank), ping_model (health check), start_run (execute), run_status (monitor), resume_run (retry), get_report (results), compare_runs (diff runs), export_run (dump data). There is no meaningful overlap between any pair.
Naming Consistency5/5
All eight tools follow a clean verb_noun snake_case pattern (list_questions, start_run, run_status, resume_run, get_report, compare_runs, export_run, ping_model). The convention is applied uniformly with no style drift.
Tool Count5/5
Eight tools is well-scoped for a benchmark harness, covering setup, execution, monitoring, recovery, and reporting without bloat. Every tool earns its place.
Completeness4/5
The lifecycle is well covered: inspect questions, verify the target model, launch a run, monitor, resume, report, compare, and export. Minor gaps remain, such as no cancel_run for a background run and no tool to enumerate available models/providers.
Maintenance
ActivityMaintained
ResponsivenessNo issues