ML Referee
Server Details
Independent statistical feature evaluation for machine learning and forecasting. ML Referee tests up to 1,000 candidate features in one run against a declared baseline, helping researchers and AI agents decide which inputs to pursue, which bounded feature repairs merit further testing, and where more evidence is needed—without a separate ablation run for every feature.
Use the MCP tools to discover input schemas, check access and credits, preflight a dataset, obtain a quote, submit an authorized evaluation, and retrieve structured results and reports. Results are scoped to the submitted evidence and evaluation design; they do not guarantee future model performance.
Invite-only beta with manual approval. Public tool discovery requires no authentication; account and evaluation tools require OAuth through approved client connections and available evaluation credits. Connect directly to https://api.mlreferee.com/mcp.
Website: https://mlreferee.com Request beta access: https://mlreferee.com/beta Agent setup and documentation: https://mlreferee.com/agents
- Status
- Healthy
- Uptime
- 100.0% over the last 24 hours
- Last Tested
- Transport
- Streamable HTTP · MCP 2025-11-25
- URL
- ChatGPT
- –
- Claude
- –
TDQS
Scored across 26 tools
Most tools target a clearly distinct resource+action, and descriptions distinguish the three parallel families (baseline audit, repair, campaign) and their upload/preflight/quote steps. The main risk is misselecting between the three similarly-named families (e.g. quote_audit vs quote_repair vs quote_campaign, or the three upload_* tools), though the payload descriptions help separate them.
Every name follows a strict verb_noun snake_case pattern (upload_private_dataset, preflight_repair, quote_campaign, get_audit_result, delete_audit). Prefixes are used predictably across get_, delete_, upload_, preflight_, quote_ and verify_ operations.
26 tools is on the heavy side and sits at the threshold where the surface starts to feel sprawling. The three full lifecycle families with their own upload/preflight/quote variants justify many of them, but consolidation (e.g. unified upload/preflight/quote with a mode parameter) would reduce load.
Coverage is strong: upload, preflight, quote, submit, status, result, receipt, verification, deletion, plus credits, beta access, schemas, verticals, usage, readiness and followup. Minor gaps like no explicit cancel/abort for a submitted audit or update operation are workable around.
Available Tools
26 toolsdelete_auditADestructiveIdempotentInspect
Delete retained inputs and results for an audit owned by this tenant.
Inputs: audit_id (required).
delete_uploadADestructiveIdempotentInspect
Delete a tenant-private upload that is no longer needed.
Inputs: upload_id (required).
describe_refereeBRead-onlyIdempotentInspect
Discover capabilities, privacy, and the approved public catalog.
Inputs: no named parameters.
get_audit_receiptARead-onlyIdempotentInspect
Read a tenant-private report fingerprint retained after report expiry or deletion. Older reports may have no record. This does not retrieve report contents.
Inputs: audit_id (required).
get_audit_resultBRead-onlyIdempotentInspect
Read a completed tenant-private result and signed output manifest.
Inputs: audit_id (required).
get_audit_statusBRead-onlyIdempotentInspect
Read the state of an audit owned by the authenticated tenant.
Inputs: audit_id (required).
get_beta_request_statusARead-onlyIdempotentInspect
Read the authenticated tenant's beta-access request status.
Inputs: request_id (required).
get_creditsBIdempotentInspect
Read beta or paid audit-credit availability and usage.
Inputs: no named parameters.
get_input_schemaCRead-onlyIdempotentInspect
Get neutral column roles and validation defaults for a vertical.
Inputs: vertical (required).
get_referee_readinessARead-onlyIdempotentInspect
Report connection, credential, audit-credit, and submission readiness with the next action.
Inputs: no named parameters.
get_referee_usageBRead-onlyIdempotentInspect
Read aggregate, privacy-preserving Referee activity and outcome evidence for the selected audience.
Inputs: window_days.
get_repair_schemaCRead-onlyIdempotentInspect
Get the feature-only contract for diagnosing degradation and testing bounded repairs.
Inputs: no named parameters.
list_verticalsBRead-onlyIdempotentInspect
List the three supported evaluation profiles and their discovery tags.
Inputs: no named parameters.
preflight_datasetBInspect
Validate a tenant-private upload and its baseline evidence/row identity without spending a credit.
Inputs: contract (required), vertical (required), upload_id (required).
preflight_repairBIdempotentInspect
Validate a tenant-private feature-repair matrix without spending a credit.
Inputs: contract (required), upload_id (required).
quote_auditBIdempotentInspect
Create a contract-bound quote for one audit credit; baseline mode and evidence cannot be substituted.
Inputs: contract (required), vertical (required), upload_id (required), idempotency_key (required).
quote_campaignCIdempotentInspect
Preflight all private feature shards, time a pilot, and quote credits and ETA for up to 1,000 candidates in one corrected search family.
Inputs: shards (required), vertical (required), idempotency_key (required).
quote_repairBIdempotentInspect
Preflight and quote label-based feature repair; large jobs return a measured ETA and credit requirement.
Inputs: contract (required), upload_id (required), idempotency_key (required).
record_audit_followupCIdempotentInspect
Record whether a completed audit changed a decision or was confirmed prospectively.
Inputs: kind (required), value (required), audit_id (required).
redeem_beta_inviteAIdempotentInspect
Redeem one operator-issued invite for free beta audit credits before January 1, 2027.
Inputs: invite_code (required).
request_beta_accessCIdempotentInspect
Request operator-reviewed beta access without purchasing or granting credits.
Inputs: lane (required), use_case (required), idempotency_key (required).
submit_auditCIdempotentInspect
Accept a quote and atomically reserve its disclosed audit credits.
Inputs: quote_id (required), idempotency_key (required).
upload_campaign_shardCIdempotentInspect
Encrypt one of up to 64 private feature shards for a large campaign; retain for at least 4,200 seconds.
Inputs: data (required), format, retention_seconds.
upload_private_datasetBIdempotentInspect
Encrypt a de-identified CSV/JSON payload for a short tenant-private retention window.
Inputs: data (required), format, retention_seconds.
upload_repair_matrixCIdempotentInspect
Encrypt one complete de-identified repair matrix of up to 100 MB and 1,000 diagnostic features.
Inputs: data (required), format, retention_seconds.
verify_audit_reportARead-onlyIdempotentInspect
Compare a locally computed SHA-256 with the retained tenant-private receipt. Use the exact downloaded result JSON bytes (not the API envelope), or UTF-8 result.report text with representation report_text. Never copy the expected hash as the candidate: compute it from the supplied artifact. A digest match does not validate a summary, factual correctness, or absence of harm. No report is uploaded.
Inputs: sha256 (required), audit_id (required), representation.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
MCP server for building and testing AI agents with multi-model experimentation and insights.
Pay-per-use tool marketplace for AI agents. Search, price-check, and call APIs via MCP.
The OpenRouter for tools. One MCP connection gives any AI agent 254 hosted tools, pay per call.
100+ MCP tools for AI agents: content metadata, trade intelligence, business-expertise analysis.
Related MCP Servers
- FlicenseAqualityDmaintenanceHigh-value scientific tools for AI agents — literature search (PubMed, arXiv, Semantic Scholar), chemical compound lookup (PubChem, ChEMBL), patent prior art search (USPTO, EPO), GPU spot prices across 4 providers, and real-time earth science data (USGS, NASA, OpenAQ). Per-call billing via API key. Keys issued instantly at https://mcp-site.com/keys/request6-
- AlicenseAqualityDmaintenanceDeterministic time-series statistics for AI agents. This MCP server gives any LLM agent unit-tested statistical tools — anomaly detection, changepoint detection, seasonal decomposition, stationarity/trend tests, data-quality audits, baseline forecasts — with schema-validated structured output and no arbitrary code execution.17MIT
- AlicenseAqualityBmaintenanceMCP server exposing statistical regression testing for LLM agents as a "run" tool: p-value, effect size, and confidence interval on whether agent behavior actually changed.1113 PyPIApache 2.0

HumanJudgeofficial
AlicenseNot gradedqualityCmaintenanceHuman-evaluation infrastructure for AI quality. 25,000+ blind human reviews by 200+ verified reviewers across 58 AI models — query the data via five MCP tools (get_model_scores, compare_models, get_flags, check_content, get_latest).16 PyPI2MIT
Glama MCP Gateway
Add one secure layer between your agents and this server.