AI-ML-Lab-MCP

Overview Schema Source Related Servers Discussions Admin

Benchmark Models

benchmark_models

Cross-validate candidate models on a split's training rows, rank them by average cross-validation score, and compare mean, standard deviation, and fit time without touching test data.

Instructions

Cross-validate several candidate models on the TRAINING rows of a split and rank them by their average score. More reliable than compare_models: every model is scored on several folds (mean and standard deviation), timings are included, and the test rows are never touched. Nothing is stored in Models; train the winner with its own tool (or compare_models) afterwards. The split must have only numerical, complete features. Args: split_id (str): ID of a split made by train_test_split. models (list[str], optional): Training tool names to try (same names and defaults as compare_models). task (str): "auto" (inferred from the target), "regression" or "classification". Default is "auto". cv (int): Number of folds, 2 to 10. Classification is stratified and uses at most as many folds as the smallest class has rows. Default is 5. sort_by (str, optional): Metric that ranks the models (mean over folds). Regression: r2 (default), mae, mse, rmse. Classification: f1 (default), accuracy, balanced_accuracy, precision, recall, roc_auc. include_slow (bool): Also run SVM on large data (over 10,000 training rows). Default is False. Returns: dict: leaderboard with mean and std of every metric, mean fit time, which models are statistically comparable to the best (within one standard deviation), failed/skipped models and notes.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "split_id": "<split_id>"
    },
    "name": "benchmark_models"
  }
}

Input Schema

Name	Required	Default
`cv`	No
`task`	No	auto
`models`	No
`sort_by`	No
`split_id`	Yes
`include_slow`	No

Schema Changelog

Changes observed during successful MCP inspections.

First observedv0.1.0Nov 14, 2026

TDQS

A4.7/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations supplied, the description carries the full burden and does well: it discloses that folds are used for scoring, that test rows are never touched, that timings and standard deviations are reported, that nothing is persisted, and that failed/skipped models are surfaced. The remaining gap is cost/runtime — cross-validating a model list over up to 10 folds can be slow, and only the SVM case is called out via include_slow.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The lead sentence is front-loaded with purpose and scope, and the Args/Returns blocks are well organized. The Args block is lengthy, but that is earned since the schema has no per-parameter descriptions; only minor tightening (e.g. restating metric defaults) would help.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a six-parameter, no-output-schema tool, the description covers inputs, defaults, allowed values, preconditions, downstream workflow, and an explicit Returns section describing the leaderboard payload. Nothing an agent needs to invoke or interpret it is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description fully compensates: every one of the six parameters is explained, including cv's 2–10 range and stratification cap, the exact sort_by metric sets per task, and include_slow's 10,000-row threshold. It effectively supplies the enum values the schema lacks.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Cross-validate several candidate models on the TRAINING rows of a split and rank them by their average score" is a specific verb-plus-resource statement with explicit scope (training rows only). It also directly names and contrasts with the sibling compare_models, so an agent can distinguish the two without inspecting schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit decision rule ("More reliable than compare_models") plus the follow-up workflow: "Nothing is stored in Models; train the winner with its own tool (or compare_models) afterwards." It also states a hard precondition ("The split must have only numerical, complete features"), so the agent knows when this tool is not callable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server