skforecast-ai

Official

Overview Schema Source Related Servers Discussions Admin

compare

Backtest multiple candidate configurations on the same cross-validation folds, rank them by a chosen metric, and return the winning plan for selection.

Instructions

Backtest several configurations on the same folds and rank them by metric. Reports progress per candidate and can be cancelled between candidates. Returns a comparison id and the plan of the winner in links.best_plan_id.

Input Schema

TableJSON Schema

Name	Required	Description
`cv_id`	Yes	Id returned by `create_cv`: every candidate is backtested on its folds.
`metric`	No	Metric of every candidate. Metric, or list of metrics whose first one is the primary metric (the one that ranks); only the ones given are computed. Null for the metric chosen for the plan the strategy was built for, if one was chosen, else the one selected from the data.
`baseline`	No	Whether to add a seasonal naive baseline (ForecasterEquivalentDate) to the ranking.
`interval`	No	Prediction interval computed for every candidate. Prediction interval as two quantiles [lower, upper] with 0 < lower < upper < 1, e.g. [0.1, 0.9] for 80 %. The baseline and ForecasterStats only take symmetric ones (lower + upper = 1). Null for the interval of the plan the strategy was built for.
`candidates`	No	Configurations to compare, each {'name': ..., 'config': {...}}, e.g. [{'name': 'ridge', 'config': {'estimator': 'Ridge'}}]. Null for the candidates recommended by the profile: the forecasters of its family (with several series, ForecasterRecursiveMultiSeries and ForecasterFoundation), or the estimators of the recommended forecaster when that leaves one, without those above 500 estimator fits; plus the plan the strategy was built for, in the row 'Strategy plan', when it is none of them. The summary names the row of that plan. A candidate that fails is ranked last with its error.

Output Schema

TableJSON Schema

Name	Required	Description	Default
`id`	Yes
`cost`	No
`kind`	Yes
`files`	No
`links`	No
`notices`	No
`summary`	Yes
`changeable`	No
`notices_omitted`	No
`values_included`	No
`summary_truncated`	No

Schema Changelog

Changes observed during successful MCP inspections.

First observedv0.1.0Oct 15, 2026

TDQS

A4/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare it is not read-only, not destructive and not open-world; the description goes further by disclosing that it reports per-candidate progress, that it 'can be cancelled between candidates', and that a failing candidate context matters ('a candidate that fails is ranked last' lives in the schema). The long-running/cancellable trait is exactly the kind of behavior annotations cannot express, though cost/scaling implications of many candidates are not stated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences, front-loaded with the operation and ranking rule, followed by runtime behavior and return contract. Every clause carries information and nothing is padded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be spelled out, yet the description still names the key return artifacts (comparison id, winner plan id). Combined with 100% schema coverage and modest parameter count, the definition is nearly complete; only cost/duration expectations for many candidates are missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all five parameters in depth. The description only reasserts that `metric` is the ranking key and references `links.best_plan_id` as the winner pointer, adding marginal meaning beyond the structured fields. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a concrete verb pair plus resource ('Backtest several configurations ... and rank them by `metric`'), and the scope 'several configurations on the same folds' implicitly distinguishes it from the single-run `backtest` sibling. An agent can pick this tool for candidate comparison without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied rather than stated: comparison of multiple candidate configs on shared CV folds is clearly the use case, but there is no explicit 'when to use this vs `backtest`/`refine_plan`' and no preconditions (e.g. that a `cv_id` from `create_cv` is mandatory) in the prose. The routing comes from the reader's inference, not the text.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server