canli-validation-mcp

Validate overfitting (CSCV)

validate_overfitting

Estimate backtest overfitting probability with CSCV: measure how often the in-sample best variant falls below the out-of-sample median using a returns matrix.

Instructions

Probability of backtest overfitting (0 to 1) by CSCV: how often the in-sample best variant falls below the out-of-sample median. Needs every variant's returns (periods by variants); with summary statistics only, use validate_deflated_sharpe. A deflated Sharpe or overfitting probability above or below any threshold is not admission to anything and is not a forecast.

Input Schema

TableJSON Schema

Name	Required	Description
`seed`	No	Sampling seed; default 42.
`matrix`	Yes	Returns as fractions, one row per period, one column per variant.
`n_splits`	No	Even number of blocks, at least 2; default 16.
`max_combinations`	No	Most splits evaluated, up to 2000 (default).

Output Schema

TableJSON Schema

Name	Required	Description	Default
`data`	No
`note`	No
`error`	No
`limits`	No
`receipt`	No
`computed`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed5 schema fields changedv0.13.0Oct 7, 2026

removedInput schema / properties / n_splits / maximum
```
Removed value: -9007199254740991
```
removedOutput schema / additionalProperties
```
Removed value: -{}
```
removedOutput schema / properties / data / additionalProperties
```
Removed value: -{}
```

changedOutput schema / properties / error / anyOf

Previous value: -[
-  {
-    "additionalProperties": {},
-    "properties": {},
-    "type": "object"
-  },
-  {
-    "type": "null"
-  }
-]New value: +[
+  {
+    "properties": {},
+    "type": "object"
+  },
+  {
+    "type": "null"
+  }
+]

changedOutput schema / properties / receipt / anyOf

Previous value: -[
-  {
-    "additionalProperties": {},
-    "properties": {},
-    "type": "object"
-  },
-  {
-    "type": "null"
-  }
-]New value: +[
+  {
+    "properties": {},
+    "type": "object"
+  },
+  {
+    "type": "null"
+  }
+]

Changed4 schema fields changedv0.10.1Sep 30, 2026

changedInput schema / $schema

Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"

changedInput schema / properties / matrix / description

Previous value: -"Returns as fractions: one row per period, one column per variant."New value: +"Returns as fractions, one row per period, one column per variant."

addedInput schema / properties / matrix / items / maxItems
```
Added value: +200
```

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "additionalProperties": {},
+  "description": "data.result holds the statistics (data.plain_reading states them); limits say what the result does not establish; receipt ({id, url}) is the signed record, null when none was stored; error ({code, message}) is set only on refusal.",
+  "properties": {
+    "computed": {
+      "type": "string"
+    },
+    "data": {
+      "additionalProperties": {},
+      "properties": {},
+      "type": "object"
+    },
+    "error": {
+      "anyOf": [
+        {
+          "additionalProperties": {},
+          "properties": {},
+          "type": "object"
+        },
+        {
+          "type": "null"
+        }
+      ]
+    },
+    "limits": {
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "note": {
+      "type": "string"
+    },
+    "receipt": {
+      "anyOf": [
+        {
+          "additionalProperties": {},
+          "properties": {},
+          "type": "object"
+        },
+        {
+          "type": "null"
+        }
+      ]
+    }
+  },
+  "type": "object"
+}

Changed4 schema fields changedv0.6.0Sep 26, 2026
- addedInput schema / properties / matrix / description
```
Added value: +"Returns as fractions: one row per period, one column per variant."
```
- addedInput schema / properties / max_combinations / description
```
Added value: +"Most splits evaluated, up to 2000 (default)."
```
- addedInput schema / properties / n_splits / description
```
Added value: +"Even number of blocks, at least 2; default 16."
```
- addedInput schema / properties / seed / description
```
Added value: +"Sampling seed; default 42."
```
First observedv0.5.0Sep 25, 2026

TDQS

A4.4/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With annotations already declaring destructiveHint=false and openWorldHint=true, the description still adds real context: the required input shape, the meaning and range of the output, and an explicit caveat that crossing any threshold 'is not admission to anything and is not a forecast'. It does not explain determinism/reproducibility of the sampling seed or the combinatorial cost of the run, so it falls short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences, front-loaded with the metric and method, then the input requirement, then the alternative, then a caution. Every sentence carries weight, though the final disclaimer sentence is somewhat tangential to tool selection.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return structure need not be described, and the description still supplies the output range and interpretation, the input shape, the routing alternative, and the key interpretive caveat. Nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all four parameters (matrix, n_splits, max_combinations, seed) are already documented in the schema. The description only restates the matrix orientation ('periods by variants') and adds nothing about seed, n_splits, or max_combinations, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific metric (probability of backtest overfitting, 0-1) and the exact method (CSCV), plus a one-line definition of what the number means (how often the in-sample best falls below the out-of-sample median). It is immediately distinguishable from the sibling validate_deflated_sharpe.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

States the data prerequisite ('needs every variant's returns, periods by variants') and names the explicit alternative for the other case ('with summary statistics only, use validate_deflated_sharpe'). This is a clear when-to-use / when-not-to-use routing rule.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.