canli-validation-mcp

Audit a backtest

audit_backtest

Instructions

One-call audit of a strategy's returns. Headline: one test whose false-positive rate was measured on nine return shapes (White's Reality Check with variants, luck trials with Lo's correction without). Then deflated Sharpe, minimum track record and, with variants, overfitting and out-of-sample decay, each with its receipt, plus fix_next. Send every variant tried (variants_file) so trials are counted; point returns_file at a CSV instead of pasting. A deflated Sharpe or overfitting probability above or below any threshold is not admission to anything and is not a forecast.

Input Schema

TableJSON Schema

Name	Required	Description
`returns`	No	Periodic returns as fractions (0.01 = 1%), oldest first; replaces the Sharpe, observations, skew and kurtosis.
`n_splits`	No	Even number of blocks, at least 2; default 16.
`variants`	No	Optional returns of every variant tried (this one included), one row per period, one column per variant; adds the overfitting check.
`confidence`	No	Between 0 and 1; default 0.95.
`returns_file`	No	Path to a CSV or JSON of the returns on this machine (not on the hosted endpoint), instead of returns.
`variants_file`	No	Path to a CSV or JSON with one numeric column per variant, instead of variants.
`returns_column`	No	Column name or 1-based position, when returns_file has several numeric columns.
`periods_per_year`	Yes	Periods per year: 252 daily, 365 crypto, 52 weekly, 12 monthly.
`benchmark_sharpe_annualized`	No	Annualized Sharpe to beat; default 0.
`effective_independent_trials`	No	Independent variants tried before choosing this one. Optional with variants: counted from them, and the larger number is used.
`cross_trial_sharpe_sd_annualized`	No	Standard deviation of annualized Sharpe across those trials. Optional with variants: measured from them, and the larger is used.

Output Schema

TableJSON Schema

Name	Required	Description	Default
`note`	No
`error`	No
`checks`	No
`limits`	No
`not_run`	No
`readings`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed7 schema fields changedv0.13.0Oct 7, 2026

removedInput schema / properties / n_splits / maximum
```
Removed value: -9007199254740991
```

changedInput schema / properties / returns_column / anyOf

Previous value: -[
-  {
-    "maxLength": 200,
-    "minLength": 1,
-    "type": "string"
-  },
-  {
-    "maximum": 9007199254740991,
-    "minimum": 1,
-    "type": "integer"
-  }
-]New value: +[
+  {
+    "maxLength": 200,
+    "minLength": 1,
+    "type": "string"
+  },
+  {
+    "minimum": 1,
+    "type": "integer"
+  }
+]

removedOutput schema / additionalProperties
```
Removed value: -{}
```
removedOutput schema / properties / checks / additionalProperties
```
Removed value: -{}
```

changedOutput schema / properties / error / anyOf

Previous value: -[
-  {
-    "additionalProperties": {},
-    "properties": {},
-    "type": "object"
-  },
-  {
-    "type": "null"
-  }
-]New value: +[
+  {
+    "properties": {},
+    "type": "object"
+  },
+  {
+    "type": "null"
+  }
+]

removedOutput schema / properties / not_run / additionalProperties
```
Removed value: -{}
```
removedOutput schema / properties / readings / additionalProperties
```
Removed value: -{}
```

Changed3 schema fields changedv0.12.0Oct 5, 2026

changedInput schema / properties / cross_trial_sharpe_sd_annualized / description

Previous value: -"Standard deviation of annualized Sharpe across those trials."New value: +"Standard deviation of annualized Sharpe across those trials. Optional with variants: measured from them, and the larger is used."

changedInput schema / properties / effective_independent_trials / description

Previous value: -"Independent variants tried before choosing this one."New value: +"Independent variants tried before choosing this one. Optional with variants: counted from them, and the larger number is used."

changedInput schema / required

Previous value: -[
-  "periods_per_year",
-  "effective_independent_trials",
-  "cross_trial_sharpe_sd_annualized"
-]New value: +[
+  "periods_per_year"
+]

Changed10 schema fields changedv0.10.1Sep 30, 2026

changedInput schema / $schema

Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"

changedInput schema / properties / cross_trial_sharpe_sd_annualized / description

Previous value: -"Standard deviation of the annualized Sharpe across those trials."New value: +"Standard deviation of annualized Sharpe across those trials."

changedInput schema / properties / periods_per_year / description

Previous value: -"Observations per year: 252 daily, 365 daily crypto, 52 weekly, 12 monthly."New value: +"Periods per year: 252 daily, 365 crypto, 52 weekly, 12 monthly."

changedInput schema / properties / returns / description

Previous value: -"Periodic returns as fractions (0.01 is 1%), oldest first; replaces the Sharpe, observations, skew and kurtosis fields."New value: +"Periodic returns as fractions (0.01 = 1%), oldest first; replaces the Sharpe, observations, skew and kurtosis."

changedInput schema / properties / returns_column / description

Previous value: -"Header name or 1-based position of the returns column when returns_file has several numeric columns."New value: +"Column name or 1-based position, when returns_file has several numeric columns."

changedInput schema / properties / returns_file / description

Previous value: -"Path to a CSV or JSON file of the returns on the machine running this server, instead of returns. Not available on the hosted endpoint."New value: +"Path to a CSV or JSON of the returns on this machine (not on the hosted endpoint), instead of returns."

changedInput schema / properties / variants / description

Previous value: -"Optional returns of every variant tried, this one included, as fractions: one row per period, one column per variant. Adds the overfitting check."New value: +"Optional returns of every variant tried (this one included), one row per period, one column per variant; adds the overfitting check."

addedInput schema / properties / variants / items / maxItems
```
Added value: +200
```

changedInput schema / properties / variants_file / description

Previous value: -"Path to a CSV or JSON file of every variant's returns (one numeric column per variant), instead of variants."New value: +"Path to a CSV or JSON with one numeric column per variant, instead of variants."

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "additionalProperties": {},
+  "description": "checks holds each check's full validate_ result by name; readings one sentence per check; not_run the checks skipped and why.",
+  "properties": {
+    "checks": {
+      "additionalProperties": {},
+      "properties": {},
+      "type": "object"
+    },
+    "error": {
+      "anyOf": [
+        {
+          "additionalProperties": {},
+          "properties": {},
+          "type": "object"
+        },
+        {
+          "type": "null"
+        }
+      ]
+    },
+    "limits": {
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "not_run": {
+      "additionalProperties": {},
+      "properties": {},
+      "type": "object"
+    },
+    "note": {
+      "type": "string"
+    },
+    "readings": {
+      "additionalProperties": {},
+      "properties": {},
+      "type": "object"
+    }
+  },
+  "type": "object"
+}

Addedv0.6.0Sep 26, 2026

TDQS

A3.6/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare readOnly=false, destructive=false, openWorld=true, so the description carries most of the burden. It adds real behavioral context: each result ships with 'its receipt, plus fix_next', and it explicitly warns that a deflated Sharpe or overfitting probability above/below any threshold 'is not admission to anything and is not a forecast' – an important interpretive caveat. It stops short of noting that file arguments are read from the local machine (left to the schema).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose is front-loaded, which is good, but the prose is dense and run-on, with parenthetical jargon like 'luck trials with Lo's correction without' that is hard to parse. No sentence is pure padding, yet the readability cost is real for a description this compressed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description need not enumerate return fields, and it still summarizes the payload (deflated Sharpe, minimum track record, overfitting, OOS decay, receipts, fix_next). For an 11-parameter statistical tool this is largely complete; the main omission is any hint about compute cost or runtime for large variants files.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3 and the schema already documents all 11 parameters. The description adds only emphasis – 'Send every variant tried (variants_file) so trials are counted' – reinforcing the variant/trial-counting interplay rather than supplying new format or syntax detail.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb+resource: 'One-call audit of a strategy's returns', and immediately enumerates the bundled analyses (Reality Check, luck trials, deflated Sharpe, track record, overfitting, OOS decay). This clearly places it as the aggregate counterpart to the granular validate_* siblings, though it never names them explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives useful conditional guidance ('with variants, overfitting and out-of-sample decay', 'point returns_file at a CSV instead of pasting'), which tells the agent how inputs change outputs. However, it never states when to choose this one-call audit over the many sibling validators (validate_reality_check, validate_deflated_sharpe, etc.), leaving the primary routing decision to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.