Skip to main content
Glama
Stochastly

Stochastly MCP

by Stochastly

adversarial_check

Read-onlyIdempotent

Test a finished backtest run against six controls plus a positive control to confirm whether its edge survives or is refuted by the data.

Instructions

Requires the Stochastly desktop app: this tool runs inside the app, which is not installed on this machine. Install it from https://stochastly.com/pricing, open it and activate a licence.

Run six controls that try to refute a finished run (drift against its own block-bootstrap null, an edge-free twin of the prop-firm cycle, Romano-Wolf multiplicity, concentration on the best days, frictions, stability across sub-periods) plus a positive control that checks a known edge would be detected on the same data.

Call it after any promising result. synthesis is SURVIVES, REFUTED, UNDERPOWERED (effect smaller than the sample can resolve; observations_needed says how many bars it would take) or BLIND (the positive control failed, so the sample, not the strategy, must change). Each passed control reports its floor, the smallest effect it could have rejected. Deterministic arithmetic, no language model; measured error rates in the test suite: 0.017 false positives, 0.000 false negatives. Needs the run's full PnL series with at least 8 points; an unknown run is a named error. Needs Stochastly open with an Atelier or Portfolio licence.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
seedNoRandom seed; the same seed gives the same answer.
n_bootNoNumber of bootstrap resamples.
n_simsNoNumber of simulations for the randomised controls.
run_idYesId of a finished run, from backtest_graph or backtest_status.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.2.1

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds substantial context beyond the annotations: determinism ('no language model', same seed same answer), measured error rates (0.017 FP / 0.000 FN), the four possible synthesis verdicts, the meaning of `floor` and `observations_needed`, minimum input size, and licence/desktop-app requirements. The readOnly/idempotent/non-destructive annotations are consistent with this and the description extends rather than repeats them.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the blocking constraint (desktop app not installed) before explaining the tool, which is the right priority for an agent that may not be able to call it. Dense but nearly every sentence carries information; minor redundancy in restating the licence requirement at both the opening and the close keeps it from a 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description carries the return-value burden and does so: it defines all four synthesis outcomes and the per-control `floor` field. Combined with prerequisites, input minimums and determinism guarantees, nothing an agent needs to invoke and interpret this tool is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents seed, n_boot, n_sims and run_id, including the same-seed-same-answer property. The description only adds the run_id provenance and the sample-size precondition, which is marginal over what the schema already establishes; baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a concrete verb and resource ('run six controls that try to refute a finished run') and enumerates the six controls plus the positive control, so the agent knows exactly what work is performed. The subject matter is specific enough that it is unmistakable against siblings like red_flag_check or diagnostic_explain even without an explicit contrast sentence.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit timing ('Call it after any promising result') and prerequisites (needs the run's full PnL series with at least 8 points, an unknown run is a named error, Stochastly open with an Atelier or Portfolio licence). It stops short of naming a when-not condition or pointing to an alternative sibling test, so it is clear context but not full routing guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.