Skip to main content
Glama
simoncoombes

tradefloor

by simoncoombes
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

Rank strategies across many seeds

rank_strategies
Read-onlyIdempotent

Score and rank trading strategies across multiple simulation seeds after evaluation, comparing each against baselines with paired sign tests to avoid one-seed luck.

Instructions

Score strategies across MANY seeds, beside the baseline agents, and rank them with a paired sign test on each pair. Use it after evaluate_strategies, because one seed's ordering is often luck. Costs about one evaluate_strategies call per seed: 2 to 12 seeds (default six), days 1 to 60 here, up to 252 through start_job. Returns each entrant's record across the seeds (median P&L, seeds ahead of buy-and-hold) and each pair's sign test. Deterministic.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
daysNoTrading days to run: 1 to 60 in a direct call, up to 252 (the certified horizon) through start_job.
seedsNoSimulation seeds, 2 to 12 of them. Every entrant trades the same market on each seed. Omit for [1, 2, 3, 4, 5, 6].
universeNoA roster document, usually the `universe` field of a build_universe result. Either {"size": n, "seed": s, "sectors": [...]} or {"instruments": [...]}. When given it replaces universe_size, universe_seed and universe_sectors.
strategiesYesStrategies to run, keyed by a name you choose. Each value is a strategy spec, for example {"signal": {"kind": "momentum", "lookback_days": 1.0}, "portfolio": {"top_k": 5, "gross": 1.0}}. Signal kinds: hold, random, momentum, mean_reversion, oracle, blend. At most 8. The baseline names (buy_and_hold, random, momentum, mean_reversion, oracle) are taken. Check a spec with validate_strategy before running it.
max_leverageNoCap on gross exposure as a multiple of net worth. null removes the cap, and the result then warns that trading size alone can win.
steps_per_dayNoDecision points per trading day, 1 to 22. Each entrant is asked for orders at each one. A step is 65 minutes, so 6 cover the trading session, and days times steps may be at most 360 in a direct call.
universe_seedNoSeed that generates the roster, separate from the simulation seed. Ignored when `universe` is given.
universe_sizeNoNames in a generated roster, 2 to 120. Ignored when `universe` is given.
universe_sectorsNoLowercase sector ids to concentrate a generated roster on, for example ["technology", "energy"]. The ids: technology, financial_services, healthcare, energy, consumer_discretionary, consumer_staples, industrials, materials, real_estate, utilities, telecommunications, transportation. A concentrated roster is a named envelope gap, and the result says so.

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault

No arguments

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly, idempotent, non-destructive), so the bar is lower; the description adds genuine extra context: cost ('about one evaluate_strategies call per seed'), the 2-12 seed range, the days cap, and 'Deterministic.' It stops short of describing what happens on a failed entrant or partial-run behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the action, then the rationale, then the cost and limits. Dense but every clause carries information; the run-on construction of the limits sentence is the only mild inefficiency.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read-only tool with a full output schema and complete parameter docs, the description supplies the missing operational context (cost per seed, seed/day ceilings, determinism, sequencing after evaluate_strategies). Nothing an agent needs to invoke it correctly appears to be missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents every parameter, and the description mostly restates those bounds (seeds default six, days 1-60/252, at most 8 strategies). Baseline 3 is appropriate; it adds little beyond what the schema carries.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Score strategies across MANY seeds ... rank them with a paired sign test') and explicitly positions itself against the baseline agents and the sibling evaluate_strategies. An agent can distinguish it from evaluate_strategies and start_job without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit trigger ('Use it after evaluate_strategies, because one seed's ordering is often luck') and names the escalation path for longer horizons ('up to 252 through start_job'). This is when-to-use plus alternatives, not just implied context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools