codecalc

Overview Schema Source Related Servers Discussions Admin

Verify Optimization

verify_optimization

Prove an optimization is both correct and significantly faster: runs candidate against original for output equality, then times both to confirm a minimum speedup passes statistical significance.

Instructions

PROVE an optimisation: same outputs, measurably AND SIGNIFICANTLY faster.

Two gates, in order. Correctness: runs candidate against original on shared inputs — a faster-but-wrong candidate fails here and is never timed. Speed: times both at increasing sizes; accepts only when the median ratio clears min_speedup AND a one-sided Mann-Whitney U test rejects "not faster" at every counted size (2-3), or a Bonferroni-corrected majority above that — one size never accepts alone. See inference for the per-size U statistic, p-value, effect size.

A rejection names which gate failed and by how much, e.g. "correct, 1.3x median, but only 1/4 sizes significant."

Accepted grades cross_checked; any rejection — wrong, not faster enough, not significant — grades ungraded: correctness alone earns no grade for the speed claim this tool answers.

Input Schema

TableJSON Schema

Name	Required	Description
`sizes`	No	Input sizes to time both programs at (2-3+ sizes needed for significance); omit for defaults
`language`	Yes	Language both `original` and `candidate` are written in
`original`	Yes	Baseline program to compare against
`candidate`	Yes	Optimised version of `original`, to prove correct and measurably faster
`min_speedup`	No	Minimum median speedup ratio required to accept the optimisation; default 1.15 (15% faster)
`test_inputs`	No	Inputs to confirm both programs still agree on; omit to use the default set

Output Schema

TableJSON Schema

Name	Required	Description	Default
No arguments

Schema Changelog

Changes observed during successful MCP inspections.

Changed6 schema fields changedv0.12.0Sep 9, 2026

addedInput schema / properties / candidate / description

Added value: +"Optimised version of `original`, to prove correct and measurably faster"

addedInput schema / properties / language / description

Added value: +"Language both `original` and `candidate` are written in"

addedInput schema / properties / min_speedup / description

Added value: +"Minimum median speedup ratio required to accept the optimisation; default 1.15 (15% faster)"

addedInput schema / properties / original / description
```
Added value: +"Baseline program to compare against"
```

addedInput schema / properties / sizes / description

Added value: +"Input sizes to time both programs at (2-3+ sizes needed for significance); omit for defaults"

addedInput schema / properties / test_inputs / description

Added value: +"Inputs to confirm both programs still agree on; omit to use the default set"

Changed1 schema field changedv0.11.0Sep 9, 2026

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "additionalProperties": true,
+  "title": "verify_optimizationDictOutput",
+  "type": "object"
+}

Addedv0.2.0Aug 19, 2026

TDQS

A4.4/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses far more than the sparse annotations provide (readOnlyHint: false, openWorldHint: true). It details the two-gate process, the Mann-Whitney U test, the Bonferroni correction, the acceptance threshold, and the grading consequences. It also gives a concrete rejection example. This is thorough behavioral disclosure that exceeds what annotations imply.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is relatively long but well-structured: a bold statement of purpose, then two gates, a rejection example, and grading outcomes. Every sentence adds operational detail; there is no fluff. The length is justified by the complexity of the statistical verification, though it could be tightened slightly without losing meaning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 6 parameters, an output schema, and a complex decision process, the description covers all necessary aspects: correctness gate, speed gate, statistical significance, grading behavior, and an example rejection message. The output schema handles the return format, so the description does not need to. Nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds significant meaning by explaining how parameters work together: 'runs candidate against original on shared inputs', 'times both at increasing sizes', and 'accepts only when the median ratio clears min_speedup'. It also clarifies the role of sizes and test_inputs, enriching the bare schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'PROVE an optimisation: same outputs, measurably AND SIGNIFICANTLY faster,' which names a specific verb (prove), resource (optimisation), and precise success criteria. This clearly separates it from siblings like benchmark (which only measures speed) or compare_execution (which likely checks equivalence without the statistical significance requirement).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains when to use the tool implicitly (when you have an original and an optimised candidate to verify) and details the process, but it does not explicitly name alternative tools or state when not to use this one. An agent could infer the usage, but there is no explicit guidance like 'use benchmark for speed-only comparisons'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.