codecalc

Overview Schema Source Related Servers Discussions Admin

Benchmark

benchmark

Measure code time complexity by running it at increasing input sizes and fitting the growth curve to estimate Big-O class.

Instructions

Empirically measure time complexity by running code at increasing input sizes.

Contract: the code must read an integer N from stdin (first line) and do work sized by N. codecalc runs it at each size in sizes and fits the growth curve to estimate Big-O (O(1), O(log n), O(n), O(n log n), O(n^2)...). 3 sizes (the accepted minimum) can only return noise-floor O(1) or "inconclusive" — never a growth class, not even exponential; 4+ sizes are needed to trust the curve fit, 5+ doubling sizes for a robust ratio median — estimate_basis names which estimator decided. Example python: 'import sys\nn=int(sys.stdin.readline()); s=0\nfor i in range(n): s+=i\nprint(s)'

Input Schema

TableJSON Schema

Name	Required	Description	Default
`code`	Yes	Program that reads integer N from stdin's first line and does work sized by N
`sizes`	No	Comma-separated input sizes to run at, e.g. '100,1000,10000,100000' (this default: 4 sizes 10x apart, decided via curve-fit). 3 is the accepted minimum but can only return noise-floor O(1) or an inconclusive estimate — never a growth class, not even exponential, which needs 2+ agreeing doubling ratios (so 4+ sizes) and cannot decide on one; 4+ sizes are needed to trust the curve fit; 5+ DOUBLING sizes (n,2n,4n,8n,16n) are needed for a robust ratio median	100,1000,10000,100000
`timeout`	No	Wall-clock seconds allowed per size before that run is killed
`language`	No	Language `code` is written in; default 'python3'	python3

Output Schema

TableJSON Schema

Name	Required	Description	Default
No arguments

Schema Changelog

Changes observed during successful MCP inspections.

Changed1 schema field changedv0.13.0Sep 23, 2026

changedInput schema / properties / sizes / description

Previous value: -"Comma-separated input sizes to run at, e.g. '100,1000,10000,100000'"New value: +"Comma-separated input sizes to run at, e.g. '100,1000,10000,100000' (this default: 4 sizes 10x apart, decided via curve-fit). 3 is the accepted minimum but can only return noise-floor O(1) or an inconclusive estimate — never a growth class, not even exponential, which needs 2+ agreeing doubling ratios (so 4+ sizes) and cannot decide on one; 4+ sizes are needed to trust the curve fit; 5+ DOUBLING sizes (n,2n,4n,8n,16n) are needed for a robust ratio median"

Changed4 schema fields changedv0.12.0Sep 9, 2026

addedInput schema / properties / code / description

Added value: +"Program that reads integer N from stdin's first line and does work sized by N"

addedInput schema / properties / language / description

Added value: +"Language `code` is written in; default 'python3'"

addedInput schema / properties / sizes / description

Added value: +"Comma-separated input sizes to run at, e.g. '100,1000,10000,100000'"

addedInput schema / properties / timeout / description

Added value: +"Wall-clock seconds allowed per size before that run is killed"

Changed1 schema field changedv0.11.0Sep 9, 2026

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "additionalProperties": true,
+  "title": "benchmarkDictOutput",
+  "type": "object"
+}

Changed7 schema fields changedv0.2.0Aug 19, 2026
- removedInput schema / additionalProperties
```
Removed value: -false
```
- addedInput schema / properties / code / title
```
Added value: +"Code"
```
- addedInput schema / properties / language / title
```
Added value: +"Language"
```
- addedInput schema / properties / sizes / title
```
Added value: +"Sizes"
```
- addedInput schema / properties / timeout / title
```
Added value: +"Timeout"
```
- addedInput schema / title
```
Added value: +"benchmarkArguments"
```
- changedOutput schema / (root)
```
Previous value: -{
-  "additionalProperties": true,
-  "type": "object"
-}New value: +null
```
First observedv0.1.0Aug 7, 2026

TDQS

A4.3/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations, it discloses that the tool actually executes the submitted code, fits a growth curve, can return only noise-floor O(1) or 'inconclusive' at 3 sizes, and names the deciding estimator via estimate_basis. This gives an agent an unusually precise model of the tool's behavior and limitations. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose is front-loaded and every section contributes, including the example. It is slightly dense and repeats some size-threshold details already present in the schema, keeping it from a top score.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description need not describe return values; it covers the code contract, acceptable sizes, timeout semantics, and language. The estimator caveat and example leave an agent with everything needed to construct a valid call.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds value by stating the stdin contract explicitly and giving a complete Python example, clarifying what the code parameter must do. The sample-size caveats largely duplicate the sizes parameter description, so it does not go higher.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence names a specific action ('Empirically measure time complexity') and resource ('running code at increasing input sizes'), making the tool's function clear. It does not explicitly name an alternative such as analyze_complexity, so it stops short of full sibling differentiation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives concrete context for when results are trustworthy: it specifies the code contract and explains the minimum sizes, when curve fits can be trusted, and when a robust ratio median is available. It does not explicitly say when to prefer this tool over siblings like analyze_complexity, but the empirical framing and thresholds imply the intended use.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.