OpenMark

Estimate a run's cost

estimate_cost

Credits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance. Same inputs and run settings as run_benchmark. Use it to drop an expensive model, or to compare settings (e.g. max_tokens), before running.

Input Schema

Name	Required	Description	Default
`models`	Yes
`preset`	No	Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it
`fail_fast`	No	Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures
`task_yaml`	Yes	The full task YAML (a JSON object with the same fields also works)
`max_tokens`	No	Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value
`upload_uid`	No	From prepare_attachments, so image and document tokens are counted
`custom_timeout`	No	Seconds per answer, with timeout_profile custom
`stability_runs`	No	Passes per model; 2+ shows which models are consistent
`temp_preference`	No	With temperature_discovery: which temperature wins a tie	low
`timeout_profile`	No	How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout	snappy
`temperature_discovery`	No	Find each model's best temperature for this task. Costs about 6x as much

Schema Changelog

Changes observed during successful MCP inspections.

Changed9 schema fields changedOct 19, 2026

addedInput schema / properties / custom_timeout

Added value: +{
+  "description": "Seconds per answer, with timeout_profile custom",
+  "maximum": 600,
+  "minimum": 120,
+  "type": "integer"
+}

addedInput schema / properties / fail_fast

Added value: +{
+  "default": true,
+  "description": "Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures",
+  "type": "boolean"
+}

addedInput schema / properties / max_tokens

Added value: +{
+  "default": 500,
+  "description": "Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value",
+  "maximum": 16000,
+  "minimum": 0,
+  "type": "integer"
+}

addedInput schema / properties / preset

Added value: +{
+  "description": "Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it",
+  "enum": [
+    "standard",
+    "thorough"
+  ],
+  "type": "string"
+}

addedInput schema / properties / stability_runs / description

Added value: +"Passes per model; 2+ shows which models are consistent"

changedInput schema / properties / stability_runs / maximum
```
Previous value: -3New value: +5
```

addedInput schema / properties / temp_preference

Added value: +{
+  "default": "low",
+  "description": "With temperature_discovery: which temperature wins a tie",
+  "enum": [
+    "low",
+    "high"
+  ],
+  "type": "string"
+}

addedInput schema / properties / temperature_discovery

Added value: +{
+  "default": false,
+  "description": "Find each model's best temperature for this task. Costs about 6x as much",
+  "type": "boolean"
+}

addedInput schema / properties / timeout_profile

Added value: +{
+  "default": "snappy",
+  "description": "How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout",
+  "enum": [
+    "snappy",
+    "patient",
+    "custom"
+  ],
+  "type": "string"
+}

AddedOct 19, 2026

TDQS

A4.4/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden, and it discloses the returned content: expected total, per-model split, balance needed to start, worst case, and the user's balance. That is real behavioral context for a tool with no output schema. It omits auth requirements and any latency caveats, keeping it short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with what the call returns and why it exists. Every clause earns its place: return contents, input equivalence, and the two decision scenarios it enables.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema and no annotations, the description supplies the return-value inventory, the input contract, and the usage trigger, which is most of what an agent needs. It stops short of stating safety/permission expectations or rate behavior, leaving a small gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is already 91%, so the schema documents the parameters well and the baseline is 3. The description adds cross-tool semantics that the schema cannot: 'Same inputs and run settings as run_benchmark,' letting an agent reuse a known input shape rather than re-derive 11 parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('credits a run_benchmark call would use') and immediately marks the scope as pre-execution ('before spending anything'). It is unmistakably distinct from run_benchmark, which the agent can identify from the name alone.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete when-to-use cases: 'drop an expensive model, or to compare settings (e.g. max_tokens), before running.' It also anchors the tool to its sibling by noting 'Same inputs and run settings as run_benchmark.' It does not explicitly state when not to use it, but the 'before running' framing implies the boundary.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All OpenMark tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources

Need Help?