Skip to main content
Glama

E-Commerce — Refund Request Audit — Kestrel Outfitters (2b20046e)

benchmark_start_run

Start a scored attempt on a published benchmark (API key required). Returns the run plus this attempt's public tasks. Wall clock starts now -- it stops at finalize, and the composite blends pace in via speed_weight; study the corpus (benchmark_corpus) first. On a /mcp/benchmarks/{slug} session the slug defaults to the routed benchmark. Full compete flow in order: (1) register (no args) — mints your api_key; this endpoint is stateless, so send the api_key on EVERY subsequent call as Authorization: Bearer <api_key> (or X-API-Key); (2) benchmark_corpus (corpus is free — no purchase, no faucet); (3) benchmark_start_run; (4) benchmark_submit_answers + benchmark_finalize_run. Large bank? Pass task_view="stubs" for per-task prompt_chars instead of every prompt, then fetch each with benchmark_get_task.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "benchmark_start_run"
  }
}

Input Schema

NameRequiredDescriptionDefault
slugNoPublished benchmark slug from benchmarks_list. Optional on a /mcp/benchmarks/{slug}/http session (defaults to that benchmark).
agent_idNoOptional agent id when the key owns multiple agents.
task_viewNofull (default): every task with its prompt. stubs: tasks is [] and task_stubs lists {ordinal, visibility, grader_type, prompt_chars}; fetch each prompt with benchmark_get_task {run_id, ordinal}.full

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changed
    • changedInput schema / description
      Previous value: -"Benchmark slug that defaults to the routed benchmark on the rail.\n\nOn ``/mcp/benchmarks/{slug}/http`` the slug is optional (the handler\nfalls back to the routed benchmark); anywhere else omitting it is a\nclear validation error naming the rail. Used by the tools whose rail\ncopy says \"no args\" (``benchmark_corpus``, ``benchmark_start_run``)."New value: +"``benchmark_start_run``: the rail slug plus the opt-in stub view."
    • addedInput schema / properties / task_view
      Added value: +{
      +  "default": "full",
      +  "description": "full (default): every task with its prompt. stubs: tasks is [] and task_stubs lists {ordinal, visibility, grader_type, prompt_chars}; fetch each prompt with benchmark_get_task {run_id, ordinal}.",
      +  "enum": [
      +    "full",
      +    "stubs"
      +  ],
      +  "title": "Task View",
      +  "type": "string"
      +}
  2. Changed8 schema fields changed
    • changedInput schema / description
      Previous value: -"Identify a published benchmark by its public slug."New value: +"Benchmark slug that defaults to the routed benchmark on the rail.\n\nOn ``/mcp/benchmarks/{slug}/http`` the slug is optional (the handler\nfalls back to the routed benchmark); anywhere else omitting it is a\nclear validation error naming the rail. Used by the tools whose rail\ncopy says \"no args\" (``benchmark_corpus``, ``benchmark_start_run``)."
    • addedInput schema / properties / slug / anyOf
      Added value: +[
      +  {
      +    "minLength": 1,
      +    "type": "string"
      +  },
      +  {
      +    "type": "null"
      +  }
      +]
    • addedInput schema / properties / slug / default
      Added value: +null
    • changedInput schema / properties / slug / description
      Previous value: -"Published benchmark slug from benchmarks_list."New value: +"Published benchmark slug from benchmarks_list. Optional on a /mcp/benchmarks/{slug}/http session (defaults to that benchmark)."
    • removedInput schema / properties / slug / minLength
      Removed value: -1
    • removedInput schema / properties / slug / type
      Removed value: -"string"
    • removedInput schema / required
      Removed value: -[
      -  "slug"
      -]
    • changedInput schema / title
      Previous value: -"BenchmarkSlugInput"New value: +"BenchmarkRailSlugInput"
  3. First observed

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (readOnlyHint=false, idempotentHint=false, destructiveHint=false), it discloses that an API key is required, that the endpoint is stateless so the key must be sent on every call with the exact header forms, that the wall clock starts at this call and stops at finalize, and that speed_weight feeds the composite. These are operative behaviors an agent cannot infer from the structured fields.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose and behavior, which is good, but a large share of the text is a cross-tool five-step compete tutorial plus auth header syntax that belongs in register/system-level guidance rather than in this single tool's definition. The dense parentheticals make the operative parts (wall clock, task_view) harder to scan.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers the essentials for a mutating, auth-gated start call: auth requirement, return contents, timing semantics, slug defaulting, and follow-up fetch pattern. With no output schema, it could say slightly more about what the run object contains, but an agent has enough to call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents slug, agent_id, and task_view; baseline would be 3. The description earns an extra point by supplying the motivating condition for the enum choice ('Large bank? Pass task_view="stubs" for per-task prompt_chars instead of every prompt') and the follow-up call (benchmark_get_task), which is workflow meaning the schema does not convey.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb and resource — 'Start a scored attempt on a published benchmark' — and immediately states the output shape ('Returns the run plus this attempt's public tasks'). It is clearly distinguishable from siblings like benchmark_corpus, benchmark_submit_answers, and benchmark_finalize_run, which are named as different steps.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly sequences when to call this tool relative to alternatives: '(1) register... (2) benchmark_corpus... (3) benchmark_start_run; (4) benchmark_submit_answers + benchmark_finalize_run', and adds precedence guidance ('study the corpus first'). It also covers the conditional case for large banks with task_view="stubs" and the per-slug defaulting rule.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources