Skip to main content
Glama

Declare run expectations

declare_run_expectations
Idempotent

Specify the criteria a run's final ping body must meet, so the run doesn't self-grade and failures are logged as assertion errors.

Instructions

Requires the write scope or higher. Declares, at the start of a run, the criteria its success ping body is judged by when the run closes, so the run does not grade itself: a success whose body fails any declared criterion is recorded as a failed run with cause 'assertion', whatever the exit code. One declaration per rid, immutable (a second call is a conflict). For use right after the run's /start ping; get_ping_instructions' expectations_how_to has a worked example.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
ridYesThe run id exactly as sent on this run's /start ping, the same rid used on every step and the terminal ping.
check_idYesMonitor UUID (from create_monitor or list_monitors).
assertionsYesThe run's complete criteria, as a JSON array string, e.g. '[{"kind":"json_path","path":"result.rows_processed","op":"gt","value":"0"}]'. Fields: kind (required), value, path, op; no name. kind: contains, not_contains, matches (RE2, max 1000 bytes) or json_path (the value at a dotted path, e.g. 'a.b.c', compared with op: eq, ne, gt, gte, lt, lte). Only not_contains entries, or a matches pattern that accepts an empty body ('.*', '^$'), pass on empty output; a contains, json_path or empty-rejecting matches entry catches a run that produced nothing. At most 20; one malformed entry rejects the whole declaration. Runs with no declaration keep the monitor's own assertions.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changedv0.1.17
    • changedInput schema / properties / assertions / description
      Previous value: -"The run's complete set of expectations, declared ONCE at the start of the run -- criteria the ping BODY of THIS run's eventual success ping must satisfy when the run closes, checked instead of letting the run grade itself. IMMUTABLE: a second call for the same rid is rejected with a conflict error and the first declaration stands unchanged -- there is no way to edit, add to, or replace it once made, so decide the whole set before you start work. Declaring nothing is allowed and always has been: simply never call this tool for a run, and the monitor's own check-level assertions (if any) stay in force unchanged. INCLUDE AT LEAST ONE POSITIVE CRITERION -- a 'contains', 'matches' or 'json_path' entry -- in every declaration. A declaration made ENTIRELY of 'not_contains' entries is self-satisfying on empty output: a run that produces nothing at all still passes, because there is nothing for the pattern to find. That is precisely the evasion this feature exists to close, so a purely negative declaration defeats its own purpose. A 'matches' entry only counts as positive if its pattern REJECTS an empty body: '.*', '(?s).*' and '^$' all accept one and are validated as perfectly legal patterns, so a declaration resting on one of those is no better than a purely negative declaration. Supply a JSON ARRAY as a string, e.g. '[{\"kind\":\"json_path\",\"path\":\"result.rows_processed\",\"op\":\"gt\",\"value\":\"0\"}]'. Fields per entry: kind (required), value, path, op -- no name; a run's declared criteria have none, unlike a monitor's own output assertions. kind is one of 'contains' (body contains value as a substring), 'not_contains' (body does not contain it), 'matches' (body matches value as a Go RE2 regexp, max 1000 bytes), or 'json_path' (parse the body as JSON, read the value at path, compare it against value with op). contains/not_contains/matches require value; json_path requires path and op. path is a DOTTED path only ('a.b.c') -- the query syntax of a real JSONPath library ('[', '*', '$') is rejected. op is one of 'eq', 'ne', 'gt', 'gte', 'lt', 'lte'. At most 20 assertions per run. A malformed entry (uncompilable regexp, a path carrying query syntax, an unknown kind or op) is rejected before anything is written, and nothing is stored if any entry fails."New value: +"The run's complete criteria, as a JSON array string, e.g. '[{\"kind\":\"json_path\",\"path\":\"result.rows_processed\",\"op\":\"gt\",\"value\":\"0\"}]'. Fields: kind (required), value, path, op; no name. kind: contains, not_contains, matches (RE2, max 1000 bytes) or json_path (the value at a dotted path, e.g. 'a.b.c', compared with op: eq, ne, gt, gte, lt, lte). Only not_contains entries, or a matches pattern that accepts an empty body ('.*', '^$'), pass on empty output; a contains, json_path or empty-rejecting matches entry catches a run that produced nothing. At most 20; one malformed entry rejects the whole declaration. Runs with no declaration keep the monitor's own assertions."
    • changedInput schema / properties / rid / description
      Previous value: -"The run id exactly as sent on this run's /start ping — the same rid used on every step and the terminal ping."New value: +"The run id exactly as sent on this run's /start ping, the same rid used on every step and the terminal ping."
  2. First observed

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations it discloses a lot: the required write scope, the grading consequence (a success failing any criterion is logged as a failed run with cause 'assertion' regardless of exit code), immutability and one-declaration-per-rid, and the dependency on the /start ping. This is exactly the behavioral context annotations cannot convey.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the scope requirement, then behavior, then constraints and usage in a few dense sentences, each earning its place. Slightly long/compound sentences keep it from a 5, but there is no wasted filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a declaration tool with no output schema, the description covers scope, timing, semantics, immutability and a pointer to an example, which is nearly everything needed to invoke it correctly. It does not describe the success response or error surface beyond the conflict case, a minor gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the schema already exhaustively documents rid, check_id and assertions (kinds, ops, limits, malformed handling). The description adds framing ('criteria its success ping body is judged by') but no parameter syntax or format beyond the schema, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: it 'Declares, at the start of a run, the criteria its success ping body is judged by when the run closes.' An agent can immediately tell this is a run-scoped assertion-registration tool, distinct from siblings like get_run, list_runs or get_ping_instructions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear timing ('right after the run's /start ping') and points to 'get_ping_instructions' expectations_how_to' for a worked example, which is strong routing context. It stops short of an explicit when-not-to-use or alternative-tool exclusion, so it falls just below the top band.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.