Skip to main content
Glama
Sign In
caveat-lang

caveat-lang

by WSattazahn
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Rust
Local

caveat_test

Read-onlyIdempotent

Run scenario assertions and runtime invariant checks against inline Caveat source, with a fresh session per scenario, to verify expected event outcomes and snapshot values.

Instructions

Run scenario assertions and runtime invariant checks against inline source, with a fresh session per scenario. Use to verify expected event outcomes and snapshot values; caveat_validate only checks loading, caveat_check reports diagnostics, and caveat_explain inspects history without assertions. report (caveat-scenario-report/0.1) includes files, per-scenario results/failures, check diagnostics and passed/failed totals. exitCode: 0 all passed; 1 scenario failure; 2 invalid scenario document (report has valid:false and error) or blocking authoring diagnostics. Every source reference must be inline.cav. Returns the same JSON in structuredContent and text: {schema:"caveat-authoring/0.1",operation,sourceSha256,runtime,exitCode,report}. Read exitCode even when isError is false. Bridge failures instead set isError:true with {schema:"caveat-authoring-error/0.1",error:{kind,message}}.

Input Schema

NameRequiredDescriptionDefault
sourceYesComplete inline Caveat program text, available as the virtual file inline.cav. No file path or URL lookup. At most 1 MiB of UTF-8 source; each call uses a fresh runtime.
scenariosYesScenario document object, not JSON text. Example: {"schema":"caveat-scenarios/0.1","source":"inline.cav","scenarios":[{"id":"initial","title":"Initial sequence","steps":[{"expect":{"/sequence":0}}]}]}. Each scenario needs a unique nonempty id, a title and nonempty steps. Every source reference, including optional per-scenario source, must be inline.cav. Each step has exactly one operation: send (event name; optional payload, rejected, repeat), expect (JSON Pointer-to-value map into the snapshot; a binding declared as `bind install.state` is at /bindings/install/state), checkpoint (state name), same_as (stored state name, with paths), resume:true, or size (save/snapshot byte bounds). Sends expect acceptance unless rejected specifies a policy message, true for any policy rejection, or an object with origin and optional code/message. Object expectations match listed members; arrays match in order; $exact, $set, $includes and $absent matchers are supported. Limits: 1 MiB serialized UTF-8 JSON, 32 scenarios, 2,000 total steps and 1,000 sends after repeat expansion.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv1.2.3
    • changedInput schema / properties / scenarios / description
      Previous value: -"Scenario document object, not JSON text. Example: {\"schema\":\"caveat-scenarios/0.1\",\"source\":\"inline.cav\",\"scenarios\":[{\"id\":\"initial\",\"title\":\"Initial sequence\",\"steps\":[{\"expect\":{\"/sequence\":0}}]}]}. Each scenario needs a unique nonempty id, a title and nonempty steps. Every source reference, including optional per-scenario source, must be inline.cav. Each step has exactly one operation: send (event name; optional payload, rejected, repeat), expect (JSON Pointer-to-value map), checkpoint (state name), same_as (stored state name, with paths), resume:true, or size (save/snapshot byte bounds). Sends expect acceptance unless rejected specifies a policy message, true for any policy rejection, or an object with origin and optional code/message. Object expectations match listed members; arrays match in order; $exact, $set, $includes and $absent matchers are supported. Limits: 1 MiB serialized UTF-8 JSON, 32 scenarios, 2,000 total steps and 1,000 sends after repeat expansion."New value: +"Scenario document object, not JSON text. Example: {\"schema\":\"caveat-scenarios/0.1\",\"source\":\"inline.cav\",\"scenarios\":[{\"id\":\"initial\",\"title\":\"Initial sequence\",\"steps\":[{\"expect\":{\"/sequence\":0}}]}]}. Each scenario needs a unique nonempty id, a title and nonempty steps. Every source reference, including optional per-scenario source, must be inline.cav. Each step has exactly one operation: send (event name; optional payload, rejected, repeat), expect (JSON Pointer-to-value map into the snapshot; a binding declared as `bind install.state` is at /bindings/install/state), checkpoint (state name), same_as (stored state name, with paths), resume:true, or size (save/snapshot byte bounds). Sends expect acceptance unless rejected specifies a policy message, true for any policy rejection, or an object with origin and optional code/message. Object expectations match listed members; arrays match in order; $exact, $set, $includes and $absent matchers are supported. Limits: 1 MiB serialized UTF-8 JSON, 32 scenarios, 2,000 total steps and 1,000 sends after repeat expansion."
  2. Changed2 schema fields changedv0.1.1
    • addedInput schema / properties / scenarios / description
      Added value: +"Scenario document object, not JSON text. Example: {\"schema\":\"caveat-scenarios/0.1\",\"source\":\"inline.cav\",\"scenarios\":[{\"id\":\"initial\",\"title\":\"Initial sequence\",\"steps\":[{\"expect\":{\"/sequence\":0}}]}]}. Each scenario needs a unique nonempty id, a title and nonempty steps. Every source reference, including optional per-scenario source, must be inline.cav. Each step has exactly one operation: send (event name; optional payload, rejected, repeat), expect (JSON Pointer-to-value map), checkpoint (state name), same_as (stored state name, with paths), resume:true, or size (save/snapshot byte bounds). Sends expect acceptance unless rejected specifies a policy message, true for any policy rejection, or an object with origin and optional code/message. Object expectations match listed members; arrays match in order; $exact, $set, $includes and $absent matchers are supported. Limits: 1 MiB serialized UTF-8 JSON, 32 scenarios, 2,000 total steps and 1,000 sends after repeat expansion."
    • changedInput schema / properties / source / description
      Previous value: -"Inline Caveat source; no file or URL lookup."New value: +"Complete inline Caveat program text, available as the virtual file inline.cav. No file path or URL lookup. At most 1 MiB of UTF-8 source; each call uses a fresh runtime."
  3. First observedv0.1.0

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, but the description adds substantial context beyond them: exitCode semantics (0/1/2), the fact that a valid exitCode must be read even when isError is false, and the separate error envelope for bridge failures. It stops short of saying how scenario authoring diagnostics block execution beyond the exitCode=2 note, but the behavioral surface is well disclosed and consistent with readOnlyHint.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose and sibling routing before detail, and every sentence carries real signal for a tool with this much surface area. It is dense but the only mild waste is restating schema constraints already present in the parameters.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description takes on the full return-value burden and does so: report schema id, per-scenario results, totals, exitCode ladder, the structuredContent/text equivalence, and the distinct isError error shape. An agent has everything needed to interpret results correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and both parameters are richly documented in-schema, including the inline.cav virtual-file rule, scenario step grammar and all limits. The description largely restates those constraints (inline.cav, limits) rather than adding new parameter meaning, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Run scenario assertions and runtime invariant checks against inline source') plus a scoping rule ('fresh session per scenario'). It then explicitly distinguishes itself from all three relevant siblings by naming what each does instead, so an agent can route without opening any other schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit 'use to' clause and contrasts it against caveat_validate (loading only), caveat_check (diagnostics) and caveat_explain (history without assertions). The 'Every source reference must be inline.cav' constraint further bounds correct invocation.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools