Skip to main content
Glama
io.github.justrunme/agent-security-contract

io.github.justrunme/agent-security-contract

by justrunme

verify_corpus

Run an abuse-case corpus against an ASC policy to measure critical false-ALLOW counts, benign false-BLOCK rates, and per-scenario outcomes.

Instructions

Verify/run the abuse-case corpus against an ASC policy.

Runs all scenario JSON files in the corpus directory and returns metrics including critical false-ALLOW count, benign false-BLOCK rate, and per-scenario results.

Args: policy_path: Path to the ASC policy YAML file corpus_dir: Directory containing scenario JSON files output_results: If True, return full per-scenario results (may be large)

Returns: Mapping with passed (bool), metrics, and optionally scenarios list

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "policy_path": "<policy_path>",
      "corpus_dir": "<corpus_dir>"
    },
    "name": "verify_corpus"
  }
}

Input Schema

NameRequiredDescriptionDefault
corpus_dirYes
policy_pathYes
output_resultsNo

Output Schema

NameRequiredDescriptionDefault

No arguments

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.3.1

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden, and it does disclose what gets executed (all scenario files in the directory) and that output_results can produce large payloads. It stops short of stating whether the run is read-only, whether any files are written, how long a full corpus run takes, or how failures mid-run are handled.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action, then a one-sentence behavior note, then Args and Returns blocks. Every line adds information and nothing is repeated from the schema without added meaning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so the brief Returns line is more than sufficient, and all three parameters are explained in prose. The remaining gap is operational: no mention of runtime, timeout, or partial-failure semantics for a tool that executes an entire directory of scenarios.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does: policy_path is identified as the ASC policy YAML, corpus_dir as the directory of scenario JSON files, and output_results as the switch that returns full per-scenario results which may be large. Format hints (YAML vs JSON) and the size warning are genuine additions beyond the bare schema titles.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: "Verify/run the abuse-case corpus against an ASC policy," then clarifies it executes every scenario JSON file in the corpus directory. The corpus-versus-single-scenario framing implicitly separates it from verify_scenario, though no sibling is named explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It implies when to use it (you have a whole corpus directory and want aggregate metrics), but never states alternatives or exclusions such as using verify_scenario for a single case. No prerequisites, ordering, or cost guidance is given for choosing this over the cheaper single-scenario tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.