Skip to main content
Glama

testpass_run

Start a TestPass run against an MCP server to seed deterministic and agent checks and get a run id with verdict summary. Retries with the same task_id stay idempotent.

Instructions

Start a TestPass run against an MCP server. Seeds deterministic and agent checks from the given pack and returns the run id plus an initial verdict summary. Response includes was_duplicate: boolean indicating whether the row was already present (idempotent retry).

Input Schema

NameRequiredDescriptionDefault
pack_idNoPack slug (default: testpass-core)
profileNoRun profile (default: smoke)
task_idNoClient-generated idempotency key. The API requires UUID format; non-UUID values are deterministically converted to a stable UUIDv5-shaped id, so any string is safe and retries stay idempotent. Required for safe retry. If omitted, the server creates a fresh row and you lose retry safety; sending the same task_id twice returns the original run_id with was_duplicate=true instead of creating a duplicate.
target_urlYesHTTP URL of the MCP server to test

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv0.1.11
    • changedInput schema / properties / task_id / description
      Previous value: -"Client-generated idempotency key (UUIDv5 from thread_id + prompt_hash + time_bucket recommended). Required for safe retry. If omitted, the server creates a fresh row and you lose retry safety; sending the same task_id twice returns the original run_id with was_duplicate=true instead of creating a duplicate."New value: +"Client-generated idempotency key. The API requires UUID format; non-UUID values are deterministically converted to a stable UUIDv5-shaped id, so any string is safe and retries stay idempotent. Required for safe retry. If omitted, the server creates a fresh row and you lose retry safety; sending the same task_id twice returns the original run_id with was_duplicate=true instead of creating a duplicate."
  2. Addedv0.1.9
  3. Removed
  4. Addedv0.1.5

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden, and it does add meaningful behavior: it seeds checks, returns a run id plus an initial verdict summary, and explains the was_duplicate idempotency-retry semantics. It stops short of disclosing auth, rate limits, or whether re-running mutates prior state, so it is strong but not complete.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, front-loaded with the primary action and outcome, with the was_duplicate detail last. No wasted phrasing, though the third sentence is somewhat redundant with the task_id schema description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema and no annotations, the description compensates by naming the return values (run id, verdict summary, was_duplicate) and the seeding behavior. For a 4-param mutation tool this is nearly complete; only auth/permission expectations are unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the task_id field already documents the idempotency behavior in detail, so the schema does the heavy lifting. The description only alludes to the pack and idempotent retry, adding little beyond the structured fields. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Start) and resource (a TestPass run against an MCP server), and describes the core mechanic of seeding deterministic and agent checks from a pack. An agent can distinguish it from siblings like testpass_status or testpass_report_md without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The context of starting a run implies this is the entry-point tool, and the mention of the pack and idempotent retry implies usage, but the description never explicitly says when to call this versus testpass_status or the report tools. Usage is inferable rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools