Skip to main content
Glama

IA-QA — 152 QA & Dev Tools for AI Agents

validate_agent_trajectory

Read-onlyIdempotent

Run declarative assertions on an agent trace (OpenAI tool calls, Anthropic tool_use, LangChain run trees, ReAct logs). Deterministic, no LLM. Assertions: order (tool A before B), must_call, must_not_call, max/min calls, no_error, recovery (a successful step after the last error), no_injection, no_pii. A step is errored when the trace marks it (is_error, status error/failed, success:false) or its text starts with Error:/Traceback — prose is never keyword-scanned. Verdict PASS / FAIL / INCONCLUSIVE (0 steps parsed, never PASS).

Input Schema

NameRequiredDescriptionDefault
traceYesThe trace: JSON (messages array, run tree) or a Thought/Action/Observation text log.
formatNoDefault auto.
assertionsYes

Output Schema

NameRequiredDescriptionDefault
stepsNo
resultsNo
summaryNo
verdictNo
warningsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed14 schema fields changed
    • addedInput schema / examples
      Added value: +[
      +  {
      +    "assertions": [
      +      {
      +        "tool": "search",
      +        "type": "must_call"
      +      }
      +    ],
      +    "trace": [
      +      {
      +        "role": "assistant",
      +        "tool_calls": [
      +          {
      +            "function": {
      +              "arguments": "{}",
      +              "name": "search"
      +            }
      +          }
      +        ]
      +      }
      +    ]
      +  }
      +]
    • removedInput schema / properties / assertions / description
      Removed value: -"List of assertions to validate against the trace."
    • changedInput schema / properties / assertions / items / properties / after / description
      Previous value: -"[order] Tool that must be called after."New value: +"[order] Called after."
    • changedInput schema / properties / assertions / items / properties / before / description
      Previous value: -"[order] Tool that must be called first."New value: +"[order] Called first."
    • removedInput schema / properties / assertions / items / properties / id / description
      Removed value: -"Optional assertion identifier."
    • changedInput schema / properties / assertions / items / properties / max / description
      Previous value: -"[max_calls] Maximum number of allowed calls."New value: +"[max_calls]"
    • changedInput schema / properties / assertions / items / properties / min / description
      Previous value: -"[min_calls] Minimum number of required calls."New value: +"[min_calls]"
    • changedInput schema / properties / assertions / items / properties / sensitivity / description
      Previous value: -"[no_injection] Default low = critical patterns only (override, jailbreak, fake role delimiters). Higher catches more and fires on ordinary text (\"act as\", {{templates}}) — pattern matching, not a classifier."New value: +"[no_injection] Default low (critical patterns only); higher also fires on ordinary text."
    • changedInput schema / properties / assertions / items / properties / tool / description
      Previous value: -"Tool name to check (for must_call, must_not_call, max_calls, min_calls)."New value: +"For must_call, must_not_call, max_calls, min_calls."
    • changedInput schema / properties / assertions / items / properties / type / description
      Previous value: -"Assertion type. no_injection runs prompt_injection_scan on every tool result (indirect injection: text a third party wrote that the agent then reads); no_pii runs redact_pii on the final answer (a leak to the user). Both report types and counts, never the matched values."New value: +"no_injection scans tool results (indirect injection); no_pii scans the final answer. Both report types and counts, never values."
    • changedInput schema / properties / assertions / items / properties / types / description
      Previous value: -"[no_pii] Comma-separated PII types (email, phone, ssn, credit_card, ip_address, jwt). Default: all."New value: +"[no_pii] Comma-separated: email, phone, ssn, credit_card, ip_address, jwt. Default all."
    • changedInput schema / properties / assertions / items / properties / where / description
      Previous value: -"[no_injection, no_pii] What to scan. Defaults: no_injection → tool_results, no_pii → answer (PII in a tool result is often legitimate — an order carries the customer email)."New value: +"[no_injection, no_pii] Override what is scanned."
    • changedInput schema / properties / format / description
      Previous value: -"Trace format. auto (default) detects automatically."New value: +"Default auto."
    • changedInput schema / properties / trace / description
      Previous value: -"Agent execution trace as JSON (OpenAI messages array, LangChain run tree) or plain text log (Thought/Action/Observation format)."New value: +"The trace: JSON (messages array, run tree) or a Thought/Action/Observation text log."
  2. Changed5 schema fields changed
    • addedInput schema / properties / assertions / items / properties / sensitivity
      Added value: +{
      +  "description": "[no_injection] Default low = critical patterns only (override, jailbreak, fake role delimiters). Higher catches more and fires on ordinary text (\"act as\", {{templates}}) — pattern matching, not a classifier.",
      +  "enum": [
      +    "low",
      +    "medium",
      +    "high"
      +  ],
      +  "type": "string"
      +}
    • changedInput schema / properties / assertions / items / properties / type / description
      Previous value: -"Assertion type."New value: +"Assertion type. no_injection runs prompt_injection_scan on every tool result (indirect injection: text a third party wrote that the agent then reads); no_pii runs redact_pii on the final answer (a leak to the user). Both report types and counts, never the matched values."
    • changedInput schema / properties / assertions / items / properties / type / enum
      Previous value: -[
      -  "order",
      -  "must_call",
      -  "must_not_call",
      -  "max_calls",
      -  "min_calls",
      -  "no_error",
      -  "recovery"
      -]New value: +[
      +  "order",
      +  "must_call",
      +  "must_not_call",
      +  "max_calls",
      +  "min_calls",
      +  "no_error",
      +  "recovery",
      +  "no_injection",
      +  "no_pii"
      +]
    • addedInput schema / properties / assertions / items / properties / types
      Added value: +{
      +  "description": "[no_pii] Comma-separated PII types (email, phone, ssn, credit_card, ip_address, jwt). Default: all.",
      +  "type": "string"
      +}
    • addedInput schema / properties / assertions / items / properties / where
      Added value: +{
      +  "description": "[no_injection, no_pii] What to scan. Defaults: no_injection → tool_results, no_pii → answer (PII in a tool result is often legitimate — an order carries the customer email).",
      +  "enum": [
      +    "tool_results",
      +    "answer"
      +  ],
      +  "type": "string"
      +}
  3. Changed7 schema fields changed
    • removedOutput schema / properties / assertions
      Removed value: -{
      -  "items": {
      -    "type": "object"
      -  },
      -  "type": "array"
      -}
    • removedOutput schema / properties / failed
      Removed value: -{
      -  "type": "number"
      -}
    • removedOutput schema / properties / passed
      Removed value: -{
      -  "type": "number"
      -}
    • addedOutput schema / properties / results
      Added value: +{
      +  "items": {
      +    "type": "object"
      +  },
      +  "type": "array"
      +}
    • addedOutput schema / properties / summary
      Added value: +{
      +  "additionalProperties": true,
      +  "type": "object"
      +}
    • removedOutput schema / properties / total
      Removed value: -{
      -  "type": "number"
      -}
    • addedOutput schema / properties / warnings
      Added value: +{
      +  "items": {
      +    "type": "string"
      +  },
      +  "type": "array"
      +}
  4. Changed1 schema field changed
    • addedOutput schema / properties
      Added value: +{
      +  "assertions": {
      +    "items": {
      +      "type": "object"
      +    },
      +    "type": "array"
      +  },
      +  "failed": {
      +    "type": "number"
      +  },
      +  "passed": {
      +    "type": "number"
      +  },
      +  "steps": {
      +    "items": {
      +      "type": "object"
      +    },
      +    "type": "array"
      +  },
      +  "total": {
      +    "type": "number"
      +  },
      +  "verdict": {
      +    "type": "string"
      +  }
      +}
  5. Changed1 schema field changed
    • changedInput schema / properties / trace / oneOf
      Previous value: -[
      -  {
      -    "type": "string"
      -  },
      -  {
      -    "type": "object"
      -  },
      -  {
      -    "items": {},
      -    "type": "array"
      -  }
      -]New value: +[
      +  {
      +    "type": "string"
      +  },
      +  {
      +    "additionalProperties": true,
      +    "type": "object"
      +  },
      +  {
      +    "items": {},
      +    "type": "array"
      +  }
      +]
  6. Changed1 schema field changed
    • changedInput schema / properties / trace / oneOf
      Previous value: -[
      -  {
      -    "type": "string"
      -  },
      -  {
      -    "type": "object"
      -  },
      -  {
      -    "type": "array"
      -  }
      -]New value: +[
      +  {
      +    "type": "string"
      +  },
      +  {
      +    "type": "object"
      +  },
      +  {
      +    "items": {},
      +    "type": "array"
      +  }
      +]
  7. Changed1 schema field changed
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "additionalProperties": true,
      +  "type": "object"
      +}
  8. First observed

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, and the description adds substantial context beyond them: deterministic with no LLM, the exact rule for what counts as an errored step (is_error, status error/failed, success:false, or Error:/Traceback prefix), the guarantee that prose is never keyword-scanned, and the safety property that no_injection/no_pii report types and counts but never values. It also defines the INCONCLUSIVE verdict and the 0-steps-never-PASS rule.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action and format support, then compressed into a dense assertion catalogue and verdict rules with no filler. The telegraphic fragment style and long parenthetical list are efficient but slightly harder to parse than plain prose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return shape needn't be restated, yet the description still supplies the verdict semantics and the empty-trace edge case an agent needs to interpret results. Combined with the error-detection rules and assertion semantics, nothing material is missing for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

At 67% schema coverage the schema already documents most fields, but the description adds real semantics: what 'recovery' means (a successful step after the last error), what no_injection vs no_pii scan by default and how `where` overrides it, and how error state is detected. It does not cover `format` behavior or the 30-assertion cap, leaving a small gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ("Run declarative assertions on an agent trace") and enumerates the trace formats it accepts (OpenAI tool calls, Anthropic tool_use, LangChain run trees, ReAct logs). The assertion catalogue and the PASS/FAIL/INCONCLUSIVE verdict make it clearly distinguishable from siblings like function_call_validate or validate_mcp_response.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The assertion list implicitly tells an agent when this tool is applicable (post-hoc trace auditing), but there is no explicit when-to-use/when-not statement and no routing to alternatives such as prompt_injection_scan or function_call_validate, which overlap on no_injection and call-shape checks. Usage must be inferred from the capability list.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.