Skip to main content
Glama

IA-QA Security — 30 tools

validate_agent_trajectory

Read-onlyIdempotent

Run declarative assertions on an agent trace (OpenAI tool-call messages, Anthropic tool_use/tool_result blocks, LangChain run trees, or plain text ReAct logs). No LLM call — deterministic. Assertion types: order (tool A before B), must_call, must_not_call, max_calls, min_calls, no_error, recovery (agent completes a successful step after its last error). A step counts as errored when the trace says so — is_error/isError, status/state in {error,failed,exception,…}, an error field, a JSON body with error/success:false — or when its text payload STARTS with an error marker (Error:, Traceback, TypeError:). Model prose is never scanned for keywords, and every errored step reports error_signal naming what flagged it. Returns per-assertion PASS/FAIL, parsed steps, warnings (a trace parsing to 0 steps is flagged — assertions passing on emptiness prove nothing), and an overall verdict. Use this to gate CI/CD on agent behavior correctness.

Input Schema

NameRequiredDescriptionDefault
traceYesAgent execution trace as JSON (OpenAI messages array, LangChain run tree) or plain text log (Thought/Action/Observation format).
formatNoTrace format. auto (default) detects automatically.
assertionsYesList of assertions to validate against the trace.

Output Schema

NameRequiredDescriptionDefault
stepsNo
resultsNo
summaryNo
verdictNo
warningsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly, idempotent, non-destructive), yet the description adds real behavior beyond them: deterministic/no-LLM, the precise error-detection heuristic (is_error fields, error markers, leading-token markers), and the guarantee that model prose is never keyword-scanned. It omits auth/rate-limit context but materially enriches the behavioral picture.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the mechanism and the assertion catalogue before the usage recommendation, and is dense with content. It is long and the 'Returns...' clause partially duplicates the output schema, which keeps it off a 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, return values needn't be re-explained, and the description still adds the empty-trace warning and error_signal behavior. Format auto-detection and assertion semantics are covered, leaving it complete for a 3-param tool, though the CI/CD integration angle is asserted rather than explained.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents format, assertions, and every nested field. The description restates assertion types and semantics (e.g., recovery) but adds little syntax detail beyond what the schema/enum already provide, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Run declarative assertions on an agent trace') and immediately scopes the input formats it accepts. It is unmistakably distinct from siblings like function_call_validate, guardrail_test, or run_semantic_tests because it names its exact mechanism (deterministic assertions, no LLM call).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Ends with an explicit use case — 'gate CI/CD on agent behavior correctness' — and the 'No LLM call — deterministic' line signals when to prefer this over LLM-based evaluators. It does not name a specific alternative or a when-not condition, so it falls short of the top anchor.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.