Skip to main content
Glama

Detect trace tool-policy violations

detect_trace_tool_policy

Analyze an agent trace for the Gray Swan Wave 16 class: untrusted retrieved/tool output causing a tool call outside the user-declared per-turn allowlist. Returns trace counts, unauthorized tool-call evidence, canon mapping VC-AI-TOOL-0001, and claim-boundary guardrails. Backed by T-IB-25/T-IB-29/T-IB-36.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
traceNoSingle agent trace with user_prompt, allowed_tools, and events[].
tracesNoOptional batch of agent traces.
targetNameNoOptional display name for the assessed target.

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
modeYes
tracesNo
canonIdNo
summaryYes
verdictYes
probabilityNo
theoremRefsNo
claimBoundaryNo
reviewPriorityYes
recommendedActionYes
explainabilityTokenNo
customerSystemProvedNo

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It discloses the tool's return values (trace counts, unauthorized tool-call evidence, canon mapping, claim-boundary guardrails) and backing policy references, which gives the agent a clear sense of what to expect. It does not explicitly state side-effect or permission requirements, but 'Analyze' strongly implies read-only behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three sentences, front-loaded with the primary purpose, and includes return-value details and backing references. It is efficient but the backing reference (T-IB-25/T-IB-29/T-IB-36) is cryptic and may not add immediate value for an AI agent, preventing a perfect score.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with an output schema and a well-described input schema, the description covers the core purpose, return categories, and backing context. The main gap is the absence of explicit usage guidelines relative to the sibling detect_injection, but overall it is sufficiently complete for the agent to invoke correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% description coverage for all three parameters, so the baseline is 3. The tool description adds no additional semantic detail about the parameters beyond what the schema already provides, so no extra credit is warranted.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the tool's function with a specific verb ('Analyze') and resource ('agent trace'), and names the exact violation class ('untrusted retrieved/tool output causing a tool call outside the user-declared per-turn allowlist'). This level of specificity distinguishes it from the sibling detect_injection.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the tool is for analyzing Gray Swan Wave 16 tool-policy violations, but does not explicitly state when to use it versus detect_injection or when not to use it. There is no explicit alternative guidance or exclusion criteria, only an implied context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A4.2/5.0
Disambiguation5/5

The two tools target completely distinct aspects: one screens untrusted input for injection patterns, while the other analyzes agent traces for policy violations. There is no overlap in their purpose or output, making misselection unlikely.

Naming Consistency5/5

Both tools follow a consistent 'detect_' + noun phrase naming convention (detect_injection, detect_trace_tool_policy). The pattern is clear and predictable, and there is no mixing of styles.

Tool Count4/5

With only two tools, the set is slightly under the typical 3-15 range, but it feels appropriately scoped for a specialized security detector. Each tool covers a distinct and essential part of the domain, so the small count is justified.

Completeness4/5

The two tools cover the primary detection needs: input screening and trace analysis. Minor gaps exist (e.g., no explicit tool for sanitization or for analyzing output exfiltration), but the core injection-detection workflows are well covered with no dead ends.