AgentLeak

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It adds genuine framing by enumerating the many leakage channels it inspects, which signals breadth (a behavioral trait not visible elsewhere). However, it does not disclose whether the operation is read-only, what cost/latency to expect, or what permissions are needed to read the trace.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the core action and scope, followed by the trigger condition. Every clause earns its place with no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be explained. But for a tool whose sole input is a loose, undocumented nested trace object, the description omits any guidance on input format, leaving a real gap given the complexity of the analysis being requested.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single 'trace' parameter has 0% schema description coverage and is typed as a fully open object (additionalProperties: true), so the schema explains nothing. The description never describes what a trace must contain or its expected shape, leaving the agent unable to construct the argument from either source.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Score') and resource ('one agent run for privacy leakage') and enumerates the channels examined (tool calls/responses, shared memory, inter-agent messages, logs, files, final answer). It implicitly differentiates from siblings like privacy_scan_code and privacy_redact by scoping to a captured run, but never names them, so an agent must infer the boundary.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Use when you have captured a run and want to know whether data escaped on a channel nobody reads' gives a clear trigger condition tied to having a trace in hand. There are no explicit exclusions or named alternatives to distinguish it from privacy_preflight or privacy_scan_code.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It usefully discloses incremental behavior (NEW findings since the last check) and optional trace scoring across eight channels, but it does not state whether the tool is read-only, what it accesses, or what permissions or state it requires.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, front-loaded with the main action and followed by the usage condition and optional trace behavior. There is little waste and the ordering is agent-friendly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers what the tool does, when to call it, and the optional trace mode, and an output schema exists for return values. However, with no annotations and 0% parameter schema coverage, it leaves gaps around the path parameter and the tool's safety/state model.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate for two undocumented parameters. It partially explains the trace parameter as a run trace used for eight-channel scoring, but the path parameter is only implied by 'this project' and is not described at all.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: checking a project for privacy problems before shipping, and reporting findings that are NEW since the last check. The scope is clear, but it does not explicitly distinguish itself from sibling tools such as privacy_scan_code or privacy_check_trace.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a concrete usage trigger: call after writing or changing code that touches user data, secrets, logging, or outbound calls. It also explains the optional trace use case, but it does not name alternatives or say when to prefer them over this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses the available output styles, but not whether the operation is a pure transformation returning new text, whether the original is mutated, whether redaction is reversible, or whether it can fail on unsupported input. Useful but incomplete for a tool with zero annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, both earning their place: the core action leads, the usage trigger follows, and the style enumeration closes compactly. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a low-complexity two-parameter utility with an output schema (so return values need not be described), the description covers purpose, usage context, and style options. The remaining gaps about mutation/reversibility are minor given the tool's simplicity, though the absence of annotations means more disclosure would help.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the style parameter has no enum, so the description's list 'placeholder, masked, asterisk, category, hash, remove' does add the set of valid values. However, it never ties the list to the 'style' parameter by name and explains none of the styles' semantics, so it only partially compensates for the coverage gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

'Remove sensitive values from text' states a specific verb and resource, and it is clearly distinct from the sibling scan/detect tools (privacy_preflight, privacy_scan_code, privacy_check_trace) because it performs the redaction rather than the detection. It does not name a sibling explicitly, but the action framing differentiates it well enough.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a concrete trigger context: 'Use before writing something to a log, an issue, a commit message, or any other place a secret should not end up.' That tells the agent when to reach for it, but it never states when NOT to use it or which sibling to use instead (e.g. when to scan rather than redact).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It describes what is detected but not how the tool behaves: nothing about whether it is read-only, what permissions or network access it needs, whether it writes a report, or any limits. For a no-annotation tool this is a real gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, no filler, and the capability statement is front-loaded ahead of the usage condition. Every clause earns its place by naming a distinct detection category.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be described. However, with no annotations and no path documentation, the description leaves gaps around invocation and behavior that an agent would need before calling it confidently.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the single 'path' parameter (default ".") is never explained in the description. The description says "scan source code" but does not clarify whether path is a file, directory, or glob, so it fails to compensate for the empty schema documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific verb (Scan) and resource (source code) and enumerates the three detection categories: hardcoded secrets, PII in logs, third-party leakage. It does not name or contrast with any of the siblings (privacy_preflight, privacy_check_trace, privacy_redact), so sibling differentiation is only implied.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

"Use when you only need the current state and not the change since last time" gives an explicit selection condition and implies a diff-based alternative exists. It stops short of naming that alternative tool, so the agent must still infer which sibling handles the change-since-last-time case.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.