Server Quality Checklist

Tool Scores

• aletheia_interceptB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It says the tool evaluates and blocks, but does not state what the tool returns when a call is allowed or blocked, whether interception has side effects, how mandate overrides affect behavior, or what failure modes exist. The mention of 'block destructive, out-of-boundary, or compromised actions' is helpful but too abstract for an agent to predict the actual result of calling this tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that communicates the tool's role efficiently. It wastes little space, though phrases like 'sub-millisecond' and the signature identifiers are somewhat ornamental rather than essential. Overall, it is concise without being vague.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no annotations and no output schema, so the description is the sole source of behavioral and return-value context. It does not describe what the tool returns, how a blocked invocation is represented, when an override is permitted, or how this relates to the sibling mandate tools. For a safety-critical interception tool with nested parameters, this is a significant completeness gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the parameters are already documented by the schema. The description adds only general context that tool_name and tool_args represent a proposed invocation; it does not enrich the meaning of mandate_override or provide any additional format, constraints, or usage details beyond the schema. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as a gatekeeper that evaluates proposed tool invocations and blocks unsafe actions before execution. It states the verb ('evaluates'/'block'), the resource ('any proposed tool invocation'), and the 'universal' scope, which distinguishes it from sibling tools like aletheia_safe_bash and aletheia_safe_sql. The internal signature labels 'Signature S3' and 'S2b' add jargon, but the core purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Evaluates any proposed tool invocation ... before execution' clearly signals when the tool should be used: as a pre-execution safety check for any tool call. The 'universal' qualifier implies no exclusions. However, it does not explicitly contrast itself with sibling tools or describe when to bypass interception, so alternatives are not directly addressed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• aletheia_get_mandateA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral disclosure burden. The verb 'Retrieve' implies a read-only, non-destructive operation, and the description names the returned content areas (mandate, permissions, boundary constraints). However, it does not disclose possible error conditions, whether a mandate must already exist, or any return format details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no wasted words. It states the action and the resource immediately, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter, no-output-schema tool, the description is nearly complete: it says what is retrieved and implies the result categories. It lacks minor context such as whether the mandate is always available and how the result relates to set_mandate, but these are not needed to invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and the schema coverage is 100%, so parameter documentation is not a concern. The description adds meaning by clarifying what the empty invocation returns, which is exactly what an agent needs for a no-argument retrieval tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Retrieve') and identifies a clear resource: the current operational safety mandate, permissions, and active boundary constraints. It is readily distinguishable from sibling tools like aletheia_set_mandate and aletheia_get_telemetry, though it does not explicitly name them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no guidance on when to use this tool versus alternatives. It does not mention that aletheia_set_mandate should be used for modifying the mandate, nor does it clarify whether this is a prerequisite or companion call to other safety tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• aletheia_safe_sqlA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It does disclose that the tool can audit or execute and detects specific unsafe SQL patterns, which is useful, but it does not say what happens when an unsafe query is detected (block, warn, audit report), whether execution can mutate data, or what the response looks like.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tight sentences, front-loaded with the core purpose and then specific safety detections. Every sentence adds information, with no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Although the parameter schema is fully described, there is no output schema and the description does not clarify the ambiguity between 'audit' and 'execute' modes, the default behavior, or the result format. Since this tool can have side effects (executing a query), an agent lacks enough context to know exactly what will happen when it invokes it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both 'query' and 'database_type' documented. The description adds no additional meaning beyond restating that this is a database query, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb phrase ('Audit or execute a database query') and resource type (SQL), then names the concrete safety checks it performs (destructive DDL, unbounded DML, privilege tampering). This clearly distinguishes it from sibling aletheia tools that target mandates, bash, telemetry, and interception.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for safely auditing or executing SQL queries but does not explicitly state when to prefer this tool over an alternative or mention any exclusion criteria. An agent can infer the domain from the name and context, but there is no comparative timing or selection guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• aletheia_get_telemetryA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of explaining behavior. It does disclose the main response contents and 'Retrieve' implies a read-only operation, but it does not explicitly state whether there are side effects, auth requirements, rate limits, or how failures surface. The mention of latency percentiles under 1ms is also slightly ambiguous.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one front-loaded sentence that uses a colon-delimited list to pack in the exact payload of metrics. There is no filler, and the structure makes the return contents easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the headline telemetry metrics well but omits any mention of audit records, even though the auditLogLimit parameter indicates the response includes recent audit records. Since there is no output schema, the description should have explained this component to be fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the single optional parameter auditLogLimit is already fully documented in the schema with its default value and purpose. The description adds nothing about this parameter, but it does not need to; a baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and object ('Retrieve runtime safety telemetry') and then enumerates the exact metrics returned, so there is no ambiguity about what the tool does. Its focus on telemetry also clearly separates it from sibling tools like aletheia_get_mandate or aletheia_intercept.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'runtime safety telemetry' implies a monitoring/observability use case, and the sibling names make alternatives visible, but the description never explicitly states when to choose this tool over aletheia_get_mandate or aletheia_intercept, nor does it mention when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• aletheia_set_mandateA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations present, the description carries the full disclosure burden. It transparently reveals that loosening restrictions requires operatorSecret and that the tool declares scope boundaries. It does not, however, disclose side effects such as overwriting an existing mandate, session persistence, or what happens if an even stricter mandate is already in effect.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three purposeful sentences with zero filler. The main action is front-loaded, the second sentence summarizes the parameter landscape, and the third captures a critical authorization constraint. It is efficient and well-ordered for an agent scanning multiple tool definitions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with ten parameters, no annotations, and no output schema, the description provides a solid high-level purpose and the key auth constraint. However, it is silent on what the tool returns (confirmation vs. full mandate object), whether the mandate persists for the whole session, and how conflicts like allowedTools overlapping disallowedTools are resolved. These gaps matter for a tool that fundamentally governs agent behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of parameters with descriptions, so the baseline is 3. The description adds modest value by grouping parameters into conceptual categories (boundaries, allowed tools, directories, write permissions, risk tolerance) and by linking operatorSecret to the loosening case, but it does not deepen the semantics of any individual parameter beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb-resource pair ('Establish or update the active operational safety mandate') and immediately enumerates what the mandate declares: permissible boundaries, allowed tools, filesystem directories, write permissions, and risk tolerance. This clearly distinguishes it from sibling tools like aletheia_get_mandate (retrieval) and aletheia_safe_bash/aletheia_safe_sql (execution).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description conveys that this tool is for establishing or updating the session mandate and notes the operatorSecret requirement for loosening restrictions. However, it does not explicitly mention when not to use it or point to aletheia_get_mandate as the alternative for reading the current mandate, leaving usage routing implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• aletheia_safe_bashA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must explain behavior, and it does: it lists several categories of commands that will be blocked (rm -rf, formatting, fork bombs, credential harvesting, privilege escalation, unauthorized network egress). It could add what happens when a command is blocked or what the shell/environment looks like, but the core safety behavior is disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the identifying verb and resource. The phrase 'inline sub-millisecond Signature S3 scope creep protection' is somewhat jargon-heavy, but the rest of the sentence is dense and useful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple three-parameter executor, the schema plus the safety block list cover most invocation needs. However, there is no output schema and no description of return/error behavior when a command is blocked, which is a meaningful gap for a shell-execution tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the schema already documents command, cwd, and timeoutMs clearly. The description adds no parameter-level detail beyond restating that commands are safely evaluated, so the baseline score applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Opens with a specific verb and resource ('Execute a shell/bash command') and goes on to list the protected operations, so an agent immediately knows what the tool does. The 'bash' focus differentiates it from clearly distinct siblings such as aletheia_safe_sql.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes the tool's intended use obvious: run shell/bash commands under a safety wrapper. It does not explicitly name alternatives or state when not to use it, but the bash-versus-SQL distinction among siblings plus the explicit safety scope gives clear context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.