Server Quality Checklist

Tool Scores

• guard_contentA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full transparency burden. It discloses the return format (JSON with decision, findings, pathViolation) and behavioral expectations for blocked writes. It includes an example of protected paths (.git/hooks), adding meaningful context beyond schema. It doesn't elaborate on 'warn' semantics but is otherwise transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: three sentences covering purpose, return structure, and usage guidance. It is front-loaded with the primary action and resource, and every sentence earns its place with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a moderate-complexity tool with no output schema, the description is sufficiently complete. It explains what the tool does, what it returns (decision, findings, pathViolation), and how to act on the result ('On block, do not write'). It lacks full detail on 'warn' and 'allow' handling, but an agent can infer safe usage from the provided information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides detailed descriptions for both parameters: content ('Full proposed content — the new file content, or the replacement string for an edit') and file_path ('Path of the file about to be written'). The tool description adds little parameter-specific meaning beyond the schema, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Check a proposed file write against the Umbra guard engine BEFORE writing.' The specific verb 'Check' and resource 'proposed file write' are precise. It differentiates from siblings (scan_repo, get_score) by focusing on pre-write validation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage context with 'BEFORE writing' and a direct instruction: 'On block, do not write — fix the content and re-check.' This tells the agent when and how to use it, though it doesn't explicitly mention alternatives or when not to use it relative to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_scoreA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the operation is static, safe (no network, no sandbox), and limited to SAFE and CLEAN axes. This is strong behavioral context for a read-only scoring tool, though it doesn't detail error behavior or computation specifics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, entirely purposeful. The first sentence states the core function and constraints; the second references the richer alternative. No filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (one param, no output schema), and the description covers its purpose, scope, and limitations. It lacks details like whether local paths are accepted or what happens on error, but given the low complexity, it's adequately complete. The explicit sibling pointer further completes the context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the only parameter 'path', whose description already states '(required) Path to the repository to score'. The tool description adds no additional parameter meaning beyond pointing to 'repository', which matches the schema. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool computes a 'Fast static Umbra Trust Score (0-100) for a repository' with a specific scope ('SAFE and CLEAN axes only'). It explicitly differentiates itself from the sibling tool scan_repo by noting that scan_repo provides the full report. This is a specific verb+resource+scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'Fast static... no network, no sandbox' implies quick local assessment. It also names the alternative: 'Use scan_repo for the full report with findings.' This clearly distinguishes from siblings and tells the agent which tool to choose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_repoA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses key behaviors: static SAFE/CLEAN findings, deep mode verifying RUNS and HONEST in a Docker sandbox, and the threshold interpretation. It does not mention permissions or error cases, but it offers substantial behavioral context beyond a simple definition.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose and enriched with usage context. Every sentence earns its place; no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (deep mode, Docker sandbox, scoring threshold) and lack of output schema, the description adequately explains what the tool returns and when to use it. It could elaborate on the meaning of SAFE/CLEAN/RUNS/HONEST, but those are likely domain terms. It is sufficiently complete for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (both parameters have descriptions), so baseline is 3. The description adds meaningful detail for 'deep' (Docker sandbox, slower, needs Docker) beyond the schema's simple description. It also clarifies that 'path' is required despite the schema's empty required array, which is valuable semantic information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool scans a repository and returns a full Umbra Trust Score report as JSON, with a specific verb ('scan') and resource ('repository'). It distinguishes itself from siblings (get_score, guard_content) by focusing on the scanning action and producing a complete report.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit usage guidance is provided: 'Call before declaring a task done — a score below 50 means the work is not done.' This gives a clear conditional trigger for when to use the tool, which effectively differentiates it from alternatives in the workflow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.