Server Quality Checklist

Tool Scores

• umbra_verify_proposalA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (readOnlyHint=false, destructiveHint=false), the description reveals key behavioral traits: it retains only a test that genuinely fails or errors, rejects malformed concrete_input or expected_property fields with correction reasons, and discards passing proposals. This substantive detail helps the agent anticipate outcomes and side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is five sentences long, each contributing distinct information: primary action, input source, retention rule, rejection behavior, and follow-up step. There is no fluff or redundancy, making it compact and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's purpose, behavioral rules, prerequisites, and successor action, offering a solid operational flow for a complex tool with a nested object. However, it omits details about return values or error response structure (no output schema exists), leaving some gaps for agents needing precise interface knowledge.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at 0%, the description must compensate for parameter meanings, but it only briefly mentions proof_index and the malformed fields. It does not explain the expected format of concrete_input, expected_property, or the role of module_path, tests_path, and plain_explanation beyond the schema's bare type declarations.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Run one user-approved Codex proposal through Umbra's execution gate.' This clearly distinguishes the tool from siblings like umbra_audit and umbra_write_report by focusing on executing a proposal rather than auditing or reporting.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides workflow guidance by instructing to use proof_index from umbra_audit's structured proof_requests and to re-run umbra_audit after a test write. It establishes clear context for when to use the tool, though it does not explicitly state exclusions or alternatives beyond the implicit sequencing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• umbra_auditA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations, the description discloses important behaviors: deterministic gap detection, an execution gate that keeps only runnable tests that genuinely fail or error, explicit statement that it 'never edits the test file,' and that the OpenAI backend reads only a locally configured key after confirmation. This provides rich context about side effects and security.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is lengthy but well-structured and front-loaded with the core purpose. Each sentence contributes useful details (capabilities, workflow, constraints). It could be slightly trimmed, but the overall structure is appropriate for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Even though there is no output schema, the description explains what the tool returns (coverage report, Silence index, proof requests, HTML and Markdown report artifacts) and describes execution side effects and backend behavior. It is complete for a complex tool with no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the schema already explains all six parameters. The tool description adds workflow context (e.g., using only module_path and tests_path for report generation) but does not significantly enrich the meaning of individual parameters beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific action: 'Locate test-suite blind spots' and names the exact languages and gap types (unexercised branches, untested input regions, unasserted return contracts, unasserted exception paths). This clearly distinguishes the tool from siblings like umbra_verify_proposal, umbra_write_report, and umbra_glossary.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context and a workflow: 'After it returns, write and verify a failing-test proof for each returned request, then generate both reports with module_path and tests_path only.' It also provides safety constraints like 'never pass a key as tool input.' However, it does not explicitly name alternative tools for when-not-to-use scenarios, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• umbra_write_reportA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations, the description discloses determinism ('deterministic action'), privacy ('sends no data to any API'), and what it does not do ('does not run proof generation'). It also explains the server's automatic proof inclusion behavior, adding rich context about side effects and state.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences, each earning its place: the first states the main action and required params, the second explains the default proof source, and the third discloses deterministic and privacy behavior. No fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, usage timing, parameter override, and behavioral traits. It does not mention return values, but since the tool writes files and has no output schema, this is a minor gap. Overall, it is sufficient for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50%, and the description adds context for retained_proofs (explicit override behavior) but does not explain what module_path and tests_path are or their expected formats. It clarifies the usage pattern ('with module_path and tests_path only') but leaves the required parameter semantics to be inferred from names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'write both Umbra report artifacts locally with module_path and tests_path only.' It specifies the verb, resource, and scope, and distinguishes from siblings by referencing umbra_verify_proposal as the source of proofs. The phrase 'After the proof loop' also situates it in the workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear temporal context ('After the proof loop') and indicates that the primary usage is with module_path and tests_path only, while retained_proofs is an optional override. However, it does not explicitly name alternatives or specify when not to use the tool, relying on implicit workflow positioning.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• umbra_glossaryA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (readOnly, non-destructive, closed-world), the description reveals that the tool is 'deterministic local' and 'makes no model call and sends no data anywhere,' adding privacy guarantees. It also commits to 'never guessed' for unknown terms, setting accuracy expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, each earning its place: the function, the usage trigger, and behavioral safeguards. No redundant phrasing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter glossary tool with no output schema, the description covers the two operation modes, usage scenario, and behavioral guarantees. It is complete enough for an agent to select and invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already describes the optional `term` parameter and its default listing behavior (100% coverage), so the baseline is 3. The description adds context that the term is 'precise, non-standard' and that unknown terms are handled plainly, enhancing the semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Return Umbra's authoritative fixed definition for one precise, non-standard Umbra term, or list all defined terms with short glosses,' which clearly identifies the verb and resource. It also distinguishes itself from sibling tools (audit, verify, write) by focusing exclusively on term definitions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs 'Use this whenever a user asks what an Umbra term means,' providing a clear trigger. It does not explicitly name alternative tools or when not to use it, but the context is unambiguous given the sibling tools' distinct purposes.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.