Server Quality Checklist

Tool Scores

• prepare_code_reviewA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that files stay local and selection criteria, but no annotations are present. Lacks details on side effects, idempotency, or whether any modifications occur. Basic transparency but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences that efficiently convey purpose, selection logic, and workflow. Front-loaded with action. Minor redundancy could be trimmed, but overall well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Explains the tool's role in upgrading dimensions from inferred to verified, but lacks detail on the output format (file contents and checklist). No output schema, so description should compensate. Adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for path and max_files. The description does not add extra semantic information beyond the schema, meeting the baseline but not exceeding it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it selects highest-signal files (security-relevant, entry points, large files) and returns their contents with a review checklist. Distinguishes from sibling tools like scan_project and score_build_readiness by its specific role in the pipeline.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context: use this to gather files for review, then the model produces an assessment passed to score_build_readiness. Implies when-not by describing the workflow, though no explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_projectA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full weight. It discloses the read-only nature ('never your source') and safety features (redacted secrets scan). No contradictions or undisclosed side effects are apparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences: first details output, second clarifies safety and usage order. No wasted words; every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a single optional parameter and no output schema, the description sufficiently explains the return contents. A minor improvement could specify the output format (e.g., JSON object), but current detail is adequate for the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only one parameter 'path' with schema description already covering its purpose and default. The tool description adds no further semantic detail beyond what the schema provides. Baseline 3 due to 100% schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it reads a local project and returns build readiness signals, listing specific signals like file counts, languages, test files, CI config, lockfile, docs, and secrets scan. Instructs 'Run this first,' distinguishing it from sibling tools that perform different actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly tells the agent to 'Run this first,' establishing it as a preliminary step. Implicitly advises against expecting source code or secrets in the output. No alternatives or exclusions needed due to the tool's single-purpose nature.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_dependenciesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint and openWorldHint. The description adds specific behavioral traits: queries real, current CVE data, and sends only package names/versions (privacy guarantee). No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (three sentences) with front-loaded action verb 'Query'. Every sentence adds value: purpose, method, privacy, data freshness, and usage guidance. No redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one optional parameter, no output schema), the description fully covers what the agent needs: what it does, how it works, privacy, and how to use results. No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the description does not add further semantics for the single parameter 'path'. The schema already documents its purpose and default behavior, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Query OSV.dev'), the resource ('locked dependencies'), and the specificity ('using exact versions from package-lock.json'). It distinguishes itself from sibling tools like 'scan_project' by focusing on CVE vulnerabilities in locked dependencies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (to check for CVEs in lockfile) and provides guidance on privacy ('Only package names and versions are sent'). It suggests using results in security assessment. However, it lacks explicit when-not-to-use or direct alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• score_build_readinessA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that the scan runs automatically and writes a report to .obra/ (via write_report parameter). It also explains how supplying test numbers and qualitative assessments affects evidence grading. No destructive actions are implied; the description is sufficiently transparent for a scoring tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The main description is two sentences with no waste, clearly stating the output and a critical workflow hint. Parameter descriptions are not repeated in the main text but are part of the schema, keeping the overall description concise and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (7 parameters, nested objects, no output schema), the description provides enough context to use it correctly: it explains the output format (score, breakdown, risks), the automatic scan, and how to leverage prior tool outputs. However, it could be slightly more explicit about the return structure (e.g., JSON object) but overall is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds substantial value beyond the schema: it explains that stage defaults to a detected guess, that qualitative input from prepare_code_review upgrades dimensions to grade A, and that test numbers come from a prior run_tests call. Each parameter's purpose and cross-tool dependency are clarified.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool produces an Obra CTO Score (0-100) with a per-dimension breakdown, evidence grades, and Top Risks register. This specific verb+resource+output distinguishes it from sibling tools like scan_project and run_tests, which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises the agent to pass test numbers from a prior run_tests call to upgrade reliability to grade-A evidence. It also mentions that the qualitative input can come from prepare_code_review, implying a workflow. However, it does not explicitly state when not to use this tool or list alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_testsA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that the tool executes the project's own test command, which carries risk. Explains that the host asks before execution. Annotations set readOnlyHint=false, which is consistent, but the description adds important context about trust and sandboxing that goes beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four sentences, each adding value. Front-loaded with the core action and parsing behavior. No redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (executes commands, parses output, security implications) and absence of an output schema, the description is complete. It explains the return value (pass/fail counts) and its intended downstream use, leaving no significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with detailed parameter descriptions. The description does not add new parameter meaning beyond the schema but integrates parameters into the narrative. Baseline of 3 is appropriate as schema fully documents parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs the test suite and parses pass/fail counts. It distinguishes itself from siblings like score_build_readiness by stating the output should be passed to that tool, and from others like scan_project which are unrelated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says when to use (for reliability evidence) and when not (only approve for trusted projects). Advises on risk acknowledgment and mentions the host asks before running. Directs to pass results to score_build_readiness, providing clear usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.