Server Quality Checklist

Tool Scores

• nrf_readA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool reads text files from a specific repo, but lacks details on permissions, rate limits, error handling, or output format. While it adds context about supported file types, it doesn't fully cover behavioral traits needed for a read operation without annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by supported file types and usage guidelines with examples. Every sentence adds value without redundancy, making it efficient and well-structured for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 1 parameter with full schema coverage and no output schema, the description is adequate but has gaps. It explains what the tool does and when to use it, but lacks details on return values, error cases, or behavioral constraints, which are important for a read tool without annotations or output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the parameter 'path' well-documented in the schema. The description adds minimal value beyond the schema by providing examples of paths, but doesn't explain semantics like path format constraints or edge cases. Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Read') and resource ('contents of a file from the nRF Connect SDK repo'), specifying it works for various text file types. It explicitly distinguishes from sibling nrf_list by stating 'Use nrf_list to discover paths first,' making the purpose specific and differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('Use nrf_list to discover paths first') and implies alternatives by mentioning sibling tools (nrf_list for discovery, nrf_search likely for searching). It sets clear context for file reading versus path discovery.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• nrf_listA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: the tool lists directory contents, returns directories first then files, provides full paths, and those paths can be used with nrf_read. It doesn't mention error conditions, rate limits, or authentication needs, but covers the core operational behavior well.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized. The first sentence states the core purpose, followed by a practical 'Useful starting paths' section that provides immediate utility, and ends with important behavioral details about the return format. Every sentence earns its place with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with no output schema, the description provides good completeness. It explains what the tool does, how to use it with examples, and describes the return format. The main gap is the lack of error handling information, but given the tool's simplicity and the provided context, it's reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the single 'path' parameter. The description adds some value by providing concrete examples of paths (e.g., 'samples/bluetooth') in the 'Useful starting paths' section, but doesn't add significant semantic meaning beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('List the contents of a directory') and resource ('nRF Connect SDK repo'), distinguishing it from sibling tools nrf_read (likely reads file contents) and nrf_search (likely searches within files). The phrase 'Returns dirs first, then files, with full paths you can pass to nrf_read' further clarifies its role in the toolchain.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (listing directory contents in a specific repo) and includes a helpful list of 'Useful starting paths' to guide initial usage. However, it does not explicitly state when NOT to use it or mention alternatives like nrf_search for different operations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• nrf_searchA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key traits: it's a read-only search operation (implied by 'search'), returns up to 20 file paths, requires authentication for reliability, and is rate-limited without it. However, it lacks details on error handling or exact response format, which could be beneficial for completeness.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded, starting with the core purpose. Each sentence adds essential information: search capabilities, qualifier examples, return details, and authentication requirements. There is no wasted text, and the bulleted examples are concise and illustrative, making it easy to scan and understand.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (search with qualifiers, authentication needs) and lack of annotations and output schema, the description is largely complete. It covers purpose, usage, parameters, and behavioral traits. However, it doesn't specify the exact output format (e.g., JSON structure) or error scenarios, which could aid the agent in handling responses, preventing a perfect score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the baseline is 3. The description adds significant value beyond the schema by providing examples of GitHub search qualifiers (e.g., 'path:doc/nrf', 'extension:c'), which clarify how to construct the query parameter effectively. This enhances understanding but doesn't fully cover all possible qualifiers, keeping it from a perfect score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search for code or documentation across the nRF Connect SDK repo using GitHub code search.' It specifies the verb ('search'), resource ('code or documentation'), and scope ('nRF Connect SDK repo'), and distinguishes it from sibling tools by mentioning 'Use nrf_read to fetch the content,' indicating it returns file paths rather than content.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool versus alternatives. It states 'Returns matching file paths (up to 20). Use nrf_read to fetch the content,' clearly differentiating from nrf_read for content retrieval. It also specifies 'Requires GITHUB_TOKEN for reliable results (unauthenticated search is heavily rate-limited),' indicating prerequisites and performance considerations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.