Server Quality Checklist

Tool Scores

• setup_dbC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that it fetches from GitHub and stores in a local DB, but it omits side effects like network usage, potential overwrites, auth requirements, or the default behavior of skipping already-indexed repos unless force is set. This is insufficient for a setup tool with significant external effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and front-loaded with the main verb and resource. However, the phrase 'Initialize the local SQLite FTS5 index and index repositories' is slightly redundant by using 'index' twice, and the second sentence could be merged. Still, it is efficient overall.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a setup tool with side effects and no output schema, the description lacks important context: it doesn't mention required authentication (likely via login), the duration or impact of re-indexing, or the default skip-if-already-indexed behavior. Given the sibling tools, this could cause an agent to misuse the tool or miss prerequisites.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, as both 'force' and 'repos' have descriptions in the schema. The description adds no parameter-level detail beyond what the schema already provides, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Initialize') and names the resource ('local SQLite FTS5 index') and the action ('index repositories'), making the core purpose clear. It distinguishes itself from sibling query/search tools conceptually, though it does not explicitly name alternatives or contrast with them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no guidance on when to use this tool versus alternatives like login, refresh_auth, or query. It does not state prerequisites (e.g., must log in first) or when to use force/repos parameters. Usage context is entirely implied by the word 'Initialize.'

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_source_filesB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility for behavioral disclosure. It only says 'List files' without mentioning whether this is read-only, requires authentication, returns directories vs files, pagination, or any side effects. This is a significant gap even for a simple read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that conveys the essential action and optional filter without any wasted words. It is perfectly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with only two simple parameters and no output schema, the description is minimally adequate. However, it does not clarify whether it returns file paths, metadata, or directory contents, and the name suggests 'source files' while the description says 'files'. There is room for context about output structure or limitations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides full descriptions for both parameters (repo as 'owner/repo' and path as 'directory prefix filter'). The description's mention of 'directory prefix' is redundant with the schema, adding no extra meaning. Since schema coverage is 100%, the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists files in a repository with an optional directory prefix filter. It uses a specific verb and resource, and the optional filter adds clarity. It doesn't explicitly distinguish from sibling tools like list_sources or get_file_tree, but the core purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives such as list_sources, get_file_tree, or search_file. There are no explicit usage contexts, exclusions, or mention of alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_fileB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It accurately conveys a read operation ('Get the content') and optional line range, but it does not disclose potential behaviors like what happens on file-not-found, how large files are handled, or the exact meaning of line range inclusivity (which is only in the schema). Nothing contradicts annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence, front-loaded with the core action and resource. Every word contributes value, and there is no redundant or fluffy content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read tool, the description is minimally adequate: it states the purpose and optional line range. However, it lacks explicit mention of the return format (e.g., raw text vs JSON) and any error/edge-case behavior, which would be helpful given there is no output schema or annotations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all four parameters. The description adds no extra meaning beyond saying the line range is optional, which is already inferable from the required fields. This is baseline 3 since the description does not hamper parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (get content) and the resource (a file from a repository), and mentions an optional line range. It distinguishes the tool from siblings like get_file_tree (which returns tree structure) and search_file/search_code (which search for matches), though it does not explicitly contrast it with those alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives such as search_file or list_source_files. It only states what the tool does, leaving the agent to infer the appropriate use case without explicit when/when-not guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• select_repositoriesA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It adds useful context by mentioning 'retrieval evidence scores' and the restricted candidate list, but it does not describe the output format, potential no-match behavior, or whether any side effects occur. This is a moderate disclosure for a selection tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is front-loaded with the action and resource, and every phrase adds value: the candidate list and scoring method are both mentioned. This is appropriately concise with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the essential purpose and scope but omits practical details like the return format or behavior when no repositories match. Since there is no output schema and no annotations, these details would be helpful for an agent to know what to expect, leaving the description moderately complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and every parameter has a description, so the baseline is 3. The tool description adds overall context but does not enrich individual parameter semantics beyond what the schema already provides, such as the meaning of max_repos or the optional repos subset.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'select' and names the resource ('repositories from REFERENCE_REPOS'), explicitly listing the allowed candidates (octocat/Hello-World, facebook/react). This clearly distinguishes it from sibling tools like search_code or search_implementation, which focus on searching within code rather than selecting from a fixed repository set.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for choosing repositories given a document/query, but it does not explicitly state when to use this tool versus alternatives or mention exclusions. No sibling tool is referenced as a fallback, so the guidance is implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_file_treeA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of transparency. It adds behavioral details such as the output being an ASCII tree and the optional subdirectory scoping, which go beyond the raw name. However, it does not explicitly state that it is a read-only operation, mention authentication, or describe behavior for invalid paths — though 'get' implies a read action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that immediately states the action and object, with no superfluous words. Every part earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter read tool with full schema coverage, the description is reasonably complete. It specifies the output format (ASCII tree), the target (repository), and the optional scope. Minor aspects like ordering, hidden-file inclusion, or invalid-path behavior are not covered, but the tool's simplicity keeps these from being critical gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides clear descriptions for both parameters (repo as owner/repo and path as subdirectory scope), giving 100% schema coverage. The description's phrase 'optionally scoped to a subdirectory' mirrors the path parameter's optionality but does not add substantive new meaning on top of the schema. A baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Get') and resource ('ASCII directory tree'), and clearly indicates the repository context and optional subdirectory scope. It distinguishes itself from sibling tools like get_file or search_code, which operate on file contents rather than the directory structure.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes the primary use case clear (obtaining a directory tree, optionally scoped to a subdirectory), but it does not explicitly mention when not to use it or point to alternatives such as list_source_files or search_file. There is enough context to infer usage, but no exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sourcesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must stand alone. It indicates a read-only listing operation but does not disclose auth requirements, network access, or side effects. For a list tool, the behavior is straightforward, but the description carries the full burden and remains minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One sentence, directly states the tool's purpose with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description names the output concept ('indexing status') but no output schema exists. It's sufficient for a simple list, though it omits details like whether pagination or filtering is needed. Given the low complexity, it's reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the schema is empty and fully covers semantics. The description adds no parameter-specific information, which is acceptable given the 4 baseline for zero-param tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a clear verb ('List') and identifies the resource ('configured reference repositories') and adds detail ('their indexing status'), distinguishing it from sibling file-level tools like list_source_files and get_file_tree.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this vs alternatives, but the phrase 'configured reference repositories' provides clear context. Absence of exclusions or alternative tool references lowers the score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refresh_authA
  4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It only says 'reloads' without detailing side effects (e.g., overwriting current tokens), whether the operation is idempotent, or failure modes if tokens are missing. This is minimal behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the core action and followed by actionable usage guidance. Every word earns its place with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with no output schema, the description adequately covers purpose and usage. It could add more about success/failure behavior, but it is sufficiently complete for typical use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline of 4 applies per the instructions. The schema correctly shows no properties, and the description naturally doesn't need to explain any parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('Reloads authentication tokens from disk'), identifying the resource (authentication tokens) and distinguishing it from siblings like login (which creates tokens) and query (which consumes them). The phrasing is specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to call ('after re-authenticating or if queries start failing'), providing clear context. However, it lacks explicit exclusions or named alternatives, though the tool's distinct purpose makes this less critical.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_codeA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses search modes, ranking algorithm, and return type, but omits details like authentication requirements, scope (all repos vs. selected), or error handling. This is useful but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the main purpose. Each phrase adds useful information (search engine, ranking, modes, return type), with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Without an output schema, the description offers a high-level summary of results but does not detail the result object structure. However, it covers the core search behavior, modes, and ranking, which is adequate for a search tool. Missing details like result fields are not critical given the schema's parameter coverage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value by specifying the substring mode's minimum 3-character requirement, which is not in the schema, and by explaining the return type (ranked results with snippets). This goes beyond the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('Search indexed code') with technical detail (FTS5, BM25 ranking). It distinguishes from siblings like search_file by specifying indexed code and search modes, making it unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for general code search but does not explicitly state when to use this tool over siblings like search_implementation or search_file. No exclusions or alternatives are mentioned, leaving the context implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_fileA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavior. It does reveal case-insensitivity and the return format (matching lines with context), but it omits details like error handling, read-only nature, or pagination. This is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, compact sentence that front-loads the main verb and resource. Every phrase adds meaning, with no filler or redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple file-search tool with four straightforward parameters and a clear result description, the description is largely complete. It doesn't cover edge cases like missing files or empty matches, but the basic behavior and output are clearly stated.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already explains all parameters. The description adds little beyond repeating 'case-insensitive substring' and implying context lines. Baseline 3 is appropriate because the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies a clear action ('Search within a specific file'), defines the pattern type ('case-insensitive substring match'), and states the output ('return matching lines with context'). This distinguishes it from sibling tools like search_code, which likely searches across multiple files.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'within a specific file' sets a clear context for use, implying this tool is for file-scoped searches rather than repo-wide searches. However, it does not explicitly mention when not to use it or name alternatives such as search_code, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• loginA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly discloses that a Chrome window will open, the user must complete Google sign-in, and tokens are saved automatically. This adds useful context beyond a simple 'login' and sets accurate expectations, though it could mention potential side effects like requiring interaction downtime.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the primary action, and contains no filler. Every sentence earns its place by explaining how authentication works and what the user should expect.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple login tool with no parameters or output schema, the description covers the key context: what happens (launches Chrome), what the user must do (complete Google login), and the outcome (tokens saved). It is complete enough for an agent to invoke correctly, though it could briefly mention that this is a one-time setup and that refresh_auth may be used later.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline score is 4 per the rubric. The description does not need to explain parameter semantics, and it appropriately focuses on the authentication process.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Authenticates with NotebookLM by launching Chrome for Google sign-in.' It uses a specific verb ('Authenticates') and resource ('NotebookLM'), and distinguishes itself from sibling refresh_auth by emphasizing initial login vs. saved tokens.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage (needed for authentication before using the product) and states that tokens are saved automatically, suggesting it is for initial login. However, it does not explicitly mention when not to use it or contrast with alternatives like refresh_auth, so guidance remains implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• queryA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses an authentication prerequisite and mandates verbatim presentation of the response, which is important behavioral guidance. It doesn't explicitly state read-only nature, but 'Ask a question' implies a safe query operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise, front-loaded sentences cover purpose, authentication prerequisite, and output handling. Every sentence provides necessary information with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter query tool with no output schema, the description is complete: it defines scope, prerequisite, and how to handle the response. No additional behavioral or context cues are needed for an agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already fully describes the sole parameter (`question`), and the description adds only marginal context like 'research question'. Since schema coverage is 100%, the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool asks a research question about Commonware via NotebookLM, with a specific verb and resource. It distinguishes itself from sibling tools like search_code or search_implementation by focusing on natural-language research questions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs the user to run `nlm login` first if not authenticated, giving clear prerequisite context. It does not explicitly contrast with alternatives, but the purpose and sibling set make when-to-use reasonably clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• factcheck_validateA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explains that the tool fetches code evidence and returns it for assessment, and it imposes a critical constraint on relying only on this tool's output. This goes beyond the schema and gives the AI a clear mental model of the tool's behavior, though it does not explicitly state non-destructiveness or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded. The first sentence states the core purpose, the second provides usage instructions, and the third delivers an essential warning. Every sentence earns its place with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, so the description must convey return behavior. It does so by stating that the tool returns code evidence alongside each claim for assessment. It also lists available repositories, which is critical for correct invocation. While it doesn't address edge cases like missing code, the core context needed for use is covered sufficiently.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with detailed descriptions for `items` and `document`. The description adds minimal semantic value beyond the schema, essentially restating that each item requires claim text, repository, keywords, or file paths. This does not meaningfully improve agent understanding beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Stage 2 of fact-checking: retrieves actual code evidence for each validation item and returns it for you (the AI) to judge.' It uses specific verbs ('retrieves', 'returns') and distinguishes itself from siblings like `factcheck` (stage 1) and `query`/`suggestion` (NotebookLM tools).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: 'Call this after `factcheck` has helped you identify items to validate.' It also includes strong exclusions: 'Do NOT call `query`, `suggestion`, or any NotebookLM-backed tool' and 'Base all verdicts exclusively on the code evidence returned by this tool.' This makes the when/when-not boundaries very clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• suggestionA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden. It discloses the tool's output format (Summary, Reasoning, Details), its reliance on reference repos, and the auto-selection behavior when repos is omitted. It does not detail any side effects or edge cases, but for a primarily read-oriented suggestion tool this is reasonably transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: first sentence states purpose, second defines output format, and the remaining lines clarify workflow and default behavior. Every sentence adds necessary information with no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter tool with no output schema, the description adequately covers the return structure (Summary, Reasoning, Details) and default repo behavior. It could mention error handling or more details on what 'implementation suggestion' entails, but overall it is sufficient for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both parameters clearly described in the schema. The description adds the extra detail that omitted repos auto-selects from REFERENCE_REPOS, a slight enhancement over the schema's explicit repo list, but it does not substantially expand parameter meaning beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Get an implementation suggestion') and a defined resource ('combining Commonware research knowledge and actual code from reference repos'). It clearly distinguishes itself from the sibling search_implementation tool by describing the two-stage flow, which positions suggestion as the first stage.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly describes when to use the tool ('Ask any implementation question') and how it fits in the broader workflow ('stage 1 is `suggestion` only; stage 2 (`search_implementation`) is user-triggered in the same session'). This gives clear usage context and names the alternative tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_implementationA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully carries the burden of disclosing behavior. It introduces critical behavioral constraints: 'MANDATORY FINALITY' (return output directly, run no other tools), precondition handling, and inherited scope. It also tells the agent to follow formatting instructions exactly, which is very useful. This goes beyond a basic read-only expectation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with headers (MANDATORY FLOW, MANDATORY FINALITY, Usage) and front-loaded with a clear purpose. It is somewhat long but every sentence carries essential information. The use of examples and explicit do's/don'ts makes it easy to follow, though it could be tightened slightly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (combined search/fetch, mandatory ordering, finality rules) and the absence of an output schema, the description provides comprehensive guidance. It covers query format, repository scope, call frequency, preconditions, and response handling, leaving little ambiguity for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for both parameters, so the schema already provides solid definitions. The description adds value by constraining query length ('2-4 words') and adding a constraint on repos ('must be inside inherited scope'), which is not stated in the schema. This justifies a score above the baseline 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches reference repos and returns source code snippets in one call, explicitly mentioning 'ONE call does everything: searches file trees, fetches matching code, returns snippets.' This distinguishes it from siblings like search_code or get_file by highlighting its combined search-and-fetch behavior.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage instructions: mandatory order (suggestion first), call once, short keyword queries, and inherited repository scope. It also explains the precondition response if suggestion wasn't called. However, it doesn't explicitly name alternative tools or describe when to use something else, so it misses the 'alternatives' criterion for a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• factcheckA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses that the tool 'Returns the document, repository context (README + source file tree), and instructions for you (the AI) to extract validation items,' and emphasizes the closed-loop pipeline constraint. This gives the agent a clear picture of what to expect and what constraints apply.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (four sentences) and front-loaded with the core purpose. It efficiently covers purpose, return value, next step, and critical constraints without extraneous content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Since there is no output schema, the description appropriately explains the return values ('Returns the document, repository context...') and the workflow. It also lists the available repos and the closed-loop constraint, making the tool's context fully understandable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description does not add detailed syntax or format info beyond the schema; it merely mentions 'document' and 'Available repositories,' which are already described in the schema. Thus, it adds no significant meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Stage 1 of fact-checking: analyzes a document against reference repositories,' which specifies a clear verb (analyzes), resource (document against reference repositories), and stage. It also distinguishes from sibling `factcheck_validate` by indicating the next step, making the tool's role in the pipeline unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs to call `factcheck_validate` after analysis and states 'Do NOT call `query`, `suggestion`, or any NotebookLM-backed tool during fact-checking.' It also says 'All evidence must come from `factcheck_validate` only,' providing clear when-to-use and when-not-to-use guidance with named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.