Server Quality Checklist

Tool Scores

• edit_pageC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are absent, so the description must carry the full burden of behavioral disclosure. It mentions validation and warnings but does not clarify whether changes are persisted, if permissions are needed, or if the tool is a dry-run. The phrase 'Proposes changes' is ambiguous and the return dictionary's 'success' field implies a potential side effect, but no details are given.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, relatively concise, and starts with the core action. However, the second sentence mixes validation and return details, and the third sentence lists return fields, which could be shortened. It is adequate but not optimally structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, no annotations, and 0% schema coverage, the description is insufficient. It does not explain how the page to edit is identified (likely from context but not stated), what the edit should look like (markdown?), or what errors may occur. The return dictionary is mentioned but its fields are not described in context. Significant gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, yet the description adds no explanation for the two parameters ('edit', 'description'). It only mentions a 'description' field in the return value, which could confuse with the input parameter. The description fails to clarify the format or purpose of the parameters beyond their names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states it proposes changes to an Evidence markdown page and validates them, but it is ambiguous whether the changes are actually applied or if this is a dry-run. The verb 'proposes' suggests a suggestion, but the return of content implies modification. Purpose is vague and does not clearly distinguish from a simple validation tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus its siblings (debug_code, get_metadata, read_docs). There is no mention of prerequisites, context, or alternatives. The agent receives no help deciding whether to invoke this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• debug_codeC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description should disclose behavioral traits; it mentions returning a dictionary with suggestions but does not confirm if the tool modifies state, requires permissions, or has side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description has some redundancy (first sentence rephrased in second) and could be more concise, but it is structured with a summary and return list.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description is incomplete; it does not explain the error format, page content expectations, or details of the analysis dictionary.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not add meaning beyond parameter names—'errors' and 'page_content' are not clarified in structure or format.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool analyzes validation errors and suggests fixes, differentiating it from siblings like edit_page, get_metadata, and read_docs which perform different functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when validation errors are present with page content, but lacks explicit when-to-use or when-not-to-use guidance compared to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_docsB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description does not disclose behavioral traits such as read-only nature, rate limits, or authentication needs. It only states what is returned, lacking full transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is moderately concise but includes a lengthy list of categories that could be summarized. It is structured with a clear purpose statement and a list, but the list is verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the return format and doc_type options well but lacks details on the 'component' parameter. Given no output schema and no annotations, it is adequate but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description partially compensates by listing categories for the 'doc_type' parameter with examples. However, the 'component' parameter is not described at all, leaving its semantics ambiguous.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it retrieves Evidence documentation via hierarchical lookup. It lists categories and subcategories, and specifies the return structure with 'title', 'content', and 'related_docs'. The purpose is specific and distinct from siblings like debug_code or edit_page.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for looking up documentation but does not explicitly say when to use this tool vs alternatives or provide usage exclusions. Siblings are sufficiently different, so guidance is minimal but adequate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_metadataA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. Clearly describes return format (JSON with tables and columns). Does not explicitly state non-destructive nature, but read-only schema retrieval is implied. Lacks mention of side effects, but they are unlikely.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each purposeful. Front-loaded with main purpose, then return format, then usage guidance. No fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no parameters, no output schema, no annotations, description fully explains what the tool does and returns, with usage context. Complete for its complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Zero parameters, so baseline is 4. Description adds meaning beyond empty schema by explaining output structure and use case.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it returns database schema from Evidence's DuckDB connection, listing tables and columns. Verb 'returns' with specific resource 'database schema', distinct from sibling tools like debug_code or read_docs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use this to understand what data is available for queries.' Provides a clear use case but no when-not or alternatives, though context with siblings makes it clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.