Server Quality Checklist

Tool Scores

• get-chunk-with-contextC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool retrieves a chunk with context, but doesn't describe what 'surrounding context' entails, whether it's read-only, potential errors, or response format. This is a significant gap for a tool with no annotation coverage, as it leaves key behavioral traits unspecified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and front-loaded with a single sentence that directly states the tool's function. There is no wasted language or unnecessary elaboration, making it efficient for quick comprehension. Every word earns its place by conveying the core purpose without fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of retrieving chunks with context, no annotations, and no output schema, the description is incomplete. It doesn't explain what a chunk is, how context is provided, or what the return value includes. For a tool with 3 parameters and no structured support, this leaves too many gaps for effective agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds minimal meaning beyond the input schema, which has 100% coverage. It implies parameters for documentId, chunkId, and windowSize but doesn't explain their relationships or semantics (e.g., how chunkId relates to documentId, what windowSize units are). With high schema coverage, the baseline is 3, and the description doesn't significantly enhance parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool's purpose ('Get specific chunk with surrounding context'), which is clear but vague. It specifies the verb 'Get' and resource 'chunk with surrounding context', but doesn't distinguish from siblings like 'get-document-by-id' or explain what a 'chunk' is in this context. The purpose is understandable but lacks specificity about what constitutes a chunk versus a document.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention siblings like 'get-document-by-id' or 'search-documents', nor does it specify prerequisites or exclusions. Usage is implied from the name and description but not explicitly stated, leaving the agent to infer context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get-document-by-idC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states this is a retrieval operation but lacks details on permissions, rate limits, error handling, or response format. For a read tool with zero annotation coverage, this leaves significant gaps in understanding how it behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is front-loaded and wastes no space, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what a 'full document' entails (e.g., content, metadata), potential errors, or usage context. For a retrieval tool with no structured behavioral data, more detail is needed to fully inform an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the 'id' parameter clearly documented as 'Document ID to retrieve'. The description adds no additional meaning beyond this, such as format constraints or examples. With high schema coverage, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Retrieve') and resource ('full document by ID'), making the purpose immediately understandable. However, it doesn't differentiate this from sibling tools like 'get-chunk-with-context' or 'search-documents', which likely also retrieve documents but with different approaches or scopes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing a specific document ID), exclusions (e.g., not for partial documents), or comparisons to siblings like 'search-documents' for broader queries or 'get-chunk-with-context' for contextual retrieval.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search-documentsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the BM25 algorithm and that it returns document chunks, but lacks critical details: it doesn't specify if this is a read-only operation (implied but not stated), whether it has rate limits, authentication needs, or how results are ranked/ordered. For a search tool with no annotations, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise and front-loaded: two sentences that directly state the action ('Search documents using BM25 algorithm'), inputs ('Takes keyword arrays'), and outputs ('returns relevant document chunks'). Every word earns its place with no redundancy or fluff, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a search tool with 3 parameters, no annotations, and no output schema, the description is incomplete. It doesn't explain the return format (e.g., structure of document chunks), error handling, or performance characteristics. While the purpose is clear, the lack of behavioral details and output information makes it inadequate for full contextual understanding, especially without annotations to compensate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters ('keywords', 'domain', 'topN') with descriptions and defaults. The description adds minimal value beyond the schema—it mentions 'keyword arrays' and 'returns relevant document chunks', but doesn't explain parameter interactions or provide additional context like format examples beyond what's in the schema. This meets the baseline of 3 for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search documents using BM25 algorithm' specifies the verb (search) and resource (documents), and 'returns relevant document chunks' clarifies the output. It distinguishes from siblings like 'get-document-by-id' (retrieval by ID) and 'list-domains' (listing domains), though not explicitly. However, it doesn't fully differentiate from 'get-chunk-with-context' (which might retrieve specific chunks), keeping it at 4 rather than 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when to choose this over 'get-chunk-with-context' for contextual retrieval or 'list-domains' for domain exploration. There's no context about prerequisites, exclusions, or typical use cases, leaving the agent to infer usage from the purpose alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list-domainsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states the tool lists domains and their document counts, which implies a read-only operation, but doesn't address potential limitations like pagination, rate limits, authentication requirements, or whether the list is comprehensive versus filtered. This leaves significant gaps for a tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function without any wasted words. It's front-loaded with the core action ('list all available domains') and adds only essential additional context ('and their document counts'). Every part of the sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is adequate but has clear gaps. It explains what the tool does but lacks context about when to use it, behavioral constraints, or output format details. For a list operation with no structured guidance, this is the minimum viable description—it covers the basics but leaves important aspects unaddressed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so the schema fully documents that no inputs are required. The description doesn't need to add parameter information, and it appropriately doesn't mention any parameters. A baseline of 4 is applied for tools with zero parameters, as the description correctly focuses on the tool's purpose rather than unnecessary parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'list' and the resource 'domains', specifying what the tool does. It adds useful context about including 'document counts' in the output. However, it doesn't explicitly differentiate from sibling tools like 'search-documents' or 'get-document-by-id', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'search-documents' or 'get-document-by-id'. It doesn't mention prerequisites, limitations, or specific contexts where listing all domains is appropriate versus searching for specific documents.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.