Server Quality Checklist

Tool Scores

• needle_create_collectionA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that it creates a new collection and returns an ID, but lacks details on permissions, rate limits, or error conditions. It adds some context (ID requirement for future ops) but is incomplete for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with four sentences: purpose, function, usage guidelines, and parameter advice. Each sentence adds value without redundancy, and it's front-loaded with the core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a creation tool with no annotations and no output schema, the description covers purpose and usage well but lacks behavioral details like error handling or response format. It's adequate but has gaps given the mutation nature and missing structured data.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the 'name' parameter fully. The description adds marginal value by advising to choose a descriptive name for better organization, but doesn't provide additional syntax or format details beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool creates a new document collection, specifying it's for organizing and searching documents. It distinguishes from siblings like needle_list_collections (lists existing) and needle_add_file (adds to collections), with specific verb+resource.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly lists three use cases (start new organization, group related documents, set up searchable repository) and mentions the collection ID is required for subsequent operations, providing clear when-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• needle_get_collection_statsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It describes what information is retrieved (metrics like total documents, processing status, storage usage, index status) and its purpose (monitoring, verification, health checks), but lacks details on behavioral traits such as rate limits, error handling, or response format. It adds context about readiness but doesn't fully compensate for the absence of annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded, starting with the core purpose, followed by bullet points for metrics and usage guidelines, and ending with a key takeaway. Every sentence adds value without redundancy, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (statistical retrieval with 1 parameter), no annotations, and no output schema, the description does a good job by detailing metrics and usage scenarios. However, it lacks information on return values or error cases, which would be needed for full completeness. It compensates well but has minor gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the parameter 'collection_id' fully documented in the schema. The description does not add any meaning beyond what the schema provides, as it doesn't explain parameter usage, constraints, or examples. Baseline 3 is appropriate since the schema handles the parameter documentation adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('retrieve detailed statistical information') and resource ('Needle collection's contents and status'), distinguishing it from siblings like needle_get_collection_details or needle_list_collections by focusing on metrics rather than general details or listings. It explicitly lists the types of metrics provided, making the purpose highly specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes explicit usage scenarios ('Use this tool to: - Monitor collection size and growth - Verify processing completion - Check collection health before operations') and states when it's essential ('Essential for ensuring collection readiness before performing searches'), providing clear guidance on when to use it versus alternatives like needle_search or needle_list_files.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• needle_list_filesA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the return format (detailed information with specific fields like File ID, status, upload date) and hints at processing workflows, but lacks details on error handling, pagination, rate limits, or authentication needs. It adds value beyond the schema but is incomplete for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded, starting with the core purpose, followed by a bulleted list of return details and usage guidelines. Every sentence adds value without redundancy, and the information is organized for quick scanning, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter, no output schema, no annotations), the description is mostly complete. It covers purpose, usage, and return format adequately. However, without annotations or output schema, it could benefit from more behavioral details (e.g., error cases or response structure), slightly reducing completeness for a tool with no structured safety hints.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the single parameter 'collection_id'. The description does not add any additional meaning or context about this parameter beyond what the schema provides (e.g., format examples or usage tips). Baseline 3 is appropriate when the schema handles parameter documentation effectively.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('List all documents') and resources ('within a specific Needle collection'), including scope ('with their current status'). It distinguishes from siblings like needle_search (which searches content) and needle_add_file (which uploads files), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage scenarios in a bulleted list ('Use this tool when you need to:'), including specific cases like inventorying documents, checking processing status, getting file IDs, and verifying availability before searching. It also mentions an alternative ('before performing searches') and includes a final note on monitoring completion, offering comprehensive guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• needle_add_fileA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and discloses key behavioral traits: the three-step processing pipeline (download, text extraction, indexing), supported file formats, and the important latency consideration ('Documents require processing time before they're searchable'). It doesn't mention error handling or authentication needs, but covers core operational behavior well.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with clear sections (supported formats, processing steps, usage guidelines, important note) and front-loaded core purpose. Every sentence adds value, though the bullet-point formatting for file formats could be slightly more concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 3-parameter mutation tool with no annotations and no output schema, the description provides comprehensive context: purpose, usage guidelines, behavioral transparency, and important limitations. It doesn't describe return values or error cases, but given the tool's complexity and lack of structured metadata, it's reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters. The description adds context about what the 'url' parameter represents ('Public URL where the document can be downloaded from') and implies the purpose of 'name' ('descriptive filename'), but doesn't provide additional semantic meaning beyond what's in the schema. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Add a new document'), target resource ('Needle collection'), and mechanism ('by providing a URL for download'). It distinguishes from siblings like needle_create_collection (creates collections) and needle_list_files (lists files) by focusing on document ingestion.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly provides three use cases ('when you need to: - Add new documents to a collection - Make documents searchable - Expand your knowledge base') and mentions an alternative tool ('Check processing status using needle_list_files before searching new content'), giving clear guidance on when to use this tool versus others.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• needle_list_collectionsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does well by disclosing key behavioral traits: it's a read operation (implied by 'List'), returns up to 20 results, supports pagination via offset, and includes metadata required for other operations. It doesn't mention rate limits or auth needs, but covers essential functionality adequately for a listing tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with key information (purpose, limit, pagination), followed by use cases. It could be slightly more concise by merging some sentences, but every sentence adds value, such as clarifying the response content and operational context. No wasted words, but minor tightening is possible.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 optional parameter, no output schema, no annotations), the description is quite complete. It covers purpose, usage, behavioral traits like pagination and result limits, and contextual value for other operations. It doesn't detail exact return formats, but for a listing tool without an output schema, this is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the parameter 'offset' fully documented in the schema. The description adds context by explaining how to use offset ('Get more results by increasing the offset'), which provides practical guidance beyond the schema's default value. This meets the baseline of 3 since the schema handles most of the documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('Needle collections'), distinguishing it from siblings like 'needle_create_collection' (creation) and 'needle_get_collection_details' (detailed view). It specifies the scope ('maximum of 20 results') and purpose ('overview of available document collections'), making it highly specific and differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly lists three use cases: getting an overview, finding IDs for subsequent operations, and verifying existence before operations. It also implies when to use alternatives by noting that 'detailed information' is returned, suggesting 'needle_get_collection_details' for more specifics, though it doesn't name alternatives directly. This provides clear, actionable guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• needle_searchA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behavioral traits: it's a search operation (implied read-only), uses semantic matching rather than keywords, returns ranked results with source information, and handles natural language queries. However, it doesn't mention potential limitations like rate limits, authentication requirements, or performance characteristics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, usage guidelines, effectiveness, returns) and uses bullet points for readability. While comprehensive, it could be slightly more concise by combining some bullet points or reducing redundancy in the effectiveness section.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with 2 parameters, 100% schema coverage, and no output schema, the description provides good context about what the tool does, when to use it, and what it returns. It explains the semantic nature of the search and provides usage scenarios. The main gap is the lack of output format details beyond 'matching text passages with their source file IDs' - more specificity about the return structure would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, providing clear parameter documentation. The description adds some context by mentioning 'natural language queries' which aligns with the query parameter description, and 'collection' context which matches collection_id. However, it doesn't provide significant additional semantic meaning beyond what's already in the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs 'intelligent semantic search across documents in a Needle collection' with specific details about using embedding technology, finding relevant content based on meaning, and distinguishing it from keyword search. It explicitly differentiates from sibling tools like needle_add_file (adds files), needle_create_collection (creates collections), and needle_list_files (lists files) by focusing on search functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance with a dedicated 'Use this tool when you need to:' section listing four specific scenarios (find information, answer questions, research topics, locate passages) and a 'More effective than traditional keyword search for:' section with three cases (natural language questions, conceptual queries, finding related content). This clearly indicates when to use this tool versus alternatives like keyword-based search methods.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• needle_get_collection_detailsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the tool's behavior by stating it 'returns detailed collection metadata' and specifies error conditions ('Will error if collection doesn't exist'), though it lacks details on rate limits, authentication needs, or response format specifics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core purpose, followed by usage guidelines in bullet points, and ending with prerequisites and error conditions. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (1 parameter, no output schema, no annotations), the description is largely complete, covering purpose, usage, prerequisites, and error handling. However, it could be enhanced by including details on the metadata structure returned or any performance considerations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already documents the single parameter (collection_id). The description adds minimal value beyond the schema by mentioning 'Requires a valid collection ID' but does not provide additional syntax or format details, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('fetch comprehensive metadata') and resources ('specific Needle collection'), distinguishing it from siblings like needle_get_collection_stats (which likely provides statistics rather than configuration metadata) and needle_list_collections (which lists collections rather than fetching details of a specific one).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly provides when-to-use guidance with three bullet points (e.g., 'Verify a collection's existence and configuration', 'Check collection metadata before operations'), and it specifies a prerequisite ('Requires a valid collection ID') and an error condition ('Will error if collection doesn't exist'), offering clear alternatives and exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.