Server Quality Checklist

Tool Scores

• list_documents_toolA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full disclosure burden. Successfully adds behavioral context: explains ID format variants (PDF filenames, video IDs, GitHub paths), discloses total chunk counting behavior, notes server config dependency for vector store location, and specifies return dictionary structure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with clear sections: purpose statement, Returns block, and Args block. Information is front-loaded. Minor verbosity in Returns section documenting fields, though this adds value by example ID patterns. 'ctx' parameter mention in Args is extraneous (not in schema) but marked as unused.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Appropriately complete for a 1-parameter listing tool. Output schema exists (per context signals), so verbose return documentation in description is optional but helpful. Includes concrete examples of document ID formats which aids LLM comprehension of return values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% description coverage for the single 'tag' parameter. Description includes Args section that essentially duplicates schema information. Baseline 3 is appropriate since schema is self-documenting; description adds no new semantic meaning beyond schema but accurately reflects it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States specific verb 'List' with resource 'indexed documents' and scope 'PinRAG index'. Clear distinction from siblings: contrasts with add/remove by focusing on retrieval of existing index, and implicitly differs from query_tool by emphasizing enumeration ('all indexed documents') rather than semantic search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides implicit usage context through detailed return value description (document IDs, chunk counts), but lacks explicit when-to-use guidance. Does not clarify when to choose this over query_tool (listing/enumeration vs. content search) or whether to check existence before calling add_document_tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• remove_document_toolA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. Explicitly states destructive scope ('Deletes all chunks and embeddings'), mentions server config dependency for vector store location, and documents return structure. Could clarify if operation is reversible.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Front-loaded with clear action statement. Uses docstring-style Args/Returns sections which provide structure but repeat schema information. Each sentence adds value (mechanism, prerequisite, config dependency, examples).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Appropriate for a single-parameter destructive tool. Covers prerequisites (list_documents_tool), explains what gets destroyed (chunks/embeddings), documents return values, and notes configuration dependencies. Complete given the tool's limited complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% coverage with document_id well-described. Description adds valuable examples of ID formats (e.g., 'bwgLXEQdq20'). However, erroneously documents 'ctx' parameter in Args section that does not exist in the input schema, potentially causing confusion.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clear specific verb (Remove) + resource (document and chunks) + scope (PinRAG index). Distinguishes from siblings: contrasts with add_document_tool/add_url_tool (ingestion), list_documents_tool (listing), and query_tool (search).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly directs users to use list_documents_tool first to obtain document_ids, providing clear workflow guidance. Includes concrete examples of ID formats (e.g., 'mybook.pdf', 'owner/repo/path'). Lacks explicit 'when not to use' exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• add_document_toolA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. Discloses critical behaviors: auto-format detection per path, partial failure tolerance (one bad path doesn't fail batch), server config dependency for vector store, and parameter applicability constraints (GitHub params ignored for other formats). Missing idempotency or rate limit disclosures.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with front-loaded purpose statement, followed by behavior details, formatted Args section, and Returns section. Minor redundancy between Args section and schema descriptions, but earns its place by presenting information in readable narrative form with practical examples.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Comprehensive coverage of multi-format input complexity (PDF, Discord exports, GitHub, YouTube), vector store configuration context, and return value summary appropriate given output schema exists. Adequately prepares agent for the tool's multi-modal indexing behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, establishing baseline 3. Description adds value by providing concrete syntax examples in prose ('paths=["/path/to/file.pdf"]'), clarifying the ordering constraint between tags and paths ('one per path, same order'), and noting that ctx is injected/unused—context not fully captured by schema descriptions alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Opens with specific verb 'Add' and enumerates exact resource types (files, directories, YouTube videos, GitHub repos) and target destination 'to the index'. Implicitly distinguishes from sibling add_url_tool by specifically listing content types that require format detection, while add_url_tool likely handles generic URLs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides concrete usage examples ('Single path: paths=[...]') and explains batch error handling ('Returns both successful and failed entries'), but lacks explicit when-to-use guidance versus add_url_tool or prerequisites like file access permissions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_toolA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must carry the full disclosure burden. It successfully explains the RAG mechanism and citation behavior but omits safety-critical context (read-only nature, error handling when documents missing, rate limits) that annotations would typically cover.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with clear Args/Returns sections and front-loaded summary. The Args section largely mirrors schema properties but justifies its length by adding cross-references and usage examples not present in the structured schema. Slight verbosity in the Args list, but every parameter earns its documentation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (8 parameters, multiple filter types, RAG architecture), the description comprehensively covers: the retrieval mechanism, all filter parameters with domain-specific notes (PDF page ranges, GitHub paths), response style options, and return structure (answer + sources). Complete for a sophisticated query tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 100% schema coverage (baseline 3), the description adds substantial value: cross-references to list_documents for document_id/file_path, domain constraints ('PDF only' for page ranges), specific GitHub path examples, and enumerated style options. The only minor issue is documenting 'ctx' as an injected argument not present in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description provides a specific verb ('Query') and resource ('indexed documents'), clearly distinguishing this RAG-based search tool from sibling add/remove/list operations. Opening sentence immediately establishes the retrieval-and-answer paradigm.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context by repeatedly referencing list_documents for obtaining valid document_id and file_path values, establishing a workflow dependency. However, it lacks explicit 'when not to use' guidance or contrast with the sibling list_documents_tool for browsing vs querying.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• add_url_toolA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. While it identifies the content sources (YouTube/GitHub), it lacks operational details: no mention of idempotency, whether existing entries are overwritten, async/sync processing, or failure modes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficiently structured sentences. First establishes purpose and supported resources; second provides sibling alternative. No redundancy or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Complete for tool selection: covers supported resources, indexing action, and sibling differentiation. With output schema present, return values need not be described. Minor gap: does not mention that this is a write/mutation operation or its side effects given lack of annotations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (baseline 3). Description adds value by specifying the two distinct URL types (YouTube vs GitHub), which provides necessary semantic context for the domain-specific parameters (branch, include_patterns, exclude_patterns) that would otherwise seem arbitrary.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States specific action ('Add'), resource types ('YouTube videos/playlists or GitHub repos'), destination ('to the index'), and input method ('via URL'). Clearly distinguishes scope from sibling add_document_tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly directs users to alternative: 'For local files or directories, use add_document_tool instead.' Provides clear when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.