Server Quality Checklist

Tool Scores

• searchA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnly, idempotent, non-destructive. Description adds behavioral details: RRF fusion when searching all sources, per-hit source tagging, outline mode for cheaper triage. No contradictions. Adds context beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with clear sections: function, usage hierarchy, best practices, parameter summary, advanced outline usage. Every sentence serves a purpose, though slightly verbose. Efficient for the amount of information conveyed.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers main search behavior, source handling, filters, and alternative return format (outline). Doesn't detail errors or output structure, but output schema likely covers that. Given the tool's complexity and features, it is fairly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, baseline 3. Description adds value by clarifying the intent of each parameter (e.g., query should be natural language, source omission implies all sources RRF-fused, outline use cases). These nuances go beyond schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it performs semantic + lexical hybrid search over an indexed source. It positions itself as the PRIMARY search tool, distinguishing from siblings like deep_search and grep for exact identifiers. The scope (all sources vs. single source) and best practices are explicit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance: 'use it FIRST for any question', recommends natural language over identifiers, advises when to use outline mode vs full bodies. Does not directly contrast with deep_search beyond implying it is secondary, but still strong guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_rag_statusA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Description aligns with annotations (readOnlyHint=true, idempotentHint=true) by indicating a non-modifying status check. Adds context that it's a read operation, but annotations already cover safety profile. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Very concise: one sentence for purpose, then usage scenarios. No fluff, front-loaded with key action, and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (1 param, no required, output schema exists), the description covers purpose, when to use, and parameter behavior. Complete for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema already covers 100% of parameters with a clear description for 'source'. The tool description reiterates the same info (source name, omit for all), adding no new semantics beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool reports the state of the RAG index for one or all sources. Uses specific verb 'report state' and identifies the resource. Distinguishes from sibling by mentioning checking before update_source_index.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: before deciding to call update_source_index and for debugging stale search results. Implicitly suggests alternative tool (update_source_index). Provides clear context for usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• update_source_indexA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that the tool is expensive, blocks until complete, and that the watcher normally handles syncing. These traits go beyond annotations (readOnlyHint false, idempotentHint true, destructiveHint false) and provide crucial operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: first line gives purpose, followed by context, guidance, warning, then Args. It is concise but includes some repetition (Args section mirrors schema). Still efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema and annotations, the description covers intent, when to use, caveats, and parameter details. It is complete for an AI agent to decide when and how to invoke the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and the description largely restates schema info for both parameters (source and force). It adds minor context like using 'list_sources' for source names, but does not significantly enhance understanding beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action ('Force a full rebuild of a specific source's index') and the specific resource ('source's index'). It also distinguishes itself from sibling tools like 'get_rag_status' by noting the watcher normally keeps the index in sync.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly tells when to use (after complex merge, bulk rename, drift detection) and when not to (not routinely, expensive, blocks). Also advises checking 'get_rag_status' first and using 'list_sources' to discover names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• feedbackA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that feedback is appended to a local log file (rag_storage/_feedback/feedback.jsonl) and notes it is never uploaded. Since annotations provide no behavioral hints (all false), the description fully informs about side effects and privacy.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single paragraph with clear structure: purpose, usage, privacy note, argument explanations. Every sentence adds value, no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (3 string params, no nested objects), the description covers purpose, usage, behavior, and parameters. Output schema exists, so lack of return value explanation is fine.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds meaningful context by explaining each parameter's purpose in the feedback scenario, going beyond the schema's basic descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool is for reporting failure to find needed information. Distinguishes from sibling search tools by specifying it's a feedback mechanism after exhausting search/deep_search/graph_query.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Call this BEFORE giving up after exhausting search / deep_search / graph_query,' providing clear usage context. Lacks explicit when-not-to-use, but the situational guidance is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sourcesA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, covering safety. The description adds value by specifying the returned fields (type, location, chunk count, drift status) and the usage flow, which aids agent decision-making without contradicting annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary purpose, and every sentence adds essential information. There is no wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that the tool has zero parameters, annotations fully cover behavioral traits, and an output schema exists (not shown but implied), the description is complete. It explains the tool's role as a precursor to other tools, making the agent well-equipped to use it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters, and schema coverage is 100%, so the description naturally adds no parameter information beyond what the schema shows. With zero parameters, the baseline is 4, and the description provides useful context about the output.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all configured sources with their type, location, chunk count, and drift status,' providing a specific verb and resource. It distinguishes itself from sibling tools by noting that the listed names are used as arguments for other tools like search and deep_search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises 'Call this first when you don't know which sources are available' and instructs to pass a name as the source argument to other tools. This provides clear when-to-use guidance and distinguishes it from alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deep_searchA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only and idempotent behavior. Description adds detailed execution flow: ordered trial of query variants, threshold-based stopping, and fallback to strongest weak set with warning. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single dense paragraph front-loaded with purpose and escalation note, followed by mechanism, best practices, and a compact arg summary. Every sentence earns its place without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 10 parameters and output schema existence, the description covers all critical aspects: main behavior, parameter details, edge cases (weakness threshold, warning), and source targeting. No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 100% schema coverage, the description adds significant value: explains `queries` with best practices and examples, clarifies `source` behavior (omission = all sources RRF-fused), and notes single-source restrictions for `mode` and `return_all_variants`.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Multi-query fallback search' and immediately distinguishes from sibling `search` by labeling it an 'ESCALATION TOOL' with explicit when-to-use instructions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description provides explicit when-to-use ('only when `search` returned weak or empty results'), when-not-to-use (slower than `search`), and best practices for constructing queries with concrete good/bad examples.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.