Server Quality Checklist

Tool Scores

• search_corpusA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses the retrieval algorithm (BM25 + dense + RRF), the targeted corpus, and the exact return payload (document title, heading path, URL, verbatim content). While it does not explicitly state side-effect freedom, the language is consistent with a non-mutating search operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a tight two-sentence structure. The first sentence states the method and corpus; the second defines the return format. Every sentence adds critical information, and the most distinguishing element (retrieval over specific guidance sources) is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the corpus scope, the retrieval method, and the exact contents of the returned chunks, which is sufficient given the simple two-parameter schema and existing output schema. It omits explicit usage guidance and caveats, but these are minor for a straightforward read-only search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides no parameter descriptions, so the tool description must compensate. The phrase 'top-k matching chunks' clarifies the top_k parameter's role, and 'retrieval' implies query is the search text. However, the description does not explain query formulation, permissible input types, or top_k bounds, leaving the agent to infer most semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Hybrid (BM25 + dense + RRF) retrieval over CIS AWS Foundations and OWASP CI/CD Top 10 guidance.' This clearly differentiates the tool from siblings like scan_terraform_plan (scanning actions) and list_sources (listing sources), making the tool's purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by describing what the tool searches and returns, but it never explicitly states when to use this tool versus the siblings or any exclusions. The purposeful phrasing 'retrieval over CIS AWS Foundations and OWASP CI/CD Top 10 guidance' makes the use case inferable, yet explicit guidance is absent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_terraform_planA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It adds useful context: it invokes an LLM-dependent agent, defaults to a stub provider when no API key is configured, and returns structured findings. However, it does not disclose potential side effects, network usage, or what happens when the stub provider is used.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the core action and input source, followed by return-value summary and configuration default. Every sentence adds distinct, useful information with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is a single-parameter operation with an output schema, so the description need not detail return values. It covers input provenance, output summary, and a key configuration default. The only minor gap is not elaborating on the stub-provider implication or environment expectations, but overall it is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description compensates by explaining that plan_json must be the output of 'terraform show -json'. Since there is only one parameter, this is sufficient to understand what to pass, despite the schema itself carrying no semantic description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Run the LangGraph scan agent') against a specific resource ('a Terraform plan JSON'), and clarifies the expected input format with the command that produces it. It clearly distinguishes the tool from its siblings, which search or list sources rather than scan.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly implies when to use the tool: when you have a Terraform plan JSON and need security findings. It also provides the exact command to generate the input. However, it does not explicitly compare with the sibling tools or state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sourcesA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the behavioral transparency burden. It clearly indicates a read-only aggregation behavior by describing returned counts, but it does not mention potential edge cases such as empty corpora, data freshness, or permission requirements. The core behavior is disclosed, but some context is missing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler. The first sentence states exactly what the tool returns, and the second gives practical usage context. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter list tool with an output schema, the description is fully sufficient. It tells the agent what data to expect and why the tool matters for grounding verification. No additional context is needed to call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4. The description correctly focuses on what the tool returns rather than trying to document nonexistent arguments.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: it returns document and chunk counts grouped by source. This clearly distinguishes it from the sibling tools search_corpus and scan_terraform_plan, which perform different functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear use case: verify the ingested corpus before trusting an agent's citations. It does not explicitly name alternatives or state when not to use the tool, but the context provided is sufficient for a zero-parameter listing tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.