Server Quality Checklist

Tool Scores

• checklist_runA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It mentions output items but also says 'against a description of your AI agent or deployment', implying an input that is not in the schema. This creates ambiguity about how the description is provided and whether the operation has side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear action sentence, output summary, usage context, and an Args section. Every sentence adds value and there is no unnecessary prose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, usage, and parameter examples, but the disconnect between the 'description of your AI agent or deployment' and the schema lacking such a parameter is a notable gap. It also omits any error or permission details that might be relevant.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has a single 'name' parameter with no description, but the Args section adds examples of valid values and explains it as a 'Checklist name', which helps the agent understand what to provide.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Run' and the resource 'a named checklist', and explains the output structure ('Returns all items...'), making it distinct from siblings like checklist_list and checklist_get.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context: 'Use this to assess whether your deployment meets security controls from NIST, CISA, OWASP, CUSTODY, and LASM.' However, it does not explicitly mention when not to use it or name alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• checklist_getA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description details the return content (threat, control, severity, verification, source references), which adds transparency beyond the bare action. However, with no annotations, it does not disclose error behavior, permissions, or side effects, leaving some behavioral aspects unclear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two purposeful sentences plus an Args list, front-loaded with the primary action. Every sentence adds value with examples and return details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple getter with an output schema, the description covers the core function, parameters, and return fields. It lacks explicit usage guidance and error handling, but is largely sufficient given the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no parameter descriptions (0% coverage), but the description explains both parameters with concrete examples (e.g., 'agent-containment' and 'AC-001'). This compensates for the schema gap, though it doesn't enumerate valid values or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves a specific checklist item by ID, with examples of ID format. This distinguishes it from siblings like checklist_list (which lists items) and checklist_diff (which compares).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying it gets a single item by ID, but does not explicitly state when to prefer this over checklist_list or other siblings. No alternatives or exclusions are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• threat_searchA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the full burden. It adds the scope ('all checklists') and typical use cases, but does not disclose whether it is read-only, how results are returned, or any limitations. For a search tool this is acceptable but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-organized: a one-line summary, a usage sentence with examples, and a parameter definition. No filler; every sentence contributes.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With only one simple parameter and an output schema available, the description covers the essential purpose and usage. It could mention result ordering or that it searches only controls, but the output schema likely covers return structure. Adequate for the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description includes an 'Args' section clarifying 'query' as 'Threat or control to search for'. This compensates for the missing schema-level description and adds practical meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Search all checklists for controls that mitigate a given threat', using a specific verb ('Search'), resource ('all checklists'), and objective. This clearly differentiates it from sibling tools like checklist_list or checklist_get.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides concrete usage guidance with examples ('prompt injection', 'credential theft', etc.) in the sentence 'Use this to find relevant controls for specific threats...'. However, it does not mention when not to use or alternatives, so it lacks explicit exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• protocol_getA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses that the tool returns the full protocol document and enumerates its sections, which is useful behavioral context. It does not mention auth, error behavior, or side effects, but the read-only semantics of 'Get' are straightforward.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise overview sentences plus a short Args block; every sentence adds value. No redundant or filler text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (one parameter) and has an output schema, so the description doesn't need to document return structure. The provided context covers purpose, argument semantics, and expected response contents.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only 'name' as a string with no description; the description compensates by giving concrete examples ('agent-deployment', 'incident-response') and clarifying that it is a protocol name. This is sufficient for a single-parameter tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly identifies a get operation for a named security protocol, enumerating document contents (prerequisites, steps, verification, rollback). This distinguishes it from sibling tools like checklist_get by specifying what a protocol document contains.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when a protocol document is needed by name, but it does not explicitly compare to sibling tools or specify when not to use. No alternative tools are mentioned, so guidance is only implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• mapping_lookupA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description alone must disclose behavior. It does state that the tool returns all matching checklist items and implies a read-only lookup operation. Yet it does not mention failure modes, authorization requirements, or any limitations beyond the listed framework names, leaving the behavioral profile somewhat incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: a clear opening sentence states the purpose, followed by a short explanation of the return value and a compact 'Args' section listing the parameter and allowed values. Every sentence serves a purpose with no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a simple lookup tool with one parameter and a present output schema. The description covers the tool's purpose, the parameter's allowed values, and the nature of the return data, which is sufficient for an agent to select and invoke it correctly. No additional context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema for 'framework' is only a string with no enum or description, giving 0% description coverage. The description fully compensates by specifying the exact allowed values ('nist-rmf', 'owasp-llm', 'atlas', 'custody', 'lasm') and explaining what the parameter selects, providing complete parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with the verb 'look up' and identifies the resource as 'security controls mapped to a specific framework.' It explicitly enumerates the supported frameworks (NIST AI RMF, OWASP, CUSTODY, LASM, MITRE ATLAS), which distinguishes it from sibling tools like checklist_list or protocol_get that serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description conveys a clear usage context by explaining that the tool enables cross-reference between specific frameworks and returns corresponding checklist items. However, it does not explicitly name alternative tools or provide 'when not to use' exclusions, so it stops short of a fully explicit contrast.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• checklist_listA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the transparency burden. It explicitly states the return fields (name, version, description, item count, frameworks) and that no arguments are required, making the expected behavior clear. For a read-only listing tool, this is sufficient, though it does not mention potential authorization requirements or response size.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two short paragraphs, with the first sentence stating the core purpose and the second enumerating return fields. Every sentence adds value, and the structure is front-loaded for quick agent parsing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter list tool with an existing output schema, the description is complete. It specifies the exact output fields and explicitly notes there are no arguments, covering all necessary context for invocation. No additional caveats or edge cases are relevant for this simple operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4. The description adds the explicit statement 'No arguments,' which removes any ambiguity about parameter usage, fully addressing the empty input schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with a specific verb and resource ('List all available security checklists'), clearly defining the tool's scope as an exhaustive listing. It also differentiates from siblings like checklist_get by emphasizing 'all available' and including aggregate details like item counts and frameworks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description establishes clear context: it is a zero-argument command that returns all checklists, implying it is the discovery/overview entry point. It does not explicitly name alternatives or when-not-to-use scenarios, so it falls just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• checklist_diffA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden. It discloses key behaviors: case/whitespace-insensitive matching, duplicate detection, and output of missing items in both directions. This goes beyond the schema and gives the agent accurate expectations. It doesn't mention side effects explicitly, but 'compare' implies read-only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, front-loaded with the main purpose, and uses a clear Args block for parameters. No wasted words; each sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a simple 2-parameter tool, the description effectively explains purpose, behavior, and usage. The output schema exists so return values are covered separately. It doesn't address error cases but nothing indicates they are critical; overall it's a complete description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has no descriptions (coverage 0%), so the description fully compensates. It defines each parameter with examples: 'a: First checklist name, e.g. agent-containment.' and clarifies that names are used, not IDs. This provides meaningful semantics beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Compare two checklists and show coverage gaps and duplicated controls.' It explicitly mentions the matching logic (normalized control text, not item ID) and distinguishes it from sibling tools like checklist_list and checklist_get.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides context for when to use it: 'Use this to keep the catalog maintainable as it grows.' This indicates a maintenance/diff scenario. It doesn't explicitly mention alternatives or exclusions, but the purpose is clear enough to guide selection among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• evidence_verifyA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure, and it delivers substantial detail. It reveals that self-attested collection, unsigned bundles, and unverifiable controls all surface as findings, and that 'unknown' observations remain unreviewed rather than counting as enforced. This goes beyond a generic summary and gives the agent accurate expectations for how edge cases are handled.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and well-structured, opening with a one-line summary then providing behavioral details and an Args section. Every sentence adds value, and there is no redundant or filler content. The formatting makes it easy to scan for key information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity, the description covers all necessary aspects: the action, the input format, the behavioral quirks, and the output categories. An output schema exists, so return structure is handled separately. The description is complete enough for an agent to select and invoke the tool correctly, leaving no significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides no descriptions for the two parameters (0% coverage), so the description is the sole source of meaning. It adds essential context: bundle_json must be a 'cinch-evidence/1' bundle as returned by evidence_collect, and deployment is an optional override. This enables correct invocation despite the schema being empty.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb 'Grade' and a clear resource ('an evidence bundle'), then enumerates the outputs: score, letter grade, insights, recommendations, plan. This distinguishes it from sibling tools like evidence_collect (which gathers bundles) and checklist_run (which executes checklists), leaving no ambiguity about its function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The tool's usage context is clear: it is meant for grading evidence bundles, and the input is explicitly tied to the output of evidence_collect. The description also states it applies the same rubric a human reviewer sees, which signals when it is appropriate. However, it does not explicitly exclude other tools or mention when not to use it, so it falls short of an explicit when-to-use versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• evidence_collectA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure, and it excels. It explicitly states that the tool runs read-only probes, returns the actual observed statuses, counts 'unknown' as never enforced, and flags self-attestation as a critical finding (provenance.self_attested). It also reveals the side effect of sending adversarial input to an endpoint when `authorized` is true. This is rich, honest, and highly transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a one-sentence summary and then well-structured sections for output details, self-audit warning, and arguments. It is longer than average, but every sentence adds necessary information for a tool with complex side effects and seven parameters. It earns its length, though a few rephrasings could tighten it slightly without losing meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (7 parameters, no annotations, high-risk behavioral probes) and the presence of an output schema, the description covers the essential context: what the tool returns, how to interpret results, the self-attestation pitfall, and the recommended follow-up (evidence_verify). The description is self-sufficient for an agent to decide whether and when to invoke this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for the lack of parameter documentation in the schema, and it does so completely. The 'Args' section explains the meaning and purpose of all seven parameters, including the role of `authorized` as a consent flag, the distinction between `pid` omission leading to self-inspection, and what `project_path` covers (MCP tool grants, manifests, etc.). This is exactly what the description should add over the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Probe a running agent deployment and return an evidence bundle.' It goes beyond a generic verb+resource by explaining the output format (pass/fail/unknown per control) and explicitly differentiates itself from the sibling tool evidence_verify by directing users to feed the bundle to it for grading. This makes the tool's distinct role in the workflow unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage context: it tells users when to use this tool (to collect evidence) and, importantly, provides a clear caveat about when not to rely on it (self-attestation) with a named alternative ('run cinch collect out of band'). It also specifies when the `authorized` parameter must be true, which doubles as a prerequisite for endpoint probing. This level of when/when-not guidance is exemplary.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.