Server Quality Checklist

Tool Scores

• regionsA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility for behavioral disclosure. It only states the operation is a list of enabled regions and requires a prior call. It does not describe what happens if the inventory is not loaded (e.g., error behavior), nor does it mention any other traits like idempotency, rate limits, or whether the output is a simple list of region names. For a tool with zero annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no wasted words. The first sentence front-loads the core purpose, and the second immediately provides the critical prerequisite. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, so the description should explain return values. It states 'List enabled AWS regions' but does not describe the format of the output (e.g., list of region names, objects, or something else). It also does not mention error conditions or edge cases. Given the tool's simplicity, the missing output description is a gap, but the prerequisite and purpose are covered, making it minimally adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, so the input schema is trivially fully covered. The description does not need to add parameter semantics because none exist. According to the rules, 0 params gives a baseline of 4, which is appropriate here.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's verb and resource: 'List enabled AWS regions from the loaded inventory.' This explicitly distinguishes it from siblings like 'scan' (data ingestion), 'load' (data loading), and 'export' (data output), as it is a read-only retrieval operation on already-loaded data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a clear prerequisite: 'Requires "scan" or "load" to be called first.' This tells the agent when to use the tool (after data loading) and implicitly not to use it before. However, it does not mention alternatives or when-not scenarios beyond the prerequisite, which keeps it from being a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• loadA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries a heavier burden. It usefully discloses that no AWS credentials are needed. However, it does not mention side effects (e.g., whether loading overwrites previous state), error handling, or what happens if the path is invalid. The description adds some value but falls short of full transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences: purpose, key behavior (no credentials), and follow-up guidance. No redundancy, each sentence adds distinct value. Perfectly concise and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a single parameter and no output schema, the description addresses the tool's purpose, offline context, and post-load workflow. It lacks detail on return values or error conditions, but for a simple data-loading tool the coverage is mostly sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (the only parameter 'path' is fully described in the schema). The description adds no additional parameter-level detail beyond what the schema already provides. Baseline 3 is appropriate given the high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('load') and resource ('cached resources from a previously dumped directory'), clearly distinguishing from sibling tools like 'scan' (live scanning) and 'dump' (creating dumps). It explicitly mentions offline mode, which differentiates it from live operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: 'No AWS credentials needed' and suggests subsequent tools ('regions', 'detect', 'export'). It implies use when a dump directory exists, but doesn't explicitly state when NOT to use it or list alternatives for equivalent functionality.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• detectA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since there are no annotations, the description fully carries the behavioral burden. It discloses that the tool operates on a 'loaded graph', requires prior commands, and optionally exports to file. It does not describe error behavior or performance characteristics, but for a detection tool these are reasonable gaps. The file format details (.json or .md) are a nice touch.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with no wasted words. The prerequisite condition is front-loaded. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple detection tool with one optional parameter and no output schema, the description is nearly complete. It covers purpose, prerequisite, and output format. It could mention what happens if the graph is empty or what 'unused/orphaned' means exactly, but the title helps clarify.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with one well-described parameter. The description reiterates the purpose of outputPath (optional export vs. returning text). This matches the schema, so baseline 3 is appropriate. No additional semantic nuance beyond the schema is provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The title 'Detect Unused Resources' combined with description 'Detect structurally unused/orphaned resources in the loaded graph' provides a specific verb–resource pair. It clearly states the scope (loaded graph) and distinguishes from siblings like 'scan' (which loads data) and 'export' (which exports already-known data).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states a prerequisite: 'Requires scan or load to be called first.' This tells the agent when not to use this tool. However, it does not mention alternatives or when to choose this over checking resource status via 'status' or 'regions', leaving a slight gap.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scanA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries full burden for behavioral disclosure. It notes the operation is live and long-running, builds an in-memory graph, and hints at the data's ephemeral nature. It could mention if it modifies anything or requires specific permissions, but for a scan tool, this is strong coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The two sentences are informative and efficient, with no wasted words. The first sentence states the core purpose, and the second provides usage timing and lists sibling tools for subsequent steps. Perfectly front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple input schema (one flat param, no output schema), the description covers the essential context: what it does, that it's long-running, and what tools to use next. It could mention failure modes (e.g., invalid profile) but is sufficient for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the single parameter, so the description doesn't need to repeat parameter details. The description adds context by mentioning 'AWS profile' without duplicating schema info. Baseline 3 is appropriate as the schema fully documents the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Scan AWS resources live using an AWS profile' and explains it builds an in-memory graph. It distinguishes itself from siblings by explicitly listing what to do after scanning (regions, detect, export, dump), showing it's the initial data-gathering step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says this is a 'long-running operation (may take several minutes)', setting expectations. It tells the agent to use this first, then lists sibling tools for post-scan actions, providing clear when-to-use and next-step guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• statusA
  4.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but the description discloses it reports server state including resources, graph size, and available actions. No mention of return format, latency, or whether state could change between calls, but the read-only nature is implicitly clear. Reasonably transparent for a simple status tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, succinct and front-loaded with purpose, then usage advice. Every sentence adds value with zero waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given this is a simple zero-parameter status check with sibling context like 'scan' and 'load', the description fully covers what the agent needs to decide to call it first. No output schema exists but the description lists what will be returned (resource load state, graph size, actions).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters to document, schema coverage is 100%. Description adds no parameter info but also has no parameters needing explanation. Baseline 3 for no parameters applies; a slight bonus for mentioning what the tool checks (graph size, resources) which adds meaning beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description uses specific verb ('check') with a concrete resource (the server state: resources loaded, graph size, available actions). Clearly distinguishes itself from siblings like 'scan' or 'load' by establishing this as the initial status/introspection tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises calling this tool first to understand what tools are available, providing clear when-to-use guidance. Absence of when-not-to is acceptable given this is a no-parameter status check with no destructive side effects.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• exportA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the key behavioral trait: the tool requires prior calls to scan or load, and it supports three named formats. However, it does not describe side effects (e.g., does exporting modify the resource graph?), error conditions, or performance implications. That said, the lack of annotations means the description does not contradict any, so annotation_contradiction is false. A slight deduction for missing mutation details, but still strong.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three short sentences, each earning its place: first states the core action, second gives a critical prerequisite, third lists formats with specific tools. No filler, no redundancy. Front-loaded with the action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 2 parameters, no output schema, and no annotations, the description is fairly complete: it explains the tool's purpose, prerequisites, and expected outcomes (files). However, it lacks details about return values (e.g., success message, file path returned) and error handling. Still, for a focused export tool, this is adequate and surpasses a minimum viable score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, providing baseline expectations. The description adds value by clarifying the default output file paths for each format ('Defaults to graph.gexf, graph.json, or report.md'), which is not in the schema's parameter descriptions. The format enum is also repeated in the description with context (e.g., 'json for sigma.js viewer'), reinforcing meaning. This exceeds the baseline of 3 for high-coverage schemas.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Export the resource graph to a file' with a specific verb ('Export') and resource ('resource graph'). It also distinguishes the tool from siblings by listing supported formats (gexf, json, md), which other sibling tools like scan, load, detect, or dump do not handle. This makes the purpose unambiguous and sets it apart.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states a prerequisite: 'Requires scan or load to be called first.' This provides clear when-to-use guidance and implicitly tells the agent not to use this tool before those steps. It also lists supported formats, helping the agent choose based on desired output. No alternatives are needed as the purpose is singular.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dumpA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full disclosure burden. It clarifies that the tool writes to a directory, requires a prerequisite step, and produces output consumable by 'load'. While it doesn't detail disk space or error handling, the core behavioral traits are well-covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loads the core action, and includes essential usage context. Every sentence serves a purpose with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given low complexity (1 optional param, no nested objects, no output schema) and no annotations, the description is complete. It explains prerequisites, output usage, and default behavior, leaving no major gaps for an agent to misunderstand.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds value by explaining that 'path' is the output directory path with a default of 'dump', which is not fully detailed in the schema's description. The schema says 'Output directory path. Defaults to 'dump'.' which is already clear, so the description doesn't add much extra meaning beyond the schema, but the schema itself is already good.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to dump loaded resources to a directory for offline use. It uses a specific verb 'dump' and resource 'loaded resources', and distinguishes itself from siblings by referencing 'scan' or 'load' as prerequisites.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool: after 'scan' or 'load' has been called. It also explains the downstream use case ('offline analysis' with 'load'), effectively preventing misuse and clarifying the workflow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.