Server Quality Checklist

Tool Scores

• fastly_cliA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key traits: it's a command execution tool with automatic authentication handled server-side, security warnings against passing API keys, and examples of common commands. It doesn't cover all potential behaviors like error handling or output format, but provides substantial context beyond basic purpose.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, usage examples, common commands, security note) and front-loads the core purpose. It's appropriately sized for a CLI tool, though the 'COMMON COMMANDS' section could be slightly more concise as it lists commands already implied by examples.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (CLI command execution with security considerations), no annotations, and no output schema, the description does a good job of providing necessary context. It covers purpose, usage, examples, security, and common commands. However, it doesn't describe what the tool returns (output format/behavior), which is a gap since there's no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, clearly documenting both parameters. The description adds minimal parameter semantics beyond the schema—it implies the 'command' parameter should exclude the 'fastly' prefix through examples, but doesn't explicitly state this. With high schema coverage, the baseline score of 3 is appropriate as the schema does most of the work.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Execute Fastly CLI commands securely without exposing API keys.' It specifies the exact action (execute CLI commands) and resource (Fastly CLI), and distinguishes it from the sibling 'fastly_api' tool by focusing on CLI execution rather than API calls.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (running Fastly CLI commands with automatic authentication) and includes usage examples and common commands. However, it does not explicitly state when NOT to use it or directly compare it to the 'fastly_api' sibling tool, which would be needed for a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fastly_apiA
  4.2/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure and excels at this. It explains authentication handling ('Authentication is handled automatically'), response handling ('summarize the results between calls'), status code interpretation ('Always check status codes'), and important constraints ('Base URL is automatically added', 'Full URL paths aren't needed'). It also provides detailed guidance about what the LLM should do before/after calls.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is excessively long (over 500 words) with multiple sections that could be streamlined. While the front-loaded 'IMPORTANT USAGE NOTES' is well-structured, the later sections on 'Creating Fastly Compute@Edge Sites' and 'COMMON PITFALLS' contain information that belongs in documentation rather than a tool description. Many sentences don't directly help the agent select/invoke the tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a generic API tool with 4 parameters and no output schema, the description provides substantial context about usage patterns, common endpoints, authentication, and response handling. It covers most aspects needed for effective use, though it could benefit from more detail about error responses or rate limits. The absence of an output schema is partially compensated by guidance on interpreting status codes and summarizing results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds significant value beyond the schema by providing concrete examples of paths ('/service', '/service/{service_id}/purge_all'), explaining how parameters work ('with params: service_id, from, to'), and clarifying that the body is 'JSON-encoded automatically'. However, it doesn't fully explain all parameter nuances like how 'params' object maps to URL parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Make requests to the Fastly API' with access to 'all endpoints', which is specific about the verb (make requests) and resource (Fastly API). It distinguishes from the sibling tool 'fastly_cli' by focusing on API calls rather than CLI commands. However, it doesn't explicitly contrast with the sibling tool beyond mentioning CLI in the usage notes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides extensive usage guidance including explicit when-to-use scenarios (e.g., 'When making multiple API calls', 'To create a service from scratch'), alternatives (CLI for local build/deployment), and exclusions (e.g., 'NEVER attempt to extract or use the user's API key directly'). The 'IMPORTANT USAGE NOTES' section offers comprehensive context for when and how to use this tool versus other approaches.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.