Server Quality Checklist

Tool Scores

• averra_check_usageA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds valuable behavioral context beyond annotations: it explains that usage is counted per user across all API keys (not per key), cached requests count against quota, and counters reset at the start of each calendar month (UTC). While annotations cover safety (read-only, non-destructive, idempotent, open-world), the description provides operational details that help the agent understand quota mechanics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (description, args, returns, examples, error handling) and front-loaded key information. While comprehensive, some sections like the detailed return format examples could be slightly condensed, but overall it's efficient and organized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only quota checking tool with comprehensive annotations and a simple parameter schema, the description provides excellent contextual completeness. It includes purpose, usage guidelines, behavioral details, parameter documentation, return format examples, and error handling—all without needing an output schema since return values are clearly described.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the input schema already fully documents the single optional parameter (response_format with enum values and default). The description repeats this information in the 'Args' section but doesn't add significant semantic value beyond what's in the schema, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Check the current month's Extract API usage and remaining quota for the authenticated account.' It specifies the exact resource (Extract API usage/remaining quota) and timeframe (current month), distinguishing it from sibling tools like key management or extraction tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage examples with 'Use when:' statements, including scenarios like checking remaining quota, confirming quota before batch operations, and identifying the user's plan. It gives clear context for when to invoke this tool versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• averra_extract_urlA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds valuable behavioral context beyond what annotations provide: it explains the 7-day caching with cross-user sharing, mentions the monthly quota system, details the JavaScript execution via headless browser, and provides comprehensive error handling information (specific HTTP status codes with explanations). While annotations cover safety (readOnlyHint=true, destructiveHint=false, idempotentHint=true), the description enriches understanding of operational constraints and implementation details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, usage guidance, args, returns, examples, error handling) and most sentences earn their place. However, the error handling section is quite detailed with multiple specific status codes, making it somewhat lengthy. The information is valuable but could be more concise in presentation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with rich annotations and no output schema, the description provides excellent completeness: it explains the return format in detail for both JSON and Markdown outputs, covers quota implications, references sibling tools, provides concrete usage examples, and documents error scenarios. This gives the agent comprehensive understanding despite the lack of structured output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the input schema already documents both parameters thoroughly. The description adds minimal value beyond the schema: it mentions the URL format acceptance (bare hostnames get https:// auto-added) and clarifies the default response_format, but doesn't provide additional semantic context. This meets the baseline expectation when schema coverage is complete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Convert any webpage URL into clean, LLM-ready Markdown') and distinguishes it from alternatives by emphasizing it's for extracting actual content rather than just metadata or link previews. It explicitly names the technology (Averra Extract) and processing steps (fetching with JavaScript execution, Mozilla Readability cleanup, Markdown conversion).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use ('when you need the actual *content* of a webpage for an LLM') with concrete examples (blog posts, docs pages, articles, product pages) and when not to use ('just need a link preview or metadata' or 'need JSON extraction with a schema'). It also references the sibling tool averra_check_usage for quota management.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• averra_list_api_keysA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, destructiveHint=false, idempotentHint=true, and openWorldHint=true, covering safety and idempotency. The description adds valuable context beyond this: it specifies that plaintext keys are never returned (only prefixes), includes both active and revoked keys, orders by creation date, and notes that plan/limit are identical across keys. This enriches behavioral understanding without contradicting annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (description, Args, Returns, Examples, Error Handling), but includes verbose JSON output details that could be omitted if an output schema existed. Sentences are front-loaded and purposeful, though the Returns section is lengthy for a tool without an output schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 optional parameter), rich annotations, and no output schema, the description is complete: it covers purpose, usage, behavior, parameter, return format, examples, and error handling. The detailed JSON output compensates for the lack of output schema, ensuring the agent understands the response structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the schema fully documenting the optional 'response_format' parameter (enum, default, description). The description adds minimal value beyond the schema, only restating the parameter in the Args section without new semantics. Baseline 3 is appropriate as the schema handles the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('all API keys for the authenticated account'), and distinguishes it from siblings by specifying it returns metadata only (not plaintext keys), unlike averra_create_api_key (creates) or averra_revoke_api_key (revokes).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides explicit 'Use when' examples (e.g., 'How many API keys do I have?', 'Need to find a key's ID before revoking it'), which clearly indicate when to use this tool versus alternatives like averra_revoke_api_key (for revocation) or averra_check_usage (for usage data).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• averra_revoke_api_keyA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds significant behavioral context beyond the annotations. While annotations indicate destructive and idempotent hints, the description elaborates with safety warnings (e.g., cannot revoke the currently authenticating key, returns 400 error), idempotency details ('revoking an already-revoked key is safe'), and error handling specifics (400, 401, 404). This provides actionable insights not covered by annotations alone.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (e.g., Args, Returns, Examples, Error Handling) and front-loaded key information. While comprehensive, some sections like 'Error Handling' are detailed but necessary for clarity. It avoids redundancy and each sentence adds value, though it could be slightly more concise in parts.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (destructive operation with safety constraints), rich annotations, and lack of output schema, the description is highly complete. It covers purpose, usage, parameters, return values, examples, and error handling, providing all necessary context for an AI agent to invoke the tool correctly and handle edge cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the schema already documents parameters well. The description adds minimal extra semantics, such as referencing averra_list_api_keys for obtaining the ID and noting the default output format. However, it doesn't provide significant additional meaning beyond the schema, so it meets but doesn't exceed the baseline expectation for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('revoke') and resource ('Extract API key by its ID'), distinguishing it from sibling tools like averra_create_api_key and averra_list_api_keys. It explicitly mentions the immediate deactivation and irreversibility, which clarifies the nature of the operation beyond just the name.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool (e.g., 'User explicitly asks to revoke a specific key' and 'Rotating keys — after confirming the new key works') and when not to use it ('Don't use without user confirmation — this invalidates a credential'). It also references sibling tools like averra_list_api_keys for obtaining the key ID, offering clear alternatives and context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• averra_create_api_keyA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds significant behavioral context beyond annotations: it discloses that the plaintext key is returned only once and cannot be retrieved later, explains that the key inherits plan/limits automatically from the subscription, and includes error handling details (401, 500). These are not covered by the annotations, which only indicate it's a non-readOnly, non-destructive, non-idempotent, openWorld operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (description, important notes, args, returns, examples, error handling), front-loaded with critical information, and every sentence adds value without redundancy. It efficiently communicates necessary details in an organized manner.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (creating a sensitive API key), the description is complete: it explains the tool's purpose, critical behavioral warnings, parameter usage, return format details (compensating for no output schema), usage examples, and error handling. This provides all necessary context for an agent to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage for the single parameter 'response_format', the schema already fully documents it. The description adds minimal value by briefly mentioning the parameter in the 'Args' section but does not provide additional semantic context beyond what's in the schema. However, since there's only one parameter and the schema is comprehensive, a score above baseline is warranted.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Create a new Extract API key') and resource ('for the authenticated account'), distinguishing it from siblings like 'averra_list_api_keys' and 'averra_revoke_api_key'. It precisely defines the tool's function without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool (e.g., 'User wants to create a new API key for a different integration' and 'Rotating keys'), when not to use it ('Don't use without user confirmation'), and mentions alternatives implicitly through sibling tool names. It clearly defines the context and exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.