Server Quality Checklist

Tool Scores

• agy_askA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate read/write and open world. The description adds key behaviors: non-interactive mode, use of --dangerously-skip-permissions, auto-approval of tool actions, and failure modes (binary not found, timeout). This exceeds annotation information.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: first sentence states purpose, then bullet arguments, return format, and a crucial note. Every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 8 parameters and no output schema, the description covers all parameters, return format, and error handling. However, lacking details on how to obtain project IDs or the exact workspace behavior limits completeness slightly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions. The description's Args section adds context (e.g., 'absolute dirs to add to agy's workspace'), improving understanding beyond the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool sends a prompt to the agy CLI and returns its response, and it distinguishes this from sibling tools (auth, list models) by specifying use cases like second opinion, delegation, and conversation continuation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly mentions when to use (second opinion, delegate task, continue conversation) and implies caution due to auto-approval, but does not explicitly state when not to use or list alternatives beyond siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agy_loginA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Goes well beyond annotations by detailing side effects: opens terminal, runs agy, drives OAuth, polls account store, and reports signed-in email. Explains return fields (logged_in, email, already, message) and timeout behavior. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is well-structured: begins with purpose, then process, args, returns, platform note. Each sentence adds value; no redundancy. Could be slightly more concise, but overall efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Completely covers all essential aspects: authentication flow, platform limitation, timeout handling, return format, and fallback instruction. Without an output schema, the description fully explains return values. No gaps for a single-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema already fully describes timeout_ms with min, max, default, and description. Description restates the range and adds context 'how long to wait for login to complete', but adds minimal new information beyond what schema provides. With 100% schema coverage, baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states that the tool ensures agy is logged in, explaining the OAuth flow and distinguishing from sibling tool agy_auth_status which checks status. The verb 'log in' and resource 'agy' are specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context on when to use (to ensure login), what happens if already authenticated (returns immediately), and instructions for timeout scenario (finish login and call agy_auth_status). Also notes macOS exclusivity and alternative for other platforms. Lacks explicit 'when not to use' statement but overall guidance is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agy_list_modelsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, openWorldHint, idempotentHint, destructiveHint=false. Description adds value by revealing the underlying CLI command, return format (JSON with models array), and example output, which are beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three clear, front-loaded sentences. No redundancy or unnecessary details. Every sentence adds value (purpose, usage, return format).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, description includes return format (JSON with models array and example). Combined with rich annotations, it fully informs an agent about purpose, usage, safety, and output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters, schema coverage is 100%. Description states 'Takes no arguments', which is sufficient. Baseline for 0 params is 4, and description adds no extra param info needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Explicitly states 'list the models available to the agy CLI' with the command `agy models`. Clearly differentiates from siblings (agy_ask, agy_auth_status, agy_login) by focusing on model listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use before agy_ask to discover valid values for its model argument', providing clear when-to-use and linking to a sibling tool. Also states it takes no arguments.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agy_auth_statusA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations (readOnlyHint, idempotentHint), description adds that it reads local account store and never returns auth token, providing security-relevant behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is detailed and includes JSON structure, but front-loads purpose well. Could be slightly more concise but still efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 0 parameters and no output schema, description provides full return schema and usage context, including security note. Complete for a status-check tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist (schema coverage 100%), baseline 4 applies; no additional parameter info needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it reports login status and per-model quota, distinguishes from siblings by specifying use before agy_ask and referencing agy_login for alternative when not logged in.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says when to use (before agy_ask) and when to call alternative (agy_login if logged_in false). Also implies not to use for token retrieval.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.