Server Quality Checklist

Tool Scores

• agy_list_jobsB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint=true and idempotentHint=true, so the safety profile is covered. The description adds the scope of 'running and recently finished', but this conflicts somewhat with the lifecycle parameter which allows other states like 'queued' and 'canceling' – the description implies a fixed subset when the tool actually supports filtering. No contradiction with annotations, but behavioral details like default ordering or limits are absent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no filler words, making it efficient and easy to parse. It front-loads the primary action and scope. Minor deduction for not using the available sentence to clarify the ambiguous 'recently finished'.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema and 5 parameters, the description is too sparse to fully specify behavior. It doesn't mention default limit, result ordering, whether the response is a list of job summaries or details, or how 'recently finished' is computed. Sibling tools like agy_cancel suggest mutation, but this list tool's interaction with them is not addressed. The annotations cover read-only, but contextual completeness for correct invocation is lacking.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 60%, with 3 of 5 parameters described. The description does not add meaning beyond the schema; it only vaguely aligns with the lifecycle parameter. limit and session_id have no schema descriptions, and the tool description doesn't compensate for these gaps. Baseline 3 due to moderate schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and resource ('jobs') and scopes to 'running and recently finished', making the core purpose clear. However, it does not differentiate from sibling tools like agy_result or agy_logs, and the meaning of 'recently finished' is ambiguous without a time frame.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given about when to use this tool versus alternatives such as agy_cancel or agy_wait. The description only states what it lists, leaving the agent to infer when this is the right tool. There is no mention of exclusions or conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agy_sessionsB
  3.1/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description claims the tool can 'close' sessions, which is a mutating operation. However, annotations include readOnlyHint=true, indicating the operation should be read-only. This is a direct contradiction, making the behavioral information misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded: the first sentence lists the actions and resource, and the second clarifies domain vocabulary. No unnecessary words or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core actions but leaves parameter semantics incomplete, especially for limit and state. It also conflicts with the readOnly annotation, undermining reliability. Since there is no output schema, the description should clarify return behavior, but it does not. The tool is not fully specifiable from this description alone.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 50%, and the tool description fails to compensate. It adds no meaning to 'limit' or 'state', which remain undocumented, and merely echoes 'action' and 'session_id' without additional insight. An agent would not know what values to pass for those parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's actions ('List, inspect, or close') on a specific resource (agy conversations/sessions). It also distinguishes sessions from jobs, helping disambiguate from sibling tools like agy_list_jobs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a useful context cue by defining sessions vs. jobs, but it does not explicitly state when to use this tool instead of siblings or when not to use it. Alternatives and exclusions are absent, so usage is only implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agy_cancelA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide idempotentHint and destructiveHint. The description adds concrete behavioral detail by specifying that it kills the entire process group, which goes beyond the generic destructive hint and clarifies the blast radius.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence that conveys the core action and scope without any redundant words. It is highly efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simple nature, the description is adequate but incomplete. It lacks usage context, parameter explanations for job_id and reason, and any indication of return behavior, but the destructive/idempotent hints and process-group detail provide some context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only grace_ms has a schema description; job_id and reason rely on name inference. The tool description does not explain any parameter meanings or relationships, and with only 33% schema coverage, this leaves important gaps for the agent.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Kill' with the resource 'a running job' and clarifies the scope as 'its whole process group.' This clearly distinguishes it from siblings like agy_start and agy_wait, making the tool's purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not state when to use this tool versus alternatives, nor does it mention any exclusions or prerequisites. Usage must be inferred from the word 'running,' but no explicit guidance or sibling comparison is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agy_logsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and idempotentHint=true, so the safety profile is covered. The description adds useful behavioral context (stream types, byte cursor/tail modes) but doesn't disclose defaults (e.g., normalized stream), pagination behavior beyond cursor semantics, or response structure. This is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence conveys the core purpose, stream options, and access methods with zero redundancy. Every word contributes information, and the most important distinctions are front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity, no output schema, and annotations covering readonly/idempotent behavior, the description provides the essential information needed to understand what the tool does. Minor gaps like default stream selection are already captured in the schema's stream description. Overall, it is sufficiently complete for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 60% (stream, tail_lines, after_cursor have descriptions) while job_id and max_bytes lack descriptions. The description loosely maps to parameters via 'byte cursor or tail' and lists stream types, but adds no significant meaning beyond what the schema already provides for covered parameters and doesn't compensate for undocumented ones.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Read' with a clear resource 'raw events, normalized human-readable lines, or stderr for a job'. It distinguishes itself from sibling tools like agy_send or agy_cancel by focusing on reading logs, and mentions distinct stream types and access methods.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like agy_result or agy_list_jobs. There is no mention of exclusions or prerequisites; the description only states capabilities, leaving the agent to infer appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agy_sendA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide only a title, so the description carries the full burden. It discloses key behavioral constraints: the queueing nature, that it takes effect only after the current turn finishes, and that there is no way to interrupt a running turn. This goes beyond typical schema details and informs the agent about timing and irreversibility of interruption. It does not mention whether the action is reversible or if it has side effects on job state, but the disclosure is sufficient for safe invocation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences of clear, front-loaded prose. It states the core action first, then provides the key constraint without any fluff or redundant details. Every word contributes to understanding the tool's behavior, making it an exemplar of conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with two optional parameters and one required parameter, the description covers the essential aspects: the action, the scope (session-mode job), and the timing/limitation. It does not explain the expected result or return value, but there is no output schema and the operation is simple enough that an agent can infer the outcome. The description is complete enough for correct invocation in most scenarios.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 67% (two of three parameters explicitly described). The description does not add meaning beyond what the schema already provides for `text` and `close`, and `job_id` remains undocumented. However, the description's mention of 'session-mode job' gives context that the job must be in session mode, which is useful. With moderate coverage, this is an adequate score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies the action ('Queue a follow-up turn') and the resource ('a session-mode job'), clearly distinguishing it from siblings like agy_cancel (which interrupts) and agy_start (which starts). The verb-noun combination is unambiguous and leaves no room for confusion with the other tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states that it applies to session-mode jobs and that it only takes effect after the in-flight turn finishes, which implies when it should be used. However, it does not explicitly name alternatives or state when not to use this tool (e.g., if you need to cancel or interrupt a running turn). Given that siblings exist for cancellation and waiting, a clearer directive would elevate this to a 4.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agy_capabilitiesA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and idempotentHint=true, covering the safety profile. The description adds no extra behavioral context beyond listing the report contents, which is minimal but not contradictory.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, compact sentence that immediately states the tool's purpose and output contents. Every word earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter read-only tool, the description fully specifies what information will be returned. No output schema exists, but the listed items (models, profiles, limits, project root, version) are sufficient for an agent to know what to expect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, so the baseline for this dimension is 4. The description does not need to elaborate on parameter meanings since none exist.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reports specific items (models, profiles, limits, discovered project root, server version) with a precise verb 'Report'. This differentiates it from sibling tools like agy_logs or agy_send, which handle other concerns.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is used to retrieve server capability information but does not explicitly state when to use it versus alternatives. Since it is a zero-parameter read-only capability query, usage context is intuitive but not explicitly articulated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agy_waitA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations only provide readOnlyHint and idempotentHint. The description adds substantial behavioral detail: internal polling at 200ms, no early return on intermediate transitions, wait_ms=0 as an immediate snapshot, how after_cursor applies to the log tail, and that only the judgement packet is returned. This far exceeds what annotations convey.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but every sentence carries operational information: blocking behavior, polling interval, wait_ms=0 semantics, return content scope. It is a bit long, but it avoids redundancy and front-loads the most critical behavioral caveat (does not return early on intermediate transitions) early. Slight over-length keeps it from a 5.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no output schema, the description adequately explains the return payload ('judgement packet only, not full logs or the response text'). Combined with annotations covering safety, the description is sufficient for an agent to invoke it correctly. It lacks explicit mention of error handling or edge cases (e.g., job not found), but those are not essential for basic usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 67% schema coverage, the description compensates by explaining the behavioral semantics of wait_ms (blocking budget, 0 returns immediate snapshot) and after_cursor (byte offset applied to in-progress log tail, finished job returns full packet). This adds meaning beyond the schema descriptions, though job_id is not described beyond its presence in the required field.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool's purpose: 'Long-poll a job until it FINISHES or wait_ms runs out'. It specifies a precise verb (long-poll), a resource (job), and the termination condition. It also differentiates from siblings by noting it returns only the judgement packet, not logs or response text, helping distinguish from agy_result and agy_logs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains the blocking behavior and the meaning of wait_ms, which implies when to use it (e.g., to wait for a job to finish). However, it does not explicitly state when to use this tool versus alternatives like agy_result or agy_logs, or when not to use it. The usage context is present but left to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agy_resultA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint, so the safety profile is known. The description adds value by disclosing the paged nature, the sections available, and the structure of blockers (source/actionable/remedy), which goes beyond simple read-only hints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One compact sentence packed with useful information: it states the resource type, the paged nature, the three main components, and the blocker detail structure. No filler words; every phrase earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the parameter schema and annotations cover safety and parameter meanings, the description adds the essential conceptual model (what each section contains) and confirms paging capability. It doesn't explicitly mention return format (no output schema exists), but the description's breakdown of sections gives a good picture. It's slightly less complete because it doesn't state default section behavior, but the schema's property description already says 'Defaults to summary.'

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 75%, with limit, offset, and section having descriptions. The description adds meaning by explaining the 'response' section is paged and that blockers have specific fields, which helps understand the section parameter's impact. It doesn't fully compensate for the undocumented job_id parameter, but that parameter's meaning is obvious from context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns the full, paged result of a finished job, enumerating specific components: broker summary, agent self-report, and verification with blockers. This specific verb and resource distinguishes it from siblings like agy_list_jobs (listing jobs) and agy_logs (logs).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for finished jobs ('finished job'), which sets context for when to use it, but it doesn't explicitly contrast with siblings or state when not to use it (e.g., when job is still running, use agy_wait). The 'finished job' qualifier provides some exclusion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agy_startA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses key behaviors beyond the openWorldHint annotation: it returns job_id immediately, never blocks, and includes policy_summary and blockers[] in the response. It also explains that dry_run avoids spawning and quota usage. This gives the agent a clear model of the tool's execution and side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, each carrying essential information: purpose, behavior, and dry_run alternative. It is front-loaded with the core action and avoids any filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (17 parameters, no output schema), the description provides crucial return-value context (job_id, policy_summary, blockers[]) that would otherwise be unknown. It also clarifies the non-blocking nature and cost behavior, covering the main aspects an agent needs for correct invocation and expectation setting.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 71%, so the schema already documents most parameters. The description adds general context about the job start flow but does not elaborate on the undocumented parameters (e.g., mode, model, requested_by, parent_task_id). It does not significantly augment the schema descriptions, so a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description begins with a specific verb and resource: 'Begin a new agy job.' It clearly differentiates this from sibling tools by noting immediate return and non-blocking behavior, and it contrasts with dry_run. This makes the tool's purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance on when to use dry_run versus actually starting a job, including the cost/quota implication. It implies the primary use case is starting a new job, but it does not explicitly contrast with continuing via sibling tools like agy_send or when to use session_id. Still, it provides sufficient context for typical invocation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.