Server Quality Checklist

Tool Scores

• get_session_summaryB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; description only states it returns a summary. Fails to disclose any side effects, auth requirements, or error behavior for missing sessions, which is needed for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single, front-loaded sentence clearly stating purpose with no redundant information. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple tool (one param, output schema exists), description is adequate but could briefly mention the output format or content to provide fuller context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema already fully describes the parameter (session_id) with default behavior. Description adds no extra meaning beyond what the schema provides, resulting in baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool gets a formatted summary for documentation purposes. It uses specific verbs and resources but does not explicitly differentiate from siblings like get_session_stats.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implied usage for copying summaries into Jira/PRs, but no explicit when-to-use or when-not-to-use guidance, nor mention of alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_session_statsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It clearly indicates idempotent reading via 'Get' and lists returned fields. It does not disclose potential side effects (unlikely) or authorization requirements. The active session fallback is described in the parameter schema but not in the description itself.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: a single sentence with a bulleted list of return values. No extraneous text, immediate purpose statement, and structured enumeration of outputs.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read tool with one optional parameter and an output schema, the description covers the primary purpose and all return fields. It does not detail the optional parameter's default behavior (covered by schema) or compare to siblings, but completeness is adequate given low complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides 100% coverage for the single parameter session_id, describing its dual type, default, and behavior ('Uses active session if not provided'). The tool description only restates 'chat session' without adding semantic nuance. Baseline 3 is appropriate given schema completeness.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with a clear verb ('Get') and resource ('current metrics for a chat session'), and the bulleted list of return values specifies exactly what is retrieved. This distinguishes it from siblings like get_session_summary (likely a text summary) and get_plan_usage (different resource).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states the tool's purpose but does not explicitly say when to use it over alternatives like get_session_summary. There is no when-not or conditional guidance, only a general statement of what it does.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_plan_usageA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses what is returned (plan limit, credits, etc.) and that it is a read operation, but does not explicitly state it is read-only or mention any other behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the main action. No redundant words, clearly structured with purpose followed by output details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity (no parameters, output schema exists), the description is largely complete. It covers what the tool returns and its scope, though it could mention any rate limits or authentication needs.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no parameters, so schema coverage is 100%. The description adds meaning by detailing the output and clarifying the scope (current billing cycle, not session-specific), which enhances understanding beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves overall Kiro plan usage for the current billing cycle, with specific output details. It distinguishes from sibling tools by noting it is not session-specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for full-cycle billing data and contrasts with session-specific views, but does not explicitly state when to use it over alternatives or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• log_interactionA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that omitting thinking_seconds records a zero-duration ping for timestamp updates. This adds meaningful behavioral context beyond the schema, though it doesn't discuss idempotency or data persistence.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long and front-loaded with the main purpose. Every sentence adds value with no wasted words. It is highly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 optional parameters, no required fields) and the presence of an output schema, the description adequately covers the tool's behavior and usage. It is complete enough for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both parameters described. The description adds value by explaining the effect of providing or omitting thinking_seconds, which is not apparent from the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Log an agent interaction (turn) within the session.' It uses a specific verb ('Log') and resource ('agent interaction'), and the context of sibling tools (get_plan_usage, get_session_stats, etc.) distinguishes it well.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Call this after each agent turn' and explains the fallback behavior when thinking_seconds is omitted. While it does not specify when not to use it or name alternatives, the context is clear and sufficient for a logging tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_sessionA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It discloses that the tool captures a snapshot of plan credits and wall-clock time, and returns session_id and starting credit balance. It does not mention error conditions or idempotency, but for a simple init tool, this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise at three sentences. The first states purpose, the second provides usage guidance, and the third explains behavior and return values. Every sentence earns its place with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, output schema exists), the description adequately covers purpose, usage, behavior, and return values. It might benefit from noting idempotency or error handling, but overall it is sufficiently complete for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and schema coverage is 100% (trivially). The description adds value by explaining why the tool exists (to enable delta calculations for get_session_stats), which goes beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Start tracking a new chat session.' It uses a specific verb and resource, and explicitly distinguishes the tool from siblings by noting it should be called 'ONCE at the beginning of every chat session,' setting it apart from query tools like get_session_stats.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context: 'Call this ONCE at the beginning of every chat session.' It explains why (to capture snapshots for later deltas) but does not explicitly clarify when not to use it or compare directly to siblings. However, the context is sufficient for an agent to understand its role.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.