Server Quality Checklist

Tool Scores

• kafka_list_topicsA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description should fully disclose behavior. It mentions listing with details but omits potential side effects, authentication needs, performance implications, or handling of large topic lists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise two-line description followed by an Args section; no wasted words. Front-loaded with main functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given low complexity (one param, output schema exists), the description adequately covers functionality. Lacks comparison to siblings or mention of limitations, but output schema handles return values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter include_internal is explained in the Args section with the detail that internal topics start with '_', adding value beyond the schema's default and title. Schema coverage is 0%, so the description compensates well.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists all Kafka topics with partition count, replication factor, and descriptions, differentiating from siblings like kafka_topic_info which likely targets specific topics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus siblings (e.g., kafka_topic_info). No when-not-to-use or alternative recommendations; the description only explains the include_internal parameter.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kafka_topic_infoA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the burden of behavioral disclosure. It states it is a read operation (Get topic details) and includes specific health diagnosis logic, but does not explicitly confirm non-destructiveness or any additional behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is front-loaded with the core purpose, followed by organized health check details and parameter documentation. No wasted sentences, highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema (not shown but flagged), the description need not detail return values. It covers parameter, health diagnosis, and retention policy. Slight gap on error handling but adequate for this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 0% description coverage, but the description's Args section clearly explains the 'topic' parameter as 'Topic name to inspect', compensating for the schema gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it retrieves detailed topic information including partitions, replicas, ISR, configs, and health diagnosis, distinguishing it from sibling tools like kafka_broker_status (broker-level), kafka_consumer_lag (consumer lag), and kafka_list_topics (listing).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description implicitly guides usage by detailing what health checks are performed, but does not explicitly mention when to use this tool over siblings or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kafka_broker_statusA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses behavioral traits by detailing what it detects (missing brokers, cluster ID mismatch, under-replicated partitions), providing context beyond a simple health check. However, it does not explicitly state that the action is read-only or non-destructive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loaded with the main purpose, and uses bullet points for clarity. Every sentence provides value with no extraneous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and an output schema present, the description adequately covers what the tool does. It could be improved by noting the return format (e.g., list of issues or overall status), but the core detection capabilities are well described.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and 100% schema description coverage. The description adds significant meaning by explaining exactly what the tool checks, compensating for the lack of parameters. It provides context that enriches the tool's purpose beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool checks Kafka cluster health, specifying brokers, controller, and under-replicated partitions. It lists specific detection items like missing brokers and ClusterIdMismatch pattern, distinguishing it from sibling tools like kafka_consumer_lag and kafka_topic_info.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for cluster health monitoring but does not explicitly state when to use versus alternatives, nor does it provide 'when not to use' guidance. The distinction from siblings is implicit through the description's focus, but no direct usage advice is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kafka_consumer_lagA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It explains that the tool checks offsets and generates diagnosis based on incident patterns, which is helpful. However, it does not explicitly state that it is read-only, nor does it mention any side effects, rate limits, or error handling. This leaves some behavioral uncertainty.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a concise opening line followed by bullet points for diagnosis patterns. While it contains detailed incident-specific examples, each sentence adds value. It is not overly verbose but could be slightly tighter.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one optional parameter) and the presence of an output schema (not shown but indicated), the description provides sufficient context. It covers parameter usage and operational diagnosis, making it complete for an agent to understand and invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter 'group' has no description in the schema (0% coverage). The description adds crucial meaning: 'Consumer group ID. Empty = all groups.' This fully clarifies the parameter's purpose and default behavior, compensating for the schema gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to monitor consumer group lag and provide operational diagnosis. The verb 'Monitor' and resource 'consumer group lag' are specific, and the mention of 'diagnosis' adds clarity. It distinguishes from siblings like kafka_broker_status and kafka_topic_info by focusing on consumer lag.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly guides usage by detailing diagnosis patterns for different scenarios (Flink vs non-Flink groups). However, it lacks explicit guidance on when to use this tool over siblings or when not to use it. The context is clear but not exhaustive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.