Server Quality Checklist

Tool Scores

• crm_pipeline_summaryA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full behavioral disclosure burden. It states 'Live distribution' and 'Reflects outreach progress', implying a read-only query, but it does not explicitly confirm idempotency, safety, or data freshness. The behavioral traits are hinted but not fully transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, each sentence serving a distinct purpose: the first explains the core functionality, the second adds contextual value about the tool's insight. No word is wasted.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and moderate complexity, the description covers the main functionality but omits details about the return format (e.g., counts, proportions). It provides sufficient context for a simple aggregation tool but leaves ambiguity about output structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% coverage with a clear description for the only parameter ('state'). The description repeats the optional scoping but adds no new semantic meaning, format guidance, or behavioral constraints beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool provides a live distribution of operators across clearly enumerated CRM stages (NEW/CONTACTED/REPLIED/DEMO/CLOSED/DEAD), with optional state scoping. This specific verb-resource combination clearly distinguishes it from sibling tools like compare_operators or market_summary.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for understanding outreach progress, but it does not explicitly state when to use this tool versus alternatives, nor does it provide exclusions or prerequisites. Some guidance is inferred from the stage list, but no direct comparison is offered.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• competitive_landscapeA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It enumerates output elements but does not mention data source, freshness, authentication needs, or any side effects. The description is transparent about content but not about execution behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, dense sentence that front-loads the core purpose and lists specific output components. No wasted words; every element adds value. It is both concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema), the description provides a comprehensive list of output metrics. It is almost self-contained, though it could mention data limitations or interpretative guidance for the HHI.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'state' is fully described in the schema (100% coverage), and the description adds no additional semantic context beyond what the schema provides. Baseline of 3 is appropriate as the description does not need to compensate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the tool's purpose: analyzing the competitive structure of a state's Part 135 market. It lists specific metrics (operator count, fleet-size distribution, concentration indices, HHI interpretation, leaders), distinguishing it from sibling tools like compare_operators or fleet_ranking.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for competitive analysis, but lacks explicit guidance on when to use this tool versus alternatives. No conditions, prerequisites, or exclusions are provided, so the agent must infer context from the tool name and sibling list.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• market_summaryA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, description carries full burden. It discloses 'Computed live over all operators in the state,' indicating real-time computation and scope. Does not mention destructive actions or authentication, but for a read-only summary, this is reasonable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence packs many details (9 metrics) efficiently, but could be slightly dense. Front-loads key verb 'summary.' No wasted words, earning a 4.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, description lists all included metrics (operator count, tails, avg/median fleet, largest operator, top-5 tail share, HHI, top operators) and states live computation. This is comprehensive for the tool's purpose.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so parameters are well-documented. Description adds no extra meaning beyond schema: 'state' and 'topN' are already clear from schema descriptions. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it provides an analyst summary of a state's Part 135 charter market, listing specific metrics (operator count, total tails, etc.). It distinguishes from siblings like compare_operators (which compares operators) and fleet_ranking (which ranks fleets).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description implies usage for summary of single state's market but does not explicitly state when to use this tool over siblings or provide exclusions. No guidance on when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fleet_rankingA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavioral traits. It identifies the ranking metric (FAA tail count) but omits details like sorting order (likely descending), data freshness, or that it is a read-only operation. This is adequate but not thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, zero waste. The first sentence introduces the primary purpose, and the second adds key modifier. Efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description should hint at return structure. It mentions operators ranked by fleet size but does not specify whether the actual tail count is included, or the sorting order. For a ranking tool with only two parameters, this is a notable gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds practical context: state is optional and defaults to whole US, limit has default (20) and max (100). This goes beyond the schema's bare parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns operators ranked by fleet size (FAA tail count), with optional state scoping. This verb+resource structure is specific and distinguishes it from sibling tools like compare_operators or market_summary.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit context on when to use state parameter vs omitting for entire US universe, and mentions total operator count. Does not explicitly list exclusions or alternatives, but the guidance is clear and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_operatorsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the comparison includes specific fields and outputs which operator is largest and whether they share a state. It also notes client-side matching. However, it does not mention error handling, rate limits, or data freshness, leaving some behavioral aspects unspecified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two sentences, no wasted words. The first sentence states the core purpose and output, and the second provides a critical implementation detail. Every sentence adds value, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity of the tool (single parameter, no output schema, no annotations), the description covers the essential information: inputs, output highlights (largest, share state), and a key implementation detail. It is sufficient for an agent to decide when to use the tool and what to provide, though it could mention potential errors if a name is not found.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already describes the 'names' parameter with 100% coverage. The description adds meaningful context beyond the schema by stating that names are matched client-side and that at least 2 names are required, and it outlines the comparison outputs, which helps the agent understand what values to provide and what results to expect.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states it performs a side-by-side comparison of 2+ named operators, listing specific fields (state, fleet size, certNo, CRM status) and outputs (largest, share state). This clearly distinguishes it from sibling tools like competitive_landscape or market_summary, which focus on broader analyses rather than direct operator comparison.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly indicates when to use the tool: when comparing 2+ named operators. It also gives a key constraint: names are matched client-side (no server text search), guiding the user on input format. However, it does not explicitly provide exclusions or suggest alternatives for different use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.