Server Quality Checklist

Tool Scores

• find_compliant_flightsA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description provides good behavioral context: it performs a search, applies policy filtering, and returns a summary. However, it does not disclose potential rate limits, authentication needs, or behavior if the policy file is missing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single informative sentence that packs key details. It is front-loaded but could be improved with structured bullet points. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and 7 parameters, the description covers core functionality but lacks return value structure details. It mentions a summary of rejections but no specifics. Sibling differentiation is absent, leaving some gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 57% (4 of 7 parameters have descriptions). The description itself does not add significant parameter details beyond the schema, such as mentioning policyPath. Baseline 3 is appropriate as schema does moderate work.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it searches Google Flights, evaluates against a policy file, returns only compliant options with a rejection summary, sorted by price. This specific verb+resource+output distinguishes it from siblings like search_flights.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when policy compliance is needed, but lacks explicit when-to-use or when-not-to-use guidance. Siblings like search_flights and check_policy suggest alternatives, but no comparison is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_flightsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes basic behavior (search, return options) and limitations (no policy/booking). With no annotations, it should disclose more (e.g., read-only, API dependencies).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose, no unnecessary words. Efficient and clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Mentions return structure (airline, times, stops, price) since no output schema. Lacks pagination or error details, but adequate for a search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 67% parameter descriptions. The description adds no extra parameter guidance; it only mentions output fields. Baseline score applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it searches Google Flights and returns structured flight options. Distinct from sibling tools like check_policy and start_booking.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says it does not check policy or book anything, clarifying its scope. However, it doesn't explicitly direct to find_compliant_flights for policy-compliant searches.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• close_browserA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It states the tool closes the browser, which is simple and expected. However, it does not mention error handling if no browser is open, nor any side effects like invalidating handles.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with a brief usage instruction, highly concise and front-loaded with the action. Every word is necessary.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters, no output schema, and no annotations, the description covers the basic purpose and usage. However, it lacks details about error states or idempotency, which would be useful for a complete understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters and 100% coverage, so the description does not need to add parameter details. The baseline is 4, and no additional info is required.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Closes the Playwright browser') and the resource ('opened by this server'). It distinguishes itself from siblings like start_booking by implying it is the teardown step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Call when done with a booking session,' providing clear context. While it doesn't mention when not to use it, the simplicity of the tool makes this sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_policyA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully convey behavior. It states what the tool returns but does not specify side effects (likely none), permissions, or rate limits. It adds some behavioral context beyond schema but could be more explicit about non-modification.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, well-structured sentence that front-loads the key purpose and outputs. No redundancy or unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description adequately lists the main return values (compliance, violations, approval). It does not detail error conditions or return format, but it covers the essential outputs for an intermediate-complexity tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and the description adds value by clarifying that flightId must come from the last search_flights call. The default policyPath is also noted in the schema, but the description reinforces it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it evaluates a specific flight from search_flights against a policy, returning compliance/violations/approval status. It distinguishes itself from find_compliant_flights and other siblings by focusing on a single flight vs. policy compliance.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly ties the flight ID to the last search_flights call, giving clear context for when to use. It does not provide explicit when-not-to-use or alternatives, but the prerequisite is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_bookingA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully covers behavior: it never enters payment info, re-validates policy, best-effort fills details, and returns a report. It also explains the out-of-policy refusal condition.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, front-loaded with the core action, no wasted words. Each sentence adds critical information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's workflow, return value (report with URL/screenshot), and policy interaction. Without an output schema, it provides sufficient expectations for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds context beyond the schema for key parameters like fareChoice (explains fare tile selection) and acknowledgeOutOfPolicy (explains its role in overriding policy refusal). For traveler and flightId, it supplements the schema by describing their roles.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool re-validates the flight, fills traveler details, and stops before payment. This differentiates it from sibling tools like check_policy or search_flights, which handle specific sub-tasks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use the tool (after selecting a flight, before payment) and conditions for refusal (out-of-policy without acknowledgeOutOfPolicy). It does not explicitly mention alternatives but implies that a human must take over after this step.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.