Server Quality Checklist

Tool Scores

• transit_find_stopsA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint=false. The description adds meaningful behavior: server-side filtering (no quota savings), unpaginated response from 511, and the 60-request/hour rate limit. It also clarifies that stop codes are operator-specific—valuable context beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer but well-structured with clear sections (Args, Returns, Examples, Error Handling). Each section contributes distinct value: the main purpose is front-loaded, examples are concise, and error handling is pertinent. No fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is fully self-contained: it includes the return object structure, examples mapping to actual operator IDs, error-handling notes, rate limits, and the relationship to sibling tools. With an output schema present and annotations covering safety, nothing an agent needs to invoke correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description's Args section largely repeats schema descriptions (e.g., 'operator_id' from transit_list_operators, 'query' substring match and 'strongly recommended'). It adds no new per-parameter semantics; the 'strongly recommended' phrasing already appears in the schema. The unpaginated response note is behavioral, not parameter-specific.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool's purpose: 'Find an agency's stops by name, and get the stop codes the real-time tools need.' It explicitly ties the tool to transit_next_departures, distinguishing it from other transitive tools. The purpose is specific and actionable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage context: 'This is the bridge between "Downtown Berkeley" and the code transit_next_departures wants.' It also states a clear when-not: 'Don't use when: you already have the stop code.' Examples illustrate typical queries, making the intended conditions unambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transit_list_linesA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already declare readOnly, idempotent, and non-destructive hints. The description goes further by disclosing the response shape, the truncation flag, the possibility of many results for large agencies, and the 60-request-per-hour rate limit across all endpoints. This adds behavioral context beyond the annotations and contradicts nothing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-organized with clear sections (Summary, Args, Returns, Examples, Error Handling). It front-loads the purpose and usage, and every sentence adds value—examples, exclusions, rate limits, and return format. Despite covering many aspects, it remains concise and skimmable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 4 parameters, an output schema, and sits among 5 siblings. The description covers all essentials: purpose, parameter examples, return structure, error handling, and even the rate limit. There is nothing an agent needs to know to call this correctly that is missing, and the output schema already documents the return shape.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers all parameters with detailed descriptions (100% coverage), so the description does not need to duplicate that. It adds a few examples (operator_id='BA', 'SF') and clarifies that query is a case-insensitive substring, but these are minor enhancements. The baseline of 3 is appropriate because the schema already carries the main semantic load.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('List') and resource ('an agency's routes'), and immediately explains its practical use (resolving a line name into an id, answering route questions). It explicitly distinguishes itself from a sibling tool ('Don't use when: you want stops on a line (use transit_find_stops)'), which is a clear differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit 'Use this to...' guidance and an explicit 'Don't use when...' with a named alternative. Examples illustrate common queries, and the error handling section advises when to prefer query narrowing over raising limit. This leaves no ambiguity about when to invoke this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transit_list_vehiclesA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already signal read-only, idempotent, and open-world behavior. The description adds valuable, non-obvious behavioral details: coordinates arrive as strings and may be empty, agencies with no real-time feed return nothing (with a pointer to check 'Monitored'), and a 60-requests-per-hour rate limit. This goes well beyond the annotations and prepares the agent for realistic edge cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than strictly necessary because it repeats the Args and Returns sections that already exist in the schema. However, it is well-structured with clear sections (purpose, args, returns, examples, error handling) and the most important guidance (purpose and alternative) is front-loaded. Every section adds unique value (examples, error handling, rate limits), so the length is justified despite some redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a list tool with an output schema and read-only annotations, the description covers all essential context: proper use cases, alternatives, parameter semantics, edge cases (missing coordinates, empty feeds, rate limits), and concrete examples. Nothing an agent needs to call it correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema fully documents each parameter. The description adds some meaning beyond that: it clarifies that 'line' matches on name or id, and provides concrete examples (operator_id='SF', line='N') that illustrate how parameters map to real queries. Defaults for limit and response_format are repeated, but the examples and matching detail provide modest added value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear, specific statement of what the tool does: 'Live positions of an agency's vehicles currently in service.' It names the resource (vehicles) and action (list positions), and explicitly differentiates from a sibling tool ('For "when does one get to me", use transit_next_departures instead'), so an agent can distinguish it without opening schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use guidance: answers 'where are the trains right now' and 'how many buses are running on this line', and gives a direct exclusion: 'Don't use when: you want arrival times at a stop' with the alternative named. Examples reinforce correct usage and operator/line selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transit_list_operatorsA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, openWorldHint=true, idempotentHint=true, and destructiveHint=false. The description adds substantial behavioral context beyond those: the Monitored flag semantics, the known TimeZone bug (inaccurate 'America/Vancouver' values), the filtering of pseudo-agencies (5E, 5F, 5O, 5S), and the rate limit (60 req/hour across all endpoints) with caching advice. This is exactly the kind of operational detail an agent needs and the annotations do not provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is thorough but efficiently organized into clear sections (Args, Returns, Examples, Error Handling). Every sentence adds value: the opening line is the purpose, the 'Start here' directive is front-loaded, examples are concise, and error handling covers real-world quirks. No redundancy or filler—each element earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 3 optional parameters and an output schema, the description covers everything needed to call it correctly: it provides default values, return shape (matching the output schema), examples, and critical edge cases (TimeZone bug, filtering, rate limit). Even though the output schema exists, the description's inclusion of the return shape is redundant but not harmful; the error-handling notes are essential and not available anywhere else. The description is fully sufficient for an agent to use this tool without surprises.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description goes beyond the schema by explaining the semantic impact of monitored_only (supports live departures/vehicle positions vs schedule-only), giving concrete usage examples for each parameter, and clarifying the response_format default. It also mentions the default for limit in prose. While the schema already has descriptions, the description adds contextual meaning that helps an agent decide parameter values appropriately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb-resource pair: "List Bay Area transit agencies and their operator codes." It immediately differentiates from siblings by noting "Every other tool needs an operator code, and this is what produces them," which is unique among the sibling tools. The specific examples (BART='BA', Muni='SF') further cement the purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs when to use: "Start here." and "Call this first whenever the user names an agency, to resolve its code." It also explains the monitored_only flag's relevance for choosing between live vs schedule-only agencies, effectively guiding the agent on when to apply different parameter values. No alternatives are named, but the tool is clearly positioned as the entry point, so exclusion guidance is unnecessary.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transit_list_service_alertsA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations only indicate read-only, idempotent, and non-destructive behavior. The description adds crucial behavioral nuance: no alerts means normal service (not an error), timestamps are epoch seconds unlike the ISO strings in other API endpoints, active_only=false surfaces future planned work, and rate limits. This goes well beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Structured with Args, Returns, Examples, and Error Handling sections. The purpose and primary use cases are front-loaded in the first sentence, and every sentence adds value—examples, rate limits, or format distinctions. No redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The output schema is present and the description includes a return format example with field types. It covers error handling (no alerts ≠ error), rate limits, timestamp format, and sibling routing. For a 5-parameter tool with no required parameters, this is fully complete for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3, but the description adds substantial meaning: operator_id omission sweeps all agencies and is 'kinder' against quota, active_only=false is tied to 'weekend track work' examples, and query is illustrated with a practical use case. These contextual insights help an agent choose parameter values effectively.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('List') and resource ('Transit Service Alerts'), and grounds it with concrete questions it answers ('is BART running normally', 'why is my line delayed'). It explicitly differentiates from the sibling transit_next_departures by naming when not to use it, making the purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use the tool (answering delay/outage questions) and when not to ('Don't use when: you want a specific stop's arrivals (use transit_next_departures)'). Also gives operational guidance, such as omitting operator_id to sweep all agencies and conserve quota—practical context an agent needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transit_next_departuresA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnly, idempotent, and non-destructive hints. The description adds substantial behavioral context: real-time vs static, minutes-from-now plus Pacific clock time, 'scheduled only' rows, a 90-minute prediction horizon, empty results meaning (end of service or arrival-only terminal), and a 60-requests-per-hour rate limit. This goes far beyond the annotations, providing critical operational details an agent needs to interpret results and avoid misuse.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Structured with clear sections (Purpose, Args, Returns, Examples, Error Handling) and front-loaded with the core purpose. Every sentence adds value; error handling is critical for correct agent behavior. It is detailed but not flabby — the length is justified by the operational caveats and edge cases.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 5 parameters and an output schema, the description covers all essential usage aspects: parameter semantics, error scenarios (empty results, truncated predictions), time horizon, rate limits, and a full output structure in the Returns section. Even though an output schema is provided, the description's explicit sample JSON and explanation of fields further aid the agent. Nothing needed for correct invocation is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Even though schema coverage is 100%, the description enriches every parameter: it provides real examples (operator_id 'BA'/'SF', stop_code from transit_find_stops), warns that stop codes are operator-specific and not interchangeable, clarifies line matching ('matched on name or id'), and explains response_format options. The Args section adds value beyond the schema's terse descriptions, especially for stop_code and line.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states a specific verb and resource: 'Live arrival predictions for the next vehicles at a stop' and explicitly positions it as 'the tool for when is my next train/bus'. It distinguishes from the printed timetable and clarifies scope (real-time, not static), making it clearly different from sibling tools that list operators, stops, lines, vehicles, or alerts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use guidance via examples ('When's the next N Judah?' maps to specific parameters), an explicit 'Don't use when' exclusion for scheduled timetable, and references prerequisite tools (transit_list_operators, transit_find_stops) for parameter sourcing. It also includes rate-limit guidance ('never poll this in a loop'), which is directly relevant to usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.