Server Quality Checklist

Tool Scores

• search_stopsA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses partial/full name matching and the limit parameter, but does not mention case sensitivity, sorting, or whether the operation is read-only. For a simple search this is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: one purpose sentence followed by a clear Args block. Each line earns its place, with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with only two parameters, and an output schema exists, so return values need not be explained. The description covers purpose and parameters adequately, though it lacks explicit use-case guidance and edge-case notes.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description fully compensates by explaining both parameters: query as 'Partial or full stop name to search for' with examples, and limit as 'Maximum number of results to return.' This adds meaning well beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Search for public transport stops/stations by name,' which is a specific verb+resource+scope. The resource (stops/stations) and scope (by name) clearly distinguish it from sibling tools like list_routes and get_next_departures.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies its use for stop-name lookups but does not explicitly state when to use this tool vs alternatives, nor does it offer any exclusions or prerequisites. It is functionally distinct from siblings, but the guidance is not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_next_departuresA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It usefully reveals that results reflect the static timetable rather than real-time positions, and clarifies that all routes serving the stop are included. It does not cover edge cases like multiple partial matches or sorting, but the core behavior is transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences plus a compact Args list. It is front-loaded with the purpose, uses a separate note for an important caveat, and has no unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter read tool with an output schema, the description covers the essential context: what it returns (scheduled departures), scope (all routes), and static nature. It could add sorting order or why one might first use search_stops, but the core information is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The Args section adds meaningful semantics beyond the schema: stop_name is a name or partial name, and limit is the maximum number of departures. Since schema description coverage is 0%, this compensation is important; it could additionally mention default behavior or allowed range.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('Get upcoming scheduled departures') and clearly identifies the resource (a given stop across all routes serving it). This distinguishes it from sibling tools like search_stops and list_routes, which target stops and routes rather than departures.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It states a clear use case—upcoming scheduled departures for a stop—and adds a contextual note that it reads static GTFS, not live GPS, which implies it should not be used when real-time delays matter. However, it does not explicitly name alternative tools or provide comprehensive when-to-use/when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_routesA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses important behavior: mode values are enumerated, omitting mode returns all modes, and limit controls max results. It does not mention potential ordering or edge cases, but the listed behaviors add meaningful context beyond the bare schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: a one-sentence purpose followed by a compact Args section. Every sentence adds value, and the format is easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 optional parameters), the description fully covers invocation details. An output schema exists, so return values are pre-defined. The mode and limit parameters are completely documented, and the purpose is clear. No critical gaps for selection or invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, so the description must compensate. It does so thoroughly by enumerating all valid mode values, explaining that omitting mode lists all modes, and defining limit as the maximum number of results. This adds significant meaning beyond the schema's bare property names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and resource ('public transport routes'), with a clear optional mode filter. It distinguishes itself from sibling tools like search_stops and get_next_departures by focusing on routes rather than stops or departures.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by stating what the tool does, but it does not explicitly mention when to use this tool over alternatives (e.g., search_stops, get_next_departures, run_sql_query). No exclusions or when-not-to-use guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_sql_queryA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the read-only nature, the SELECT-only constraint, available tables, and points to ptv://schema for column details. This covers key behavioral traits, though it omits details about limits or error handling, which are less critical given the output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: a clear one-line purpose, followed by constraints and an args section. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description provides essential context: available tables, the schema resource for columns, and the intended scope (other tools don't cover). Since an output schema exists, return values are documented elsewhere. Minor gaps like query limits or examples prevent a perfect score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description compensates by explaining that the query parameter must be a single SQL SELECT statement. This adds essential meaning beyond the property name, though it could benefit from syntax examples or clarifications.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it runs a read-only SQL SELECT query against the GTFS database, distinguishing itself as a direct query tool for questions other tools don't cover. This is a specific verb+resource with explicit differentiation from siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'for questions the other tools don't cover' provides clear context for when to use this tool, and the constraint that only SELECT statements are permitted implies a read-only use case. However, it does not explicitly name alternatives or list when-not-to-use conditions beyond the general other-tools note.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.