Server Quality Checklist

Tool Scores

• get_ovfietsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It only states what the tool does without detailing behavioral traits like response format, error handling, rate limits, or whether it's a read-only operation. This leaves significant gaps in understanding how the tool behaves in practice.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence that efficiently conveys the tool's purpose without unnecessary words. It is front-loaded and appropriately sized, making it easy to understand at a glance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete for a tool that likely returns availability data. It does not explain what the output includes (e.g., number of bikes, timestamps) or handle potential complexities, leaving the agent with insufficient context to use the tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, so the schema already documents the 'stationCode' parameter with an example. The description does not add any additional meaning or context beyond what the schema provides, such as explaining the format of station codes or availability data, resulting in a baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get') and resource ('OV-fiets availability at a train station'), making it immediately understandable. However, it does not explicitly differentiate from sibling tools like 'get_station_info', which might also provide station-related data but for different aspects.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention sibling tools or specify contexts where this tool is preferred, such as for checking bike availability versus other station information tools like 'get_arrivals' or 'get_station_info'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_pricesC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions that the tool 'Returns detailed pricing information with conditions and validity,' which gives some indication of output behavior. However, it doesn't address important behavioral aspects like whether this is a read-only operation, potential rate limits, authentication requirements, error conditions, or what happens when parameters are invalid. For a 10-parameter tool with no annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured in two sentences that cover purpose and return value. There's no wasted language, and it's appropriately sized for the tool's complexity. While it could be slightly more detailed given the lack of annotations, what's present is well-organized and front-loaded with the core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 10-parameter tool with no annotations and no output schema, the description is incomplete. It doesn't explain the relationship between parameters (e.g., how routeId interacts with station parameters), doesn't mention error conditions or validation rules, and provides minimal guidance on the return format beyond 'detailed pricing information.' Given the complexity and lack of structured metadata, the description should do more to help an agent understand how to use this tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description mentions 'different travel classes, ticket types, and discounts' which maps to some parameters (travelClass, travelType, isJointJourney). However, with 100% schema description coverage, the schema already documents all 10 parameters thoroughly. The description adds minimal value beyond what's already in the parameter descriptions, providing only high-level context without additional syntax or format details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get price information for domestic train journeys' with specific details about what's included (travel classes, ticket types, discounts). It distinguishes from sibling tools like get_arrivals or get_departures by focusing on pricing rather than schedules or status. However, it doesn't explicitly differentiate from potential pricing-related siblings that might exist in other contexts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. While sibling tools like get_travel_advice might include pricing information, there's no explicit comparison or context about when this specific pricing tool is preferred. The description mentions what the tool returns but not when it should be selected over other available tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_station_infoC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states it 'gets' information (implying a read operation) but doesn't mention any behavioral traits like rate limits, authentication needs, error handling, or what 'detailed information' entails (e.g., format, completeness). This leaves significant gaps for an agent to understand how to use it effectively.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without any wasted words. It's appropriately sized and front-loaded, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no annotations and no output schema, the description is incomplete. It doesn't explain what 'detailed information' includes (e.g., station facilities, location, services), how results are structured, or any limitations (e.g., data freshness, availability). For a tool with 3 parameters and no structured output, this leaves too much ambiguity for reliable agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, meaning all parameters are documented in the schema itself. The description doesn't add any meaning beyond what the schema provides (e.g., it doesn't explain how 'query' interacts with station codes vs. names or clarify 'detailed information' in relation to parameters). Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('detailed information about a train station'), making the purpose immediately understandable. However, it doesn't explicitly differentiate from siblings like 'get_arrivals' or 'get_departures' that also involve station data, which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With siblings like 'get_arrivals' and 'get_departures' that might overlap in station-related queries, there's no indication of when this tool is preferred or what specific 'detailed information' it provides that others don't.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_disruptionsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds some context by mentioning the tool 'can filter for active disruptions and specific disruption types,' which hints at filtering capabilities beyond the schema. However, it lacks details on rate limits, authentication needs, or what happens if no disruptions exist, leaving gaps in behavioral understanding for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core purpose and followed by details on returns and filtering. Every sentence earns its place by adding value, such as listing returned information and filtering options, without redundancy or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is somewhat complete but has gaps. It covers the purpose and filtering but lacks details on output format, error handling, or how disruptions are structured, which could hinder an agent's ability to use it effectively without further context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already fully documents the two parameters (isActive and type). The description adds marginal value by mentioning filtering for 'active disruptions and specific disruption types,' which aligns with the schema but does not provide additional syntax or format details beyond what the schema specifies. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('Get comprehensive information') and resources ('current and planned disruptions on the Dutch railway network'), distinguishing it from sibling tools like get_arrivals or get_departures which focus on different railway data. It explicitly lists the types of information returned (maintenance work, unexpected disruptions, etc.), making the scope unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool by specifying it returns 'comprehensive information about current and planned disruptions,' implying it's for disruption-related queries rather than arrivals, departures, or other railway data. However, it does not explicitly state when not to use it or name specific alternatives among siblings, such as get_travel_advice for route planning, which could be related.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_arrivalsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the tool's function and output format, but lacks details on potential limitations (e.g., rate limits, data freshness, error handling) or prerequisites (e.g., authentication needs). The description does not contradict any annotations, as none are provided.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with the first sentence clearly stating the core purpose. Both sentences earn their place by adding value: the first defines the tool's function and scope, and the second specifies the return format. There is no redundant or wasted information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (5 parameters, no output schema, no annotations), the description is reasonably complete. It covers the tool's purpose, scope, and return format, but could improve by addressing behavioral aspects like rate limits or error handling. The lack of an output schema means the description's detail on return values is helpful, though not exhaustive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema already documents all parameters thoroughly. The description does not add any parameter-specific semantics beyond what the schema provides (e.g., it doesn't clarify station code formats or date-time usage). However, it implies the tool's purpose aligns with the parameters, maintaining the baseline score for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('Get real-time arrival information') and resources ('trains at a specific station'), distinguishing it from siblings like get_departures (which handles departures) and get_station_info (which provides static station details). It explicitly mentions what information is included (platform numbers, delays, origin stations, travel notes) and the return format (list of upcoming arrivals).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying 'real-time arrival information for trains at a specific station,' which helps differentiate it from siblings like get_disruptions (disruption info) or get_travel_advice (route planning). However, it does not explicitly state when NOT to use this tool or name specific alternatives, such as using get_departures for departure data instead.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_departuresA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool returns real-time information with a list of upcoming departures, including timing, destination, and status, which covers basic behavior. However, it lacks details on rate limits, authentication needs, error handling, or data freshness, which are important for a real-time API tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by additional details in a second sentence. Every sentence adds value without redundancy, and it is appropriately sized for a tool with 5 parameters and no annotations.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (real-time data fetching with 5 parameters) and lack of annotations or output schema, the description provides a good overview of what the tool does and returns. However, it could be more complete by mentioning potential limitations (e.g., data availability for foreign stations) or response structure details, which would help compensate for the missing output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description does not add any parameter-specific semantics beyond what the schema provides (e.g., it doesn't explain station code formats or date-time usage further). Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Get real-time departure information') and resource ('trains from a specific station'), distinguishing it from siblings like get_arrivals (arrivals vs. departures) and get_station_info (station metadata vs. departure data). It specifies the scope includes platform numbers, delays, route details, and travel notes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying 'real-time departure information for trains from a specific station,' which distinguishes it from tools like get_disruptions (general disruptions) or get_travel_advice (route planning). However, it does not explicitly state when not to use it or name alternatives, such as clarifying that get_arrivals is for incoming trains.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_travel_adviceA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses key behavioral traits: returns multiple route options, includes status and crowding information, supports real-time updates, and handles both departure and arrival time optimization. However, it doesn't mention rate limits, authentication requirements, error conditions, or pagination behavior for the returned routes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured in two sentences: the first establishes core functionality, the second adds important contextual details about timing and output. Every phrase adds value with zero redundant information, making it appropriately sized and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 4-parameter tool with no annotations and no output schema, the description does well by explaining what the tool returns (multiple route options with status/crowding info) and key behavioral aspects. However, it doesn't fully compensate for the lack of output schema by detailing the structure of returned data or error handling, leaving some gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 4 parameters thoroughly. The description adds marginal value by mentioning 'immediate departure or specific future time' and 'optimize for arrival time,' which helps contextualize the dateTime and searchForArrival parameters, but doesn't provide syntax or format details beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get detailed travel routes') and resource ('between two train stations'), specifying the scope includes transfers, real-time updates, platform info, and journey duration. It distinguishes from siblings like get_arrivals/get_departures (which focus on single stations) and get_disruptions (which doesn't provide routes).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('plan trips for immediate departure or for a specific future time') and mentions optimization options. However, it doesn't explicitly state when NOT to use it or name specific alternatives among the sibling tools, though the differentiation is implied through the detailed functionality described.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_current_time_in_rfc3339A
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly states this is a read operation that returns current time in a specific format and timezone, but doesn't mention potential limitations like rate limits, authentication requirements, or whether the time is synchronized with atomic clocks.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with zero waste - the first states the core functionality with precise specifications, the second provides practical usage context. Every word serves a clear purpose in helping an agent understand and use this tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with no annotations and no output schema, the description provides excellent context about what the tool does, the format it returns, and its practical application. It could be slightly more complete by explicitly stating the return type or format details, but it's very effective given the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters with 100% schema description coverage, so the schema already fully documents the lack of inputs. The description appropriately doesn't discuss parameters, maintaining focus on the tool's purpose and output format.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Get'), resource ('current server time'), and format ('RFC3339 format') with precise timezone specification ('Europe/Amsterdam timezone'). It distinguishes itself from sibling tools by focusing on time retrieval rather than transportation data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool ('This can be used as input for other tools that require date-time parameters'), but doesn't explicitly mention when not to use it or name specific alternative tools for time-related functions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.