Server Quality Checklist

Tool Scores

• freight_logistics_guideC
  2.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and openWorldHint=false. The description adds 'DEMO', signaling that the tool is illustrative or non-production, which is useful context beyond the annotations. However, it does not disclose other behavioral traits such as output format or data limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  At six words, the description is extremely short, but this is under-specification rather than effective conciseness. The phrase 'options' is vague, and 'DEMO' adds a caveat without explaining its implications.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With two required parameters (origin, destination) and an optional cargo_type, an agent needs context about expected inputs and usage. The description provides almost none, leaving the tool's purpose and invocation details severely underdetermined even with an output schema present.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has three parameters (origin, destination, cargo_type) with 0% description coverage, and the description does not mention any of them. It fails completely to compensate for the schema's lack of semantic meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description identifies the domain (freight logistics for Kenya SMEs) but lacks a specific verb or action, making it unclear whether the tool lists options, provides guidance, or performs a calculation. It separates the tool from transport siblings by broad topic but not by specific function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like matatu_route_finder or ntsa_services_guide. The 'Kenya SMEs' framing implies an intended audience but not a decision rule for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transport_rights_queryC
  2.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, so the agent knows this is a safe read operation. The description adds 'DEMO' which hints at limited production readiness, but otherwise adds no extra behavioral detail beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very brief, with two short fragments, which keeps front-loading good. However, it omits essential information about the parameter and usage, making it under-specified rather than efficiently concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity and presence of an output schema, return values are covered. However, the description is incomplete for effective use because it doesn't clarify what 'topic' should contain, leaving a significant gap in how to invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'topic' has no schema description and the tool description provides no guidance on what values or topics are valid. With schema coverage at 0%, the description fails to compensate, leaving the agent to guess what to pass.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the resource and scope: passenger rights for Kenya public transport. It is distinguishable from sibling tools such as matatu_route_finder or boda_licensing_guide, but it lacks an explicit verb like 'query' or 'retrieve', relying on the tool name for that.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives, nor any context about appropriate topics or scenarios. The description merely states what it covers without any usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• boda_licensing_guideB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and openWorldHint=false, so the safety and scope boundaries are set. The description adds the word 'DEMO', which hints at limited or sample data but is not explained. No other behavioral details (e.g., response format, error cases, county-specific differences) are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two short fragments totaling fewer than ten words. It is front-loaded with the core resource and adds the DEMO label for context. The structure is minimal but acceptable, though the second fragment feels like an annotation rather than a fully formed sentence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple optional parameter, readOnly annotation, and presence of an output schema, the description covers the essential purpose. The ambiguous 'DEMO' label raises questions about data authenticity or completeness, and the description does not clarify what the tool actually returns or its limitations, so it falls slightly short of complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema describes the single 'county' parameter with 100% coverage, and the default/null behavior is clear. The description does not add any additional meaning about how the county parameter affects the results, so it does not go beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies a concrete resource ('Boda boda (motorcycle taxi) licensing requirements') and adds geographic scope ('in Kenya'), which distinguishes it from sibling tools like matatu_route_finder or freight_logistics_guide. It lacks an explicit verb (e.g., 'lists' or 'provides'), but the intent is clear from context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The usage context is implied by the resource name: use this tool when you need boda boda licensing information for a specific county. However, there is no explicit guidance on when not to use it or how it relates to the sibling tools (e.g., matatu_route_finder). No exclusions or alternatives are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ntsa_services_guideA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The readOnlyHint annotation already establishes this is a safe read operation. The description adds only modest context: the 'DEMO' flag indicates it may be a sample/limited tool, and the topic list provides scope. No contradictions with annotations are present, but no deeper behavioral details (e.g., return format, limitations) are given.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise and front-loaded with the tool name and topic list. Every phrase adds value, including the 'DEMO' note. It could be slightly more structured to separate the tool's purpose from its parameter hints, but overall it is efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple one-parameter schema and presence of an output schema, the description covers the main purpose and parameter domain. However, it does not mention whether any default service applies, how many services exist, or what to expect in the response (e.g., a static guide vs. dynamic data). The DEMO note hints at limitations but doesn't fully explain behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at 0% and no enums, the description is the primary source of meaning for the 'service' parameter. It lists the likely acceptable values ('driving licence, vehicle inspection, permits'), which is essential for an agent to construct a valid request. This substantially compensates for the sparse schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies a resource ('NTSA services guide') and lists the topics it covers (driving licence, vehicle inspection, permits). It partially distinguishes from sibling tools by specifying this is the NTSA-related guide, though it could state its function as a verb (e.g., 'provides information about').

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for questions about NTSA services like driving licences, vehicle inspection, or permits. However, it does not explicitly say when to use this guide over sibling tools like matatu_route_finder or boda_licensing_guide, leaving some ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• matatu_route_finderA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, so the read-only nature is covered. The description adds 'DEMO' and 'between Kenyan towns', which gives some behavioral context (limited/demonstration scope and geographic restriction), but it does not elaborate on what 'DEMO' implies (e.g., sample data, incomplete coverage).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single front-loaded sentence with no wasted words. 'Find matatu routes between Kenyan towns. DEMO.' delivers the core purpose and a key caveat efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity, an output schema, and readOnly annotation, the description is mostly adequate. However, it does not explain what 'DEMO' entails or possible edge cases (e.g., no route found, unsupported towns), leaving some ambiguity for a production-minded agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description says 'between Kenyan towns', which implies that origin and destination are town names in Kenya. However, it offers no format, examples, or additional constraints beyond the schema's basic string type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Find'), resource ('matatu routes'), and geographic scope ('between Kenyan towns'). It distinguishes itself from sibling tools like 'ntsa_services_guide' and 'boda_licensing_guide', which are informational guides rather than route finders.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for route finding but does not explicitly state when to use this tool versus the sibling guides, nor does it mention any exclusions or alternative tools. It provides clear context but lacks explicit guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.