Server Quality Checklist

Tool Scores

• libraryB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must disclose behavioral traits. It mentions 'Read or mutate' but does not describe side effects, persistence, or required permissions for mutations. Schema's 'active_only' hints at soft-delete but description omits this context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no redundant words. First sentence defines scope, second adds constraint. Efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Tool has 5 parameters, multiple actions, and no output schema. Description is too minimal to cover behavioral details (return values, action-specific outcomes). Lacks completeness for an agent to reliably invoke.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, baseline 3. Description adds that IMDb ID and type are required for add/remove, providing context beyond schema. But other parameters (query, action, active_only) are not elaborated.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states 'Read or mutate the Stremio library' which clearly defines the verb and resource. However, it does not distinguish from sibling tool 'search' which may overlap with the library's search action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description provides a usage hint for add/remove actions (require IMDb ID and type) but lacks explicit guidance on when to use this tool versus alternatives like search or play.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that results include IMDb IDs but omits other behavioral traits such as read-only nature, pagination, search scope (e.g., full text vs. title), or limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that efficiently conveys the tool's purpose and key output detail. No redundant words, front-loaded with essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 3-parameter search tool with no annotations or output schema, the description is too sparse. It lacks usage guidelines, behavioral details (e.g., read-only, case sensitivity), and context on how search results integrate with sibling tools like play.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 100% description coverage, so baseline is 3. The description adds value by mentioning that results contain IMDb IDs, which is not in the parameter descriptions, but does not further elaborate on parameter usage beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Search for movies or TV shows' with a specific verb and resource, and mentions returning IMDb IDs, which distinguishes it from sibling tools like play, library, tv_control, and playback_status that serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives, or when not to use it. The description merely states what it does without contextualizing usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tv_controlA
  3.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, the description discloses key behavioral details: the 'stop' action only succeeds when playback is verified ended, not on mere key event delivery; volume range 0-15 for 'set'; and enumerates valid actions per category. This provides transparency beyond a simple action listing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (single paragraph) and organized by category, though bullet points would improve readability. Every sentence provides useful information; no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers main functionality and a critical behavioral nuance (stop). However, lacks information on prerequisites (e.g., device must be on), error handling, what happens on invalid actions, and whether other actions besides volume set require values. Partial completeness for a tool with no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds value by explicitly listing valid actions for each category and clarifying that 'value' is used for 'set' actions (e.g., volume). The schema's action description merely refers to tool description, so the description fills that gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it controls Android TV across volume, playback, navigation, and power categories. It specifies verbs (up/down, play/pause, etc.) and resource (Android TV). However, it does not explicitly differentiate from sibling tools like 'play' or 'search', which could lead to ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives such as 'play' (for starting playback) or 'search'. The description lists actions but does not specify contexts where this tool is preferred or not suitable. No exclusions or alternatives mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• playA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must disclose behavior. It states it 'plays' content but omits side effects (e.g., starting playback, handling multiple results, authentication requirements). Adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose, then usage guidance. No filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 7 parameters (none required) and no output schema, the description covers only the two primary modes. Does not explain parameters like year, season, episode, or source in context, nor describe expected output or errors.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%; description adds value by explaining the two main invocation modes (query vs imdb_id) and their relationship, going beyond individual parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states verb ('play') and resource ('movies or TV episodes'), and distinguishes from sibling tools like 'search' and 'library' by implying play action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit use cases: search by title or use IMDb ID. Lacks explicit when-not-to-use or comparison with siblings, but offers clear guidance on invocation methods.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• playback_statusA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully explains that it returns current playback status and details the 'state' field including the nuanced meaning of 'stalled'. It does not disclose authentication or rate limits, but for a read-only tool, the behavioral context is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, with the first stating the core purpose and the second enumerating returned fields. No wasted words; efficient and clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters and no output schema, the description fully covers what the tool does and returns, including state values with special explanation for 'stalled'. Complete for a simple read-only tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so the description need not explain parameter semantics. It adds value by describing the return fields, which compensates for the lack of an output schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get' and the resource 'current playback status', and lists the specific fields returned (app, title, state, position, duration). It distinguishes from sibling tools like play, search, library, tv_control by focusing on status retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage context is implied by the nature of the tool (getting status) compared to siblings like play (control) and search (finding content). However, no explicit guidance on when to use this vs alternatives, or when not to use it, is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.