Server Quality Checklist

Tool Scores

• style_packC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the tool includes packs that may be 'off-rubric' or 'restricted' based on parameter docs, but the description itself doesn't disclose what gets generated, whether any mutation occurs, license/reuse implications, or output format. The 'attribution block that must travel' hints at a licensing expectation but doesn't explain its mechanics or consequences.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence and compact, which is good. However, it packs in several undefined jargon terms ('exemplars', 'liked/disliked calibration', 'attribution block') that force the reader to parse marketing-style language for meaning. It's concise but not maximally clear; front-loading a plain-verb statement of the operation would serve the agent better.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-style tool with full schema coverage and no output schema, the description is adequate but leaves gaps. It doesn't clarify the return format (is it text, structured data?), how restricted packs are handled in output, or why/when to enable the hidden and restricted flags. Given four boolean/integer parameters influencing output content and licensing-sensitive behavior, the description could meaningfully expand on when those flags matter, which it does not.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so each parameter (n, topic, include_hidden, include_restricted) is documented in the schema itself. The description adds no parameter-specific meaning beyond what the schema provides; it references 'exemplars' and 'calibration' generically but doesn't clarify how 'n' maps to sides, or what 'liked/disliked calibration' means operationally. Per baseline for full schema coverage, a 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description reads like a marketing blurb ('compact, paste-ready style brief') using jargon ('exemplars', 'liked/disliked calibration', 'attribution block') that isn't precisely defined. It conveys a general sense that this produces a style summary with examples, but doesn't clearly state the tool's verb-resource operation (e.g., 'Generate a style brief'). The jargon and stylistic phrasing obscure rather than clarify the core purpose, distinguishing it only weakly from siblings like taste_profile and preference_pairs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is minimal guidance on when to use this tool. The description references an 'attribution block that must travel with any output,' implying a downstream requirement, but doesn't explicitly state when to prefer this over search_humor, taste_profile, preference_pairs, or breakdown. No exclusions or alternatives are mentioned, and the context of 'style' vs. sibling concepts like 'taste' or 'humor' is left implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• corpus_statsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It reveals the output nature (counts by pack/kind, disk size) but doesn't disclose whether this is read-only, how it handles an empty corpus, execution cost, or what 'pack' and 'kind' refer to. For an analytical tool this is thin coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single short sentence that efficiently conveys the tool's informational scope. No wasted words, though it could be slightly more explicit about the verb (report, summarize).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter reporting tool, the description is reasonably complete about what information is returned (counts by pack and kind, disk size). However, lacking an output schema and any guidance on how 'pack' and 'kind' are defined relative to siblings like 'breakdown' or 'sources' leaves ambiguity in a toolset with several aggregation options.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0 parameters (100% schema coverage trivially) and no params to document, the baseline of 4 applies. The description appropriately describes what the output reveals since there are no input parameters requiring explanation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'What is loaded: counts by pack and by kind, size on disk' states what the tool reports on (loaded corpus, counts, size), giving reasonable purpose clarity. However, it lacks a clear verb-resource structure and doesn't differentiate from siblings like 'breakdown' or 'sources' which could plausibly return similar aggregate data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus siblings. Given siblings like 'breakdown', 'sources', and 'style_pack', the agent has no way to know which aggregation tool to choose. The description gives no context about scenarios where this tool is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• top_ratedB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It does disclose the value scale nuance (single-word lexicon entries use a different scale and are excluded unless kind='word'), which is useful. However, it doesn't describe default behaviors like the min_score=2 gate, the include_hidden/include_restricted defaults, the limit cap, or the source override behavior — all of which meaningfully affect what gets returned.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence that states purpose and the key exception in one breath. Efficient and front-loaded with the essential fact. Minor opportunity: it could note the default gate values, but the sentence is well-constructed and wastes no words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 6 parameters, no output schema, and no annotations, the description leaves gaps on default behavior (min_score filter, hidden/restricted exclusions, limit cap, source override priority). The kind='word' exception is the one piece of nuance disclosed. For a listing tool with array of defaults, the description is adequate but doesn't fully orient the agent on what makes results appear or disappear from the listing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema documents all 6 parameters well. The description adds marginal value by explaining the special-case exclusion rule tied to kind='word', which supplements the schema's bare enum list. Since coverage is high, the description doesn't need to re-document parameters; credit for the one meaningful extra it does provide.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear purpose: returns the highest human-rated lines in the corpus, with a noteworthy exclusion for single-word lexicon entries unless kind='word'. It identifies the resource (highest-rated lines) and a distinguishing condition. However, it doesn't explicitly differentiate from siblings like search_humor or preference_pairs, relying instead on the inherent uniqueness of a 'top rated' listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context (fetching top-rated content) but provides no explicit guidance on when to use this vs alternatives like search_humor, taste_profile, or preference_pairs. The kind parameter hints at different categories but the description doesn't articulate when this tool is the right choice over sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• taste_profileB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. The description doesn't disclose how results are returned, the meaning of the 'liked vs disliked' structure, sorting, pagination, or what happens with edge cases like an rater with no judgements. For a tool without annotations, this is a thin behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tight sentences that front-load the primary purpose and then give actionable guidance on when to use it. No wasted words or redundant detail. Could arguably add a note on return format but remains efficiently concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 5 optional parameters and no output schema, so the description carries the burden for return semantics. The description explains the conceptual purpose well but leaves the actual output structure (what a 'score distribution' looks like, how liked vs disliked items are presented) unspecified. Adequate but with a notable gap given the absence of an output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and all five parameters have inline descriptions in the schema, so the baseline is 3. The description adds no parameter-level meaning beyond what the schema already documents; the schema's 'kind' and 'rater' descriptions are serviceable. The description doesn't compensate further but isn't required to at this coverage level.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool shows what the corpus rater liked vs disliked with a score distribution, using a specific verb ('calibrate') plus resource ('register'). It distinguishes itself from siblings by its focus on the rater's taste profile versus other tools like search_humor or breakdown. However, it doesn't explicitly compare against sibling distinctions beyond the implicit calibration framing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly frames this as a calibration tool before writing ('Use this to calibrate register before writing'), which gives clear context for when to invoke it. It doesn't name specific alternatives or exclusions, but the calibration-when-before-writing guidance plus sibling names like top_rated and breakdown provide implied differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• breakdownA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. The description explains the conceptual output but doesn't disclose return format, pagination behavior, how results are ordered (beyond the query-empty browse mode), or whether results include the raw text. For a tool that returns structural analysis, the agent is left guessing about response shape and ordering semantics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a tight single sentence that front-loads the core purpose. It's efficient with no wasted words. It could arguably add a touch more behavioral detail, but for its length it earns its place. Not verbose, not under-specified to the point of confusion.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema and no annotations, plus 4 parameters and conceptual complexity (categories of structural breakdown), the description is reasonably complete on the 'what' but thin on the 'how results come back' — no ordering default beyond score-browse, no return shape, no inclusion/exclusion semantics beyond what's in the parameter docs. Adequate but with clear gaps for a moderately complex tool with zero annotation coverage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and the description adds meaningful conceptual context beyond the schema: it frames 'query' as words to search for, connects empty-query to score-based browsing (aligning with the top_rated sibling), and distinguishes include_hidden packs as 'on-rubric but judged unrepresentative' and include_restricted as 'licence bars redistribution'. The description enriches each parameter with real behavioral meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description identifies a specific verb+resource ('structural breakdowns ... for lines matching a query') and clarifies the category ('mechanism, setup/turn, technique'). It contrasts with siblings like search_humor and sources by emphasizing it's about joke construction, not text retrieval. However, it doesn't explicitly name which sibling to distinguish from, so it misses the top marks for differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is a search-and-browse mechanism ('lines matching a query', 'empty = browse by score'), and the contrast with top_rated (score browsing) is hinted at. But it doesn't give explicit guidance on when to choose this over search_humor, top_rated, or sources, nor does it state exclusions or prerequisites. Usage context is implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_humorA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations to rely on, so the description carries the transparency burden. It discloses return content (lines with credit) and a heuristic behavior on whole_lines. However, it doesn't explain pagination behavior, how the sort/ordering works, or what happens with special inputs — moderate disclosure but with gaps for a read/search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences that front-load the core purpose and immediately list the key filters. Zero waste — every clause carries meaning. The return description ('lines with their credit attached') is folded neatly into the first sentence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a moderately complex tool with 8 parameters but no output schema. The description covers the search-and-filter story well but doesn't explain the default ordering when query is empty, the interaction between source and the gating flags, or how results are ordered/ranked. For a search tool with no output schema, slightly more about result shape and ordering would improve completeness, though the parameters are well-covered by the schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema documents all 8 parameters. The description adds semantic value by grouping parameters into meaningful categories (source pack, kind, minimum human score), which helps the agent reason about which filters apply together. It also labels whole_lines as 'heuristic' and 'strict about a final full stop,' adding behavioral nuance beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it performs 'full-text search the humor corpus' and returns 'lines with their credit attached.' It also names the filter dimensions (source pack, kind, minimum human score), distinguishing it from siblings like top_rated, corpus_stats, and sources which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies search/filtering use cases via the filtering verbs, but doesn't explicitly state when to choose this over alternatives like top_rated, taste_profile, or breakdown. There's no when-not-to-use guidance, though the 'browse by score' hint in the query param schema partially indicates an alternative to top_rated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• sourcesA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided at all (no readOnlyHint, no destructiveHint), the description carries the full burden of behavioral disclosure. This is clearly a read-only informational tool with zero parameters, so the risk profile is low and inherently safe. The description accurately conveys it's a passive reporting tool, which is sufficient given the tool's trivial side-effect surface. It doesn't hide any destructive or mutating behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, dense sentence that packs in exactly what's needed: coverage scope ('every corpus pack loaded') and the four dimensions reported (authorship, license, redistribution/commercial use, citation). Zero wasted words, appropriately sized for a parameterless reporting tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with no output schema, the description is reasonably complete. It tells the agent the scope ('every corpus pack loaded') and the content dimensions. It doesn't describe the return format or structure, but with no output schema and a simple informational purpose, the description covers the essential usage context well enough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema description coverage is 100% (trivially, since there are no properties to document). The no-parameter nature is consistent with the description's framing as a holistic report of all loaded corpus packs. With no parameters to explain, there's nothing missing in the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states what the tool does: it reports information about every loaded corpus pack covering authorship, license, redistribution/commercial use rights, and citation format. The verb is implied (it 'reports' or 'shows' this information), and it names a specific resource (corpus packs). It's distinguishable from siblings like corpus_stats since it focuses on provenance/legal metadata rather than usage statistics, though it doesn't explicitly differentiate itself.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The context is implied: one would use this to learn about licenses, citation, and redistribution rights for loaded corpus packs. However, there's no explicit guidance on when to use this vs. alternatives, no stated exclusions, and no mention of how this differs from corpus_stats or breakdown. The usage context is understandable but not explicitly framed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• preference_pairsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. It discloses that off-rubric packs are withheld by default and reported in `withheld`, and notes the licence fine/relevance caveat for hidden packs. However, it doesn't describe the output structure beyond the `withheld` field or clarify what 'restricted' packs mean for returned data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single tight sentence that conveys core purpose plus an important default behavior (withholding off-rubric packs). No wasted words. The parameter descriptions in schema carry the detail. Could arguably be slightly richer but earns a high score for efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 5 parameters, no enums, no output schema, and no annotations - so the description needs to cover a fair amount of behavioral ground. It explains the default withholding behavior and the `withheld` reporting field. The sibling context (search_humor, top_rated, breakdown) suggests this is a specialized comparison tool, and the description sets it apart reasonably well given the constraints.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds value by explaining that naming a source pack 'overrides both gates below,' and clarifies the 'empty query = browse by score' behavior. It also explains the hidden/restricted filters in plain language, going beyond the raw schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns chosen vs rejected pairs from head-to-head comparisons, with a mechanism for off-rubric packs being withheld. It uses a specific verb and resource, and the 'what won head-to-head and what lost' phrasing distinguishes it from siblings like top_rated or search_humor. However it doesn't explicitly name sibling tools for differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies browsing by score when no query, and mentions off-rubric packs are 'withheld by default.' The filter semantics of the parameters (source overriding gates, include_hidden, include_restricted) are present in the schema descriptions. It gives clear context for when to use, though it doesn't name explicit alternatives among the siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.