Server Quality Checklist

Tool Scores

• list_councilA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral burden and does so thoroughly. It discloses that the tool makes no network calls, explains the meaning of each output field, and covers edge cases like backups, proxy routing, password masking, and conditional columns.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but each paragraph explains a distinct cluster of output fields and edge cases. It front-loads the core action and field list, then uses compact and consistent formatting for backups, routing, and usage guidance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no parameters and an output schema, the description is complete: it names every visible field, defines non-obvious values, explains backup behavior and proxy routing, and gives concrete use cases. There are no meaningful gaps an agent would need filled to call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters and 100% schema description coverage trivially, so there is no parameter semantics for the description to add. The baseline of 4 for zero-parameter tools applies, and the description does not need to compensate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: "List the council's members" and enumerates the exact fields returned. It also positions itself against siblings by saying it reveals which ids may be passed to ask and ask_all, so an agent can distinguish it from the other tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear usage context: "Cheap and local — makes no network calls" and "Use this to find out which ids you may pass to ask and ask_all, or to explain a configuration problem." It names the relevant alternatives, though it does not explicitly spell out when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• askA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral burden. It discloses statelessness, lack of conversation context, how materials are delivered to every member/round, image handling, and caching implications — all beyond the minimal description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Every sentence earns its place: the core action is front-loaded, then parameters are explained in natural order. The materials rationale is detailed but directly serves correct usage, not padding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  An output schema exists, so return-value details need not be repeated. The description covers the tool's action, model selection, prompt construction, and material handling — sufficient for an agent to invoke it correctly without additional documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description fully compensates. It explains model as a short ID, prompt as needing to carry full context, system as optional steering, and materials' path/text/label semantics including when to prefer paths. This adds essential meaning the schema omits.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Opens with a specific verb and resource: 'Ask ONE member of the council a question and return its answer.' The 'ONE member' phrasing immediately distinguishes it from the sibling ask_all, and the purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives clear contextual guidance: the member is stateless so prompt must be self-contained, and materials should be passed by path when possible. It does not explicitly name alternatives like ask_all or revise, but the 'ONE member' framing makes the boundary clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ask_allA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full disclosure burden and meets it. It reveals hidden mechanics: members are stateless, round 2 only works because previous answers are carried back verbatim, steelman assignments are hidden from members, and weights are never shared. It also states what the transcript will and won't say (e.g., a retired steelman is labeled as retired by configuration, not as a retraction).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but deliberately organized: it front-loads the core behavior, then devotes one section to each advanced feature in a consistent order. Some conceptual rationale could be trimmed, but for a seven-parameter tool with this many interaction effects, the length earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and the presence of an output schema, the description covers the full decision space: defaults, rounds mechanics, guest behavior, steelman tenure and concealment, weights, and material handling. Nothing an agent needs to decide whether to call it or how to shape a call is left unexplained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds substantial meaning beyond the 0%-coverage schema for `models`, `rounds`, `guests`, `steelman`, and `materials`, including defaults and interaction effects. However, it never mentions the `system` parameter, and the guest `weight` field is absent from the prose even though it exists in the schema, leaving small gaps that the schema cannot fill at 0% coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence names a specific verb and resource — 'Ask several members the SAME prompt in parallel' — and states the output shape ('answers side by side and labeled by model'). This clearly separates it from siblings like `ask` and `revise` without needing to open their schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit decision rules: one round is a survey, two rounds are for disagreement, and 'when you would rather look first, ask with rounds=1 and then call revise'. It also tells when to use `guests` (answers already obtained elsewhere) and `steelman` (a council that mostly agrees), so an agent can route itself correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• probe_modelsA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden. It discloses the HTTP-route behavior, the difference between omitting and passing model, and the important nuance that an endpoint refusing the listing does not mean it cannot answer. This is substantive behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, front-loaded with the core action, and each sentence earns its place: usage, parameter behavior, motivation, and an important caveat. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given one optional parameter, an output schema, and no annotations, the description covers all essential context: what the tool does, which parameter selects what, when to rely on it, and how to interpret a refusal. Nothing critical is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must supply meaning for model. It does so clearly: pass model to probe one member, omit it to probe every configured member. This adds real semantic value beyond the bare enum/default in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb + resource: ask a member's endpoint which model ids it exposes via the /models route. It clearly separates probe_models from siblings like ask or ask_all by focusing on endpoint/model-id discovery rather than answer generation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to use the tool: after an unknown-model error or to discover provider offerings. It also explains the parameter-driven choice between probing one member or all members, though it does not explicitly name alternatives or exclusion cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reviseA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and delivers: it discloses that members are stateless, that answers must be passed verbatim, that model-named answers are shown back as the member's own so they revise rather than start over, and that each round is paid for. It also walks through the expected loop so an agent knows exactly what will happen.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than average but earns its length with dense, useful information: a summary sentence up front, two motivating use cases, a numbered loop, and parameter-specific warnings. Every paragraph adds value, and the structure makes navigation easy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity, lack of annotations, and zero top-level schema coverage, the description covers almost everything needed to call it correctly: when to use it, how to construct `answers`, what to pass every round, and the statelessness caveat. The only gap is the undocumented `models` and `system` parameters, which keeps it just short of fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The top-level schema has 0% description coverage, so the description compensates well for the key parameters: `answers`, `round`, `materials`, and `prompt` all receive operational meaning. However, `models` and `system` remain undocumented, so the compensation is not complete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence states the exact action: 'Run ONE more round of an existing discussion: show every member what was said last round and ask it to revise.' This clearly distinguishes it from ask_all (which commits to rounds in advance) and ask (one-off), and the rest of the description reinforces that distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly contrasts with ask_all: 'This is ask_all's rounds turned inside out' and explains two concrete use cases: deciding after reading each round, and injecting a subagent or one's own voice as a full member. It also gives the loop steps and notes there is no round ceiling unlike ask_all. This is exemplary when-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• revision_promptA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the tool 'makes no network calls', that re-running a subagent on the original question would reproduce the previous answer, and that material is named rather than pasted. It also warns about the consequence of opening files when preparing the prompt, which is important behavioral context beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose and then adds rationale, parameter mapping, and material-handling guidance in a logical order. It is somewhat verbose and repeats the 'transcript will not show' idea twice, but most sentences earn their place given the tool's subtle behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no annotations and no top-level schema descriptions, this description is unusually complete. It explains what the tool does, why it is needed, how to call it, what arguments to pass, what side effects are absent, and how materials should be handled. An output schema exists, so the lack of a described return format is not a gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage for top-level parameters is 0%, and the description compensates by mapping parameters to the sibling call: 'Pass the same prompt, answers, round and materials you are passing to revise, plus seat.' It also clarifies that 'seat' is 'the label (or member id) of the seat to write for.' It does not independently explain prompt, answers, or round, but the cross-reference to revise makes the semantics usable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and object: 'Build the exact prompt a seat should be given for the next round.' It immediately contrasts itself with 'revise' ('revise asks the members'), so an agent can distinguish this prompt-builder from the council-asking sibling. It also names the exact target seat ('the one seat this server cannot ask itself: yours').

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says which sibling to compare against and exactly what to pass: 'Pass the same prompt, answers, round and materials you are passing to revise, plus seat.' It even gives timing guidance: 'run it alongside revise rather than after it.' This is explicit, actionable usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.