Server Quality Checklist

Tool Scores

• query_postingsB
  3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description reveals a key behavior: the tool only returns postings with at least one score, and that an empty result is valid. However, since there are no annotations, the description should also disclose aspects like idempotency, side effects, or required permissions, but it does not. The disclosed behaviors are useful but incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two sentences with no redundant words. It states the core action, a critical behavioral detail, and a note about empty results. Every sentence adds value, achieving high efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema (reducing the need to describe return values), the description still leaves gaps. It does not elaborate on the filter criteria parameters, nor does it explain the output beyond 'latest score'. The tool has 4 parameters, all undocumented in schema descriptions; the description fails to compensate adequately. The behavioral note is good but insufficient for a complete understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the input schema lacks any explanatory text for parameters. The description only mentions 'filter criteria' generically, without explaining how each parameter (limit, min_score, date_after, role_anchor) works. The parameter names offer some clues, but for an AI agent, more explicit guidance is needed to ensure correct invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists postings with filter criteria and includes the latest score. The verb 'List' and resource 'postings' are specific. Although it doesn't explicitly differentiate from siblings, the sibling tools (rescore_posting, tag_mismatched_score, regenerate_digest) are obviously different in function, so no confusion.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. The description mentions a behavioral constraint (only postings with at least one score) but doesn't explain under what circumstances an agent should choose this over other list or search tools. Absence of usage context or exclusions limits its helpfulness.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tag_mismatched_scoreB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Reveals the tool writes to a database (side effect) and that reason must be non-empty (constraint). However, without annotations, it omits idempotency, rate limits, error behavior, and what happens on duplicate calls. The output schema is present but not referenced.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences; each adds value: purpose, database context, constraint. No redundancy. Slightly better front-loading would help, but acceptable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Fails to cover return values (despite output schema), error conditions, expected_band format, or duplicate handling. With 3 required params and no schema descriptions, the description is too sparse to support confident invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only reason is given a constraint (non-empty). posting_id and expected_band are not described at all, leaving their semantics unclear. With 0% schema coverage, the description should explain all parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool records mismatches between stored scores and operator judgment, with specific side effects (writing to a database) and intended use (rubric refinement). The purpose is distinct from siblings like rescore_posting.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies usage when an operator disagrees with a stored score, and mentions downstream aggregation by tune_rubric. However, no explicit guidance on when not to use it vs. alternatives like rescore_posting, and no prerequisites or trigger conditions are stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• regenerate_digestA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses output path safety and file style, but lacks details on side effects (e.g., overwrites existing file?), rate limits, or required permissions. Not comprehensive for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three efficient sentences, each adding distinct information: purpose, output path safety, and file styling. No fluff, front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequate for a 0-param tool with output schema, but missing guidance on when regeneration is appropriate and how 'current' top-20 is defined. Could mention if it replaces an existing file.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters in schema, so description adds value by explaining output path security and file characteristics, which the empty schema cannot convey. Baseline is 4 for 0-param tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool renders a digest HTML of the current top-20 postings. The action 'render' and resource 'digest HTML' are specific and distinguish it from sibling tools like query_postings or rescore_posting.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives. Does not mention prerequisites, timing, or comparison with query_postings or other siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rescore_postingA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It details the steps (load posting, load rubric, score via LiteLLM, persist result) and error conditions (ValueError for missing posting, ScoringError for LLM failure). This provides good transparency into side effects and behaviors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with four sentences front-loaded by the purpose. It efficiently covers purpose, steps, and errors without unnecessary words. Slightly more conciseness could be achieved by combining steps, but it is well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the existence of an output schema and single parameter, the description is nearly complete. It covers the main actions and errors, though it could mention idempotency or implications of re-scoring (e.g., overwriting previous score). Overall, it provides sufficient context for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not elaborate on the posting_id parameter beyond its use. However, the single parameter is self-explanatory from the context. The description could add value by explicitly stating the meaning of posting_id.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it re-scores a posting using the configured LLM, with a specific verb (re-score), resource (posting), and method (via LLM). It is distinct from sibling tools like tag_mismatched_score, query_postings, and regenerate_digest.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (when needing to rescore a posting) but does not provide explicit guidance on when not to use or contrast with alternative tools. The sibling tools are different in function, but no exclusions are stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.