Server Quality Checklist

Tool Scores

• search_logsA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It meaningfully adds constraints beyond the schema: searching must be scoped by an ID, substring queries are optional, results are capped at 25 lines, and regex/glob/unscoped searches are unsupported. It does not describe output shape or error behavior, but what it shares is substantive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very compact and front-loaded with the core purpose. Every sentence adds relevant information and there is no filler or repetition of schema details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The definition covers scope and operational constraints well, but some context is missing: no guidance on sibling tool selection, no output/return semantics, and no explicit statement that at least one ID is required despite the schema listing no required fields. The phrase 'one incidentId or paymentId' partially covers this, but ambiguity remains for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It effectively explains the semantic role of all four parameters: incidentId/paymentId are the required scoping IDs, query is the optional substring, and limit corresponds to the 25-line cap. It does not explicitly map 'limit' by name, but the meaning is clear enough.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Search') and resource ('log lines'), and clearly scopes the tool to one incidentId or paymentId. It is easy to distinguish from sibling tools like list_incidents or get_payment by resource type, even though no sibling is named explicitly.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to prefer this tool over list_incidents, get_payment, list_job_attempts, or retry_job. The description provides constraints (no regex, no glob, no unscoped search) but no alternatives, exclusions, or selection criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_job_attemptsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden. It explicitly says 'Read-only,' which is a useful safety signal, but it does not disclose what happens when both parameters are provided, neither is provided, or whether the response is paginated. The read-only claim is meaningful but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single focused sentence plus a short read-only qualifier. Every word adds value, and the core behavior is stated up front without filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter list tool the description is mostly adequate, but the missing output schema and sparse annotation coverage leave room for ambiguity about required parameter combinations and return shape. An agent could call it without knowing whether both jobId and paymentId may be supplied.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for the schema's lack of parameter documentation. It names both jobId and paymentId and indicates they are alternative filters via 'one ... or,' but it does not clarify whether exactly one is required, what each identifier refers to beyond its pattern, or how the two interact.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'List settlement-job attempts' and scopes it to 'one jobId or paymentId'. This clearly distinguishes it from siblings like retry_job or get_payment by focusing on a read-only list of attempts rather than a mutation or a single payment lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states when to use the tool: when you have a jobId or paymentId and need to see settlement-job attempts. It does not explicitly name excluded alternatives or conditions for choosing siblings, but the context is clear enough for straightforward routing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• retry_jobA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility, and it does well: it discloses mutation, the confirm/audit behavior, and that an audit event is written even on rejection. It does not mention permissions, idempotency, or success/failure return behavior, leaving some side-effect transparency gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely tight: three or four short clauses, each adding a distinct fact, and the most important differentiator ('only mutating tool') is front-loaded. There is no filler or repetition of schema constraints.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  It covers purpose, the required confirmation/reason workflow, and the audit side effect, which is solid for a small tool. Yet with no output schema and no annotation coverage, it omits return value/error behavior and does not clarify how jobId identifies a failed job, so an agent still has unresolved operational questions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, but it only paraphrases confirm=true and 'written reason' from the schema. It never explains what jobId refers to or clarifies the meaning/format of reason beyond the schema's minLength/maxLength constraints, providing little added value for parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the exact action ('Re-enqueues a failed job') with a specific resource, and the opening 'only mutating tool' distinguishes it from the read-style sibling tools (list_incidents, get_payment, etc.). No ambiguity remains about what this tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It clearly signals that this is the mutating option among read-only siblings, which points an agent to use it when a retry action is needed rather than a query. However, it does not explicitly name alternatives or describe when-not-to-use conditions (e.g., after investigating with list_job_attempts), so it stops short of full routing guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_paymentA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral disclosure burden. It explicitly states that the operation is read-only and that SQL/wildcards are not accepted, which is meaningful safety and input-behavior context. It does not describe not-found or error behavior, but for a simple get-by-id tool this is a minor gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two terse sentences with the core action front-loaded. Every clause adds value: fetching by id, read-only safety, and input restrictions. There is no fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a low-complexity tool with one required parameter and no output schema, the description plus the schema pattern are sufficient for an agent to invoke it correctly. The read-only and no-wildcard notes prevent common misuse, and no additional context is necessary.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains that the paymentId parameter is a pay_… id and clarifies that it is not a SQL expression or wildcard, adding practical meaning beyond the raw regex pattern in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb, resource, and retrieval method: fetch exactly one synthetic payment by its pay_… id. It also differentiates itself from search/list siblings by explicitly noting it does not accept SQL or wildcards.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The 'by id' phrasing and the explicit rejection of SQL/wildcards signal exact-match retrieval rather than search-style querying, which helps distinguish it from search_logs. However, it does not explicitly name alternative tools or provide an exhaustive when-to-use/when-not-to-use breakdown.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_incidentsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing behavior. It explicitly states 'read-only' and notes the data is synthetic and seeded, which sets expectations about data type and side effects. It does not describe the return shape, but for a simple parameterless list operation this is reasonable disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences with no filler. The main action is front-loaded, and the read-only and no-filter constraints are stated immediately afterward. Every word adds value, and the structure is easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with zero parameters and no output schema, the description is complete enough for an agent to invoke it correctly. It communicates what is listed, the nature of the data, the lack of filters, and the read-only safety profile.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, so the schema already fully covers parameter semantics. The description reinforces this by explicitly stating 'No filters,' which prevents an agent from assuming hidden query options. This matches the baseline of 4 for a no-parameter tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and clearly identifies the resource as 'seeded synthetic payment incidents.' It also reinforces the scope with 'No filters' and 'catalog is small on purpose,' which distinguishes it from sibling tools like search_logs and retry_job without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly states the tool is read-only and takes no filters, making it obvious this is for fetching the full small catalog of seeded incidents. It does not explicitly name alternative tools, but the context strongly implies the appropriate usage and signals that filtering is not supported here.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.