Server Quality Checklist

Tool Scores

• get_solve_jobA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description bears full responsibility and does well: it explains the meaning of each status ('running', 'completed', 'failed'), what completed includes (solution and usage), and that failed includes error.code. It also mentions being free of charge, which is helpful context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with the purpose, followed by practical status details. Every sentence earns its place—no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple polling tool, the description covers the key behavioral aspects: when to poll, what statuses mean, and expected output content. It lacks details on invalid job_id or authentication, but given the simplicity and lack of output schema, it is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already fully describes job_id (100% coverage), including that it comes from the 202 async response of solve_shift. The description adds no additional parameter semantics beyond restating this, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool checks the status of an asynchronous solve job created by solve_shift, using a specific verb (check) and resource (job). It also references the sibling tool solve_shift, distinguishing it from validate_shift_input and solve_shift.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context: this is the polling tool for async jobs from solve_shift, and explicitly advises waiting ~15 seconds and calling again while status is running. It implies the tool should be used after receiving a 202 response, though it doesn't state explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• solve_shiftA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosure. It clearly states the call is synchronous, returns within ~30 seconds, is charged per call, and requires an API key with quota. This covers latency, cost, and authentication—key behavioral aspects beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three concise sentences: the first states the core purpose, the second adds behavioral constraints (synchronous, timeout), and the third gives a usage tip and a link to the full schema. No wasted words; information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one complex parameter and no output schema, the description provides adequate context by referencing the external OpenAPI schema for request/response details, covering prerequisites (validation, API key), and describing latency/cost. It does not spell out the return structure directly, but the external link fills that gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already thoroughly describes the single `request` parameter (100% coverage) including its contents (schedule_period, shifts, duties, etc.). The description's additional mention of 'solver_timeout_seconds' is useful but not required to understand the parameter, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Run the Shiftyasan solver on a shift scheduling request payload,' using a specific verb ('Run') and resource ('Shiftyasan solver') that clearly distinguishes it from sibling tools like validate_shift_input. The intent to actually solve a shift schedule is unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The 'Tip: call validate_shift_input first' gives an explicit alternative to use for validation before invoking this tool. It also states the synchronous behavior and mention of charging helps agents decide when to call it, though it does not explicitly mention when to use get_solve_job.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_shift_inputA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool is 'Free of charge (does not run the solver),' implying no side effects, and describes the return format: '{valid: true, ...summary} or an RFC 7807 problem detail.' This provides solid transparency, though it doesn't elaborate on potential errors or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with purpose, then usage guidance, and then return format. Every sentence earns its place with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema), the description covers everything needed: purpose, usage, behavior, and return value. It points to the external schema for the full payload structure, making it complete for the target use case.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% coverage of the single parameter (`request`) with a detailed description pointing to the OpenAPI schema. The tool description adds no extra parameter-level meaning, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Validate a shift solver request payload against the Shiftyasan public API schema.' It uses a specific verb ('validate') and resource ('shift solver request'), and distinguishes itself from siblings by noting 'does not run the solver' and pointing to usage before `solve_shift`.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says when to use it: 'Use this before `solve_shift` to confirm the request shape is correct, especially when constructing the payload programmatically.' This names the alternative tool and provides clear contextual guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.