Server Quality Checklist

Tool Scores

• job_preflightB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description must carry the full behavioral burden. It discloses that the tool can produce defer decisions and that they should be obeyed. However, it does not state whether calling it is safe/read-only, whether it reserves resources, whether repeated calls are allowed, or what the actual response will look like.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one tight sentence with no filler, and the primary usage instruction is front-loaded. It is appropriately short, though it is terse enough to omit important behavioral and parameter context. Still, the structure is clean and scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no annotations and no output schema, the description should explain what a preflight result looks like, what a 'defer decision' means, and how it relates to sibling tools like defer_until_reset or quota_status. None of that is present, so an agent is left without enough context to fully act on the result.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 25%, with workspaceRoot documented. The description adds no guidance about jobClass choices, the purpose of the required description field, or why estimatedMinutes matters. The enum values 'small/medium/long' and numeric bounds give minimal hints, but the description fails to compensate for the low schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names the action context: call before costly boundaries such as builds, deploys, migrations, and training jobs. The phrase 'Obey defer decisions' signals that the tool returns a go/defer judgment, which distinguishes it from the sibling tools. It does not fully state what preflight checks or computes, but it is clear enough for an agent to know it is a job-approval gate.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use guidance: call before long builds, tests, deploys, migrations, training, or packaging jobs. The examples give concrete triggers. It stops short of saying when not to call and does not explicitly name an alternative tool such as defer_until_reset, though 'obey defer decisions' implies that follow-up path.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• defer_until_resetB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden. It does disclose key behavior: creating a checkpoint, preparing a prompt, and not creating the automation itself. Still, it omits side effects, prerequisites, persistence, or response behavior, so transparency is partial rather than complete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences carry the full description with no filler. The main action is front-loaded, and the caller responsibility is stated separately and clearly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has nine parameters, no annotations, and no output schema, so the description must compensate. It explains the high-level purpose but not how to populate required inputs, what the prepared prompt should contain, what the tool returns, or what the five-hour reset workflow expects. This is not enough for reliable invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 22% (taskId and workspaceRoot have descriptions), yet the tool description names none of the nine parameters. Required fields like objective, completed, and pending are left for the agent to infer from context, which is insufficient for a tool with four required parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific action — 'Create a checkpoint and prepare a same-task Codex heartbeat automation prompt' — and gives the context ('for the five-hour reset'). It is clear about the tool's role and distinct enough from sibling checkpoint_create by mentioning the automation prompt, though it never names the sibling explicitly.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'for the five-hour reset' and 'The caller must create the automation' provide clear usage context and a caller responsibility. However, it does not explain when to choose this over checkpoint_create, checkpoint_get, or quota_status, leaving alternative selection mostly implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• checkpoint_createA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Because no annotations are present, the description carries the full disclosure burden. It does add useful behavioral context: checkpoints must be redacted, must not include credentials/full prompts/full responses, and live in shared local state. However, it does not disclose side effects like overwriting an existing checkpoint, idempotency, or failure behavior, so disclosure is only partial.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences that front-load the primary purpose before adding the security constraint. Every sentence earns its place, and there is no unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with nine parameters, no output schema, no annotations, and four required fields, this description leaves too much unstated: how to fill required parameters, whether the checkpoint is keyed or overwritten, and what the tool returns. It provides the core purpose and redaction guideline, but an agent would still need to infer or probe much of the contract.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With only 22% schema description coverage, the description is expected to compensate for undocumented parameters, but it names none of them. The redaction warning provides generic content guidance but does not clarify the meanings of objective, completed, pending, lastTest, gitStatus, resumeNotes, pendingCommand, or taskId beyond what their names imply.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'persist' with the resource 'checkpoint' and the destination 'shared local state', so it clearly states what the tool does. It also distinguishes the create/persist behavior from the sibling checkpoint_get by naming the action explicitly.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is used when an agent wants to save a resumable checkpoint, but it does not state when to choose this tool over the sibling tools or provide any exclusion criteria. There is no explicit mention of checkpoint_get for retrieval, so usage guidance remains implicit rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• checkpoint_getA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the full burden. It does convey that this is a read operation and that omitting checkpointId returns the latest checkpoint, which is useful behavioral information. However, it does not disclose return format, error behavior for missing checkpoints, or any permission/authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no filler. The core action, resource, and key distinction are front-loaded and immediately actionable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple, the required parameter is present, and the main selection behavior is explained. There is no output schema, so a bit more detail on what the returned checkpoint contains would improve completeness, but it is not necessary for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning beyond the schema by explaining that checkpointId is optional and that its absence means 'latest checkpoint.' It also ties workspaceRoot and taskId to the scoping of the read. The schema covers two parameter descriptions, but checkpointId itself has no description, so the tool description fills that gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Read') and resource ('checkpoint'), and differentiates the two access modes: a specific checkpoint or the latest checkpoint. This makes it obvious how it differs from sibling checkpoint_create even without naming it.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The usage context is implied: it is the tool to call when you need to retrieve a checkpoint for a workspace/task. It does not explicitly discuss when not to use it or compare with alternatives, but the sibling tools are not direct alternatives for this read operation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• quota_statusA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral disclosure burden. It meaningfully reveals that the server enforces cache TTL, lease, and backoff and that callers cannot force refresh. This goes beyond the empty schema and helps set expectations for callers.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two compact sentences with no wasted words. The key action and resource appear first, and the behavioral caveat follows directly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has zero parameters, no annotations, and no output schema, the description covers the core purpose and the most important runtime constraints. It doesn't state the return value format, but for a simple quota snapshot read this is a minor gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so the schema is already fully descriptive and there is nothing for the description to add. Baseline 4 applies as the tool has no parameter surface requiring explanation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('Read') and a specific resource ('shared adaptive Codex quota snapshot'), clearly distinguishing this tool from the sibling tools, which involve preflight checks, checkpoints, and deferral. Its purpose is immediately obvious.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when a caller needs the current quota snapshot and clearly states that a forced refresh is not possible. It does not explicitly name alternatives or when-not conditions, but none of the sibling tools appear to overlap with quota status, so the guidance is adequate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.