Server Quality Checklist

Tool Scores

• costwright_certifyA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses key traits: tamper-evident, Ed25519-signed, server-side re-run, logged to public transparency log. With no annotations, description carries burden and does well, though prerequisites (auth, network) are omitted.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with action and key features. No extraneous information; every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose, key features, and return values (cert_id, signed cert, verify_url). No output schema, but description compensates. Missing detail on 'policy' but overall sufficient for a certification tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 67% but description adds little beyond 'same input as check plus label'. Does not explain 'policy' enum values (default vs strict) or the meaning of 'label'. Relies heavily on schema which is minimal.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it issues a tamper-evident Ed25519-signed budget certificate. Distinguishes from siblings by mentioning same input as costwright_check plus optional label and specific returns (cert_id, signed cert, verify_url).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies use for auditable proof of spend ceiling and compares to costwright_check, but does not explicitly contrast with costwright_pubkey or costwright_verify. Lacks explicit when-to-use vs when-not-to-use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• costwright_checkA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses that the tool builds a .py-only archive, performs static analysis, and returns a summary with counts and verdict. Some details like file size limits or performance are missing, but the key behaviors are transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences cover purpose, operation, and usage context with no fluff. Front-loaded with the core action ('Statically analyze') and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema and two simple parameters, the description provides sufficient context: output includes categorized counts and a verdict. Could be slightly more explicit about output format, but overall complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so both parameters have descriptions. The description does not add additional semantics beyond the schema, but the baseline of 3 is appropriate since it doesn't contradict or confuse.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: static analysis of LLM-agent repos for runaway-budget risk, specifying supported frameworks and output summary categories. This differentiates well from siblings (certify, pubkey, verify).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use ('before deploying a workflow') and that it runs without executing code. However, no explicit when-not-to-use or alternative tools are mentioned, so it loses a point.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• costwright_verifyA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description fully covers behavior. It clearly states the operation is a read-only verification (no key needed), returns state, signature check, and certified result. It does not mention idempotency or error handling, but for a simple verification tool, this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single well-structured sentence that efficiently conveys purpose, scope, and output. No extraneous words; every part contributes value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple verification tool with one parameter and no output schema, the description provides sufficient context: what the tool does, how it is used (public, by ID), and what it returns (state, signature, result). It is complete for its complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a clear description 'The certificate id to verify.' The tool's description adds no additional meaning to the parameter beyond what the schema provides, thus baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'verify', the resource 'costwright certificate by id', and adds context 'public, no key'. It distinguishes from sibling tools like costwright_certify (issuance) and costwright_pubkey (key retrieval).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description indicates this tool is for verifying a specific certificate ID and notes it is public without a key, implying no authentication needed. However, it does not explicitly state when to use this tool versus its sibling 'costwright_check', which might have overlapping functionality.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• costwright_pubkeyA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavior. It mentions 'active Ed25519 public keys (PEM)' and 'public, no key' indicating no authentication. It does not specify number of keys or format details, but for a simple fetch it is sufficiently transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, well-structured sentence with no filler. Every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters and no output schema, the description covers purpose, format, and usage. It could be more specific about the return structure, but is adequate for a simple public key fetch tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, and schema description coverage is 100% trivial. The description adds no parameter info, but that is appropriate given no parameters exist. Baseline for 0 params is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Fetch', resource 'costwright's active Ed25519 public keys (PEM)', and purpose 'for offline signature verification'. It distinguishes itself from sibling tools (e.g., costwright_verify) by indicating it provides the keys needed for verification.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when needing public keys for offline verification, and contrasts with siblings (certify, check, verify). However, it does not explicitly state when not to use it or provide alternative scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.