Server Quality Checklist

Tool Scores

• get_compliance_reportC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool generates and aggregates, implying a read operation that processes data, but lacks details on permissions, rate limits, output format, or whether it's a safe read-only operation. This is a significant gap for a tool with no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with two clear sentences that directly explain the tool's function without unnecessary words. Every sentence earns its place by stating the action and scope efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of generating a report, lack of annotations, and no output schema, the description is incomplete. It does not cover behavioral aspects like data processing, error handling, or what the report contains, leaving gaps for an AI agent to understand the tool fully.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with the parameter 'framework' fully documented in the input schema. The description adds minimal value beyond the schema by implying the framework is used to tag receipts, but does not provide additional syntax or format details. Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Generate a compliance report mapped to a specific framework. Aggregates all receipts tagged with the specified framework.' It specifies the verb ('generate'), resource ('compliance report'), and scope ('framework'), but does not explicitly differentiate from sibling tools like 'query_receipts' or 'verify_trustatom', which might have overlapping or related functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions aggregating receipts tagged with a framework, but does not specify prerequisites, exclusions, or compare it to sibling tools such as 'query_receipts' (which might retrieve receipts without generating a report) or 'verify_trustatom' (which could involve validation).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_receiptsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions querying for audit/compliance reporting but doesn't disclose critical traits like whether this is a read-only operation, if it requires specific permissions, rate limits, pagination behavior, or what the response format looks like. For a query tool with zero annotation coverage, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, stating the purpose in the first clause. It efficiently lists filtering options in a single sentence without unnecessary details. However, it could be slightly more structured by separating purpose from usage context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what a 'receipt ledger' contains, the return format, or behavioral aspects like safety and limitations. For a query tool with four parameters and no structured output documentation, more context is needed to guide effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds minimal value beyond the input schema, which has 100% coverage. It lists filterable fields (time range, action, actor, decision) but doesn't provide additional semantics like format examples or constraints. Since the schema already documents all parameters thoroughly, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Query the receipt ledger for audit or compliance reporting.' It specifies the verb (query) and resource (receipt ledger) with a clear use case (audit/compliance reporting). However, it doesn't explicitly differentiate from sibling tools like 'get_compliance_report' or 'verify_trustatom,' which might have overlapping audit/compliance functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions filtering capabilities but doesn't specify prerequisites, exclusions, or compare it to sibling tools like 'get_compliance_report' for reporting or 'verify_trustatom' for verification tasks. This leaves the agent without clear usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_trustatomB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden but provides minimal behavioral context. It states the verification purpose but doesn't disclose critical traits like required permissions, rate limits, error conditions, or what happens on failure (e.g., returns error vs null). The mention of 'Ed25519 signature' adds some technical context, but overall disclosure is inadequate for a security-focused tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with zero waste: the first states the purpose, and the second adds technical specificity. It's front-loaded and appropriately sized for a single-parameter tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (security verification), lack of annotations, and no output schema, the description is incomplete. It doesn't explain return values (e.g., success/failure indicators, timestamps), error handling, or dependencies on other tools. For a tool with no structured behavioral data, more context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the 'receipt_id' parameter. The description implies the parameter is used for verification but doesn't add syntax, format, or validation details beyond what the schema provides (e.g., examples of valid IDs). Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('verify integrity'), target resource ('existing TrustAtom receipt'), and technical details ('Ed25519 signature authenticity', 'tampering check'). It distinguishes from siblings like 'create_trustatom' (creation vs verification) and 'query_receipts' (listing vs validation).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing a receipt ID from another operation), exclusions, or comparisons to siblings like 'get_compliance_report' or 'query_receipts' for related verification tasks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_trustatomA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds useful context about performance ('Signing takes <3ms'), cryptographic details ('Ed25519-signed + SHA-256-hashed'), and the auto-computation feature for risk_score. However, it doesn't cover error conditions, authentication requirements, rate limits, or what happens if the signing fails.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured with three sentences: purpose statement, importance rationale, and technical/performance details. Every sentence adds value without redundancy, and it's front-loaded with the core function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, cryptographic operations) and lack of annotations/output schema, the description is moderately complete. It covers the purpose, importance, and some behavioral traits but lacks details on error handling, response format, or integration constraints that would help an agent use it correctly in varied scenarios.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description doesn't add any parameter-specific semantics beyond what's in the schema (e.g., it doesn't explain format examples for 'action' or 'context' beyond the schema's descriptions). Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('sign an AI decision', 'return a cryptographic receipt') and distinguishes it from siblings by focusing on creation/signing rather than retrieval (get_compliance_report, query_receipts) or verification (verify_trustatom). It also explains why this matters ('Every AI decision that matters should have a receipt').

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool ('Every AI decision that matters should have a receipt') and implies usage for creating signed records of decisions. However, it doesn't explicitly contrast with alternatives like query_receipts for retrieval or verify_trustatom for verification, nor does it specify exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.