Server Quality Checklist

Tool Scores

• search_webC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It only says 'Search the web' and implies payment via two methods, but does not disclose whether payment is mandatory, if there are limits, authentication needs, or any side effects. Minimal behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, concise and to the point. It could be improved by placing the main action first, but it is not overly verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema (which relieves explanation of returns), the description lacks crucial details: how payment works, what happens without payment, how to choose payment method, and any constraints on max_results. Incomplete for a tool with 4 parameters and payment implications.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description must compensate. It adds meaning by explaining that payment_hash is for Lightning and tx_hash for Arbitrum ETH. However, it does not explain query or max_results, so compensation is partial.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Search the web' which is a specific verb+resource. It distinguishes from siblings like search_news (different domain) and invoice tools. However, mentioning payment methods adds potential confusion about the primary action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool vs alternatives. The description mentions payment methods but does not clarify when payment is required or how to choose between them, leaving the agent without decision context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_newsB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description adds value by disclosing payment requirements (Lightning or Arbitrum). However, it omits other behavioral traits such as error handling, rate limits, or output structure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise with only two sentences, front-loading the core purpose and then adding the payment detail. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 4 parameters (1 required) and an output schema, the description covers the payment mechanism but lacks detail on query meaning, result limits, and when to use. Adequate but incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains 'payment_hash' and 'tx_hash' but does not clarify the 'query' or 'max_results' parameters, missing crucial semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Search recent news,' which is a specific verb and resource. However, it does not differentiate from sibling tool 'search_web,' leaving potential ambiguity about scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool vs alternatives like 'search_web' or 'get_invoice.' The description only mentions payment methods, not usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reportA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility for behavioral disclosure. It only says 'Helps Giskard improve', implying a write operation, but does not mention side effects, permissions, idempotency, or whether the operation is reversible. For a feedback tool, this is a notable gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise—four short lines total. It front-loads the purpose, then succinctly defines parameters. No fluff or redundant phrasing. Each sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple feedback tool, the description covers purpose and parameters adequately. However, it does not mention the output (even though an output schema exists per context), nor does it provide any usage notes or prerequisites. It is minimally complete but lacks some context about the return value.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has titles but no descriptions (0% coverage). The description compensates by clearly explaining each parameter: 'useful: True if the result helped you, False if it didn't' and 'note: optional — what was missing or what worked well'. This adds meaningful semantics beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Report whether the search was useful.' This distinguishes it from sibling tools which are about retrieving data (invoices, news, web). The explicit verb-resource pair and mention of improving Giskard make the intent unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied: the tool is for providing feedback after a search. However, there is no explicit guidance on when to use it versus alternative tools, nor any when-not-to-use conditions. Given the simple nature, the implicit guidance is adequate but not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_arbitrum_invoiceA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility. It only states 'Get payment info' without disclosing behavioral traits like idempotency, safety, or authentication needs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence that earns its place with clear purpose and no extraneous words. Perfectly concise for a simple tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters and an output schema exists, the description adequately conveys the tool's core function. However, it could briefly note that this is an alternative payment method to Lightning.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so schema coverage is 100%. The description does not add parameter info, but none is needed; baseline for zero parameters is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves payment info for paying with ETH on Arbitrum, distinguishing it from the sibling 'get_invoice' which is implied for Lightning.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for ETH-on-Arbitrum payments versus Lightning, but lacks explicit when-to-use/when-not-to-use guidance or alternative tool references beyond the implicit contrast.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_invoiceA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses pricing tiers and the optional agent_id parameter's effect on cost, which is helpful. However, without annotations, it fails to specify whether the tool is read-only, if it creates state (e.g., a pending invoice), or any side effects like rate limits or authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief at two sentences, conveying purpose and key pricing details efficiently. However, the tier list formatting is slightly dense and could be structured more clearly (e.g., as a table).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose and pricing but omits details about the output format, the relationship between karma and Giskard Marks, and any prerequisites (e.g., having a Giskard Marks account). The presence of an output schema mitigates the need for return value explanation, but behavioral context like potential side effects is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description explains the agent_id parameter's purpose (identity in Giskard Marks) and its behavioral impact on pricing. Given the schema has 0% coverage (no parameter descriptions), the description fully compensates, adding meaning beyond the schema's type and default.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves a Lightning invoice for payment before searching, which is a specific verb and resource. It distinguishes itself from sibling get_arbitrum_invoice by implying a different blockchain (Lightning vs Arbitrum).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description indicates when to use the tool ('before searching'), providing clear context for invocation. However, it lacks explicit guidance on when not to use it or alternatives, such as other payment methods or prerequisites like having a Giskard Marks account.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.