Server Quality Checklist

Tool Scores

• deep_analysisB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must fully disclose behavioral traits. It mentions the output includes 'percentile rankings, risk signals, and behavioral patterns', but does not state whether the operation is read-only, if it requires any authentication, or if it has side effects. This is insufficient for a tool with no annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the action and resource, followed by a use case. Every word is functional with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Although the tool is simple with one parameter and no output schema, the description could be more complete by elaborating on how the 'deep trust analysis' differs from sibling tools like get_agent_reputation or get_agent_trust. The current text is adequate but leaves room for ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% coverage with a clear description for the 'address' parameter. The description adds no additional meaning beyond the schema, justifying the baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves 'deep trust analysis' for an ACP agent address, distinguishing it superficially from siblings like get_agent_reputation and get_agent_trust by implying a more thorough analysis. However, it does not explicitly differentiate from these similar tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises using this tool for 'thorough due diligence', providing clear context for when it's appropriate. However, it lacks explicit guidance on when not to use it or direct comparisons to alternatives, leaving the agent to infer usage from the name and context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_agent_trustA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must disclose behavior. It describes output but does not explicitly state that the tool is read-only or mention any permissions or side effects. The instruction to 'report outcome' hints at usage but not behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences: first defines function and output, second gives a procedural instruction, third a suggestion. Concise, though the procedural advice might be considered out of place for a tool description.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with no output schema, the description covers the essential purpose, output, and follow-up actions adequately. Missing details like error handling or data format are minor given the simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 100% coverage for the single parameter, so baseline is 3. The description adds no additional parameter semantics beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it gets the Maiat trust score for an ACP agent address, returning score, verdict, and breakdown. However, it does not differentiate from the sibling 'get_agent_reputation' which might have similar function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides post-usage instructions (report outcome, leave review) but lacks explicit guidance on when to use this tool versus alternatives like 'get_agent_reputation'. The context for use is implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• report_outcomeA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must fully disclose behavior. It mentions feeding the oracle but omits details like idempotency, side effects, or escalation on failure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with verb and resource, no fluff. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with 4 params and no output schema. Description covers purpose but lacks details on constraints, defaults, or behavior variations. Adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so parameters are well-documented. Description adds context about the trust oracle but does not enhance understanding of individual parameter semantics beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the verb 'report' and resource 'outcome of a job', and explains its role in feeding the trust oracle. Differentiates from sibling tools like deep_analysis or list_agents.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies usage after job execution but does not provide explicit when-not-to-use or alternatives. No guidance on prerequisites or conflicts with siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_scarab_balanceA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It implies a read-only query but does not explicitly state safety, permissions, or potential side effects beyond the obvious balance check.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no waste. First sentence immediately states purpose, second adds context about the token's role.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Simple tool with one parameter and no output schema. Description is sufficient for an agent to understand the token and the needed input, though it could mention the return type (e.g., number) explicitly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, baseline 3. Description does not add meaning beyond the schema's 'address' parameter; no examples or format hints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states verb 'Get' and resource 'SCARAB token balance for an address'. Distinguishes from sibling tools like get_agent_reputation and get_token_forensics by specifying the exact token and operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives, no mention of when not to use it, and no explicit context for preferred usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_token_forensicsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It describes the tool as reading data and lists analysis types, but does not disclose potential side effects, error behavior, or permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One sentence covering purpose and key outputs, no wasted words. Could improve structure by separating usage notes.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 2-param tool with no output schema, description provides enough context about what it does. It lacks detail on result format or edge cases but is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds no extra parameter meaning beyond the schema's field descriptions, which are already clear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get forensics and safety data for a token contract address.' It lists specific outputs (honeypot detection, rug pull risk, liquidity analysis), distinguishing it from siblings like deep_analysis or get_agent_reputation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (when needing token safety analysis) but lacks explicit guidance on when not to use or alternatives. No mention of prerequisites or handling invalid addresses.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• trust_swapA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses two key behaviors: 'Checks both tokens for safety' and 'returns a Uniswap quote with calldata'. However, it does not cover potential failure modes, gas costs, or other side effects, leaving gaps in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise: two sentences that immediately state the tool's action and key differentiator. Every word is necessary—no redundancy, no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema, the description covers the main return value (Uniswap quote with calldata) and the safety check behavior. It lacks details on quote format, error conditions, or potential limitations, but for a simple swap quoting tool, the provided information is largely sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% parameter description coverage, so baseline is 3. The description does not add new semantic information beyond what the schema already provides (e.g., token addresses, amount in wei, slippage tolerance). The safety-check behavior is mentioned but does not directly enrich parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: getting a trust-verified swap quote with calldata, including safety checks on tokens. It explicitly distinguishes from raw DEX quotes with 'Use this instead of raw DEX quotes', which provides strong differentiation from sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear guidance to use this tool instead of raw DEX quotes for safety. While it does not explicitly list when not to use, the context suggests alternatives (raw DEX quotes) and the safety benefit is highlighted. More explicit exclusions would improve the score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_agentsA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the full burden. It explicitly states 'browse' and 'returns', indicating a read-only operation. No hidden behaviors are disclosed, but for a simple list, this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary purpose, and contains no extraneous information. Every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one optional param, no output schema), the description fully explains what it does: browse agents with trust scores, paginated. No gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the parameter 'limit' is fully described in the schema. The tool description adds no additional context beyond what the schema already provides, so baseline score applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool browses indexed agents with trust scores and returns a paginated list, distinguishing it from sibling tools that focus on individual agents or specific operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for listing all agents, but provides no explicit guidance on when to use it versus alternatives like get_agent_trust or deep_analysis. Usage is clear but not contrasted.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submit_reviewA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must cover behavior. It discloses wallet assignment, cost (5 Scarab), and potential rewards. However, it does not mention idempotency, updateability, error cases, or confirmation of submission, which are important for a write operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, no filler. Front-loaded with purpose, then usage timing, then cost/reward. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema, so return format is missing. The tool is simple with 4 params, but missing what the agent gets back (e.g., review ID, success status). Adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% description coverage, so baseline is 3. The description adds value by advising on comment quality: 'cite tx hashes, completion rates, or concrete evidence for higher quality scores,' which enhances parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states verb and resource: 'Submit a review for an on-chain entity (agent, token, protocol).' It also specifies when to use, distinguishing it from read-only siblings like get_agent_reputation or vote_review.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use after completing a job, swap, or interaction,' providing clear context. Lacks explicit when-not-to-use or alternatives, but the context is strong enough for an agent to decide.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vote_reviewA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses a behavioral consequence (upvotes reward the reviewer +2 Scarab) beyond the basic action. Without annotations, this adds valuable transparency, though downvote effects are not stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences provide all necessary information without redundancy. Every word serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and simple schema, the description covers purpose, usage guideline, and a key behavioral effect. Lacks output details, but acceptable for this tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. Description adds no new information beyond schema; it restates the enum meanings but does not elaborate on formats or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (upvote/downvote a review) and the resource (review), distinguishing it from siblings like submit_review or report_outcome.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises voting only with firsthand experience, providing clear guidance on when to use. Does not discuss alternatives or when not to use, but sufficient for context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_agent_reputationA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full weight. It clearly implies a read-only operation ('Get') and describes the output content (reviews, rating, sentiment, market consensus). While it does not explicitly state 'no side effects' or 'read-only', the verb and context are sufficient for transparency. Minor gap: does not mention if any authentication or rate limits apply.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no wasted words: definition, usage guidance, and call to action. All information is front-loaded and essential.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has only one parameter and no output schema, the description covers purpose, usage guidance, and a follow-up action. It lists output components but does not specify the exact data structure or format. For a simple 'get' tool, this is mostly complete; a score of 4 reflects minor missing details about return format.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one parameter 'address' with a full description (100% coverage). The description does not add any additional meaning or context beyond what the schema already provides. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Get') and clearly identifies the resource ('community reputation for an ACP agent'). It lists the components (reviews, average rating, sentiment analysis, market consensus) and distinguishes itself from sibling tool 'get_agent_trust' by stating they are complementary.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells when to use this tool ('Use this alongside get_agent_trust for a complete picture') and provides a follow-up action ('After acting on this data, report the outcome via report_outcome to earn 5 Scarab'). This gives clear contextual guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.