Server Quality Checklist

Tool Scores

• check_ipA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses quota consumption, a behavioral trait, but does not mention other aspects like read-only nature, latency, or error behavior. This is partial disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no fluff. The purpose and key constraint (quota) are front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool without an output schema, the description lists the return categories and quota cost, providing adequate context for the agent. It could mention limitations or assumptions but is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers 100% of the ip parameter with a clear description, so the baseline is 3. The description adds no new parameter-level detail beyond confirming 'one IP', which does not exceed the schema's content.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: enriching and providing in-network reputation for a single IP address, listing specific data categories. This distinguishes it from check_email/check_phone siblings targeting different entities.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for IP lookups but does not explicitly say when to use it over alternatives or mention exclusions. Sibling names provide context, but the description itself lacks direct usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_eventsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses the ordering (newest first) and the quota-free nature, but does not mention the return format, pagination behavior, or any other side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose and key differentiators. No wasted words; every phrase adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with no output schema and no annotations, the description covers the core aspects: what is listed, ordering, cost, and filtering use case. Lacks response shape details, but that is often implicit for list tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 50%; event and limit have descriptions, offset and verdict do not. The description adds that verdict and event are filters for investigations, but does not explain offset or pagination semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists scored events for the tenant, with newest first. It provides specific verb and resource, but does not explicitly differentiate from siblings like triage_queue or get_stats.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives clear context: the tool is free (does not consume quota) and suggests using filters to narrow an investigation. However, it does not explicitly state when not to use it or mention alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• triage_queueA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It specifies the filter and sort, indicating a read-only query. However, it doesn't disclose the effect of the limit parameter (the description says 'every event' but limit can restrict results), and 'Free' is ambiguous. No contradictions with annotations (none present).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One sentence, highly efficient, front-loaded with the core behavior. The final 'Free' note is extraneous but not harmful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity (one optional param, no output schema), the description covers the main purpose. However, it fails to reconcile 'every event' with the limit parameter, and does not describe the return format or potential pagination. The lack of annotations and output schema places more burden on the description, but it still provides a sufficient overview for a basic list tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of the single parameter with a description (though minimal: 'Default 50'). The property name is self-explanatory, so baseline 3 applies. The tool description adds no parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: returns events with 'review' verdict, sorted by score descending. It distinguishes from sibling tools like list_events by specifying the filter and sort order. The title reinforces this.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context (daily triage job) and implies this is the tool for reviewing high-risk events. However, it doesn't explicitly mention alternatives or when not to use it, preventing a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_phoneA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly mentions consuming one row of monthly quota, which is a key operational detail (rate limit/cost). It also lists the output data points. However, it does not mention any side effects, permissions, or error conditions, which for a simple lookup may be acceptable but leaves some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary purpose, and includes a crucial quota warning without any unnecessary words. It is highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity of the tool (two params, full schema coverage, no output schema), the description adequately covers its behavior and outputs. It lists the returned fields and the quota consumption, but it could be more complete with explicit usage context relative to sibling tools, though that is mostly a usage-guideline issue.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for both parameters, so the description does not need to add parameter meaning. It adds no extra semantics beyond the schema's existing descriptions for 'phone' and 'country', so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: it returns validity, line type, carrier, and fraud score for a single phone number. This specific verb-less enumeration distinguishes it from sibling tools like check_email and check_ip, which target different entity types.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for phone number checks but does not explicitly discuss when to use it versus alternatives like check_email or check_ip. No exclusions or prerequisites are mentioned, so usage guidance is minimal but not misleading.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_eventA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses what the tool returns (event, checks, weights, raw evidence) and that it is 'Free', but it does not explicitly state whether it has side effects, requires certain permissions, or has other operational constraints. This leaves some ambiguity, though the read-only nature is strongly implied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences plus 'Free.', front-loaded with the purpose ('why was this blocked?') and then a compact, informative explanation of the output (every check, weight, evidence). Every word earns its place; no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with no output schema, the description does a good job conveying what the caller will get (event with checks, weights, evidence). It could be slightly more complete by mentioning whether the event itself is returned in full or just the analysis details, but the phrase 'Returns the event with...' sufficiently covers this.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already describes the only parameter (event_id) as 'The event id, as returned by list_events', achieving 100% schema description coverage. The tool description does not add any additional parameter-specific information beyond what the schema provides, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: explaining why an event scored as it did, specifically by returning the event with all checks that fired, their weights, and raw evidence. This specific verb-resource pairing ('explain event') distinguishes it from siblings like score_event (which likely computes the score) and investigate_entity (which sounds broader).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'The ‘why was this blocked?’ tool' gives a clear situational context for when to use this tool. It implies you should use it when you need the reasoning behind a score/block decision rather than just the score itself. However, it does not explicitly mention when not to use it or name alternative tools, so it's not a full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_configA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden. It states the tool is 'Free' and returns tenant-specific config, but it does not explicitly confirm that the operation is read-only, whether any authentication is needed, or what 'Free' means. The 'get' verb implies safety, but more explicit behavioral disclosure would improve transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no redundancy. The first sentence immediately explains the tool's output; the second adds a use case. All words earn their place, and the structure is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple zero-parameter getter, the description covers the essential context: what is returned, that it is free, and when it is useful. No output schema exists, but the description gives enough detail about the config composition. It does not overpromise or omit critical information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and the schema is empty, so there are no parameter semantics to add. Baseline for zero params is 4. The description adds value by clarifying what the configuration contains, which is more than schema alone would provide.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves the tenant's scoring configuration, specifying the exact contents: weight and threshold overrides plus the effective merged engine config. This goes beyond the title and distinguishes it from siblings like get_stats or explain_event.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear use case: 'Useful for explaining why a score landed where it did.' This implies when to use it relative to scoring-related tasks. However, it does not explicitly mention alternative tools or exclusions, so it stops short of full comparative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_emailA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It transparently reports quota consumption, explains canonical alias collapsing behavior, and notes specific return fields like reject_reason. It doesn't explicitly state read-only nature or error handling, but these are reasonably implied by the enrichment context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is moderately detailed but each sentence adds value: purpose, canonical key, specific fields, and quota. The structure is logical, though the first sentence is dense with colon-separated lists. It is appropriately sized for a tool with no output schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Since there is no output schema, the description is the sole source for return values. It lists the main fields (canonical, is_aliased, alias_tricks, reject_reason) and covers quota consumption. It could be more complete by detailing response structure or error cases, but it covers the key behavioral and output aspects for a check tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only describes the parameter as 'The email address to check' with type string. The description adds that the tool accepts one email address (not a batch) and explains the canonical key semantics, providing meaningful context for interpreting the parameter. It stops short of providing format constraints or examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as an email enrichment and reputation lookup, enumerating specific outputs such as disposable domain, deliverability, fraud score, and abuse history. It distinguishes itself from sibling tools like check_ip and check_phone by explicitly focusing on email addresses.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides a clear use case: using the canonical key to compare across accounts to detect duplicate signups. It also mentions quota consumption as a cost consideration. However, it doesn't explicitly state when not to use this tool or reference alternatives beyond the implicit sibling context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• investigate_entityA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden and it discloses key behaviors: cross-operator reputation scope, event retrieval, the one-identifier requirement, and quota costs. However, it does not mention what happens if multiple identifiers are supplied, nor does it clarify the relationship between 'every recent event' and the 'limit' parameter, which is a slight transparency gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences and front-loaded with the core benefit. The first sentence is long but information-dense, and every clause serves a purpose. It is not overly verbose, though it could be split for readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, identifier types, expected outputs (reputation, events), and quota costs. Given no output schema and no annotations, it is reasonably complete for a complex investigation tool, but it omits return format details, error handling, and the exact role of the limit parameter relative to 'every recent event'.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is only 25% (only limit has a description). The description compensates by explaining that email, ip, and device_id are mutually exclusive entity identifiers and that device_id lookups are free. It adds meaning beyond the schema, though it lacks format details or explicit behavior when multiple identifiers are passed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: it enriches an entity, provides reputation across a cross-operator abuse network, and returns recent events to uncover fraud rings. It uses a strong verb-resource pairing ('Returns enrichment for the entity, its reputation... and every recent event') and differentiates from sibling tools like check_email or list_events by emphasizing the investigation use case.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage constraints ('Supply exactly one of email, ip or device_id') and mentions quota implications. It implies when to use this tool ('what a fraud analyst actually wants') but does not explicitly name alternatives or state when not to use it, so it stops short of full exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_statsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses that the tool is free (does not consume quota) and operates over a rolling window, adding behavioral context. It does not explicitly state read-only behavior, but 'aggregate view' strongly implies it. This is useful beyond schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three short sentences: the first states the core functionality, the second adds the free/quota trait, and the third gives usage guidance. Every sentence adds value; no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one optional parameter, no output schema), the description is complete: it explains what is returned conceptually, the rolling window behavior, the free trait, and the suggested usage workflow. No critical information is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single parameter (window_hours), so the schema already fully documents it. The description does not add any extra parameter semantics, but that is unnecessary. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: an aggregate view over a rolling window with totals by verdict, average score, and most common reasons. It also differentiates from siblings by saying 'Start here... before drilling into individual events,' positioning it as the initial overview tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage guidance: 'Start here to see what changed before drilling into individual events.' This tells the user when to use it (first, for an overview) and implies that event-level tools are for subsequent drilling. It also mentions the free/quota aspect, which is a practical consideration.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• score_eventA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full weight and does an excellent job: it discloses monthly quota consumption, that it records an event, and the conditional identity block with email_canonical as a dedupe key. These are serious side effects an agent must know before invoking.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, perfectly front-loaded with the core purpose, then the email behavior, then the quota/recording warning. No wasted words or repetition of schema details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema and no annotations, the description covers the essential return fields, side effects, and usage caveat. It lacks a full per-parameter breakdown, but the schema already lists all parameters and the description focuses on the most consequential behaviors.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 29%, so the description must compensate. It adds meaningful semantics for "email" (identity block, dedupe key) and implicitly for "event" (the scoring trigger), but completely ignores ip, phone, user_id, and device_id—leaving those parameters opaque for the agent.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: "Run an event through the scoring engine and get {score, verdict, reasons, checks}". It also distinguishes this from sibling check_* tools by focusing on scoring whole events and the added email identity dedupe feature.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly advises "prefer the read-only tools when investigating history rather than testing new input", giving a clear alternative and context. It also notes the quota consumption and event recording, signaling this is for live scoring rather than historical investigation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.