Server Quality Checklist

Tool Scores

• get_screenshotA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral burden. It does disclose that no API key is needed and that it returns a PNG image block, which adds useful context. However, it omits details like failure behavior, whether the screenshot is full-page or viewport, and any resource or rate-limit implications, leaving clear gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The first two sentences are concise and front-loaded. The third sentence is somewhat wordy but provides valuable use-case rationale. Overall, it is not overly long and each sentence contributes meaningful information, though the third could be tightened.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and no output schema, the description covers the key points: what it returns (PNG image block), authentication requirements (none), and intended use case (visual brand-impersonation judgment). Missing details like error conditions and exact scope of the screenshot prevent a perfect score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description never explicitly explains the `uuid` parameter beyond the schema's bare property name. While the context implies it identifies the page/scan, the description itself adds no meaning about what value to provide or where to obtain it, failing to compensate for the low schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description begins with a specific verb and resource: 'Fetch the page screenshot as an image you can actually look at.' It clearly distinguishes the tool from siblings like get_dom by emphasizing the visual image output rather than text/DOM data. The screenshot purpose is also explicit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for use: 'so a vision-capable model can judge whether a page impersonates a brand' and 'the thing no field in the JSON captures.' This implies when the tool should be used over other scanning tools, though it does not explicitly mention sibling alternatives or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_domA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, the description fully discloses behavior: partial document retrieval, API key requirement, behavior with and without `contains`, case-insensitive matching, window and max_chars semantics, and a security warning that returned markup is untrusted and should not be interpreted as instructions. This exceeds typical transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured, front-loading the core purpose, then behavior, then security, then use cases. Every sentence adds value without redundancy; the length is justified by the tool's complexity and the absence of annotations.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present, the description doesn't need to explain return values. It covers authentication, operational behavior, security implications, and typical search patterns, making it sufficient for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With zero schema descriptions, the description compensates by explaining `contains`, `window`, and `max_chars` in actionable terms. `uuid` is not explicitly described, but the context ('DOM urlscan captured') implies it identifies the scan. Defaults and ranges are left to the schema, which is acceptable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool reads part of the DOM captured by urlscan, never the whole document. The verb and resource are specific, and the sibling context (get_screenshot, scan_url) is implicitly differentiated by emphasizing DOM content retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit use cases: locating credential form posts, searching for exfiltration hosts, payload markers, or brand names. It also mentions API key requirement and the 403 error, but doesn't explicitly name alternative tools or provide when-not-to-use conditions beyond the whole-document limitation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_scan_resultA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully carries behavioral disclosure. It explains API key requirement and 403 behavior, output size difference between summary and full, and the 404 polling behavior. All these add real behavioral context beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, then proceeds logically through prerequisites, return content, the optional full-detail behavior, and polling edge case. Every sentence adds value; no wasted phrases or repetition of schema fields.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there is an output schema and no annotations, the description covers the important contextual aspects: auth, result content, size trade-offs, and error handling. It does not need to spell out the return structure since an output schema exists, so the description is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explicitly explains the 'detail' parameter with its enum values and impact on context window. The 'uuid' parameter is implicitly clear from the context of fetching a specific scan, and the description notes the relationship to recently submitted scans.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb+resource+scope: 'Fetch a completed urlscan.io scan as a summarised verdict.' It distinguishes itself from siblings by focusing on retrieving an existing scan's summary verdict rather than submitting, searching, or providing visual/DOM outputs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear usage context: use when a scan is completed, poll again on 404 for fresh scans, and use detail='full' sparingly because it consumes context. However, it doesn't explicitly name sibling tools as alternatives for other scenarios (e.g., use search_scans to find scans).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pivotA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses pagination (one page, check total, use next_cursor), escaping of untrusted indicators, and matching semantics (domain/page.url analyzed vs exact for ip/page.asn/hash). It also warns to read hit fields before treating as match. This goes well beyond the absent annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than average but every sentence adds critical context: pagination, matching semantics, security, API key requirement, and usage example. No fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (multiple indicator types, pagination, override behavior), the description covers all essential caveats. The output schema exists, so return value details aren't needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has no descriptions, but the description explains indicator auto-detection and field mapping (domain, ip, page.asn, hash, page.url), and the indicator_type override for filename/tls_issuer. Pagination via next_cursor is mentioned, which likely maps to search_after, though size is not explicitly explained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Find scans touching one indicator, without writing the query yourself,' which clearly states the tool's function. It distinguishes from siblings like search_scans by emphasizing indicator-based pivoting and automatic field selection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states 'This is the tool to reach for repeatedly' and gives a concrete workflow (pivot on IP, then ASN, then script hash). It also explains when to use indicator_type overrides and the pagination behavior, providing clear when-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_urlA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden and does so thoroughly: requires API key, returns immediately with UUID (scan not finished), fetches URL from urlscan infrastructure (real interaction with hostile site), and explains visibility as a disclosure decision. All key behaviors are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Every sentence provides necessary operational or security context. The paragraph structure logically flows from high-level purpose to behavioral notes to parameter-specific guidance. No padding or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is complex with asynchronous behavior, security implications, and nuanced visibility choices. The description covers prerequisites, async pattern, safety warnings, and parameter semantics. Output schema exists, so return values need not be documented.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so description must compensate. It explains tags (max 10), country (requests scanner location, e.g., 'fr'/'us'), and visibility (enum meanings and default). URL is self-evident. Slightly more detail on tag format could push to 5, but coverage is very good.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Submit a URL to urlscan.io for scanning,' a specific verb+resource statement. It clearly distinguishes from siblings by referencing get_scan_result and explaining the async flow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to call get_scan_result after poll_after_seconds, explains when to use 'public' visibility, and warns not to submit URLs with session tokens or password-reset links. Provides strong when-to-use and when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_scansA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does so excellently. It reveals error behavior (unknown top-level fields return an error; dotted-field typos return zero hits without error), result-limit semantics ('A total of 10000 means "at least 10000"'), and pagination mechanics ('Page forward by passing `next_cursor` back as `search_after`'). These details go far beyond what annotations would typically provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but every sentence serves a purpose: purpose, auth, usage differentiation, field list, error nuance, examples, and pagination. It is well-structured in short paragraphs with code examples, and no information is repeated or wasted.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with ElasticSearch query syntax, the description covers the query language, supported fields, error edge cases, result cap, pagination, and even authentication. Given the output schema exists, it need not detail return values, and the provided context is sufficient for an agent to use the tool effectively across the sibling set.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It provides rich semantics for the `query` parameter through field lists and examples (e.g., 'domain:"evil.tld"'), and explains `search_after` via the pagination note. However, the `size` parameter is never mentioned, leaving its semantics entirely to the schema (which shows a default of 20 but no explanation). Minor gap, but the description makes the two most complex parameters clear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb and resource: 'Search historical urlscan.io scans with an ElasticSearch query string.' It also distinguishes itself from the sibling tool `pivot`, which is explicitly mentioned for single-indicator use cases, making the purpose and differentiation unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to use this tool ('Use this for free-form queries') and when to use an alternative ('use `pivot` when you have a single indicator and want the field chosen for you'). It also notes the auth requirement ('Works without an API key') and provides concrete query examples, giving the agent clear context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.