Server Quality Checklist

Tool Scores

• breach_statsA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint. The description adds valuable behavioral context: 'Aggregate counts only; no leaked records' clarifies data granularity, and the mention of the five largest incidents sets output expectations. This goes beyond annotations and does not contradict them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is moderately sized and well structured: a clear main purpose statement followed by a list of aggregation axes and example use cases. The examples are useful but slightly redundant; every sentence still earns its place, so it does not drop below 4.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description explains the result type (incidents and accounts exposed counts) and explicitly states that no leaked records are returned. It does not specify output formatting or pagination, but for a read-only stats tool with strong annotations and a simple parameter set, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers both parameters (sector and group_by) with descriptions and allowed values, achieving 100% coverage. The description restates the group_by options in natural language and gives usage examples, but adds no new parameter-level detail beyond what the schema already provides. Thus baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Aggregate the full breach archive into analyst-grade statistics.' It enumerates distinct aggregation axes (year, source, data_type, threat_level, actor) and mentions the five largest incidents, clearly differentiating it from siblings like breach_timeline or check_exposure.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides concrete example questions ('how has breach volume trended since 2015', 'which ransomware groups have the most victims') that signal when to use the tool. It does not explicitly name alternative tools or state when not to use it, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• breach_historyA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already signal read-only/idempotent/open-world, so the description only needs to add context beyond those hints. It does so by explaining the return boundary: 'Returns disclosure metadata only, never breach contents' and enumerating the source archives covered, giving strong transparency for a read tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three structured sentences front-load the verb/resource, then detail sources, filters, and examples without fluff. Slightly longer than strictly necessary but each clause carries information; not quite the two-sentence minimalism of a perfect score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no output schema, the description explicitly states the return type ('disclosure metadata only') and the full scope of sources, filters, and ordering options, matching the tool's 8-parameter complexity. The examples further anchor when to use it, making the description effectively complete for agent selection.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All 8 parameters have complete schema descriptions, so the tool's parameters carry the semantic weight. The description paraphrases the filter options ('keyword, year range, sector, exposed data type or minimum scale') but adds no new parameter-level details beyond the schema; baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description opens with 'Search the FULL historical breach archive'—a specific verb+resource with clear scope (every incident back to 2007, multiple named sources). The examples ('what were the biggest breaches of 2013') further clarify intent and distinguish from sibling news/stats tools, though not by explicit alternative naming.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  'Use this for questions like...' provides concrete application scenarios. It doesn't explicitly say when not to use (e.g., for current news use breach_news) or name alternatives, so it falls short of full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• assess_threatA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds meaningful behavioral context beyond the annotations: 'Pure local analysis: it collects nothing, stores nothing and reaches no network; the text never leaves the server.' This privacy/network guarantee is not visible in the annotations and is valuable for an AI agent handling sensitive data. It also lists the output fields, complementing the readOnly and idempotent hints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences and 69 words, with the core action and outputs stated first, followed by privacy guarantees and usage guidance. Every clause adds value without repetition or fluff, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description enumerates all key return elements (threat level, categories, flags, confidence, action) and clearly defines the single input parameter. The tool is simple (one required parameter, no enums/nested objects), and the description covers its purpose, behavior, and usage context fully. It is complete for an agent to decide when and how to call it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The parameter 'text' is fully described in the input schema (100% coverage) with the same phrasing used in the tool description ('security text to classify — an advisory, alert or forum post'). The description adds no new semantic detail beyond what the schema already provides, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Classify') and names both the input ('security text you supply — an advisory, alert or forum post') and the outputs ('threat level, matched categories, financial-target flags, a confidence score and a recommended action'). It also distinguishes itself from siblings by framing the tool as a triage step for findings from 'breach_news' or personal monitoring, which clearly differentiates it from feed or exposure tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear use case: 'Use it to triage findings surfaced by breach_news or from your own monitoring.' This provides context for when to invoke the tool relative to siblings, though it does not explicitly state when not to use it or name alternative tools. The contrast with sibling tools (news fetching, exposure checking) is implicit but sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• feed_sourcesA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint, and the description adds further behaviors: it includes an 'honest staleness flag', explicitly avoids credential/PII output, and clarifies the scope (public feeds only). This goes beyond the annotation baseline without contradicting it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is slightly longer than minimal but packs each sentence with value: the first sentence lists outputs, the second clarifies scope, and the third gives usage guidance. No filler, though it could be trimmed without losing meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description fully explains what the tool returns and its operational limits. For a zero-argument list tool, it provides complete guidance—including when to consult it—so an agent can invoke it correctly and interpret results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With zero parameters, the schema already fully covers inputs (100% coverage), but the description proactively states 'Takes no arguments,' eliminating any ambiguity. This adds clarity beyond the empty schema, aligning with the baseline for parameterless tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and names the resource ('public disclosure feeds'), plus enumerates the exact output fields (disclosure counts, newest item, staleness flag, cache ages). This distinguishes it clearly from sibling tools like breach_timeline or check_exposure, which focus on different data or actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly tells the agent when to use it ('Check this first if another tool's answer looks thin') and gives a rationale ('a stale live feed is a finding, not background noise'). It also states exclusions ('no .onion access, no arbitrary fetching or crawling'), preventing misuse.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• breach_newsA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark readOnly/openWorld/idempotent, but the description adds crucial safety context: 'Every row is metadata only — never the leaked data' and a redaction pass strips credential-shaped content. It also discloses output ordering ('newest first'), exceeding the minimal annotation bar without contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three dense sentences front-load the core verb and resource, then deliver critical safety and usage distinctions. Every clause earns its place with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only list tool with no output schema, the description explains the output fields (entity, date, scale, exposed data types, threat level, source), ordering, filtering, and safety redaction. It also covers alternatives for different use cases, making it fully self-contained for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with detailed parameter descriptions, so the baseline is 3. The description adds slight reinforcement ('Use sector to narrow to an industry keyword') but does not provide significant new meaning beyond the schema's already explicit filter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific action ('Read recent breach and ransomware DISCLOSURES') and resource (public threat-intel feeds) with concrete sources and ordering ('newest first'). It also distinguishes itself from siblings by explicitly pointing to check_exposure for one organization and breach_history for all-time history.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit usage context: 'Use sector to narrow to an industry keyword' and names alternatives for specific use cases ('for one specific organization use check_exposure; for all-time history use breach_history'). This gives clear when-to-use and when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• breach_timelineA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnly/idempotent annotations, the description discloses ordering ('oldest first'), the analytical outputs (first/latest incident, per-year counts, repeat victim status, worst threat level, total accounts), the metadata-only policy, and the forward-looking risk interpretation of repeat victimhood. This is substantial behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise yet information-dense. It front-loads the core purpose, then adds key output details, a risk interpretation, a data policy statement, and an explicit alternative — each sentence earning its place without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema, so the description must explain return values. It does so thoroughly: chronological incident list, summary judgments (first/latest, per-year, repeat victim, worst threat, total accounts), and the metadata-only nature. Given the single parameter and clear annotations, the description fully completes the picture.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (single 'entity' parameter with a clear description and examples). The tool description adds only minimal reinforcement ('one organization'), but does not need to add much since the schema already explains the parameter thoroughly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action: 'Build the complete incident-by-incident CHRONOLOGY of one organization across every source and all history, oldest first'. It distinguishes itself from siblings by explicitly naming 'check_exposure' for a different use case ('yes/no presence check'), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear when-to-use guidance by defining the full chronology scope, and explicitly points to 'check_exposure' as the alternative for a simple presence check. It also includes a 'Metadata only; never the leaked data' note, setting expectations for what the tool is not for.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_exposureA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and the description adds crucial non-obvious context: it returns aggregated metadata but 'never the exposed records themselves,' and it is 'a triage signal built from disclosure feeds, not proof of compromise.' This goes beyond the annotation flags with meaningful behavioral caveats.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, each earning its place: the first states purpose and outputs, the second explains the caveat and required confirmation, and the third names sibling tools. It front-loads the core action and avoids redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no output schema, the description thoroughly explains the return content (mention count, worst threat level, total accounts, data types, metadata) and what is deliberately excluded. It also covers interpretation and alternative tools, making the description complete for a 2-parameter triage tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both parameters documented in the schema. The description does not add new parameter-level detail beyond restating 'domain, company or brand' and the all-history scope, so it fails to exceed the baseline of 3 for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb phrase ('Answer whether... appears') and clearly identifies the resource ('public breach or ransomware disclosures'). It enumerates the specific outputs (mention count, worst threat level, total accounts, data types) and distinguishes itself from siblings by naming breach_timeline and breach_news as alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use this tool vs alternatives: 'For the incident-by-incident chronology of one entity, use breach_timeline; for a recent-news sweep, use breach_news.' It also provides the triage context and advises confirmation through authorized channels, guiding appropriate use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.