Server Quality Checklist

Tool Scores

• download_rapid7_exportA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that this tool downloads Parquet files and loads them into the local DuckDB database for querying, which adds side-effect context beyond the annotations. It also states the completion prerequisite. It does not detail overwrite behavior or failure handling, but idempotentHint and destructiveHint already cover safety expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and front-loaded with the core purpose. There is mild redundancy between the first sentence and the final sentence, both saying the export is loaded into a database, but overall every sentence carries useful information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the key invocation context: call after status is COMPLETE, download Parquet files, and load into DuckDB for querying. With a complete input schema and an output schema present, an agent has enough information to invoke it correctly. The only notable gap is the relationship to load_rapid7_parquet.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema fully documents both parameters with clear descriptions, so the baseline is 3. The description adds no extra parameter-level meaning beyond restating 'completed export,' which the schema already includes.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb and resource: 'Download a completed Rapid7 export' and 'load into the database.' It is distinct from siblings like start_rapid7_export or check_rapid7_export_status, but it does not explicitly differentiate itself from load_rapid7_parquet, since both involve loading Parquet into DuckDB.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says to call this after check_rapid7_export_status confirms the export is COMPLETE, giving clear timing guidance. It does not mention when not to use it or point to alternatives such as load_rapid7_parquet, but the prerequisite is concrete and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• load_rapid7_parquetA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already establish the safety profile (idempotentHint=true, destructiveHint=false, readOnlyHint=false), and the description's 'load' claim is consistent with those. The description adds a performance trait ('much faster than running a full export') and the workflow context of skipping exports. However, it doesn't disclose what happens when data is loaded into a non-empty store (overwrite vs. append), which is the key behavioral unknown for a load tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no filler: the purpose is front-loaded, the usage condition follows, and the performance rationale comes last. Each sentence earns its place, and the whole is well under any reasonable length limit.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with 100% schema coverage, an output schema, and annotations covering idempotency and destructiveness, the description is nearly complete. The only gap is conflict semantics with pre-existing data (does loading replace or merge?), which is a minor omission for an idempotent loader.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% — parquet_path's description states it accepts a file or directory, so the schema already carries the full semantic load. The description's phrase 'Parquet file(s)' reinforces the directory case but adds no new meaning beyond the schema, so the baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb and resource: 'Load vulnerability data from existing Parquet file(s).' This is unambiguous among the siblings, which are all export-management or query tools (start_rapid7_export, download_rapid7_export, query_rapid7, etc.), so there is no confusion about what this tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives an explicit trigger condition: use this when you already have Parquet files downloaded and want to skip the export process. It contrasts this path with running a full export, letting an agent infer that the export siblings (start_rapid7_export, download_rapid7_export) are the alternative. It stops short of naming a specific sibling tool, but the condition is clear enough to route correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_rapid7_export_statusA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false. The description adds useful context beyond annotations by specifying it is a fast, non-blocking call that queries the API once and does not poll or wait, which clarifies the operational behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, front-loaded with the purpose and followed by the key non-polling behavior. Every word earns its place and there is no redundant content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple: one parameter, full schema coverage, rich annotations, and an output schema present. The description adds the crucial behavioral guarantee of non-polling. Nothing material is missing for an agent to select and invoke this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the sole parameter export_id is already fully documented in the schema. The tool description does not add additional parameter meaning, but the baseline of 3 is appropriate when the schema carries the full load.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Check the current status of a Rapid7 export job.' It clearly differentiates itself from sibling tools by emphasizing this is a status check and by noting 'Does NOT poll or wait,' which distinguishes it from any waiting/polling or export-starting workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool: when you need the current status without blocking or waiting. However, it does not explicitly name alternatives or state conditions for when not to use it, such as 'if you need to wait for completion, use start_rapid7_export' or 'use download_rapid7_export when ready.'

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_rapid7_exportsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is covered. The description adds useful behavioral context: it lists only 'recent' exports, scopes them to 'tracked in the system', and specifies the metadata fields returned. This goes beyond the structured annotations without contradicting them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences with no filler. The primary action is front-loaded, and the second sentence adds valuable detail about output contents and intended use. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only listing tool with one optional parameter, a full output schema, and safety annotations already present, the description provides sufficient context. It explains what the tool returns, what the results represent, and why an agent would call it. Nothing essential is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the limit parameter already documented including its default. The description adds no additional parameter semantics, but the schema fully covers the single optional parameter. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description uses a specific verb ('List') and identifies the exact resource ('Rapid7 exports tracked in the system'). It names the metadata fields returned, which distinguishes this from sibling tools like start_rapid7_export or download_rapid7_export. The phrase 'available for reuse' clarifies its role as an inventory/reuse tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for when to use the tool: to see what exports exist and what can be reused. It does not explicitly state when not to use it or name alternatives, but the intended use case is clear enough for a simple listing tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_rapid7A
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is covered. The description adds useful context: the data comes from Rapid7 InsightVM Bulk Export Parquet files and SQL is executed with DuckDB support. This gives the agent expectations about the underlying engine and data source without contradicting the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The opening sentence is direct, and the table-by-table field lists are useful rather than filler for a SQL tool. The examples are compact and illustrative. It is long, but the length is mostly justified by the need to document the data model for query authoring.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex SQL-query tool, the description covers the available tables, representative fields, dialect, and example queries. An output schema exists, so return-value documentation is not the description's job. It does not mention result limits, timeouts, or error behavior, but those are minor given the annotations and the tool's read-only nature.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the single sql parameter, so the baseline is 3. The description adds real value by specifying DuckDB as the SQL dialect, listing queryable tables with key columns, and providing representative examples. This materially reduces ambiguity about what kinds of queries are valid.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb and resource ('Execute a SQL query against the Rapid7 database') and goes beyond the name by enumerating the four available tables and their key fields. This clearly distinguishes it from sibling lifecycle tools like start_rapid7_export, download_rapid7_export, or purge_rapid7_data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use this tool to query any of the above tables' and gives concrete SQL examples, establishing clear context for when it applies. It does not name exclusions or alternatives, but the sibling set is self-evident enough that an agent can see this is the data-querying tool versus export/schema/stats helpers.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_rapid7_schemaA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare the tool read-only, idempotent, and non-destructive. The description adds meaningful behavior beyond annotations: tables not yet loaded are omitted, and schemas are returned as formatted JSON keyed by table name. No contradiction exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, well-structured, and front-loaded with the core action. Every sentence adds useful information: scope, included tables, omission behavior, and usage intent. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter, read-only schema tool with an output schema and rich annotations, the description covers everything needed: what it returns, which tables, the omission nuance, and how to use it. Nothing critical is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the description does not need to document parameter behavior. The baseline of 4 applies because the schema already fully covers the empty parameter set.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb and resource: 'Get the schema of all database tables,' then enumerates exactly which tables are included. This makes the tool's scope unambiguous and distinguishes it from query/stats/export tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly tells the agent when to use the tool: 'Use this to understand what data is available before writing queries.' It does not discuss exclusions or compare against alternatives, but given the sibling set, no other tool claims schema discovery, so the guidance is adequate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_rapid7_statsA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark it readOnly, idempotent, and non-destructive. The description adds meaningful behavior beyond those hints: it returns row counts plus distributions, omits tables not yet loaded, and returns formatted JSON keyed by table name. There is no contradiction with the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and front-loaded with the core purpose and return shape. A small redundancy exists between 'Returns row counts...' and the later 'Returns: Summary statistics as formatted JSON...' line, but the latter adds keying detail so the duplication is minor.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a no-argument read-only tool with annotations and an output schema available, the description is complete: it identifies the covered tables, the omission behavior for unloaded tables, and the return format. No critical call-time information is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no parameters, so there is nothing to document beyond the baseline for zero-parameter tools. The description's statement of what the stats cover is the only relevant semantic context and it is adequate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource ('Get summary statistics for all database tables') and then enumerates the exact outputs ('row counts and relevant distributions') and the tables involved (assets, vulnerabilities, policies, vulnerability_remediation). This makes it clearly distinct from siblings like query_rapid7 or get_rapid7_schema, which are for detailed queries or schema inspection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives an explicit use context: 'Useful for getting an overview of the data across all loaded datasets.' It does not name alternatives or state when not to use it, so it stops short of a full routing rule, but the intended scenario is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_rapid7_exportA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds substantial behavioral context beyond the annotations: non-blocking background processing, typical 3-5 minute completion, automatic reuse of a same-day export ID, and automatic splitting of >31-day remediation exports. These are exactly the kind of runtime behaviors an agent needs to know and cannot infer from the schema or annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the essential non-blocking behavior, followed by downstream tool usage and edge-case behavior. It is slightly repetitive ('non-blocking' appears twice; returning the export ID is stated more than once), but every block of information serves a distinct purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with three optional parameters, an output schema, and annotations available, the description is fully complete. It explains what the tool returns, what happens in the background, how long it typically takes, how to check/download the result, and how edge cases like duplicates and long date ranges are handled.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already covers parameter types, defaults, and enum values at 100%, so the baseline is 3. The description adds extra value by clarifying that date parameters are remediation-focused and by explaining the 31-day chunking behavior, which directly affects how start_date and end_date should be provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Start a new Rapid7 export job (non-blocking)', giving a specific verb and resource. It clearly separates this tool from siblings like check_rapid7_export_status, download_rapid7_export, and list_rapid7_exports by focusing on job creation and immediate ID return.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear workflow: call this first, then check_rapid7_export_status, then download_rapid7_export. It also explains the existing-export deduplication behavior, which helps the agent decide whether a new call is needed. It does not explicitly name alternatives like list_rapid7_exports for checking existing exports, so it falls just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• purge_rapid7_dataA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare destructiveHint=true and idempotentHint=true, and the description reinforces and enriches this by naming the exact databases and WAL files that will be destroyed and noting that the action is 'permanent.' It clearly discloses the irreversible destructive behavior without contradicting the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tightly organized with a one-line summary, a bulleted list of affected files, a clear usage statement, and a return note. Every sentence adds useful information and there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter destructive tool with annotations covering destructiveness and idempotency, the description is complete: it names exactly what is deleted, says when to use it, warns about the need to re-export afterward, and states the return value. No critical context is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and the schema is empty, so there is nothing to document at the parameter level. The description instead clarifies that the operation acts on all local Rapid7 data, which is the only relevant 'scope' information an agent needs. Baseline 4 is appropriate for a no-parameter tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Permanently delete all local Rapid7 data and tracking databases.' It goes on to enumerate the exact files removed (rapid7_bulk_export.db, rapid7_bulk_export_tracking.db, WAL files), which clearly distinguishes it from the read/query/export sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool: 'when you are done with your analysis session, before handing off a machine, or to free disk space.' It also explains the consequence—'After purging, you will need to run a new export to query data again'—which tells the agent the practical impact and when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.