Server Quality Checklist

Tool Scores

• scan_targetA
  4.2/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description states 'writes a new snapshot, violations, and drift events (vs the prior scan of the same target) to the local twin DB', indicating that repeated calls create new snapshots and change the state used for drift comparison. This contradicts the idempotentHint=true annotation, which implies repeated calls with the same arguments have the same effect. The description does add useful context about read-only vCenter API calls and local writes, but the contradiction forces a score of 1.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but every sentence earns its place: parameter definitions, side effects, return values, coverage caveat, and performance warning. It is front-loaded with the [READ] tag and a clear summary, then structured logically with no repetition or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no output schema and only 2 parameters, the description carries the full burden and exceeds it: it explains return fields, side effects, how to inspect details, and crucial interpretation guidance about coverage. It is complete for an agent to select and invoke the tool correctly, aside from the annotation contradiction.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description fully compensates by explaining both parameters: target is 'a vCenter target name as configured in vmware-aiops', and baseline is 'a baseline id from list_baselines' with a default value. This adds meaning beyond the bare schema, clarifying where values come from.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Run a compliance scan') and resource ('vCenter target against a baseline'), clearly distinguishing this tool from siblings like list_baselines and list_violations. It also states the outcome ('persist results locally') and the return type, leaving no ambiguity about the tool's purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly references alternatives and complementary tools: baseline id should come from list_baselines, and details can be inspected via list_violations and list_drift_events. It also specifies when not to interpret violations=0 as compliance, and warns about long runtime on large inventories, effectively guiding when and how to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_remediationA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes well beyond the annotations by disclosing local DB storage path, no network calls, nothing is executed, and suggestions are advisory only. It also reveals the None-return behavior and how to generate a missing suggestion. This adds context that annotations alone do not provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is relatively long but well-structured: it starts with a [READ] tag and one-sentence summary, then details the parameter, return shape, and special cases. Every sentence adds value, though it could be slightly tightened; the density is justified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is fully complete for the tool's complexity. It explains the parameter source, return fields, the None case, and the advisory-only nature. Given the simple single-param input and the presence of an output schema, the description covers all necessary context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at 0%, the description fully compensates by explicitly defining violation_id as the required 'id' field from list_violations. This adds meaning beyond the bare schema, which only provides the parameter name and type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Fetch' and the specific resource: the persisted LLM-generated remediation suggestion for one violation. It distinguishes from sibling tools by specifying it operates on a violation's remediation data, not on baselines, rules, or drift events.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context: violation_id comes from list_violations, and if no suggestion exists, it tells the user to generate one via the vmware-harden CLI advisor. While it doesn't explicitly list alternative tools or when not to use it, the context is unambiguous and includes a conditional workflow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_baselinesA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare read-only, idempotent, and non-destructive hints. The description additionally reveals that it only parses local YAML files with no database or network access, guarantees pagination behavior (truncated always false, total exact), and explains how failed entries are represented with an 'error' field—far exceeding the annotation baseline.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than a simple two-liner but every sentence serves a purpose: scope enumeration, return envelope details, item structure, edge cases (error field), read-only behavior, and intended usage. It is well-structured, front-loaded with the main purpose, and free of filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description carries full responsibility for explaining return values and does so thoroughly: the envelope fields, item fields, error handling, and completeness guarantee (total exact, truncated false). It also provides the next-step usage context, making this description fully self-sufficient for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so semantics are minimal. The description explicitly says 'Takes no parameters' and the schema confirms an empty object, which fully clarifies the absence of inputs. Baseline for 0-parameter tools is 4, and no further parameter description is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'List all available compliance baselines' and enumerates specific built-in baselines plus user-imported YAML files, making the scope unambiguous. It also differentiates from sibling tools by noting this is the starting point for discovering baseline ids usable by get_baseline_rules and scan_target.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides a clear directive: 'Start here to discover valid baseline ids for get_baseline_rules and scan_target.' This gives strong usage context, though it doesn't explicitly state when not to use the tool or directly contrast it with siblings like list_violations or list_drift_events.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_stig_content_syncA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations, the description discloses 'Read-only, local static content — no database, network, or API call,' which adds critical detail about side effects. It also reveals that compliance_api_available is 'always false' and explains why_no_api, making behavioral traits fully transparent without contradicting the readOnlyHint.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Though the description is long, every sentence serves a purpose: purpose, parameter confirmation, return fields with explanations, behavioral guarantees, and routing guidance. It is front-loaded with the verb and resource, and the structure allows quick scanning for key facts without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is fully self-contained for an agent: it lists all return fields with in-line explanations (compliance_api_available, why_no_api, content_sources, mechanism, routing_note, importer_status), states no side effects, and provides routing guidance. Since there is no output schema, this field-by-field description is essential and complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the description explicitly states 'Takes no parameters,' which aligns with the empty input schema. Since there are no parameters to explain, the baseline score is 4, and the description adds no further parameter detail needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Explain harden's vSphere STIG integration and route continuous enforcement,' which is a specific verb (explain) and resource (STIG integration). It further distinguishes itself from the sibling tool list_stig_controls by stating 'use list_stig_controls to see the actual controls,' clarifying the different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit usage guidance is provided: 'Call this before assuming a compliance endpoint exists; use list_stig_controls to see the actual controls.' It also explains when to use VCF Operations SPM/ACC UI for fleet-wide enforcement versus harden as the API-scriptable scanner, giving clear conditions and alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_baseline_rulesA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (readOnlyHint, destructiveHint), the description reveals key behavioral traits: 'unknown ids raise a not-found error', 'truncated is always false and total is the exact rule count', and 'Read-only — parses local baseline YAML only, no database or network access.' This adds significant context that annotations do not provide, with no contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: it opens with a bracketed READ marker and purpose, then covers parameter details, return format, behavioral specifics, and usage guidance. While detailed, every sentence adds value, and the information is front-loaded for efficient parsing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description thoroughly documents the return envelope and item structure, including severity enums. It also covers error handling, performance expectations (truncated false), and resources accessed, making it self-sufficient for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates fully by explaining the parameter's required format ('a baseline id exactly as returned by list_baselines'), providing an example, and noting error behavior. This is far more useful than a schema-only description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with '[READ] Return every rule in one compliance baseline', using a specific verb and resource that clearly states what the tool does. It explicitly distinguishes itself from siblings by noting 'Use after list_baselines to preview what scan_target will check; use list_violations for actual scan findings.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use guidance: 'Use after list_baselines to preview what scan_target will check; use list_violations for actual scan findings.' It names alternative tools and clarifies the intended workflow, making it easy for an agent to select this tool correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_drift_eventsA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations (readOnlyHint, idempotentHint, destructiveHint), the description reveals important behavioral details: return envelope structure, truncated semantics, no offset/cursor, local twin DB path (~/.vmware-harden/twin.duckdb), no network calls, and the edge case where an empty envelope is returned. These details go far beyond what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but every sentence earns its place: it covers the core purpose, parameter semantics, return envelope shape, edge cases, and usage guidance without fluff or repetition. Structure is front-loaded with the primary function, then specifics.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description provides complete return structure ({items, returned, limit, total, truncated, hint}), each item's fields ({node_id, field, old_value, new_value, detected_at}), and the meaning of total/truncated. It also explains when an empty envelope is returned, making the tool fully self-contained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% and the parameter schema only shows type/default, but the description fully documents 'limit': optional int default 50, maximum rows returned, ordering by node_id then field, and no offset/cursor. This adds semantic meaning that the schema alone cannot provide.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource+scope: 'List configuration drift events from the most recent scan snapshot — fields whose values changed since the prior scan of the same target.' It clearly distinguishes from sibling list_violations by explicitly stating 'Use for change tracking; use list_violations for compliance failures.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly provides usage guidance: 'Use for change tracking; use list_violations for compliance failures.' It also states the prerequisite that a target must be scanned at least twice, helping the agent understand when an empty result is expected.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_stig_controlsA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations (readOnlyHint, idempotentHint), the description reveals that the tool parses local baseline YAML only and makes no external calls, which is critical behavioral context. It also details pagination behavior via 'total' and 'truncated', giving the agent a precise mental model of the operation's side effects and return semantics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose ('[READ] List ... controls'), then efficiently covers parameters, return envelope, and alternatives in a structured, information-dense paragraph. Every sentence adds unique value—no filler or redundant restatement of the tool name.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description takes responsibility for explaining the exact return envelope and item structure, including severity values and advanced_setting semantics. It also covers read-only behavior and sibling tool usage, making the description self-sufficient for an agent to correctly select and invoke this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only defines limit and offset with defaults, but the description explains each parameter's meaning ('max rows returned', 'rows to skip for paging') and adds actionable guidance on how to use them ('truncated tells you definitively whether to raise offset'). This fully compensates for the 0% schema description coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and resource ('vSphere 9 / VCF 9 STIG-aligned host baseline's controls') and identifies the exact baseline id ('vsphere-stig-v9-subset'). It clearly distinguishes this tool from siblings like scan_target and describe_stig_content_sync by stating its role in listing controls.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells when to use this tool (to list controls of the baseline) and provides alternatives: 'Use scan_target with baseline ... to evaluate these controls' and 'use describe_stig_content_sync for how this catalog is kept in sync.' It also clarifies the read-only, local-only scope, reinforcing appropriate use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_violationsA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide readOnlyHint, idempotentHint, and destructiveHint, but the description adds crucial behavior: sorting by severity then rule_id, pagination via has_more, the meaning of empty results (undetermined vs passing), and coverage fields. The description confirms read-only local DB query with no network calls and does not contradict annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but each sentence adds operational value: it starts with a [READ] marker, front-loads the core query and envelope, then details edge cases and coverage semantics. No filler or redundant phrasing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no output schema and only 3 simple parameters, the description covers return envelope fields, sorting, pagination, empty-scan behavior, coverage semantics, and related tools. It is self-sufficient for an agent to invoke the tool correctly and interpret results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% description coverage, but the description explains severity allowed values ('critical', 'high', 'medium', 'low', 'info'), limit default 50, offset default 0, and how to page by raising offset while has_more is true. It fully compensates for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states 'List compliance violations recorded by the most recent scan snapshot', which specifies the verb (list), resource (compliance violations), and scope (most recent snapshot in local twin DB). It clearly distinguishes from siblings by pointing to get_remediation and scan_target as related but different actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance says to run scan_target first when no scan exists, and to pass a violation id to get_remediation for a fix plan. It also warns that an empty list is not a compliance verdict and instructs to check coverage before drawing conclusions, effectively defining when to use this tool vs alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.