Server Quality Checklist

Tool Scores

• diff_architecturesA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses behavioral traits: it states 'Pure computation — no LLM, no network. Read-only. Does not modify either spec.' This clearly informs the agent about safety and side-effect-free operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and reasonably concise, with a clear purpose statement, output contents list, usage guideline, and behavior note. It is slightly verbose but still efficient, earning a 4.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description thoroughly explains what the tool returns, listing specific components of the change report. Given the complexity of nested object inputs, this provides enough context for an agent to understand the tool's full behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the schema already provides clear descriptions for both parameters (old_spec_json as baseline, new_spec_json as target). The description adds no additional parameter-specific meaning beyond the schema, so baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool diffs two architecture specs and returns a structured change report, listing specific delta items (components, connections, cost, compliance). This distinguishes it from sibling tools like compare_provider_costs or compare_providers, which cover different comparisons.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes an explicit 'When to use' section indicating the tool is appropriate for reviewing before/after spec changes for approval or ADR writing. However, it does not mention when not to use or suggest alternatives, preventing a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chat_delete_sessionA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden and excels: it discloses return values, destructive nature (history and uncommitted spec lost), irreversibility, and that it only removes a file with no network or LLM calls.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise and well-structured: purpose, return format, destructive warning, usage guidance, and behavior are each in a short paragraph or bullet, with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description thoroughly covers return values (success and error), side effects, and when to use. It provides all necessary context for an agent to safely and correctly invoke this delete tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the parameter is well-documented in the schema. The tool description adds no new per-parameter details but provides overall context about destructiveness, meeting the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with a clear verb and resource: 'Delete a conversation session.' It also provides usage context like 'Clean-up after a completed design, or abandoning a dead-end conversation,' which helps distinguish it from sibling tools like chat_create_session.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool (clean-up, abandoning a conversation) and notes that it does not affect deployed infrastructure. However, it does not explicitly state when not to use it or list alternatives, though the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_provider_costsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but description discloses it's pure computation, no LLM/no network/no API costs, uses offline catalog, does not deploy. Adds clear behavioral context beyond schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four sentences, front-loaded with purpose, then returns, then usage contrast, then behavioral note. Every sentence adds value, no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given parameter count (2) with full schema coverage, output schema exists, and no missing critical areas: returns are described, behavior clarified, limitations noted (no deploy, no API). Complete for a cost comparison tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%. Description adds context that spec_json maps services to cross-cloud equivalents, but this is mostly reinforcing schema examples. Baseline 3 appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it compares monthly cost totals across cloud providers, and that it returns numeric summaries per provider. Distinguishes from sibling compare_providers by specifying this tool only returns costs, not architectures.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to use for cost-focused provider selection. Contrasts with compare_providers (architectures vs. costs) and suggests workflow: call compare_providers first then estimate_cost, or call in parallel.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_blast_radiusA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description fully discloses behavior: 'Pure graph computation — no LLM, no network. Read-only. Does not touch cloud resources.' This covers all side effects and constraints, compensating for the lack of annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is approximately 100 words, well-structured with sections, front-loaded with core purpose, and every sentence adds value. No redundancy or wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no output schema, the description lists return values (blast-radius metrics, SPOF, tier position). It explains use case and behavior sufficiently for an agent to select and invoke correctly. Minor omission: it could clarify the expected structure of spec_json, but the schema's additionalProperties allows flexibility.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with clear descriptions for both parameters. The description adds context about how spec_json builds a directed graph and what component_id focuses on, plus lists output fields. This extra context raises the score above the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Analyze blast radius and dependency structure of an architecture.' It lists specific outputs (direct/transitive dependents, blast-radius size, SPOF status, tier position) and distinguishes itself from siblings like score_architecture and lint_architecture.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use: 'when you have a spec and want to understand coupling and failure domains before production.' It also mentions complementary tools (score_architecture, lint_architecture) but does not specify when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• design_architectureA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes internal behavior: calls an LLM provider incurring costs, deterministic post-processing applies safe defaults, and explicitly states it does not deploy or modify resources. With no annotations provided, this fully covers transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is concise and well-structured: front-loaded with purpose, then output, usage, and behavior. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose, usage, behavior, and return format despite no output schema. Mentions output is an ArchSpec with components, connections, etc. Minor omissions like error handling or rate limits keep it from a 5.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with detailed parameter descriptions including examples and defaults. The tool description adds overall context but does not significantly extend per-parameter meaning beyond the schema, hence baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Design a cloud architecture from a natural-language description.' It identifies itself as the primary entry point for greenfield architecture design, and distinguishes from siblings like modify_architecture and chat tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit 'When to use' section provides clear guidance: use when you have a requirement and need a concrete architecture. It also specifies alternatives such as modify_architecture for iteration and chat sessions for multi-turn refinement.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_providersA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that the tool calls an LLM for ambiguous mappings and does not deploy anything. This is beyond basic annotations (none provided). However, it does not explicitly state side effects like whether it modifies data or is read-only, but the nature of comparison implies no mutation. Still, good transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with clear sections: main purpose, when-to-use, behavior. Each sentence is informative and concise. No unnecessary text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers all necessary aspects: purpose, usage guidelines, behavioral traits, and parameter context. Output schema existence means return values need not be detailed. Complete for a comparison tool with complex input.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% but description adds value: explains 'spec_json' is an ArchSpec, 'providers' are target slugs with examples, and mentions the equivalence pairs. This goes beyond schema fields.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool compares service mappings across cloud providers, returning translated ArchSpecs. The verb 'compare' and resource 'service mapping' are specific, and it distinguishes itself from the sibling 'compare_provider_costs' by detailing the type of output.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly explains when to use this tool versus 'compare_provider_costs', stating this one returns full architectures while the other returns only cost totals. No alternative tool is left ambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lint_architectureA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that it's pure computation with no LLM, no network, and no cloud interaction. Lists specific checks (errors and warnings) beyond what annotations (none) provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured: uses bullet points for error/warning categories and a separate paragraph for usage guidance. Every sentence is informative and no redundant text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the single parameter and presence of output schema, the description fully explains the tool's behavior, inputs, and outputs. No gaps identified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only one parameter 'spec_json' with 100% schema description coverage. The schema already explains its purpose and what checks it runs. The main description adds no further parameter detail, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it lints architecture for anti-patterns and best-practice violations. Distinguishes itself from sibling 'security_scan' by specifying its focus on architectural hygiene.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly compares with 'security_scan', defining when to use each and recommending both for comprehensive review. Provides clear context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_servicesA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description fully discloses behavior: pure local lookup with no LLM, network, or cloud access. This addresses potential concerns about side effects, latency, or dependencies.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four succinct sentences: purpose, output details, usage guidance, behavior. No wasted words, front-loaded with key information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one optional parameter, output schema exists), the description covers all necessary context: what it returns, when to use it, and how it behaves. No gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema coverage, the description adds value by connecting the parameter to the use case of discovering service keys. It doesn't repeat schema details but reinforces the tool's purpose.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists all cloud services for a provider, specifying return fields like slug, name, category, and tiers. It distinguishes itself from sibling tools (e.g., design_architecture, estimate_cost) by being a pure lookup tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises using the tool to discover valid service keys for ArchSpecs or requirement mapping, providing clear context. It could be improved by mentioning when not to use it, but the use case is well articulated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• score_architectureA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Clearly states behavior as 'Pure computation — no LLM, no network. Read-only.' This fully compensates for missing annotations and discloses that the tool is safe and non-destructive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with no redundant sentences. Front-loaded with the core action, followed by return details, usage context, and behavioral note.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no output schema, the description sufficiently explains return values (dimension scores, overall score, letter grade, notes) and covers weights and dimension details. Complete for a simple scoring tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a description for the single parameter. The tool description adds context about scoring dimensions but does not elaborate on the parameter's structure beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool scores an architecture across five dimensions and returns a weighted score, letter grade, and notes. It distinguishes from siblings by specifying when to use and listing alternatives for specific findings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says when to use ('quick quality summary before a design review') and provides exclusions ('For specific findings, use lint_architecture, security_scan, or validate_compliance').

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• security_scanA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that it is pure computation with no network or cloud access, though read-only nature could be explicitly stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise, front-loaded, and well-structured with no unnecessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Comprehensive description covering input, output, scope, behavioral constraints, and relationship to sibling tools despite lacking output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema provides 100% coverage for the single parameter, and description adds context on what the scanner inspects, enhancing understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool scans architecture for security anti-patterns, lists example checks, and distinguishes from validate_compliance.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly directs users to use validate_compliance for specific regulatory frameworks, providing clear guidance on when to use this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• catalog_searchA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that it is a pure lookup from a bundled SQLite catalog with no LLM or network calls, and that prices are from a catalog snapshot date. This fully describes the tool's behavior, especially since no annotations are provided.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (3 paragraphs) with clear headings for purpose, usage, and behavior. Every sentence provides essential information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers all aspects: search criteria, returned data, usage context, and behavioral notes. Given that an output schema exists (context signal), the description does not need to detail return values. It is complete for an effective tool invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value by stating that all filters combine with AND semantics, which is not explicit in the schema. This clarifies how multiple filters interact, earning a 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches the cloud instance catalog by multiple criteria (provider, specs, text query) and lists the returned fields (instance family, vCPU, memory, storage, price, region, architecture). It distinguishes from siblings like compare_providers and estimate_cost by focusing on search and filtering.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The 'When to use' section explicitly outlines three use cases: right-sizing workloads, finding cheapest instance meeting hardware requirements, and discovering equivalent instance families. This provides clear guidance on when to invoke this tool over alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chat_list_sessionsA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that this is a pure disk read with no LLM or network calls, and explicitly states it is read-only. Since no annotations are provided, the description fully covers behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is extremely concise: one sentence for purpose, one for return details, one for usage, and one for behavior. No redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Fully covers what the tool does, what it returns, when to use it, and its behavior. Output schema exists, but description already details return fields.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema is empty (0 parameters). Description adds meaning about the tool's purpose and return data, which is sufficient given no parameters exist.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it lists saved conversation sessions and specifies exact metadata fields returned (session_id, creation timestamp, etc.). Differentiates from siblings like chat_create_session and chat_delete_session by focusing on listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly enumerates three use cases: resuming prior work, cleaning up sessions, and auditing token spend. Provides clear guidance on when to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chat_sendA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that it calls an LLM with API costs, history growth, session state persistence, and error handling for invalid session. No annotations present, so description fully carries the burden.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-organized with separate sections for purpose, return values, usage, and behavior. Every sentence is informative without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description fully explains return format, error cases, cost implications, and side effects, making it complete for a conversational tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema already covers both parameters (100% coverage). Description adds examples for message and clarifies session_id requirement, providing extra value beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it sends a message to an existing session and gets a response. It distinguishes from siblings like design_architecture and chat_create_session by specifying when to use each.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states 'When to use: Every turn after chat_create_session. For zero-state single-shot calls use design_architecture / modify_architecture instead.' Also implies prerequisite of an existing session.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• estimate_costA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since annotations are absent, the description fully discloses behavioral traits: deterministic computation, no LLM/network/API calls, works offline, and does not deploy. This transparency goes well beyond minimal requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured with clear sections (overview, return info, determinism, usage, behavior). Every sentence adds value; no fluff. Well front-loaded with the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (nested spec_json, pricing tiers), the description covers return structure, determinism, offline operation, and usage boundaries. No output schema is present, but the description describes the output comprehensively, making it complete for agent understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% parameter coverage, so baseline is 3. However, the description adds meaningful context: explains pricing resolution via a bundled SQLite catalog with formula dispatch and fallback, and describes the pricing tier multiplier values, which adds value beyond schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Estimate the monthly cloud bill for an architecture spec,' specifying the verb (estimate) and resource (monthly cloud bill). It distinguishes from siblings by explicitly naming `compare_provider_costs` and `compare_providers` as alternatives for multi-provider scenarios.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides a clear 'When to use' section stating the tool is for a single provider+tier combination, and explicitly names two sibling tools for multi-provider comparisons, offering both inclusion and exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• export_architectureA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; description fully discloses behavior: pure computation, no LLM/network, does not write files or deploy, and details about sensitive value handling. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with purpose, return value details, behavioral notes, and usage guideline. Each sentence adds value; no unnecessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given two well-documented parameters, no output schema but description covers return structure, usage, and behavior. Complete enough for agent to select and invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with detailed parameter descriptions. Description adds context about return format ('Returns {'format': str, 'content': str}') and guarantees (no hardcoded credentials), providing value beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool exports an architecture spec to multiple formats (Terraform, CloudFormation, Mermaid, etc.). Distinguishes from sibling tools like design_architecture and modify_architecture.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly provides when-to-use guidance: 'You have a finalized ArchSpec and need IaC code, a diagram, or an audit artifact.' Also advises calling once per format for multi-format export.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• modify_architectureA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses behavioral traits: 'Calls an LLM provider — incurs API costs. Pure function: returns a new spec without mutating the input. Does not deploy.' This informs the agent of side effects and safety.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is six sentences, well-organized with labeled sections ('When to use', 'Behavior'). Every sentence adds essential information; no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 2 parameters, no output schema, and no annotations, the description covers purpose, usage, behavior, and return value ('Returns the updated ArchSpec'). It is self-contained and sufficient for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% coverage with descriptions and examples for both parameters. The description adds value by stating that 'spec_json' is typically the output of prior tool calls and must contain specific keys, which aids agent understanding beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool modifies an existing architecture with natural-language instruction. It specifies the resource (existing ArchSpec) and verb (modify). It also distinguishes from siblings: for from-scratch use 'design_architecture', for multi-turn editing use 'chat_create_session'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit when-to-use: 'You already have an ArchSpec and want to evolve it.' Explicit alternatives: 'For from-scratch design, use design_architecture. For iterative multi-turn editing..., use chat_create_session.' This leaves no ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_terraformA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses that the tool is pure computation with no LLM, no network, does not run Terraform or touch cloud, and is safe for scanning untrusted HCL. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is well-structured: first sentence states purpose, then examples, when-to-use, and behavior. Every sentence adds value, no fluff. Length appropriate for the information conveyed.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Tool has one parameter and is security-related. Description covers purpose, usage, behavior, and parameter details. Mentions output (severity-graded findings on resource blocks). Complete for this complexity level.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. Description adds significant value by explaining the parameter is raw HCL source code, typical usage (main.tf), and how the scanner works (parses directly, no terraform binary). Just above baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it scans Terraform HCL for security misconfigurations, using specific verb 'scan' and resource 'Terraform HCL'. It distinguishes itself from sibling tool security_scan which is for ArchSpec-level audit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit 'When to use' section specifies to use this tool when you have existing Terraform code (not an ArchSpec) for an immediate security audit, and directs to security_scan for ArchSpec-level audit. Provides clear context and alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_complianceA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It details behavior: pure computation, no LLM/network, static evaluation against 30+ rules, no cloud resource access. This fully informs the agent about safety and side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with clear sections (purpose, return value, usage, behavior). Every sentence is necessary and concise, no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Even without viewing output schema, the description explains return format (one result per framework with pass/fail, evidence, remediation hints). Combined with behavior and usage guidelines, it provides full context for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 100% description coverage. The description adds practical context: spec_json is the architecture to validate, frameworks are slugs each with 5-7 checks, well_architected runs additional independent checks. This integrates schema info meaningfully.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool validates an architecture against compliance frameworks, specifying verb and resource. It distinguishes itself from sibling tool security_scan by noting the latter is for framework-agnostic anti-pattern detection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit 'When to use' section describes scenarios (verifying HIPAA/PCI-DSS/etc. compliance before proceeding) and suggests an alternative (security_scan for anti-patterns).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chat_create_sessionA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must cover behavior. It states 'Writes a new session file to the session store (persisted on disk)' and 'Does not call the LLM — the first LLM call happens on the first chat_send.' Also specifies the return value format. This fully discloses operational traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is 7 sentences, well-structured with front-loaded main purpose, then return format, usage guidelines, and behavioral details. Every sentence adds value; no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description explains the return value ('session_id'). It covers purpose, when to use, behavioral details, parameter context, and constraints. For a tool with 3 optional params, this is fully complete and leaves no gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds contextual meaning: provider can be overridden per message, budget_monthly is a cap applied across turns, and compliance frameworks are enforced across the session. This goes beyond the schema's property descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Create a new stateful architecture-design conversation session.' The verb 'create' and resource 'session' are specific. It differentiates from sibling tools by mentioning stateful and multi-turn context (vs single-shot design_architecture and modify_architecture).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly tells when to use this tool: 'Multi-turn architecture design where each turn depends on the prior one.' Provides clear alternatives: 'For single-shot design use design_architecture; for one-shot edits of an existing spec use modify_architecture.' Also explains constraint freezing at creation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.