Server Quality Checklist

Tool Scores

• check_vram_fitA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of explaining behavior. It is transparent about inputs and return values (weights + KV cache + overhead in GB, compatible cards/Macs), but does not disclose estimation caveats, approximation assumptions, or limitations of the fit calculation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long and front-loads the core purpose. It wastes no words and efficiently covers what the tool does, what it takes in, and what it returns.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only calculator-style tool with no output schema and no annotations, the description is reasonably complete: it states inputs, the computation, and the output shape. It lacks only explicit usage conditions or caveats, which are also relevant in other dimensions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already documents all three parameters with 100% coverage, so baseline is 3. The description does add useful context by linking params_b, quant_bits, and context_tokens to the returned weights and KV cache, but it does not add significant detail beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb ('work out whether') and a clear resource ('an open-weights model fits on a given GPU'), and lists concrete inputs and outputs. It does not explicitly differentiate from sibling tools like estimate_llm_cost, though the focus on VRAM/GPU memory is fairly distinctive.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for GPU memory-fit checks, but it does not state when to use it over alternative tools or mention any exclusions. A clear context is present ('whether a model fits on a GPU'), but there is no guidance about alternatives or edge cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_config_complianceA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations present, the description carries the disclosure burden. It adds a valuable behavioral detail: checks are 'deterministic regex checks with framework references', so results are reproducible between reviewers. It also enumerates what the check covers. It does not explicitly state that the operation is read-only, but the 'run ... over config' wording and the input being a config text make side effects unlikely, so this is a minor gap only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two well-structured sentences: the first announces the operation and its checklist; the second adds the determinism guarantee. The highest-value information is front-loaded, and no sentence is wasted.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a compliance checker, the description clearly defines scope and inputs, but with no output schema it does not state what the tool returns (e.g., pass/fail per check, a report, violations only). It also gives no hint about how vendor detection or the optional vendor hint affects results. These are moderate gaps for an agent that must invoke the tool and interpret its result.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both `config` and the optional `vendor` hint. The description adds no extra parameter-level detail beyond naming what the config contains, which meets the baseline but does not elevate it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb and resource ('Run a CIS/PCI starter pack over one network device config') and enumerates the precise artifacts it checks, from telnet/SSH status to idle timeout. This makes the tool's function immediately recognizable and clearly distinct from sibling tools like explain_config_diff or audit_mcp_config, which address different artifacts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the intended use — checking a single network device's config against CIS/PCI concerns — but it does not explicitly state when to prefer this tool over the sibling tools or mention any exclusions. The context is reasonably clear, yet there is no direct usage guidance or alternative routing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lookup_mcp_serverA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the full transparency burden. It explicitly discloses the lookup behavior, the data facets returned, and the important edge case that unlisted servers still receive vetting guidance. It does not describe the exact output shape, but for a simple lookup tool this is a minor omission.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no filler. The first sentence establishes purpose and result content, the second gives search modes, and the third explains unlisted-server behavior. Every sentence adds distinct value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter lookup tool with no output schema, the description covers the input semantics, the returned information, and the fallback behavior for unlisted servers. A statement about whether the result is a single entry or a list would improve completeness, but the tool is simple enough that this gap is minor.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already covers the query parameter with examples, giving 100% schema description coverage, so a baseline of 3 is appropriate. The phrase 'Search by name, capability or category' reinforces but does not add substantial meaning beyond the schema. The description also does not specify what happens when the optional query is omitted.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Look up' and the resource: an MCP server in the curated AIOps Field Notes directory. It also specifies the kind of information returned: connections, maintainer, and server classification (reference, vendor, or community). This distinguishes it from siblings like audit_mcp_config or check_config_compliance, which are about configuration auditing rather than directory lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains how to search the directory: by name, capability, or category. It also mentions what happens for unlisted servers, which helps set expectations. However, it does not explicitly state when not to use this tool or suggest any alternative sibling, leaving the differentiation to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• latest_field_noteA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and it does disclose the most important behavioral trait: this tool performs network I/O against a public RSS feed. It does not mention failure modes or rate limits, but for a simple read-only feed fetch, the disclosure is solid.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tight sentences: the first states the resource and value, the second adds the crucial network-behavior differentiator. No filler or redundant schema repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one optional parameter and no output schema, the description covers what it reads, why it matters, and its network behavior. It does not describe the exact return shape, but that is not critical for such a simple feed-reading tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the single 'count' parameter already has a description and default value. The tool description adds nothing beyond the schema, so the baseline of 3 is appropriate; no extra semantic value is provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific resource ('This Week in AIOps episodes') and a concrete action ('reads a public RSS feed'). It also distinguishes itself from siblings by being the only tool that touches the network, so an agent can tell it apart without opening any schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly indicates this is the network-dependent tool among local analysis siblings, implying use when external RSS content is needed. It does not explicitly list when-not-to-use or name an alternative, but the context is strong enough for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• estimate_llm_costA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral disclosure burden. It discloses that results are returned for every model, sorted cheapest first, and that numbers are subject to moving caveats like 'batch discounts, prompt caching, intro pricing that expires', signaling estimates rather than exact quotes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence packs in the tool's purpose, required inputs, output structure, ordering, and important caveats with no filler. The opening phrase 'Price a workload' immediately establishes the action and target.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Even without an output schema, the description explains what the caller gets: per-request and per-month costs for every model, sorted cheapest first, plus the caveats affecting the estimate. Required inputs are fully documented in the schema, and the optional filter is also described, so the agent has enough information to call the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% description coverage for all four parameters, so the description does not need to re-explain them. It lightly reinforces that tokens are per request and volume is monthly, but adds no meaning beyond what the schema already contains.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific action ('Price a workload') with a clear resource (cost estimation across 'current frontier and open-weights models'). It also states the return shape: cost per request and per month, cheapest first, which leaves no ambiguity about what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It tells the caller exactly what inputs are needed ('tokens per request and monthly request volume') and what the tool produces, making the intended use case clear. It does not explicitly state when not to use it, but the sibling tools are unrelated, so no alternative-based exclusion is necessary.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_config_diffA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full disclosure burden and exceeds it: it reveals that the operation is deterministic ('no model call'), locally contained ('nothing leaves the machine'), normalizes input ('strips volatile lines'), structures output ('groups changes by config section'), and augments it with risk tags plus rationale. This gives an agent a precise behavioral model with zero annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler: the core purpose is front-loaded, and the second sentence packs only high-value operational details (supported vendors, normalization, grouping, risk tagging, determinism, privacy). Every clause earns its place and the colon-delimited list is easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema, but the description compensates by describing the return experience: plain-English prose, grouped by section, annotated with risk level and a 'why it matters' rationale. An agent can anticipate the result well enough to invoke the tool. Minor gap: the exact return structure/format isn't specified, but for a natural-language output tool this is a modest omission.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all four parameters (before, after, device, vendor). Per baseline rules, the description need not repeat param details. The vendor list lightly reinforces the 'vendor' parameter's accepted values, but adds no meaning beyond what the schema already conveys.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Explain what changed between two network device config snapshots, in plain English.' It is immediately distinguishable from all six sibling tools (cost estimation, VRAM checks, MCP auditing, etc.), none of which perform diff analysis. The vendor list further pins down the exact domain of applicability.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear applicability context by enumerating supported config formats (EdgeOS, MikroTik RouterOS, OPNsense, Cisco IOS-style, EdgeSwitch) and states the deterministic, offline behavior that would guide an agent's choice for privacy-sensitive or reproducible analysis. It does not explicitly name when-not-to-use or name an alternative tool, but no sibling is a plausible substitute, so exclusions are less critical.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• audit_mcp_configA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does so well: it explicitly states 'Nothing is read from disk; pass the text', clarifies the input mode, and discloses that the output consists of findings 'each with the fix'. This gives an agent an accurate safety model without needing annotation hints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense, but every clause earns its place: trigger, input sources, security categories, fix output, and no-disk-read warning. The long checklist is justified because it defines the tool's scope precisely, and the critical 'pass the text' instruction is clearly stated.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a single string parameter, no output schema, and no annotations, the description is complete enough: it tells the agent what to pass, how to pass it, what the tool checks, and what the result contains. Nothing necessary for basic invocation is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and already describes the config parameter as JSON text, so the baseline is 3. The description adds value by pointing to real file names and emphasizing that the agent must paste the text rather than pass a path or object.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('Review') and resource ('MCP client config'), and enumerates the concrete security issues it flags, such as unpinned packages, plaintext secrets, and non-read-only database access. The 'before you trust it' framing and detailed checklist distinguish it from generic config or compliance siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives a clear trigger ('before you trust it') and concrete input sources (.mcp.json, claude_desktop_config.json, or equivalent). It does not explicitly mention when not to use it or route to sibling tools like check_config_compliance, so it stops short of full alternative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.