Server Quality Checklist

Tool Scores

• get_gpu_statusA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. It explicitly states 'Read-only,' which is a key safety trait, and explains the per-process VRAM tracking rationale. It adds context beyond the tool name, though it omits potential failure modes (e.g., nvidia-smi not available).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, tightly-worded sentence that front-loads the essential information ('nvidia-smi wrapper') and efficiently includes all necessary details without waste. Every word contributes to the meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only status tool with no parameters and an output schema, the description adequately covers purpose, key metrics, and safety. It does not explain return format, but the existence of an output schema makes that unnecessary. The only gap is the lack of explicit usage guidance, but this is partially covered under that dimension.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters, so the baseline score is 4. The description adds no parameter details (as none are needed) but clarifies what the tool reports, which is sufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as an nvidia-smi wrapper for GPU status, listing specific metrics (VRAM used/total, utilization%, per-process VRAM usage). This distinguishes it from sibling tools like check_health or get_service_status, which suggest broader system or service monitoring.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage context is implied through the phrase 'to catch VRAM-overshoot,' indicating a troubleshooting scenario. However, there is no explicit statement about when to prefer this tool over alternatives or any exclusions, leaving the guidance somewhat implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_service_statusA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It explicitly states 'Read-only -- never starts/stops the unit,' which is a key safety disclosure. It also details the execution context (WSL vs native systemctl) and lists the returned fields, adding transparency beyond the bare schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two compact sentences that front-load the purpose, then add the command detail and safety note. Every phrase earns its place; the WSL/native distinction is relevant without being verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with an output schema, the description is fully complete. It states what the tool does, the execution environment, the specific return data, and the critical read-only constraint. Nothing essential is missing for an agent to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the description adds no parameter-specific detail beyond mentioning the distro placeholder in the underlying command. Since the schema already fully covers parameters (none), the baseline of 4 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it reports the systemd unit status for the vLLM service, describing both the underlying command and the specific data returned. This distinguishes it from sibling tools like check_health (likely HTTP health) or get_gpu_status (hardware status).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when systemd unit status is needed, but it does not explicitly contrast with alternatives or state when not to use it. No explicit when/when-not guidance is provided, though the read-only note hints at suitable contexts.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modelsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description explicitly states this is a GET passthrough, signaling a safe read-only operation, and clarifies that it returns currently loaded/served model IDs. With no annotations, it carries the burden well, though it omits details about error behavior or data freshness; for such a simple list tool this is acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no extraneous words. It states the action and resource directly, achieving maximum efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple, has no parameters, and an output schema exists, so the description only needs to state its purpose. It does so clearly, and the sibling tool names provide additional context for placement.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has zero parameters, so description parameter guidance is unnecessary; the baseline for 0 params is 4. The description does clarify that the output consists of model IDs, complementing the existing output schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'lists' and identifies the resource as 'model IDs currently loaded/served,' clearly distinguishing it from sibling tools like check_health or get_service_status. The 'GET /v1/models passthrough' prefix reinforces the operational scope without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly suggests use when querying available/loaded model IDs, but provides no explicit when-to-use or when-not-to-use guidance. It does not name alternative tools, leaving usage timing to inference based on the tool's name and sibling context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_serve_configA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral burden and excels. It discloses that it never queries the server, sources data from /proc/<pid>/cmdline with fallback to the exec script, and redacts sensitive values (token/key/secret/password/credential). This provides the agent with a complete safety and operational profile.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each earning its place: the first establishes purpose, the second explains the data source and fallback, the third explains redaction. It is front-loaded with the core action and avoids any redundant phrases.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is complete for a parameter-less tool with an output schema. It covers the source of truth, fallback logic, API interaction avoidance, and security redaction. Given no annotations and no params, there are no critical gaps in the agent's ability to invoke and interpret results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so parameter semantics are not applicable. The description does not need to compensate for schema gaps, and the baseline of 4 for parameter-less tools is appropriate. The description's detailed behavior adds context beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: read-only introspection of vLLM launch flags. It names the specific resource (vLLM launch flags) and action (introspection), and the use of 'read-only' distinguishes it from mutation tools. Among siblings like check_health or list_models, this is uniquely about configuration, so purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when the tool prefers live process data over static fallback, which implies it should be used to get launch flags regardless of process state. However, it does not explicitly contrast with sibling tools or say when not to use it. There is no direct alternative mentioned, so usage guidance is implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_healthA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does so excellently. It discloses the two-stage behavior, the exact HTTP calls, the side effect of GPU consumption, and the complete return envelope. This is unusually transparent for a health check.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Every sentence earns its place. The description efficiently covers purpose, parameter semantics, behavioral caveats, and return value without redundancy. It is front-loaded with the main verb and resource, then details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, output schema present), the description is complete. It explains the two modes, the side effects, and the return structure. No important aspect is left unexplained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only provides a bare 'deep' boolean with no description. The description fully explains the parameter's meaning, default, and the behavioral difference between true and false, adding substantial meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Two-stage health probe for the local vLLM server.' It clearly distinguishes this from siblings like list_models or test_completion by framing it as a health check, not a utility for listing or textual generation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use deep=False (liveness check) versus deep=True (verifying actual text generation) and warns that deep=True is rate-limited and consumes GPU cycles. However, it does not explicitly name sibling alternatives or state when not to use this tool in favor of another.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• test_completionA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses key behaviors: it is a real inference call, rate-limited, consumes GPU cycles, clamps max_tokens, caps prompt size (rejects oversized), and the rate limiter bounds frequency not per-call cost. Return shape is also given. This goes well beyond a minimal description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single dense sentence front-loads the core purpose and packs in all critical constraints, alternatives, and return fields without redundancy. Every clause adds value; no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given an output schema exists, the description doesn't need to detail return values but does anyway. It covers purpose, usage, safety, rate limits, parameter constraints, and alternatives. For a tool with 2 params and moderate complexity, this is fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It explains prompt as 'caller-supplied' and adds behavioral semantics for max_tokens (clamped) and prompt (capped, oversized rejected). It doesn't mention max_tokens is optional or its default, but the schema already shows default=16. This is a strong addition, though not exhaustive.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool performs a real /v1/chat/completions call with a user-supplied prompt for manual sanity-checking. It distinguishes itself from siblings by explicitly comparing to check_health(deep=True) and noting the user-controlled nature, and from the other get/list tools which are obviously not completion calls.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit context: 'for manual sanity-checking' and directly names the alternative check_health. It also warns about rate limiting and GPU consumption, implying it should be used sparingly, and clarifies when not to rely on it (costly, rate-limited).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.