Server Quality Checklist

Tool Scores

• smoke_testA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that it sends chat prompts and reports latency/speed, but does not clarify side effects (e.g., whether it is read-only or modifies server state), auth requirements, or rate limits. The mention of 'as measured by llama.cpp' adds some context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-formed sentence that front-loads the action and resource. Every word contributes to meaning, resulting in zero waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema, the description need not detail return values. It adequately covers the tool's purpose, inputs (implied by schema), and reported outputs (latency, speed). Prerequisites (running server) are obvious from context. Nearly complete for a test tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so each parameter already has a description. The tool description adds limited extra semantic value beyond the schema, only mentioning 'batched chat prompts' and output metrics. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool sends batch chat prompts to a running llama-server via OpenAI-compatible API and reports each response with latency and tokens/second. It uses a specific verb ('smoke-test') and resource ('running server'), distinguishing it from sibling tools like start_server or stop_server.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for testing a running server but does not provide explicit when-to-use or when-not-to-use guidance. No alternatives or exclusions are mentioned, making it minimally adequate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modelsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description adequately discloses scanning behavior, file filtering, and return fields. However, it does not mention potential performance impacts or restrictions like recursion depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, informative sentence that front-loads key information with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, the description covers return values adequately. It could mention that results are returned as an array, but is otherwise complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds that the directory defaults to BELLOWS_MODELS_DIR and that scanning is recursive, providing value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool scans a directory recursively for .gguf model files, skips projector files, and returns specific fields. It is distinct from sibling tools like start_server.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for finding local models, but no explicit when-to-use or when-not-to-use guidance relative to siblings. Siblings are different enough that confusion is unlikely.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• eval_historyA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description explicitly labels the tool as 'Read-only queries', indicating no destructive side effects. It describes the output for each action, but lacks details on potential rate limits or authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, two sentences that front-load the essential information and then list the actions without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters (1 required) and no output schema, the description provides sufficient context for the three actions. It covers what each action returns, though it could elaborate on the output format.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% with detailed parameter descriptions. The tool description adds overall context but does not significantly enhance the meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it is a read-only query tool for crucible eval results, listing three specific actions (list runs, summarize, compare). This distinguishes it from sibling tools like list_models and server_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description enumerates the three actions with brief explanations, providing context for when to use each. However, it does not explicitly state when not to use this tool or compare with alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• stop_serverA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the graceful stop strategy (SIGTERM then SIGKILL) and the ownership constraint, which are critical behavioral traits. It does not detail the grace period length or error handling, but the key behaviors are transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, with only two sentences that front-load the action and method. Every word contributes meaning, and there is no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple, single-parameter tool with an output schema (not shown but present), the description provides sufficient context: the stopping method, ownership constraint, and implicit usage hint. It could mention potential error conditions or prerequisites, but given the simplicity, it is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description does not add additional parameter-specific guidance beyond what the schema already provides (the port's ownership constraint is repeated in the schema). No extra value is added for the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('gracefully stop') and the specific resource ('llama-server that bellows started'), effectively distinguishing it from the sibling tool 'start_server'. The mention of SIGTERM and SIGKILL further specifies the behavior.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly guides usage by stating it 'refuses to touch processes it does not own', indicating when not to use it. However, it does not explicitly compare to related siblings like 'server_status' or 'eval_history', leaving room for slight ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• server_statusA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that the tool reports on supervised processes and can probe a health endpoint for any server, including those not owned by bellows. Since no annotations are provided, the description adequately conveys the read-only nature and scope.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the core purpose and immediately provides key details. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the single optional parameter and the presence of an output schema, the description fully covers what the tool does, including the types of information reported (pid, port, model, uptime) and the optional probe behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds no new information about the 'port' parameter beyond what is already in the schema description (probing a health endpoint). With 100% schema coverage, the baseline is 3, and the description does not elevate it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reports supervised llama-server processes with specific details (pid, port, model, uptime) and optionally probes a health endpoint. It distinguishes itself from sibling tools like start_server and stop_server by focusing on status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  While the description does not explicitly state when to use this tool over alternatives, the sibling names and the tool's purpose imply it is for obtaining server status rather than starting/stopping. A slight improvement would be to include a direct usage note.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_serverA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavior: full arg validation, no shell, health wait, pid recording, port conflict refusal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with action and key constraints, no excess words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, the description sufficiently explains the tool's operation and constraints for a 5-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% so definitions exist; description adds 'full arg validation' but no additional parameter meaning beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the verb 'spawn' and specifies the resource 'llama-server for a model', clearly distinguishing it from sibling tools like stop_server and server_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit when-to-use or when-not-to-use guidance is given. The description implies usage for starting a model server but does not compare to alternatives like smoke_test.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.