Server Quality Checklist

Tool Scores

• omv_connection_infoA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the full burden. The text implies a read-only operation ('Show how this server connects'), but it does not disclose potential side effects, required permissions, or error behavior. It's a minimal disclosure, not covering important behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, front-loaded with purpose and immediate use cases. Every word earns its place; no unnecessary content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema, and the description does not explain the return format or what fields the connection info will contain. For a 'smoke test' tool, an agent would likely need to know what to expect in the response. The description is too sparse for full completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the parameter semantic burden is none. Per rubric, a baseline of 4 applies since there are no parameters to document.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Show how this server connects and whether that works.' This is a specific informative action, distinct from sibling tools like omv_call or omv_shell. However, it doesn't explicitly differentiate itself by naming alternatives, so a 4 is appropriate.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit use cases: 'Useful as a first smoke test and when troubleshooting connectivity.' This gives clear context for when to invoke the tool. It does not mention when not to use it or alternatives, but the identified scenarios are specific enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• omv_shellA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It identifies the tool as arbitrary shell execution but does not warn about destructive potential, permission requirements, side effects, or output behavior, which is a significant gap for an unrestricted shell tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: purpose first, then use cases, then alternative guidance. Every sentence earns its place with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For an arbitrary shell tool with no output schema, the description explains its scope and relationship to omv_call well. However, it omits timeout behavior, return/output format, and safety caveats, which are important for such a powerful tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema descriptions cover both parameters (command and timeout), so the baseline is 3. The description adds context for command types but no parameter-specific semantics beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Run an arbitrary shell command on the NAS' with a specific verb and resource. It also distinguishes itself from omv_call by explicitly positioning itself for everything outside the RPC layer.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit intended use cases (journalctl, package status, docker commands, disk details) and gives a direct alternative: 'Prefer omv_call for anything OMV manages itself.' This is clear when-to-use and when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• omv_callA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden for behavioral disclosure. It reveals an important async behavior: some heavier methods run in the background and return a filename that must be passed to omv_wait_for_task. However, it does not warn that RPC calls may be destructive or require elevated permissions, nor does it describe the normal return format beyond the background case. This is meaningful but incomplete transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is moderately sized but each sentence contributes: the main-tool claim, the scope, the async behavior, and examples. It is front-loaded with purpose and uses examples efficiently. It could be tightened slightly, but it is well-structured for a generic RPC facade.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is a generic and potentially complex RPC wrapper with no output schema and no annotations. The description explains the async path and gives examples, but it does not mention how to discover valid method/service names via omv_list_methods or omv_list_services, nor does it describe normal return values or error behavior. Given the complexity, this is only partially complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already covers all four parameters with examples, so the baseline is 3. The description adds value beyond the schema by showing full call examples, illustrating how the params object is constructed, and explicitly stating that params should be omitted for methods taking none. This extra guidance helps the agent correctly format arguments.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool calls an OpenMediaVault RPC method and is the main tool. It goes beyond a simple verb+resource by noting that 'anything the web interface can do is available here,' which effectively defines its broad scope and distinguishes it from the sibling helper tools. Concrete examples reinforce the resource and action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives strong context for when to use the tool: it is the 'main' RPC entry point covering everything the web UI can do. It also explicitly explains that heavier background methods require passing the returned filename to omv_wait_for_task, which is an explicit alternative workflow. It does not mention omv_list_methods or omv_list_services for discovery, so it lacks complete exclusion/alternative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• omv_wait_for_taskA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It transparently states that the tool polls until the background task finishes, which communicates the blocking/asynchronous nature of the operation. It does not detail failure/timeout outcomes, but the core behavior is clearly and honestly described.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded: the first sentence states the action, and the second provides necessary context. Every sentence contributes useful information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple polling tool with two well-documented parameters, the description is sufficiently complete. It explains the workflow, the trigger condition, and the polling behavior. It omits explicit return-value details, but 'collect the output' adequately conveys the tool's function given the simple context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% parameter coverage with descriptions for both 'filename' and 'max_seconds'. The tool description adds minimal new semantic value beyond referencing omv_call's result, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: collecting output from an asynchronous background task started by omv_call. It uses a specific verb ('collect') and resource ('output of a background task'), and it distinguishes itself from siblings like omv_call by explicitly connecting the workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context: use this tool when omv_call returns a filename indicating an async job. It names the prerequisite condition and the polling behavior, but it does not explicitly mention alternatives or edge cases like when a task has already finished.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• omv_list_methodsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. As a listing operation, it is implicitly read-only, and the description adds the prerequisite relationship with omv_call. It doesn't detail output format or error handling, but for a simple introspection tool, that is acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, front-loaded with the primary action, and contains no fluff. Every sentence adds value: the first states what it does, the second explains why to use it.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool of this simplicity, the description is complete: it states the action, the timing relative to omv_call, and the parameter source. Without an output schema, it doesn't describe the return format, but the name 'list_methods' makes it obvious. Slight gap around failure modes keeps it from a 5.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description covers 100% of the single parameter (service), and the description adds no additional parameter-level meaning. The baseline of 3 is appropriate since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool lists every method of an RPC service, using a specific verb and resource. It distinguishes itself from sibling tools like omv_call (which calls a method) and omv_list_services (which lists services).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs to call this tool before omv_call to get the exact method name instead of guessing. This provides clear, actionable guidance on when to use this tool, fulfilling the dimension completely.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• omv_list_servicesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that service names are case-sensitive and some are lowercase, advising to use names exactly as returned. It also mentions plugin-added services, which is useful behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three well-structured sentences: the core statement, contextual explanation, and a necessary caveat. It is concise, front-loaded, and every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with 0 params and no annotations, the description fully covers purpose, usage, and an important case-sensitivity detail. It implicitly describes the output as a list of service names, which is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters, so the description doesn't need to explain parameter semantics. The baseline for 0 params is 4, and the description adds context about the return values being service names, which is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'List every available OpenMediaVault RPC service', which is a clear verb+resource. It positions itself as the starting point for discovery, distinguishing it from siblings like omv_list_methods.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description says 'This is the starting point', indicating when to use it. It also notes that installed plugins add services, giving context. It doesn't explicitly name alternatives, but the use case is well implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.