Server Quality Checklist

Tool Scores

• simple_job_statsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the full burden of behavioral disclosure. It implies a read-only informational tool but does not explicitly state that it performs no mutations, requires no auth, or has any other behavioral caveats. The phrasing 'answers whether this server is earning its place' is not operationally informative.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief and front-loaded with the core subject, but the final clause 'Answers whether this server is earning its place' adds little operational value and is somewhat metaphorical. Still, it is concise overall with minimal waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter statistics tool, the description gives a reasonable sense of the returned dimensions: jobs by kind, check pass rate, and speed. However, it does not clarify the output format or precisely what 'speed' refers to, and it lacks any comparison to sibling tools, leaving some ambiguity for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool accepts zero parameters, so the schema already fully covers the input surface. The description does not need to explain parameter details, and the baseline for zero-parameter tools applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description communicates that this tool reports statistics about job delegation, including breakdowns by kind, check pass rates, and speed. It is clear enough to identify the tool's purpose, though it lacks an explicit verb like 'returns' or 'computes' and does not explicitly differentiate itself from siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives such as simple_job or summarize. No context, conditions, or exclusions are provided that would help an agent choose this tool correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarizeB
  3.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavior, and it does: free, local, avoids consuming Claude context, fetches URLs, and enforces a real anti-copy/shorter-than-source check. It does not cover failure modes or service dependencies, but the main behavioral traits that would affect an agent's invocation and expectation are present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the main verb and purpose. Every sentence contributes useful information, but 'on pop' appears twice without definition, slightly hurting clarity. It is appropriately sized and not padded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 11 parameters, a nested check object, and no output schema, the description covers the main usage and the default verification behavior but leaves gaps: return value shape, error/failure behavior with URLs, and what the check object supports beyond its default. The schema covers most parameters, but the missing output context is notable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is high at 91%, so the baseline is 3. The description adds a little context about text/url/urls usage and the default check, but mostly repeats what the schema already documents. It does not add meaningful detail for parameters like focus, style, model, or temperature.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the core action clearly with a specific verb and resource: 'Summarise text with ornith on pop.' It also clarifies the two input modes (direct text or URL/URLs) and differentiates itself via 'does not spend Claude context on the source.' However, 'on pop' is unexplained jargon, and it does not explicitly contrast with sibling search_and_summarize, so it is not a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains how to invoke the tool ('Pass the text directly, or a url/urls to fetch first') but provides no guidance on when to choose this tool over its siblings. search_and_summarize is a plausible alternative for search-then-summarize use cases, yet no exclusion or comparison is mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_and_summarizeA
  4.2/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility and excels: it discloses the DuckDuckGo backend, the reading and summarization pipeline, the lack of Claude context use, the spacing of requests, and the crucial behavior that throttles are reported truthfully. It even documents a known failure mode with the 'think' parameter. This is exemplary transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description front-loads the purpose but then becomes a run-on sentence mixing operational details, typos, and policy notes. While information-dense and not excessively long, it could be split into clearer sentences to improve scannability. It feels a bit stream-of-consciousness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For an 11-parameter tool with no output schema, the description covers the core use case, operational constraints, and a known gotcha ('think'). It doesn't describe the return value, but that's a reasonable trade-off given the already rich parameter documentation and the absence of an output schema. The description is sufficient for an agent to decide when to call this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is high (91%), so the baseline is 3. The description itself does not add parameter-level details beyond what's in the schema, and the schema already describes 10 of 11 parameters. No contradiction or gap that requires the description to compensate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search the web (DuckDuckGo), read the top pages... and have ornith summarise them'. It also differentiates from siblings by declaring 'THE ONLY SEARCH TOOL HERE', making it unmistakable what this tool does relative to others. The typos ('pop', 'ornith') don't obscure the core verb-resource actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly frames the tool as the only search option ('THE ONLY SEARCH TOOL HERE') and provides practical guidance on rate-limiting and throttle behavior. However, it doesn't explicitly state when *not* to use it (e.g., when you already have content to summarize, which sibling 'summarize' might handle), leaving exclusions to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• simple_jobA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses key behaviors: it verifies the result before returning, refuses tasks without a check, and mentions limitations (bad for judgment calls, code, editing). However, it does not specify what happens if verification fails, error handling, or output format, leaving some behavioral aspects undisclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: it opens with the core action, then lists requirements and use cases. Every sentence adds value without redundancy, and the format is easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there is no output schema, the description should explain the return value or error cases, but it only says 'returns it' without specifying format. It also omits details about parameter interactions (e.g., temperature affects creativity) and failure modes, leaving some operational context incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides descriptions for all parameters (86% coverage). The description adds little beyond the note that a check is required (which is already in the schema). No additional semantic value is provided for parameters like temperature or max_tokens.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: delegating a subtask to ornith and verifying the result. It explicitly lists good and bad use cases (extraction, classification, reformatting vs. judgment calls, code, editing), making the purpose and scope unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides explicit guidance on when to use the tool: it requires a verifiable check, lists suitable tasks, and explicitly warns against unsuitable ones (e.g., judgment calls, code) with 'do those yourself.' This clearly differentiates it from the alternatives (doing it manually or using other tools).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• downloadA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly states the expected return values ('path, byte count and sha256') and adds the important caveat that success without these is not verified, which is a meaningful behavioral warning. It also implies the default host is pop. However, it does not describe failure behavior beyond the overwrite rule (which is in the parameter schema), nor does it mention whether downloads are atomic or what happens on network errors. Given the lack of annotations, this is a strong disclosure but not exhaustive, so a 4 is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the primary action and destination, followed by the return-value verification note and then the host-selection guidance. Every sentence carries meaningful information; there is no filler or redundancy. It is an excellent example of concise, high-signal writing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 7 parameters, no output schema, and no annotations, the description covers the core aspects: what it does, where it downloads, what it returns, and how to decide between hosts. It also adds the verification requirement that guards against silently bad downloads. It does not mention edge cases like how max_mb is enforced or what happens when an existing file blocks the download (though the parameter schema covers that), but these are secondary. Overall, an agent can call this tool safely and correctly, so a 4 is suitable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already describes all 7 parameters with 100% coverage, so the baseline is 3. The main description adds value by explaining the strategic choice of host ('Put it on pop when pop will use it...'), which reinforces the host parameter's purpose and provides context beyond its schema description. It also emphasizes the verification requirement, which ties into the expect_sha256 parameter. While the schema covers syntax, the description adds usage logic, so a 4 is justified.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb-and-resource pair — 'Download a file' — and specifies the two possible destinations ('to pop (default) or to this Mac'), which uniquely defines the tool's action. It also distinguishes it from the sibling tools (simple_job, summarize, etc.) which are entirely unrelated, so there is no ambiguity about what this tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit, actionable guidance on when to use each host: 'Put it on pop when pop will use it (models, datasets, anything for the GPU); put it on the mac when you will open it.' This directly tells an agent how to choose the destination based on the intended use, and the default is also stated. No alternative tools are relevant, so this guidance is complete.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.