Server Quality Checklist

Tool Scores

• riddle_batch_screenshotC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It states the tool returns base64 images, which is useful, but lacks critical details: it doesn't mention whether this is a read-only operation, potential rate limits, authentication needs, error handling, or how device selection affects output. For a tool that performs external actions (screenshotting URLs), this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and front-loaded: two sentences that directly state the action and output with zero wasted words. Every sentence earns its place by conveying essential information efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (external actions, multiple parameters) and lack of annotations or output schema, the description is incomplete. It misses behavioral context like safety, performance implications, and error scenarios. While concise, it doesn't provide enough information for an agent to use the tool confidently without additional assumptions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50% (only 'urls' has a description). The description adds no parameter semantics beyond what the schema provides—it doesn't explain the 'device' parameter's purpose or impact, nor does it clarify URL format requirements. With partial schema coverage, the description doesn't compensate adequately, but it doesn't contradict the schema either.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Screenshot multiple URLs' specifies the verb (screenshot) and resource (URLs), and 'Returns array of base64 images' indicates the output format. It distinguishes from siblings like 'riddle_screenshot' by emphasizing batch capability ('multiple URLs'), though it doesn't explicitly contrast with all siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. It doesn't mention when to prefer this batch tool over 'riddle_screenshot' (likely for single URLs) or 'riddle_click_and_screenshot' (which may involve interaction), nor does it specify prerequisites or exclusions for usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• riddle_get_jobC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool retrieves 'status and artifacts,' implying a read-only operation, but doesn't specify if it requires authentication, has rate limits, or details the format of returned data. This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function without any unnecessary words. It's front-loaded and appropriately sized, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what 'status and artifacts' entail, such as possible states or artifact types, nor does it cover error handling or prerequisites. For a tool with no structured behavioral data, this leaves too many gaps for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the 'job_id' parameter clearly documented. The description doesn't add any extra meaning beyond the schema, such as explaining what a 'Riddle job' is or how to obtain the ID, so it meets the baseline score without enhancing parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get') and resource ('status and artifacts of a Riddle job'), making it easy to understand what it does. However, it doesn't explicitly differentiate from sibling tools like 'riddle_automate' or 'riddle_run_script', which might also involve job-related operations, so it misses the highest score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, such as needing a job ID from another tool, or clarify its role among siblings like 'riddle_batch_screenshot' or 'riddle_click_and_screenshot', leaving the agent without usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• riddle_run_scriptC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses the async behavior and job_id return mechanism, but lacks critical details like error handling, timeout behavior, authentication needs, rate limits, or what 'page object available' entails in the script context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with two sentences that directly convey core functionality and return value. Every word earns its place, and it's front-loaded with the primary action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 4-parameter tool with no annotations, no output schema, and 50% schema coverage, the description is inadequate. It doesn't explain the return format beyond job_id, doesn't cover undocumented parameters, and lacks behavioral context needed for safe invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50% (only 'url' and 'script' have descriptions). The description adds no parameter-specific information beyond what the schema provides. It implies 'url' and 'script' are required via context, but doesn't explain 'width' and 'height' parameters at all, leaving them undocumented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Run a Playwright script') and target ('on a page'), and specifies it's asynchronous. However, it doesn't differentiate from sibling tools like 'riddle_automate' or 'riddle_batch_screenshot' which might also involve script execution or automation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'riddle_automate' or 'riddle_batch_screenshot'. It mentions the async nature and job_id return, but lacks explicit when/when-not instructions or prerequisites for usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• riddle_screenshotA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions the API ('Riddle API') and output format, but lacks details on behavioral traits such as rate limits, authentication needs, error handling, or whether the operation is idempotent. For a tool with no annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core action ('Take a screenshot') and includes essential details (resource, API, output format) without unnecessary words. Every part of the sentence adds value, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description provides basic purpose and output format but lacks completeness for a tool with 4 parameters and potential complexity (e.g., API behavior, error cases). It is adequate as a minimum viable description but has clear gaps in contextual details needed for full understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters (url, width, height, device) with descriptions and defaults. The description does not add any parameter-specific semantics beyond what the schema provides, such as explaining interactions between device and width/height, so it meets the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Take a screenshot') and resource ('of a URL using the Riddle API'), distinguishing it from siblings like 'riddle_batch_screenshot' (multiple screenshots) and 'riddle_click_and_screenshot' (interactive screenshot). It explicitly mentions the output format ('base64-encoded PNG image'), which helps differentiate its purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for single URL screenshots but does not explicitly state when to use this tool versus alternatives like 'riddle_batch_screenshot' for multiple URLs or 'riddle_click_and_screenshot' for interactive actions. No exclusions or prerequisites are mentioned, leaving usage context somewhat vague.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• riddle_automateA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds useful context about what the tool does ('wait for completion', 'return all artifacts', 'Includes console logs and network HAR') and its scope ('Full sync automation'), but it doesn't cover important behavioral traits like error handling, performance characteristics, or whether it's idempotent. For a tool with no annotations and complex automation, this leaves gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with every sentence earning its place. The first sentence states the core functionality, the second lists included artifacts, and the third emphasizes the automation scope—all without waste. It's efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (automation with 5 parameters) and no annotations or output schema, the description is moderately complete. It covers the high-level purpose and scope but lacks details on return values, error cases, or performance implications. For a tool without structured output information, more guidance on what 'artifacts' entail would be beneficial.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description doesn't add any meaning beyond what the input schema provides, such as explaining interactions between parameters or providing additional usage examples. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('Run a Playwright script, wait for completion, and return all artifacts') and resources ('Playwright script', 'artifacts', 'console logs', 'network HAR'). It distinguishes from siblings by emphasizing 'Full sync automation - one call does everything', contrasting with tools like riddle_batch_screenshot or riddle_click_and_screenshot that likely perform more limited operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('Full sync automation - one call does everything'), implying it's for comprehensive automation tasks rather than simpler operations. However, it doesn't explicitly state when not to use it or name specific alternatives among the siblings, such as riddle_run_cript or riddle_screenshot, which might be better for lighter tasks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• riddle_click_and_screenshotA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses key behavioral traits: 'Uses force-click by default to handle animated buttons' explains a default behavior not obvious from the schema. However, it doesn't mention error handling, timeout behavior, screenshot format, or what happens if the selector isn't found. For a mutation tool (clicks change state), more behavioral context would be helpful.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise (two sentences) and front-loaded with the core functionality. Every sentence adds value: the first states the exact workflow, the second provides usage context and a key behavioral detail. There's zero wasted verbiage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and 5 parameters (with 80% schema coverage), the description is minimally adequate. It covers the basic workflow and one behavioral trait, but for a tool that performs actions (load, click, screenshot) with multiple parameters, it lacks details on output format, error conditions, and the purpose of the 'device' parameter. The context is incomplete for confident use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 80%, so the baseline is 3. The description adds minimal parameter semantics beyond the schema: it implies the 'click' parameter is for interactive elements like buttons, and mentions 'force-click by default' which relates to the 'force' parameter. However, it doesn't explain the 'device' enum options or provide additional context for 'wait_ms' beyond the schema's default value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action sequence: 'load URL, click a selector, take screenshot.' It distinguishes from siblings by focusing on single-click automation (vs. batch_screenshot, run_script, or generic screenshot). The phrase 'Good for testing button clicks, game starts, etc.' further clarifies the use case.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: 'Good for testing button clicks, game starts, etc.' It implies this is for simple automation scenarios. However, it doesn't explicitly state when NOT to use it or name specific alternatives among the sibling tools (e.g., when to choose riddle_batch_screenshot instead).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.