Server Quality Checklist

Tool Scores

• get_recipe_var_rawB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states what the tool does but doesn't describe how it behaves: it doesn't mention if it's read-only, what format the 'raw assignments' are returned in, whether it handles errors for missing files/variables, or any performance considerations. This leaves significant gaps for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that efficiently conveys the core purpose without any wasted words. It's front-loaded with the key action and resource, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 required parameters, no output schema, no annotations), the description is minimally adequate. It covers the basic purpose but lacks details on behavior, output format, and sibling differentiation. Without annotations or output schema, more context would be helpful, but it meets the minimum viable threshold.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with clear documentation for both parameters (file_path and variable). The description adds no additional parameter semantics beyond what's in the schema, such as examples of variable names beyond those listed or file path constraints. With high schema coverage, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get all raw assignments') and the target resource ('for a specific variable in a .bb or .bbappend file'), making the purpose immediately understandable. However, it doesn't explicitly differentiate this tool from its siblings (find_recipe_files, parse_recipe_file, scan_layer_recipe_files), which would be needed for a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus its siblings or alternatives. It mentions the specific file types (.bb or .bbappend) but doesn't explain why one would choose this tool over parse_recipe_file for variable assignments, leaving the agent to infer usage from context alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_layer_recipe_filesB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions listing files but doesn't disclose behavioral traits like whether it recursively searches subdirectories, handles errors, returns absolute/relative paths, or has any rate limits. The description is minimal and lacks operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero wasted words. It is appropriately sized for a simple tool and front-loads the core purpose without unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one parameter, no output schema, no annotations), the description is minimally adequate but incomplete. It lacks details on output format, error handling, and behavioral traits, which are important for a file-listing operation without structured output documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% with one parameter ('layer_path') fully documented in the schema. The description adds no additional parameter semantics beyond what the schema provides, so it meets the baseline of 3 for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('List all'), target resource ('.bb/.bbappend files'), and scope ('under a Yocto layer path'). It distinguishes from sibling tools like 'find_recipe_files' by specifying the exact file types and location context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'find_recipe_files' or 'parse_recipe_file'. The description only states what it does without indicating appropriate contexts, prerequisites, or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_recipe_filesA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions recursive searching and filename matching, but lacks details on permissions needed, error handling, output format, or performance considerations (e.g., depth limits, large directory handling). For a search tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that efficiently conveys the core functionality without redundancy. It is front-loaded with the main action and includes all essential elements (search type, file types, matching criteria, scope), making it easy to parse and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (recursive file search), lack of annotations, and no output schema, the description is adequate but incomplete. It covers the basic operation but omits details on return values, error cases, and behavioral nuances that would help an agent use it correctly in varied contexts. It meets minimum viability but has clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('root_path' and 'query') adequately. The description adds minimal value by reinforcing that 'root_path' is the directory to search under and 'query' matches against filenames, but does not provide additional syntax, examples, or constraints beyond what the schema specifies. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Search recursively'), target resources ('.bb/.bbappend files'), and scope ('under root_path'), distinguishing it from sibling tools like 'parse_recipe_file' (which parses rather than searches) and 'scan_layer_recipe_files' (which may have different scoping). It uses precise technical terminology that aligns with the tool's domain.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for finding recipe files by filename substring matching, but it does not explicitly state when to use this tool versus alternatives like 'scan_layer_recipe_files' or 'get_recipe_var_raw'. No guidance is provided on prerequisites, exclusions, or comparative contexts, leaving the agent to infer usage from the description alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• parse_recipe_fileA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses key behavioral traits: it parses a single file, returns raw assignments, and does not evaluate variables. However, it lacks details on error handling, performance, or output format, which are important for a parsing tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that is front-loaded with the core purpose and includes a critical behavioral note ('never evaluates variables'). There is no wasted text, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (parsing with specific constraints), no annotations, and no output schema, the description is adequate but incomplete. It covers the basic operation and a key limitation, but lacks details on return values, error cases, or integration with sibling tools, leaving gaps for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the 'file_path' parameter fully. The description does not add any additional meaning or context beyond what the schema provides, such as file format constraints or examples, meeting the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('parse'), the resource ('.bb or .bbappend file'), and the output ('all raw variable assignments'), distinguishing it from siblings like 'find_recipe_files' (which finds files) and 'get_recipe_var_raw' (which gets a specific variable). It explicitly notes it 'never evaluates variables,' which further clarifies its scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by specifying the file types (.bb or .bbappend) and the output (raw variable assignments), but it does not explicitly state when to use this tool versus alternatives like 'get_recipe_var_raw' or 'scan_layer_recipe_files.' No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.