Server Quality Checklist

Tool Scores

• make_ocsf_mappingA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate readOnlyHint=false and idempotentHint=true, but the description adds limited context beyond them. The phrase 'Follow the workflow instructions provided in the response' hints that the tool may return guidance rather than directly mutating the pipeline, which is useful, but it does not clarify side effects, permissions, or what happens to existing mapping. This is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, but the bullet list contains three similar items ('map security logs to OCSF', 'normalizing data...', 'make data compatible...') that are somewhat redundant. It's not wasteful enough to score lower, but it could be more tightly structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (2 params, no output schema) and the presence of annotations, the description covers the core purpose, usage scenarios, and hints at the response format. It lacks details about sample input format, edge cases, or what the workflow instructions specifically entail, but it is sufficient for basic selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50%; only 'sample' is described, and the description's mention of 'sample log events' mirrors the schema's own description. The 'ctx' parameter is unexplained. The description does not add significant meaning beyond the schema, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Add OCSF mapping to a TQL parsing pipeline.' It also elaborates with scenarios like 'map security logs to the OCSF standard' and 'normalizing data from multiple sources into a common schema,' clearly distinguishing it from sibling OCSF lookup tools (e.g., ocsf_get_class) and parser creation tools (e.g., make_parser).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description has an explicit 'Use this tool when' section listing four relevant scenarios, providing clear context for when to invoke it. It does not explicitly mention alternatives or exclusions, so it falls short of a 5, but the guidance is direct and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• make_parserA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint false and idempotentHint true. The description adds that it provides a step-by-step workflow and instructs the agent to follow the instructions in the response. This adds some context about the tool's behavior, but it does not fully disclose the nature of side effects (e.g., package creation) or clarify what happens on repeated calls. No contradiction with annotations, so a 3 is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than typical but well-structured with a clear first sentence, bulleted use cases, and a numbered workflow list. Every segment contributes meaning, though it could be trimmed without losing value. Front-loading the purpose helps; the overall organization earns a 4.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there is no output schema, the description should explain what the tool returns. It lists workflow steps and says to follow instructions, but it does not explicitly state whether the response contains generated code, a package, or a plan. The description provides a decent overview but leaves ambiguity about the exact output format and any potential errors or limitations, warranting a 3.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema descriptions cover only the 'sample' parameter (50% coverage), and the description does not add any additional meaning for either parameter. The 'ctx' parameter is completely undocumented. The description mentions log format and sample events but does not explain the parameters or their syntax beyond what the schema already provides, failing to compensate for the missing parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with 'Generate a TQL parser for the given log format', which clearly states the specific action (generate) and resource (TQL parser). The use case bullets further clarify the tool's scope, and it is distinct from sibling tools like make_ocsf_mapping, which handles mappings rather than parsers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes an explicit 'Use this tool when' section with concrete scenarios such as having sample log events, starting a parser for JSON/CSV/syslog/key-value, or needing TQL operator guidance. It provides clear context but does not mention when not to use it or explicitly name alternatives, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_pipelineA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate `readOnlyHint=false` and `openWorldHint=true`, implying potential side effects and external access. The description adds that diagnostics are enabled, providing useful context, but it fails to disclose that executing TQL pipelines may have side effects or resource implications. There is no contradiction between the description and annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a front-loaded opening statement followed by a clear bulleted list of use cases. It is slightly longer than necessary but every sentence contributes meaningful guidance, with no filler or redundant repetition of schema/annotation details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core function and diagnostics behavior, but lacks information about output/return format and potential side effects. Since there is no output schema, the description could be more complete in explaining what the agent can expect from execution, though the primary usage is adequately described.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, all parameters are already documented in the input schema. The description doesn't add parameter-level details, but this is acceptable because the schema handles the semantic burden. The baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool executes a TQL pipeline via the local `tenzir` binary, and lists specific use cases such as testing, debugging, and verifying syntax. It effectively distinguishes itself from sibling tools like `run_test` (which runs formal tests) and `package_add_test` (which adds tests to packages).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  A 'Use this tool to' section provides explicit contexts for invocation, covering testing, debugging, and verification. It doesn't explicitly state exclusions or alternative tools, but the scenarios clearly position it as a development/debugging aid, which is sufficient guidance for most cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• package_add_changelogA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=false and destructiveHint=false, establishing that this is a non-readonly, non-destructive mutation. The description adds that entries are 'timestamped and categorized,' which is minor additional context. However, it doesn't disclose important side effects such as whether the changelog file is created if missing, whether package_dir must already exist, or how existing entries are affected. The safety profile is clear, but operational behavior remains vague.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a one-sentence purpose statement, then a bulleted list of use cases. The bullet list is somewhat redundant with the opening sentence and could be tightened, but the overall length is modest and the structure aids scanning. It earns a 4 rather than 5 due to the slight redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple mutation tool with all parameters described in the schema and no output schema, the description covers the essential 'what' and 'why.' The missing detail about file creation or prerequisites is a minor gap, but the tool is straightforward enough that an agent can likely invoke it correctly with the given information. It is not as lean as the get_calls example but is still reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description mentions 'breaking changes, new features, bug fixes, and general changes,' which loosely maps to the 'type' parameter values (breaking, feature, bugfix, change), but it adds no syntax or format details beyond the schema. The description does not significantly augment the parameter-level information already present in the input schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear action: 'Add a changelog entry to a package.' It specifies the resource (package) and the operation (add changelog), and distinguishes from sibling tools like package_add_operator and package_add_test by focusing on changelog maintenance. The listed use cases (document changes, track breaking changes, etc.) reinforce the purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states 'Use this tool to' and provides a list of valid use cases, giving clear guidance on when to invoke it. It doesn't name alternatives or define when not to use it, but the sibling tool names and the changelog-specific context make the appropriate context obvious. Lacks explicit exclusions but is otherwise solid.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• package_createA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate this is a write operation (readOnlyHint false, destructiveHint false). The description adds context about creating a scaffold and directory structure, but does not disclose potential behavior such as whether existing files in the target directory would be overwritten or if the directory must be empty. More detail on side effects would be valuable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a brief opening sentence followed by a bullet list of use cases and a closing note about next steps. It is appropriately sized, front-loads the main purpose, and avoids unnecessary fluff, though a few of the bullet items overlap slightly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's purpose, usage guidelines, and integration with sibling tools. However, with no output schema, it does not explain what the tool returns after creation (e.g., a success message, package path, or file listing). It also lacks details on prerequisites or behavior with existing directories, leaving some contextual gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 50% (ctx lacks a description). The description does not mention the ctx parameter at all, nor does it add meaningful information about package_dir beyond what the schema already provides. Since the description should compensate for low schema coverage, this is a notable gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool creates a new Tenzir package scaffold, with specific actions listed (starting a project, setting up directory structure, initializing metadata). It uses a specific verb+resource ('Create a new Tenzir package scaffold') and distinguishes from sibling tools by mentioning package_add_operator and package_add_test as follow-up steps.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly lists when to use the tool ('Use this tool to: Start a new Tenzir package project...') and provides alternatives for subsequent steps ('After creation, use package_add_operator... and package_add_test...'). This gives clear context on both when to use and what to use instead for related tasks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• docs_readA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already declare readOnlyHint=true and idempotentHint=true, so the agent knows this is a safe read operation. The description adds the context that the documentation is embedded and provides workflow guidance, but it does not describe the output format or error behavior. With annotations covering safety, this is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded, with a single sentence followed by a focused bullet list. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a simple read-only tool with a fully documented parameter. The description covers its purpose and usage context, and the annotations cover its safety profile. The lack of an output schema is not a significant gap since reading documentation content implies returning the text.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 100% coverage for the single 'path' parameter, including examples like 'reference/operators/select'. The description does not add additional parameter semantics beyond what the schema provides, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Read documentation content from the embedded Tenzir documentation,' specifying the verb (Read) and resource (embedded documentation). It distinguishes itself from the sibling docs_search by focusing on reading specific paths rather than searching, and the bullet points give concrete use cases.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use guidance with bullets: 'Read operator documentation BEFORE using any TQL operator,' 'Read function documentation BEFORE using any TQL function,' and 'Study tutorials and guides for learning workflows.' It does not explicitly mention when to use docs_search instead, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ocsf_get_objectA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations (readOnlyHint=true, idempotentHint=true) already establish that this is a safe, read-only operation. The description adds useful behavioral context: it returns the 'complete definition including all fields and metadata,' and clarifies that objects are reusable components. No contradictory information exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear opening line, bulleted use cases, and a brief explanatory paragraph. It is not overly verbose, and each section has a purpose. A slightly tighter wording could be achieved, but it remains easily skimmable and informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple nature of the tool and the presence of sibling tools for context, the description covers the essential aspects. It clearly states what the tool returns (complete definition, fields, metadata), which partially compensates for the lack of an output schema. It does not go into deep detail, but that is acceptable for a reference lookup tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the schema already provides clear descriptions and examples for both parameters (name and version). The description adds some context about how the parameters are used (e.g., mapping source data) but does not substantially enhance the parameter semantics beyond the schema's baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear, specific purpose: 'Get the complete definition of an OCSF object type including all fields and metadata.' It uses a strong verb ('Get') and identifies the exact resource ('OCSF object type'). It also distinguishes from siblings like ocsf_get_class by focusing on object definitions versus class-level structures.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit use cases (understanding nested structures, field mapping, TQL construction) that clarify when to use the tool. It does not explicitly name alternative tools or say 'when not to use this,' but the context of reusable objects within classes implies a clear differentiation from class-level tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• docs_searchA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and idempotentHint=true, so the safety profile is known. The description adds behavioral context by explaining that depth 'traverses cross-references' and enables discovery of related operators/functions, which goes beyond the schema's dry parameter description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a one-line summary and bullet-point use cases, front-loading the main purpose. It is moderately concise, though the final paragraph on depth is somewhat redundant with the earlier mention of 'See Also' expansion.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, and the description does not describe what results look like (e.g., format, ordering, or how query/paths interact). It covers use cases well but omits behavior for common scenarios like supplying both query and paths or providing no parameters, which is a gap for a search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All five parameters have schema descriptions (100% coverage), so the schema does the heavy lifting. The description adds value by explaining depth's role in 'See Also' expansion and search_type as a category filter, but doesn't elaborate on limit, paths, or query syntax beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Search documentation by keyword or retrieve specific paths' with a specific verb and resource, and lists distinct use cases like finding operators/functions and discovering related docs via 'See Also' links. This distinguishes it from sibling tools like docs_read, which implies reading a specific document.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit 'Use this tool to' scenarios, covering keyword search, 'See Also' expansion, and category filtering. However, it does not explicitly mention when not to use it or name alternative tools (e.g., docs_read for direct reading), so there's no exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• package_add_operatorA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations only declare readOnlyHint=false, destructiveHint=false, which tells the agent it's a mutation but little else. The description adds meaningful behavior: automatic generation of test scaffolds, support for nested namespaces, and post-install availability as package_id::operator_name. It doesn't cover overwrite semantics or error conditions, but goes beyond annotation basics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: an opening statement, a concise bullet list of use cases, and a closing note on availability. Every sentence earns its place, no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 5-parameter tool with no output schema, the description gives enough to understand the operation and its outcomes (operator added, test scaffold, post-install usage). It doesn't mention return values or error cases, but for an 'add' operation this is adequate and matches the complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so each parameter is already documented. The description adds context around 'name' (nested namespace examples) and references test scaffold generation for 'no_tests', but largely reinforces what's in the schema. Baseline 3 is appropriate when schema carries the load.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Add a user-defined operator (UDO) to a package.' It also lists concrete use cases (custom TQL operators, nested namespaces, parsers/transformations/OCSF mappings, test scaffold generation), clearly distinguishing it from siblings like make_parser or make_ocsf_mapping.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The 'Use this tool to' bullet list provides explicit context for when to use it. It lacks explicit when-not-to-use guidance or named alternatives, but the use cases are clear and the sibling list helps. Not quite a 5 because exclusions/alternatives are absent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_testA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate idempotentHint=true, readOnlyHint=false, destructiveHint=false. The description adds meaningful context by explaining that update=True modifies baselines and passthrough=True shows actual vs expected output for debugging. It goes beyond just restating the annotations, though it does not detail all potential side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a brief introduction and a bulleted list of use cases. Every sentence earns its place, and the overall length is appropriate for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the main use cases, selection semantics, and support for fixtures in integration tests. It is complete enough for an agent to invoke the tool correctly, though it does not describe the return value or test result format, which would be helpful but is not critical given the tool's nature.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 80%, so baseline is 3. The description reinforces the semantics of `selection` (files, directories, empty list) and mentions update/passthrough effects, but these are also described in the schema. It does not clarify the `root` or `ctx` parameters beyond the schema, so it adds limited additional value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs tests for TQL pipelines using the tenzir-test framework, with a specific verb and resource. It also enumerates distinct use cases (verify operators, regression tests, baselines, debugging), which distinguishes it from sibling tools like run_pipeline or package_add_test.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit use cases: verifying package operators, running regression tests, generating baselines, and debugging failures. It does not explicitly mention when not to use the tool or name alternative tools, but the context is clear enough for an agent to choose it for test-related tasks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ocsf_get_classA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint. The description adds that it returns the complete class definition including all attributes, types, and constraints, and explains what can be validated. It doesn't cover error cases, but for a read-only lookup this is relevant context beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: a concise opening sentence, a bulleted list of use cases, and a closing summary. Every sentence adds value, and the key purpose is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple lookup tool with two parameters and no output schema, the description covers purpose, usage, and return value (complete class definition). It is sufficient for an agent to pick and use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with both name and version having examples. The description does not add significant new parameter meaning beyond the schema's own descriptions, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get the complete definition of a specific OCSF event class' using a specific verb and resource. It distinguishes from sibling tools like ocsf_get_classes and ocsf_get_object by focusing on a single class and its full schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The 'Use this tool to' list provides four concrete use cases (understand schema, see required vs optional fields, discover nested objects, validate mapping). It does not explicitly mention alternative tools or when not to use it, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ocsf_get_classesA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and idempotentHint=true, so the safety profile is covered. The description adds that it returns 'all' classes and their descriptions for a specified version, which clarifies the scope but does not disclose additional behavior such as pagination or error handling. This is adequate but not rich, consistent with the annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is succinct, front-loaded with the core function, and uses a bulleted list for usage guidance. It avoids redundancy and every sentence contributes value, including the pointer to `ocsf_get_class`. This is a model of concise, structured tool documentation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (one required parameter, no output schema). The description covers the action, the use context, and the next step, making it complete for effective selection and invocation. No gaps are apparent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides 100% coverage for this single parameter, including a clear example ('1.6.0'). The description mentions 'specific schema version' but does not add further parameter-level detail beyond the schema. Baseline of 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Get all OCSF event classes and their descriptions for a specific schema version,' identifying the specific verb, resource, and scope. It also distinguishes itself from the sibling tool `ocsf_get_class` by noting that the latter is for complete schema details, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit use cases ('before creating a mapping', 'identify which class best matches your log data', 'understand the purpose and scope') and explicitly directs the user to the alternative tool `ocsf_get_class` once a candidate is found. This clearly communicates when to use this tool versus its siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ocsf_get_latest_versionA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and idempotentHint=true, so the safety profile is known. The description adds behavioral context by explaining that only stable releases are returned and development versions (alpha, beta, rc) are excluded, which is not inferable from annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: a one-sentence purpose statement, a bulleted list of use cases, and a brief note on filtering behavior. Every sentence earns its place, and the most important information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that the tool has no parameters and no output schema, the description is complete: it states what is returned (latest stable version), what is filtered out, and when to use it. This fully covers the agent's needs for selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the description need not explain parameter semantics. The baseline for no-parameter tools is 4, and the description appropriately focuses on the return value rather than parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Return the latest stable OCSF schema version.' It uses a specific verb and resource, and distinguishes itself from sibling tools like ocsf_get_versions by emphasizing 'latest stable' and filtering out development versions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit use cases: 'Get the current recommended OCSF version for new mappings', 'Ensure you're using up-to-date schema definitions', and 'Start OCSF mapping workflows'. It also notes what it filters out, implying that a different tool (e.g., ocsf_get_versions) should be used for all versions, though it does not explicitly name the alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• package_add_testA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds context beyond the minimal annotations by explaining the tenzir-test framework and the scaffolding workflow. It implies the tool creates test files, but does not explicitly discuss side effects like overwriting existing files or permission requirements. No contradiction with annotations is present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear first sentence, bullet points for use cases, and a closing note about the framework. It is concise, front-loaded, and all content is relevant to selecting and using the tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description provides sufficient context for a tool with 7 parameters, covering core workflow and linking to run_test. It could be more complete by mentioning what happens if test_file already exists or what the tool returns, but these are not essential for basic invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although the schema covers all parameters, the description enriches meaning by explaining how input/output pairs define expectations, that fixtures can be embedded Tenzir nodes, and how omitted output can be filled later with run_test. This adds usage context beyond raw parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool adds a test to a package, with specific use cases like creating test cases, defining input/output expectations, and setting up fixtures. It is distinct from siblings like run_test and package_add_operator by focusing on test creation rather than execution or operator packaging.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool and when to use the alternative run_test: it says to use run_test with update=True to generate baselines from actual execution. It also outlines concrete scenarios (test cases, input/output pairs, fixtures) that clarify the tool's role.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ocsf_get_versionsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint, and the description adds valuable behavioral context by specifying it lists 'bundled' versions and is a read-only operation. It doesn't contradict annotations and provides useful scoping beyond the title.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is succinct, front-loaded with a clear one-liner, then uses a concise bullet list for usage, and closes with an alternative recommendation. No wasted words; every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple no-parameter listing tool with strong annotations, the description is complete. It states what the tool does, when to use it, and points to the more appropriate sibling for the common case. No output schema is needed, and the return value (a list of versions) is implied.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4. The description doesn't need to explain parameters, and the empty schema is fully consistent with the tool's purpose.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all bundled OCSF schema versions' with a specific verb and resource, and distinguishes itself from the sibling `ocsf_get_latest_version` by emphasizing 'all bundled' versions. The bullet points reinforce the exact purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly lists use cases (see available versions, choose a version, understand evolution) and provides a direct alternative: 'Typically you'll want to use `ocsf_get_latest_version` instead'. This gives clear guidance on when to use this tool vs. its sibling.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.