Server Quality Checklist

Tool Scores

• list_live_topicsA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description aligns with the annotations (readOnly, idempotent) by using the term 'inspect,' but does not add any additional behavioral details beyond the annotations themselves. The annotations already cover the key transparency aspects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence that conveys the essential purpose without any unnecessary words or details. It is well-structured and to the point.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the description indicates the general outcome (listing topics), it gives no information about the meaning of the input parameters or the expected output structure. This leaves significant gaps for a potential caller.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description provides no information about the four parameters (type_, args, host, port). Since the schema also lacks descriptions, the meaning of these parameters, especially the required type_, is entirely unclear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: inspecting a live data stream and listing available topics. It also distinguishes this from subscribing, which is helpful.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description notes it is 'helpful before starting a subscription,' which gives a clear use case. However, it does not explicitly compare or contrast with alternative tools like describe_topic or query_messages, so it could be more prescriptive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• export_for_rerunA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate the operation is non-destructive and idempotent. The description adds useful behavioral context: scalar signals become Rerun time series, the output is an .rrd file, and a specific dependency is required. However, it does not disclose output file naming/location, behavior with non-scalar topics, or possible size/scope limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three focused sentences: it opens with the core purpose, then adds the recommended usage context, then notes the dependency. It is concise and front-loaded, though the title and first sentence repeat the same 'export event window' idea.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has seven parameters and zero schema descriptions, the description provides a decent high-level overview but is not complete enough for reliable parameter use. It covers purpose, use case, output format, and dependency, yet leaves parameter semantics and output specifics largely implicit. The presence of an output schema and good annotations helps, but the large parameter surface area still demands more detail.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There is zero schema-level parameter description coverage, so the description must compensate, but it does not explicitly describe any of the seven parameters. It only hints at time window and topic filtering via phrases like 'time window of topic data' and 'scalar signal,' leaving `path`, `topics`, `start_seconds`, `end_seconds`, `signals`, `name`, and `args` under-explained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action—exporting a time window of topic data as a Rerun recording (.rrd)—and differentiates the tool from sibling export tools by naming the Rerun viewer and the .rrd format. It also mentions the concrete outcome (`rerun <file>` opens the event).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says when to use the tool: 'Use after preview_pipeline to hand an event to a human for visual inspection.' It also notes a prerequisite (optional rerun-sdk dependency), which helps the agent plan setup. It does not explicitly contrast with sibling export formats, but the unique target format and viewer make the intended context fairly clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_messagesA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the description only needs to add context beyond safety. It adds the single-topic scope and the DuckDB engine, but does not discuss query limits, errors, or authorization; without the output schema, the 'structured dictionaries' line would be the only return-value insight.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with the action first, output second, and usage guidance third; no filler. It is appropriately scaled for a moderately complex tool, though it could add a brief parameter note without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The core behavior and primary use case are covered, and the output schema plus annotations handle return values and safety. Still, for a tool with six parameters and user-supplied SQL, the description omits time-range semantics, args, and any boundary conditions such as cross-topic queries being unsupported.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at 0%, the description carries the parameter-explanation burden. It clarifies path (data source), topic (single topic), and sql_statement (DuckDB SQL capable of filtering/aggregation/downsampling), but leaves args, start_seconds, and end_seconds entirely unexplained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific action ('Run a DuckDB SQL query') and a bounded resource ('messages from a single topic in a data source'), clearly distinguishing it from sibling describe/run/export tools. It also states the intended result ('structured dictionaries').

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says to use this tool for answering questions about message data with filtering, aggregation, and downsampling, which gives clear application context. It does not, however, name alternatives or exclusion cases, such as when to use export_for_plotjuggler or run_pipeline instead.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_loggingsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already declare readOnly, idempotent, and non-destructive behavior, which the description reinforces with 'Extract' but does not add new behavioral details. The description also mentions time filtering but does not clarify edge cases or side effects, so transparency relies mostly on annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with no redundancy or irrelevant details. It directly states the action, the scope (INFO/WARN/ERROR), and the optional filtering, which is efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description provides the core purpose and usage but omits details about the return format or what happens when no time filter is applied. Since there is no output schema, it should clarify what the tool returns (e.g., a list of messages), but it only implies messages. This leaves some ambiguity for agents.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate for parameter explanations. It only hints at 'optional time filtering' related to start_seconds and end_seconds, but leaves 'path' and 'args' completely undefined. This is insufficient for an agent to correctly construct the call without additional context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool extracts INFO, WARN, and ERROR messages from a data source, which is specific and distinct from sibling tools that focus on pipelines or queries. It directly names the verb 'Extract' and the resource 'data source', making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use for debugging or diagnostics', providing a clear context for when to employ this tool. It also mentions optional time filtering, which further guides usage, but it does not compare against alternatives or state when not to use it, so it is not fully exhaustive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• export_for_lerobotA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already disclose idempotency and non-destructiveness; the description adds valuable context beyond that: data is resampled to uniform fps, windows become episodes, and feature vectors are composed. The beta caveat ('awaiting validation by real training runs') is honest about maturity. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three tightly written sentences deliver purpose, mechanism, workflow, and beta status with no filler. The most important information is front-loaded in the first sentence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is strong on the high-level transformation and workflow, and an output schema exists, but the tool is complex with nine parameters and nested objects. Given 0% schema description coverage, the description does not fully compensate by explaining required argument semantics, so completeness is only adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description does not compensate for the nine parameters. It mentions fps and signals/features at a high level, but does not explain required parameters like path, topics, episodes, features, task, or optional ones like args and robot_type. The agent would have to guess at the intended meanings and formats.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb and resource: 'Export time windows as a LeRobotDataset v3.0 for robot-learning training.' It further details the transformation semantics (each window becomes an episode, resampled to uniform fps, signals become feature vectors), which distinguishes it from sibling export tools like export_for_plotjuggler and export_for_rerun.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit workflow context: 'Use after preview_pipeline to turn detected events into a curated dataset.' This clearly indicates when the tool should be invoked. However, it does not mention when-not-to-use or name alternatives for different export formats.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_pipeline_capabilitiesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  While annotations already cover read-only and idempotent behavior, the description does not clarify the impact of the include_unavailable parameter, which could lead to misinterpretation of what 'available' means. The description's phrasing might imply only available items are returned by default, which is not fully accurate given the parameter.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, consisting of two clear sentences with no redundant information. It efficiently conveys the tool's purpose and primary use case.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description provides enough context for the tool's primary function and when to use it, but it omits explanation of the parameter and does not mention any output structure or limits. This leaves some gaps for an agent trying to fully understand the tool's behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The parameter include_unavailable has no description in the schema, and the tool description does not explain its meaning or effect. Although the name is suggestive, the lack of any explanation leaves its semantics ambiguous, especially regarding the default behavior.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool lists tasks and gates for composing a data pipeline, along with their module path, kind, constructor parameters, and summary. This is specific and distinguishes it from sibling tools like run_pipeline or describe_data_source.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'Use this before authoring a pipeline so the correct module and args are chosen instead of guessed,' giving a clear when-to-use directive. This is an explicit usage guideline that effectively replaces guessing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_poml_capabilityA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the description need not restate those. The description adds that the file specifies instructions and output formats, which is useful context. However, it does not disclose potential side effects like logging or resource usage, but the annotations cover the safety profile adequately.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, all earning their place: purpose, discovery path, and context customization. No fluff, properly front-loaded with the key action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool is high-level with an output schema and clear annotations, the description covers the essential aspects: what it does, how to find capabilities, and the optional context. It lacks details on error handling or output specifics, but the output schema likely provides that, making it complete enough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It mentions 'Optional context values can be injected to customize its behavior,' which aligns with `poml_context`, and implies `poml_path` is the location. This is a minimal addition and does not clarify the structure of `poml_context` or expected file paths.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs a predefined capability from a `.poml` file, distinguishing it from other tools like `list_agent_capabilities` and `run_pipeline`. It explains the file's role and the optional context injection, making the purpose specific and actionable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs to discover capabilities with `list_agent_capabilities`, which provides clear when-to-use guidance. However, it does not specify when not to use this tool or mention alternatives like `run_pipeline`, leaving some room for ambiguity in choice of tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_data_sourceA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, idempotent, and non-destructive behavior. The description adds transparency by explicitly stating what is included (summary, metadata, topics) and excluded (messages, detailed definitions), which helps set expectations beyond the schema. It does not add unnecessary detail, and no contradictions with annotations exist.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise: two sentences, front-loaded with the core purpose, and then a clear list of inclusions and exclusions. Every sentence earns its place with no redundant or tangential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the existence of an output schema and the moderate complexity (one required parameter, optional args), the description covers the key aspects: what it does, what it returns, and what it excludes. It could be slightly more complete by explaining the purpose of 'args' or providing an example, but it is sufficient to differentiate from siblings and understand basic usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, and the tool description does not explain the meaning of the 'path' parameter or the optional 'args'. It only implies that 'path' identifies a data source, which is inferred from the tool name. The description fails to provide adequate semantics for parameters, leaving the agent to guess about path format or additional arguments.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Summarize a data source without returning its messages.' It specifies the resource (data source) and the action (summarize), and distinguishes it from sibling tools like query_messages and describe_topic by explicitly excluding messages and detailed topic definitions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about what the tool returns and excludes, implying when to use it (when you need a high-level summary rather than messages). However, it does not explicitly name alternative tools or provide explicit 'when not to use' guidance, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_topicA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint, so safety is covered. The description adds useful behavior details: the output includes a short summary, DuckDB schema, IDL definition, and SQL guidelines, while excluding actual data. This enriches the agent's understanding without contradicting annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the action and immediately followed by a concise list of includes/excludes. Every word contributes value; no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is relatively simple (read-only metadata fetch) and the output schema exists. The description covers what the output contains (summary, schema, IDL, SQL guidelines) and what it excludes (actual data). The only gap is parameter semantics, but overall the description is complete enough for an agent to invoke correctly with reasonable assumptions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain any of the parameters (path, topic, args). The names are somewhat self-explanatory (path likely refers to data source path, topic to topic name), but the description provides no explicit guidance on their format, constraints, or relationships. The agent must rely solely on schema field names, which is insufficient for a 0% coverage situation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Generate a structured summary of a topic' with a specific verb and resource, and explicitly lists inclusions and exclusions. It distinguishes from siblings like query_messages (which returns messages) and describe_data_source (which describes the entire source, not a topic).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clarifies it does not return messages, implying it's for metadata/schema when querying topics. However, it does not explicitly name alternative tools for when to use them or provide positive guidance on when to choose this tool over peers like describe_data_source or query_messages. The context is clear but lacks explicit alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• export_for_plotjugglerA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide idempotency and non-destructiveness, and the description adds meaningful behavioral detail beyond them: the session is a flattened CSV with one scalar column per signal, the layout has curves pre-added, and opening the returned command shows the event pre-zoomed. No annotation contradiction is present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tightly packed sentences front-load the core export behavior and then give workflow guidance. Every clause adds information, with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers input window, output artifacts, return behavior, and intended human-review workflow, and an output schema exists for return values. The main gap is the meaning of path and name, but overall it is sufficient for choosing and invoking the tool in its primary context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate, but it only clarifies topics, time window, and signals. Critical required parameters like path are not explained, and optional args/name semantics are left entirely implicit. Partial compensation for 7 total parameters is insufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action: 'Export a time window of topic data as a PlotJuggler session,' and gives concrete output details (flattened CSV, layout file). It clearly distinguishes this from sibling export tools by naming the PlotJuggler format and the pre-framed/zoomed behavior.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'Use after preview_pipeline to hand an event to a human for visual inspection,' which gives clear workflow context. It does not enumerate when to prefer this over sibling exporters, but the intended post-preview inspection use case is strong enough guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• save_pipelineA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With annotations already declaring idempotentHint=true and destructiveHint=false, the description adds value by specifying the output (YAML file) and that it returns the file path. It does not contradict annotations and provides useful detail about the side effect of writing a file.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no redundancy. The first sentence immediately states the core action and purpose, and the second adds the return value. All words are necessary.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple nature of a save operation with an output schema already present, the description adequately covers the main aspects. It mentions the return path, which is the key output. The directory parameter is left to schema defaults, and the config object is self-explanatory as a pipeline configuration. No critical information is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate by explaining parameters. It only mentions 'pipeline configuration' and implicitly 'name' and 'directory' through the tool name, but provides no clarification of the config object structure, name requirements, or directory default behavior. The description adds minimal meaning beyond what the schema alone shows.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Persist a pipeline configuration to a YAML file'. It is specific (action: persist, resource: pipeline configuration, format: YAML) and distinguishes itself from sibling tools like export_for_plotjuggler or run_pipeline by focusing on saving the config for later reuse.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides context on when to use it: 'so it can be reused, edited, or run later with `run.py`'. This clarifies the intended workflow but does not explicitly mention alternatives or when not to use the tool. The contrast with run_pipeline (immediate execution) is implied but not stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• snap_hardwareA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  All annotations are false, so the description carries the full burden, and it performs well: it discloses auto-detection behavior, the side-effect of writing a WaffleForm, the dependency footprint, and the post-condition of data-source queryability. Could be stronger with failure modes (e.g., what happens if no robot is available, whether the directory is created). Not a contradiction, just an opportunity for more.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three tightly-scoped sentences: purpose, prerequisite/installation context, and side-effect/composition note. There is zero filler, and the most important information (what it does) is front-loaded. Every sentence adds distinct value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For the tool's complexity (1 optional param, no nested objects, output schema present), the description covers the essentials: core behavior, setup prerequisite, and downstream consumption model. Gaps include the role of the `directory` parameter and what happens on failure, but for a tool of this size these are minor. The description respects the line of what structured fields already convey and adds meaningful orchestration context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the burden falls on the description to explain the `directory` parameter, but it's never mentioned. The schema itself only gives a name and default ('.'), so an agent must guess whether it's the output destination, the robot's config directory, or a scan root. Given the description does zero compensation for its single parameter, a 2 is appropriate here.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb-resource pair ('Auto-detect the robot's current hardware, firmware, and software') and clearly states the output ('return the resulting hardware state' and 'writes a WaffleForm'). It clearly distinguishes this from siblings like run_pipeline or query_messages by establishing a unique outcome (queryable data source) and the experimental beta caveat in the title adds useful maturity context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Discloses a hard prerequisite ('Requires the waffle CLI on PATH (cargo install waffle-iron)') and implies when it's useful by noting the output is 'immediately queryable as a data source.' It stops short of explicitly naming alternatives or excluding contexts (e.g., 'don't use for X, use save_pipeline instead'), so it loses a point here, but the practical when-to-use context is well covered.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• subscribe_live_topicsA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare destructiveHint=true and readOnlyHint=false, so the description doesn't need to restate those. It adds useful behavioral context: messages are written to a local sink directory, usable later via `path` in SourceFactory, and optionally creates a standing pipeline. This goes beyond the annotations without contradicting them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the core purpose ('use this tool to connect to a live data stream and subscribe to one or more topics'), then adds concise details on the sink and optional pipeline. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (7 params, no enums) and the presence of an output schema, the description covers the essential aspects: purpose, side effects, and optional configuration. It omits details on some parameters (host, port, type_, etc.) which the schema alone doesn't explain, but these are likely less central. The description is reasonably complete for an agent to know when and how to use it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description carries the full burden. It mentions 'one or more topics' (topics parameter) and 'pipeline config' (pipeline parameter), but does not explain host, port, type_, args, or overwrite. With 7 parameters and only 2 partially described, the description insufficiently compensates for the total lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'connect to a live data stream and subscribe to one or more topics.' It specifies the verb (connect/subscribe) and the resource (live data stream topics), and differentiates from siblings like 'run_pipeline_batch' and 'query_messages' by focusing on live streaming and subscription.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description opens with 'Use this tool to connect to a live data stream,' providing clear context for when to use it. It also explains that the sink directory can serve as input for other tools via SourceFactory, and mentions the optional pipeline for standing pipelines. However, it does not explicitly state when not to use it or name alternatives, though the context strongly implies it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• export_for_lichtblickA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already include idempotentHint: true, readOnlyHint: false, destructiveHint: false, openWorldHint: false. The description adds that it exports to a file path, implying a creation action but not destructive. It mentions the resulting file is an MCAP with JSON-encoded channels, which is useful. However, it does not disclose potential size limits, permission requirements for writing to the path, or whether the layout is fully customizable beyond presets. With annotations covering safety aspects, the description adds moderate value, so a 3 is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, each adding value. It front-loads the main purpose, then explains the output format and compatibility, and ends with a usage directive. No fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is an output schema (not shown but indicated), so return values need not be described. The tool is moderately complex with 7 parameters, but the description covers the core workflow (preview_pipeline then export) and the output characteristics. It lacks some parameter details (units, signal vs topic), but given the output schema exists, the description is mostly complete for the agent to understand what the tool does and when to use it. A 4 is warranted.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It mentions the output includes 'plot series and time/value ranges pre-set', which hints at the 'signals' parameter (selecting which signals to include) but does not clarify the difference between 'topics' and 'signals', nor the units of start_seconds/end_seconds. It also does not explain 'name', 'path', or 'args'. With 7 parameters and 0% coverage, the description only partially compensates, hence a 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool exports a time window of topic data as a Lichtblick session, producing an MCAP file with JSON-encoded channels and a layout with pre-set plot series and time/value ranges. It names the verb 'export', the resource 'time window of topic data', and the output format, distinguishing it from sibling export tools (export_for_plotjuggler, export_for_rerun, export_for_lerobot) by specifying Lichtblick/Foxglove.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use after preview_pipeline to hand an event to a human.' This provides clear when-to-use guidance, and the sibling list includes preview_pipeline, making the sequential relationship explicit. It also indicates the tool works in both Lichtblick and Foxglove, which are open-source and share the layout format, helping the agent choose this over other export targets.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_pipelineA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint=false, destructiveHint=false, idempotentHint=false. The description adds that it produces artifact paths, implying write operations, which is useful beyond annotations. However, it doesn't disclose other side effects, rate limits, or error behaviors, so it's not exhaustive but adds some value.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose, and the second sentence provides concise actionable guidance. Every word contributes without fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is an output schema (not shown) and a single config param, but the description doesn't explain what constitutes a valid config, prerequisites, or error handling. It mentions artifact paths but not other return details (covered by output schema). For a complex pipeline execution tool, the description leaves ambiguity around configuration structure and side effects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter 'config' is an untyped object with additionalProperties true and zero description in the schema (0% coverage). The description merely says 'from a configuration' without detailing required fields, structure, or examples, failing to compensate for the lack of schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool builds and runs a pipeline from a configuration and returns produced artifact paths. This is a specific verb+resource+outcome, and it distinguishes itself from preview_pipeline by explicitly advising to use preview first.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance: prefer preview_pipeline first for event-driven reductions to audit effects before writes. This clearly situates run_pipeline as the execution step after preview, providing direct usage direction and an alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_pipeline_batchA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds meaningful behavior beyond annotations: sources are processed independently, a failure on one source is reported but does not stop the batch, and results include per-source output plus a summary. It does not detail side effects, but annotations already provide the main read/write/destructive hints and the output schema is present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four short, purposeful sentences cover the main action, input style, failure behavior, and preview guidance. There is no filler or redundant repetition of schema/annotations, and important details are front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema and annotations already cover return shape and safety hints, the description covers the essential usage context: one config, many paths, isolation of failures, batch summary, and a preview example. The main gap is that the config parameter remains opaque, but this is a single toolbox-dependent object rather than a fully documented structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds real meaning to 'paths' via explicit paths or glob patterns like 'logs/*', which the schema does not provide. However, the 'config' parameter is only described as 'one pipeline configuration' with no indication of its fields, expected structure, or constraints, and the schema has 0% description coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific action: 'Run one pipeline configuration against many data sources.' It further clarifies scope with 'explicit paths or glob patterns like logs/*', which distinguishes it from the sibling single-source tool run_pipeline.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly implies batch usage over many data sources and that sources are independent. It also gives a practical alternative/precondition: 'For an event reduction, preview a representative source first,' suggesting preview_pipeline. It does not explicitly say when to choose run_pipeline over run_pipeline_batch for a single source, but the distinction is strongly implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• preview_pipelineA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (readOnlyHint, idempotentHint, destructiveHint), the description details the operational behavior: 'Detects the rising-edge events where a SQL predicate becomes true on a topic, builds pre/post windows around them, merges overlaps, and reports how much data would be kept.' This substantially enriches understanding of what the tool does during the dry-run.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with the most critical information ('Dry-run... WITHOUT writing any files'), and each sentence adds meaningful detail—purpose, mechanism, and usage. There is no redundancy or padding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, behavior, and usage, which is strong for a moderately complex tool with an output schema. However, the lack of explanation for `path` and `args` creates a small completeness gap, especially given the schema carries no descriptions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description provides contextual meaning for key parameters: predicate ('SQL predicate'), event_topic ('on a topic'), pre_seconds/post_seconds ('pre/post windows'), and debounce_seconds ('merges overlaps'). However, it does not explain `path` (a required parameter) or `args` at all, and with 0% schema description coverage, this gap leaves the agent guessing about those parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('Dry-run') and resource ('event-windowed reduction'), and explicitly states the key scope: 'WITHOUT writing any files.' This distinguishes it from sibling tools like run_pipeline and run_pipeline_batch, which presumably execute rather than preview.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'Use this to audit a reduce/snippet pipeline before running it.' This implies the alternative (running the pipeline) and the context (auditing to verify data reduction behavior) clearly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_agent_capabilitiesA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the description doesn't need to repeat safety. However, it adds behavioral context by explaining the tool's role in discovery ('instead of guessing file paths') and provides details about output structure, which complements the annotations. No contradictions found.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the core purpose, and every clause serves a purpose: it states the subject (predefined POML capabilities), the output fields, the connection to `run_poml_capability`, and the concrete use case. Zero waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that the tool takes no parameters, has a rich output schema (implied), and has strong annotations, the description is complete. It covers discovery, output structure, and usage guidance, making it fully self-sufficient for an agent to know when and how to use it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters, and the schema description coverage is 100% (trivially). The description adds value by stating that each entry contains `name`, `path`, and `summary`, which prepares the agent for the output without needing to inspect the output schema. This is more than a baseline, justifying a score above 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists predefined POML capabilities, with specific details about output fields (`name`, `path`, `summary`). It distinguishes itself from sibling `list_pipeline_capabilities` by explicitly focusing on POML capabilities and mentions `run_poml_capability` as a related tool, making its purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes explicit guidance on when to use the tool: 'Use this to discover available capabilities instead of guessing file paths.' This tells the agent the intended use case, and the mention of `run_poml_capability` hints at a follow-up action, providing clear context for when to invoke this tool versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.