Server Quality Checklist

Tool Scores

• get_schemaA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries the burden. It states the tool is non-destructive ('Does not require loading') and returns a sql_hint with JOIN guidance. It does not disclose failure modes or exact return format, but is reasonably transparent for a schema inspection tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise, front-loaded sentences with no wasted words. Every sentence adds value: purpose, non-destructive nature, usage guidance, and return value description.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers key aspects: what it does, that it doesn't load, usage, and return. With an output schema present (though not shown), the description does not need to detail return values. Could mention prerequisites (e.g., recording must exist), but overall complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%; the description does not explain the parameters 'file' and 'topic' beyond the tool's general purpose. It mentions 'topic names' but does not clarify how the topic parameter is used. Minimal added value over the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool inspects SQL schema for a recording, listing topics, tables, columns, and types. The verb 'Inspect' and specific details distinguish it from sibling tools like load_recording and query.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this to plan SQL queries before running them,' giving clear guidance on when to use. It does not provide explicit when-not-to-use or alternatives, but sibling tools provide context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_recordingsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  States 'Does not require loading,' implying safe, non-destructive operation. With no annotations, this is valuable context. Describes return type (file information) but could mention permissions or scalability (e.g., large directories).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each serving a purpose: purpose+output, when to use, path parameter detail. No fluff, front-loaded with key information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Output schema exists but is not shown; description provides a high-level overview and one parameter detail. Missing explanations for two parameters (after, before) that are not in schema descriptions. Adequate for basic understanding but incomplete for full use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description must compensate. Only 'path' parameter is explained (absolute directory). 'after' and 'before' parameters are not described at all; their purpose and format remain unclear. This is a significant gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it discovers MCAP recording files, lists their metadata (names, sizes, durations, etc.), and does not require loading. It distinguishes from sibling tools like load_recording by explicitly noting it's a discovery step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises to 'Use this first to see what data is available before loading.' Provides context on default project directory scanning and flexibility via absolute path. Lacks explicit exclusions but sufficient for typical use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_recording_infoA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the full burden. It discloses that the tool returns full metadata, channel details, and attachments, and states it is a read-only operation ('Does not require loading'). This is sufficient for a simple one-parameter tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no unnecessary words. It front-loads the purpose and follows with usage guidance, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low complexity (one required parameter, no output schema shown but exists), the description covers what the tool does, what it returns, and when to use it. It misses potential details like error handling or prerequisites, but it is largely complete for the given context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter 'file' has 0% schema coverage. The description clarifies it is 'a specific MCAP recording file' but does not specify format or constraints. It adds some meaning but could be more precise.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and the resource ('full metadata, channel details, and attachment list for a specific MCAP recording file'). It differentiates from siblings by specifying that it does not require loading, which contrasts with 'load_recording'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by stating 'Does not require loading' and 'Use this for detailed inspection before loading data,' implying when to use it (inspection) and when not to (loading). It does not explicitly name alternatives but effectively guides the agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_versionA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It indicates the tool returns information but does not disclose any behavioral traits like authentication requirements or rate limits. However, the simplicity of the tool (no parameters, read-only) partially compensates.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the output and use case. Every word is valuable and there is no wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and an output schema exists, the description sufficiently covers the purpose and usage. It does not elaborate on response structure, but the output schema provides that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so baseline is 4. The description adds no parameter-specific information since none exist.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool returns 'the server version, supported encodings, and upgrade command', which is a specific verb and resource. It distinguishes itself from sibling tools like get_recording_info or get_schema by its unique output.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this to check for updates or diagnose compatibility issues', providing clear usage context. It does not explicitly mention when not to use it, but the sibling tools cover different use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• load_recordingA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes that decoding all messages may take seconds to tens of seconds, and that filtering reduces load time and memory usage. Since no annotations provided, the description adequately covers behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single paragraph with clear, front-loaded main purpose. Each sentence adds value, though could be slightly more structured (e.g., bullet points). No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 6 parameters and an output schema, the description covers the load process, prerequisites, performance considerations, and usage of optional parameters. Sufficient for a complex tool without needing to explain return values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema coverage, the description explains the purpose of 'file', 'topics', 'start_time', 'end_time', and 'alias', covering most parameters. Only 'downsample' is not described. Adds significant value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States specifically that it decodes an MCAP file and loads into DuckDB for SQL querying. Clearly distinguishes from sibling tools like 'query' and 'list_recordings'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'You must call this before running queries' and provides tips for large files (use topics, start_time/end_time to filter) and setting alias for multi-recording comparison, effectively guiding when and how to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• queryA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but the description details supported DuckDB SQL features (JOINs, window functions) and prerequisites. It does not mention error handling or performance impacts, but is fairly transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise at a few sentences with main purpose front-loaded. Every sentence adds value without verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given an output schema exists (not needed to explain return values), the description covers prerequisites, capabilities, and usage context thoroughly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% (no parameter descriptions in schema). The description explains the sql parameter well ('Supports full DuckDB SQL'), but the limit parameter receives no additional explanation beyond its existence.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Execute a SQL query against loaded MCAP data', specifying the verb and resource. It distinguishes from siblings like load_recording and list_recordings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states data must be loaded first via load_recording, and instructs to call load_recording if a table is missing. Provides context on supported SQL features.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.