Server Quality Checklist

Tool Scores

• session_resetA
  3.6/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description openly discloses that loaded frames and variables are gone, which is a destructive effect. This directly contradicts the annotation destructiveHint=false, which indicates the tool is not destructive. Since the description contradicts structured metadata, it scores 1.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that explains what happens (session kernel emptied) and the consequence (next data_load starts fresh). Every clause contributes meaning, with no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a no-parameter, no-output-schema tool, the description fully explains the purpose and outcome. It could mention that external state is unaffected, but the openWorldHint=false annotation already covers that, and the core behavior is described completely.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and the schema covers 100% of them, so there is no parameter documentation burden. With no parameters, the baseline is 4, and the description appropriately adds no irrelevant parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Empty') and a clear resource ('session kernel'), and explicitly states the effect ('loaded frames and variables are gone'). This clearly distinguishes it from sibling tools like session_state, which would inspect state, and data_load, which would populate it.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool ('the next data_load starts fresh') but does not explicitly state conditions, exclusions, or alternatives. It does not compare itself to sibling tools like session_state or data_load, leaving the decision partly implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chart_exportA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already supply idempotentHint, readOnlyHint=false, and destructiveHint=false, lowering the burden. The description adds helpful behavioral details: output formats, the self-contained nature of HTML (vega js inlined, offline-capable), and the collision-refusal behavior. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the action, and every phrase adds meaning (formats, self-contained HTML, collision behavior). No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a small export tool, the description covers purpose, output types, and a key safety behavior. Still, it leaves gaps: no mention of scale, no explicit relationship between out_path and extensions, and no hint about what happens beyond refusal. Given moderate complexity and no output schema, this is minimally adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate, and it does partially: mentioning formats clarifies image_format, and 'out_path' collision clarifies that parameter. However, it does not mention 'scale' or explicitly map spec_path to the input, leaving some parameter semantics unexplained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Render') and resource ('existing .vl.json') and lists output formats (svg, png, self-contained html), making the tool's primary purpose clear. It also implies differentiation from siblings like chart_spec by emphasizing 'existing', but it does not explicitly name an alternative tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'existing .vl.json' implies the tool should be used after a chart spec exists, which gives some context. However, there is no explicit guidance about when to choose chart_export over sibling tools, nor exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• session_stateA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate a safe read-only operation. The description adds valuable behavioral context beyond annotations: that wall-timeout or session_reset empties the loaded frames but leaves stored_handles, and that loaded frames include frame_handle. This clarifies the state lifecycle.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, dense sentence that front-loads the core purpose and adds essential lifecycle context. No wasted words; every phrase earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description sufficiently covers what the tool returns (loaded frames, variables, stored_handles) and the persistence behavior. It does not describe the exact return format, but for a zero-parameter state inspection tool with no output schema, this is adequate and not incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With zero parameters, the baseline is 4. The description adds no parameter details, but none are needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly explains what the tool returns (kernel state: loaded frames, user variables, stored_handles), but lacks an explicit verb like 'List' or 'Get'. It implicitly distinguishes from siblings by referencing session_reset, which is related.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The context is clear: this tool inspects the current session state. It doesn't explicitly state when to use it versus alternatives, but the purpose strongly implies it. No exclusions are mentioned, but the mention of session_reset provides lifecycle context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• data_loadA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes well beyond the sparse annotations (all false hints). It discloses critical behavioral traits: the kernel starts on first use, a durable frame_handle is returned, the handle is stored in a persistent store so it survives kernel timeouts, and the handle has a TTL (ttl_ms). This is rich, actionable information not available in annotations or schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The first sentence delivers the core purpose concisely, followed by valuable but somewhat lengthy details about the frame_handle. Each sentence adds meaningful information, though the handle explanation could be tightened. Overall, it is well-structured and not wasteful, but slightly more verbose than necessary.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the minimal schema (2 string params) and no output schema, the description thoroughly covers the tool's return values (frame_handle, profile) and handle semantics. It lacks details on file type support and error behavior, but for a data loading tool with these siblings, the description is sufficiently complete for an agent to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% coverage, so the description must compensate. It explains that `name` becomes the session variable name (DATA[name]) and `path` refers to the data file. However, it does not specify file format, path constraints (absolute vs. relative), or other parameter details, leaving some ambiguity for correct invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb and resource: 'Load a data file into the session kernel as DATA[name]'. It also distinguishes from siblings by showing how the loaded data is used downstream with py_run and chart_spec, making its role in the workflow explicit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides strong usage context: 'loaded once, use it many times' and explicitly mentions integration with py_run and chart_spec. It implies this is the initial step before those tools, but does not explicitly compare to sibling tools like data_profile or session_reset. No clear 'when not to use' is stated, but the intended workflow is well conveyed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• data_profileA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint=true, and the description adds valuable behavioral details: sample-based row counting for large CSVs ('sampled at 100k'), and a data privacy guarantee ('nothing leaves it'). These go beyond the annotation, disclosing performance and security behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences front-load the core purpose, then provide workflow and privacy context. Every sentence adds value without redundancy or wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter, read-only tool, the description is remarkably complete: it states the operation, the file types, the exact output contents, a sampling caveat, and the local-only/privacy behavior. No output schema exists, but the description enumerates what will be returned, so the agent knows what to expect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one parameter, path, with 0% description coverage. The description partially compensates by indicating the path refers to a local data file and listing supported formats, but it does not elaborate on path format, valid values, or error conditions. Since the parameter is single and arguably self-explanatory, the compensation is adequate but not thorough.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Describe a data file' with specific formats (csv/tsv/parquet/xlsx/json) and the output content (row count, dtypes, head, numeric summary). It also distinguishes from sibling tools like data_load by explicitly positioning it 'BEFORE plotting' and outlining a workflow: Profile → design the chart → load.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage context: 'BEFORE plotting it' and the workflow sequence 'Profile → design the chart → load', which tells the agent when this tool should be used relative to other steps. It does not explicitly name alternatives or exclusions, but the workflow implies its placement.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chart_specA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (idempotentHint, readOnlyHint false), the description discloses several important behaviors: the render comes back attached to the result, diagnostics include deterministic layout checks, rendering is local with Vega-Lite v6, and handles can chart even after the originating kernel dies. This adds significant value over the structured metadata.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense with every sentence providing actionable information. It's structured logically: purpose, data handling, output behavior, and rendering details. The use of capitalization and dashes improves readability without wasting words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 5 parameters, no output schema, and only three annotations, the description covers the essential behavior and outputs thoroughly. It explains the return includes the render and diagnostics, local rendering specifics, and data persistence semantics, making it adequate for an agent to select and invoke the tool confidently.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates well by explaining data (kernel frame name or handle, max_rows cap), name and out_dir (via output file naming/location), and the spec as a Vega-Lite chart. It doesn't explicitly describe the spec parameter but it's reasonably implied from the tool's purpose. This is strong but not exhaustive.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool writes a Vega-Lite chart as a source document and renders it to PNG in out_dir. It is specific about the action and the generated artifacts, but it doesn't explicitly differentiate from sibling tools like chart_export or viz_recipes, so it falls just short of a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use the artifact scratch (when the user wants to see the chart) and how to handle data (inline vs handle, with a max_rows cap and guidance to aggregate large data first). However, it doesn't explicitly state when not to use this tool or mention alternative tools for different charting scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• viz_recipesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, so no need to restate safety. The description adds behavioral context by explaining that the tool returns a "worked template" ("Templates teach the shape; adapt fields to your data") and that the output varies by topic (Vega-Lite vs Python), going beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, information-dense, and free of fluff. The first sentence front-loads action instructions and topic lists, while the second sentence adds a concise summary of the template philosophy, all without unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-optional-parameter tool, the description is sufficient to use it correctly: it explains the two call modes, lists topics, and indicates output types. It doesn't specify the exact structure of the index or template, but this is a minor gap given the absence of an output schema and the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description fully compensates by enumerating all valid topic values and describing the effect of empty vs non-empty topic. It tells the agent that no topic yields an index and a topic yields a worked template, with explicit topic names and output types, making the parameter's semantics complete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly indicates that viz_recipes provides chart templates ("Engineering chart recipes") and explains the two modes of calling: with no topic for an index, or with a topic for a specific worked template. It lists the available topics and their output formats (Vega-Lite vs Python), distinguishing it from sibling tools like chart_spec and py_run.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit instructions on when to call with or without a topic, and notes the integration with chart_spec and py_run by stating output formats ("Vega-Lite, chart_spec-ready" and "python, via py_run"). However, it doesn't explicitly say when not to use this tool, only implies the alternative contexts through integration points.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• py_runA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are all false, so the description carries the burden. It discloses state persistence ('variables and imports survive'), resource limits (memory rlimit, max 570s), return behavior (frame_handles returned even on errors), and output bounding (stdout/stderr come back bounded). This goes well beyond the structured metadata.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but every sentence adds value—covering purpose, libraries, persistence, resource limits, parameter semantics, and output behavior. It is well-structured into clear paragraphs and uses front-loaded action-oriented language.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no output schema, the description fully explains return values, error behavior, and state management. It provides all needed context for an execution tool, including recovery after timeouts, which is critical in this environment. No gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema coverage, the description fully compensates. It explains frame_handles in detail (how to pass and what it returns), wall_s (resource cap with max), and code (available libraries and usage context). Every parameter is given practical meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'EXECUTES Python in the session kernel' with a clear verb and resource, and lists exact use cases (transform/aggregate before charting, run a python recipe). This distinguishes it from sibling tools like data_load or chart_spec.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context for when to use ('transform/aggregate before charting, or run a python recipe') but does not explicitly mention when not to use or name alternative tools. Still, the guidance is specific and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.