Server Quality Checklist

Tool Scores

• reproduce_runC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description states it will 'rerun' and 'compare' but discloses no behavioral details beyond that. For a mutation tool (readOnlyHint=false), it does not explain side effects (e.g., does it overwrite outputs? create logs? require specific permissions?). The minimal description adds little beyond what annotations already provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, which is concise, but it lacks structure and does not front-load critical information. It is not overly wordy, but it fails to earn its place by providing sufficient detail. A score of 3 reflects it is adequate in length but not optimally informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 4 parameters, no output schema excerpt shown, and 17 sibling tools, the description is incomplete. It does not explain the comparison format, what 'result JSON hashes' means, or any preconditions (e.g., the previous run must exist). The agent would need additional context to use this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 4 parameters but the description adds no meaning to any of them. Schema description coverage is 0% (the schema itself has some descriptions for config_path and project_root, but the tool description provides none). The agent must rely entirely on the schema, which has gaps (e.g., run_id has no description).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: rerun a previous script and compare result JSON hashes. It uses specific verbs ('rerun', 'compare') and identifies the resource ('previous saved input script'). However, it does not differentiate from sibling tools like run_did or run_iv, which are more specific analyses; the differentiation relies on the tool name rather than the description.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for reproducing previous runs but provides no explicit guidance on when to use this tool versus creating a new run via sibling tools (e.g., run_did, run_regression_fixest). There are no when-not statements or alternative recommendations, leaving the agent to infer context from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_modelsummaryC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  All annotations are false, so description carries full burden but only mentions 'create' without detailing side effects (e.g., file creation, overwrite behavior) or permissions. Adds minimal behavioral context beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, no wasted words. However, it is somewhat under-specified for the complexity of the tool, preventing a perfect score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (multiple input types, output tables), the description lacks details about input formats, output structure, and behavior with existing files. Output schema exists but is not shown, so completeness is low.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description does not explain any of the 7 parameters. Context signals indicate 0% schema coverage, so the description should compensate, but it fails to add meaning beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool creates publication-style model tables from saved model objects or result JSON files. This is a specific verb+resource and differentiates it from sibling tools that run analyses or manage projects.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives. The description only states what it does, without mentioning when-not or providing context for choosing over other tools like run_regression_fixest.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_artifactsC
  2.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, idempotent, non-destructive. Description adds pagination behavior, but omits details like default limit/offset or filtering via run_id.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The single-sentence description is front-loaded but overly sparse for a tool with multiple parameters and pagination, missing critical details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema, the description does not cover the filter parameters (run_id, config_path) or pagination specifics (limit/offset), leaving the agent underinformed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at 0% per context, the tool description fails to explain any parameters (limit, offset, run_id, config_path), relying solely on schema which has limited descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it lists run artifacts under a project with pagination, specifying the verb 'List' and resource 'run artifacts' with scope. It distinguishes from the sibling 'read_artifact' which reads a single artifact.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives like 'read_artifact'. Pagination is mentioned but not contextualized.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_didC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are all false, so the description carries the full burden of disclosing behavior. It only says 'Run DiD workflows' without indicating side effects, state changes, or resource requirements. No mention of auth needs, rate limits, or what gets created/modified. This is insufficient for a tool that likely produces outputs or modifies state.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, making it concise. However, it is overly brief given the tool's complexity, sacrificing essential details. It front-loads the verb but does not earn its place with substantive content beyond the obvious.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex tool with a nested input schema, many required parameters, and no parameter descriptions, a single sentence is grossly inadequate. The description does not explain the purpose of parameters, output (though output schema exists), or how it relates to sibling tools. It misses critical context for correct usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the description must add meaning to the parameters. However, it does not mention any parameter names, required fields, or their meanings. The description only provides high-level tool purpose, leaving the user with no guidance on how to fill in the many parameters (project_root, data_path, outcome, etc.).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs DiD workflows, specifically TWFE event studies and Callaway-Sant'Anna estimates. It uses a specific verb and resource, and distinguishes from sibling tools like run_iv or run_regression_fixest which handle other econometric methods.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for difference-in-differences tasks but does not provide explicit guidance on when to use this tool versus alternatives like run_regression_fixest. It mentions the two methods available via the method parameter, but lacks when-to-use or when-not-to-use instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_ivC
  2.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide no hints (all false), so description must convey behavior. It mentions first-stage output and diagnostics but does not disclose potential errors, limitations, or default behavior when diagnostics are unavailable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, no extraneous words, but it omits critical information about parameters and output, making it overly terse for a complex tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of IV regression (many parameters, nested input, output schema exists), the description is too minimal. It does not summarize inputs or outputs, nor does it explain the tool's role among many sibling regression tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool description contains zero parameter descriptions; schema has some descriptions for config_path and project_root, but the description adds no value beyond the schema. With 0% schema coverage, the description fails to explain any parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Run an IV regression' with specific outputs (first-stage output and weak-instrument diagnostics). It distinguishes from sibling tools like run_did, run_panel_diagnostics, etc., which are different econometric methods.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives, no prerequisites, and no exclusion criteria mentioned. The description is purely operational.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_regression_fixestC
  2.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are present (readOnlyHint=false, destructiveHint=false) but the description adds that outputs are saved, which implies side effects. However, it does not clarify whether existing files are overwritten, where outputs are stored, or any other behavioral details. The added value beyond annotations is limited.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence with no wasted words. It is front-loaded with the primary action. However, it is arguably too terse and could benefit from additional context without sacrificing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the existence of many sibling regression tools and the presence of an output schema, the description is insufficient. It does not explain what fixest is, when to choose it, or what the output schema contains. The agent lacks enough information to use this tool effectively in context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has many parameters (like formula, fixed_effects, cluster_vars) but schema description coverage is reported as 0%. The tool description does not explain any parameter meanings or usage beyond the generic action, so it adds little meaning for parameter selection.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Run a fixest regression') and the outputs saved (coefficient tables, logs, etc.), providing a specific verb and resource. However, it does not distinguish this tool from sibling regression tools like run_did or run_iv, which is a gap for an AI agent deciding which tool to use.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It lacks any mention of context, prerequisites, or exclusions, leaving the agent without decision-making support.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_rdrobustB
  3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds some behavioral context by listing saved outputs (bandwidths, robust estimates, plot, density check). However, it does not disclose whether the tool modifies existing files, requires authentication, or has other side effects. Annotations are all false, so no contradiction, but the description carries the burden of transparency and is only partially informative.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise (one sentence), but it lacks structure and important details. While brevity is good, it sacrifices completeness. The information is front-loaded but not efficiently organized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (many parameters, output schema exists) and the absence of schema descriptions, the description should at least list key parameters or expected outputs. It mentions saved outputs but does not explain input requirements or the output schema. It is adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning no parameter descriptions exist in the schema. The tool description does not explain any of the parameters (e.g., p, cutoff, kernel, covariates). The agent must rely solely on parameter names, which may be insufficient for correct invocation. The description provides zero parameter-level guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Run rdrobust' which clearly identifies the action and resource. It also mentions what outputs are saved (bandwidths, robust estimates, RD plot, density check), making the purpose clear. However, it does not explicitly differentiate from sibling tools like run_did or run_iv, but the context of regression discontinuity is implied.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for regression discontinuity analysis, but it does not provide explicit guidance on when to use this tool versus alternatives (e.g., when to choose rdrobust over run_iv or run_regression_fixest). No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_time_series_diagnosticsB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide no safety hints (all false), and the description only adds 'where available', failing to disclose behavioral traits like side effects, permissions, or output format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that covers the core functionality, though it lacks structure like bullet points or sections for readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema and moderate complexity (many parameters), the description provides no context on prerequisites (e.g., project root, data format) or limitations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% (or low) and the description adds no information about the many parameters (e.g., data_path, time_column, value_columns), leaving the agent to infer from names alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs time series diagnostics including plots, ACF/PACF, unit-root checks, and structural-break tests, which distinguishes it from sibling tools like run_panel_diagnostics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for time series diagnostics but offers no explicit guidance on when to use this tool versus alternatives (e.g., run_panel_diagnostics) or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_panel_diagnosticsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are all false, placing full burden on the description. The description does not disclose behavioral traits such as whether the tool modifies data, reads only, or has side effects. It only lists what it checks without describing behavior or outputs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of 13 words, making it extremely concise. It is front-loaded with the key action verb ('Check') followed by the specific items, with no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 7 parameters and no parameter descriptions in the description, it lacks completeness. The output schema exists but is not described, leaving the agent uninformed about return values or how to interpret the diagnostic checks.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% according to context, and the description adds no meaning to the parameters. It does not explain what 'unit', 'time', or 'data_path' represent, leaving the agent without guidance beyond the schema's field names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific checks: panel balance, duplicate unit-time keys, missing periods, and treatment timing. It distinguishes from sibling tools like run_time_series_diagnostics by focusing on panel data diagnostics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, nor does it mention prerequisites or exclusions. Context signals show sibling tools like run_time_series_diagnostics, but no explicit comparison is made.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• init_projectB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations (readOnlyHint=false, destructiveHint=false) are minimal, and the description adds little beyond stating that it creates a layout. It does not clarify behavior for edge cases (e.g., if the project_root already exists), nor does it disclose side effects or required permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that is front-loaded with the key action and resource. It is efficient but could be slightly more comprehensive without adding significant length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema and sibling tools, the description is adequate but minimal. It does not explain what constitutes a 'reproducible project folder layout' or how the allowed workspace root is determined, leaving some gaps for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema includes descriptions for all parameters (project_root, init_renv, config_path), so the schema itself provides good coverage. The description does not add any additional meaning or context beyond the schema, making it neutral.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool creates a reproducible project folder layout, specifying the action (create) and resource (project folder layout) with a constraint (inside allowed workspace root). This distinguishes it from sibling tools like run_regression_fixest or create_modelsummary, which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, nor does it mention prerequisites or exclusions. It simply states the action without contextual direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_datasetB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide no hints (all false), and the description gives minimal behavioral info: it inspects and returns summaries. No mention of read-only nature, side effects, required permissions, or limitations. With no annotation support, the description should disclose more.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One sentence, front-loaded with the main action. Efficient, but could benefit from a bit more detail on parameters or behavior without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Output schema exists, so return format is covered. However, with 6 parameters and no parameter descriptions in schema or description, the tool is underspecified for an agent to use correctly. The description is too brief for the complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% (only project_root and config_path have descriptions in schema). The description adds no parameter details, leaving 6 parameters (including id_columns, time_column, timeout_seconds) without guidance. It only hints that data_path should refer to one of the listed file types.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the verb 'inspect' and the resource types (CSV, TSV, RDS, Parquet, Stata), and mentions returning machine-readable summaries. This clearly distinguishes it from sibling tools that run analyses or manage projects.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when inspecting data files, but provides no explicit when-to-use guidance, alternatives, or exclusions. It's left to the agent to infer from context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_artifactB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false. The description adds that it reads without executing code, which is consistent but does not disclose additional behavioral traits like permissions or size limits. No contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of 14 words, front-loaded with the key action. It is concise but sacrifices completeness; however, it earns its place by being direct.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema, the description lacks completeness for a tool with 4 parameters and no schema coverage. It provides no parameter guidance or usage context, making it inadequate for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description provides no information about parameters (max_bytes, config_path, project_root, artifact_path). The agent cannot infer parameter meaning or usage from the description alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Read' and the resource 'artifact', listing specific file types (table, log, JSON, Markdown, HTML, TeX). It distinguishes from sibling tools that execute code by explicitly noting 'without executing code'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for reading artifacts without execution, but lacks explicit when-to-use or when-not-to-use guidance compared to alternatives like run_* or render_quarto. No exclusions or alternative tool names are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• render_quartoB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide no hints about side effects or safety. The description does not disclose where output files are saved, whether it overwrites existing files, or the impact of the operation. As a rendering tool, it likely produces new files, but this is not stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One sentence with all key elements (verb, object, output formats). No extraneous text. Could be slightly expanded for clarity on usage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 5 parameters and an output schema, the description is minimal. It does not explain the purpose of each parameter or the return structure, but the output schema exists. Overall adequate but not comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool description adds no parameter details beyond the schema's inline descriptions. Since schema coverage is 0%, the description should compensate, but it only mentions the output format implicitly. Baseline 3 due to schema descriptions being present.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Render', the resource 'Quarto report', and the output formats (HTML, PDF, DOCX). It distinguishes from sibling tools that focus on other tasks like running regressions or creating summaries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus siblings like run_r_script or reproduce_run. No mention of alternatives or situations where rendering is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_r_scriptB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are minimal (all false), placing the burden on the description. It discloses headless execution and output saving but omits side effects, permissions needed, or safety implications beyond the parameter 'allow_dangerous_code' which is not elaborated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no wasted words. It efficiently conveys the core purpose, though a bit more structure could aid scanning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has multiple parameters and an output schema, the description is adequate but basic. It does not explain the safety flow (allow_dangerous_code) or the meaning of 'headlessly' in the project context. Somewhat complete but leaves gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides detailed descriptions for most parameters (coverage >80%), so the description adds limited extra meaning—such as the mention of logs, session info, and result JSON. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action 'Run', the resource 'project R script', the mode 'headlessly', and outputs saved (logs, session info, result JSON). It effectively distinguishes from sibling econometric-specific tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives (e.g., run_did, run_regression_fixest). There are no context cues or exclusion criteria, leaving the agent to infer usage without support.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lookup_econometrics_guidanceB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate a safe read-only operation. The description adds that it searches a corpus but doesn't clarify behavior when both query and method are provided or what 'curated' means. Consistent with annotations, no contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence that front-loads the purpose. Concise and efficient, though slightly more detail (e.g., relationship between parameters) could fit without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With annotations covering safety and an output schema present, the description is adequate but incomplete. It doesn't explain what to expect in the output or how parameters interact (e.g., precedence between query and method). Leaves some questions for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already has descriptions for all three parameters. The description adds value by framing 'method tag' and 'local corpus search', but omits the optional 'corpus_root' parameter. Example tags are repeated from schema, so no new meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns 'curated econometric guidance' by method tag or search, distinguishing it from sibling tools that run regressions. However, 'curated econometric guidance' is somewhat vague; specifying 'pre-written advice or notes' would improve clarity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies using a method tag or search query, but does not explicitly state when to use this tool vs. siblings (e.g., run_did for running regressions instead of guidance). No when-not or alternatives are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• install_r_packagesA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds the key behavioral detail that only allowlisted packages are installed, which is beyond the annotations. However, it does not disclose side effects (e.g., overwriting existing packages, dependency installation) or error handling. The annotations already indicate it is not read-only (readOnlyHint=false), so no contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that is front-loaded with the core action and constraint. Every word is meaningful with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that installs packages, the description omits important context such as behavior when a package is not allowlisted, dependency resolution, or response format details (though output schema exists). It is minimally adequate but could be more complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has high description coverage (all parameters have descriptions in the schema), so the tool description does not need to repeat them. The description adds no new parameter information beyond the schema, so a baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action (install), the resource (allowlisted R packages), and the distinguishing feature (only allowlisted, returns structured results). This effectively differentiates it from sibling tools like run_r_script.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives (e.g., run_r_script). It does not mention prerequisites, scenarios where installation fails, or when to prefer other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• health_checkA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate it is read-only, idempotent, and non-destructive. The description adds value by specifying exactly what is checked (R, Rscript, Quarto, packages). No contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with 'Check', no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that output schema exists, the description covers the essential action. However, it lacks guidance on when to use the optional parameters.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% per context, but the schema itself has descriptions for each parameter. The tool description does not explain the optional parameters (config_path, project_root) or how they affect the check.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it checks availability of R, Rscript, Quarto, and R packages. This is a specific verb+resource and distinguishes from sibling tools like install_r_packages or run_r_script.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit when-to-use or when-not-to-use. It implies usage before running R scripts or installing packages, but does not state alternatives or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.