Server Quality Checklist

Tool Scores

• classify_effect_modificationA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations available, the description carries the full burden. It discloses the five classification categories and their meanings, which is valuable behavioral context. However, it does not state what the function actually returns (e.g., a map, list, or report), nor does it mention whether it is read-only, what happens if the DAG lacks modifiers, or any assumptions about the input structure. 'Outputs are conditional on the encoded structure' is vague.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description begins with a clear one-sentence purpose, followed by a structured bullet list of the five classification types. While it is lengthy, the complexity of the taxonomy justifies the detail. The text is well-organized and front-loaded with the main verb and resource.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, so the description should compensate by explaining return values and behavior. It explains the classification types thoroughly but does not describe the output format, how results are per-modifier, or edge cases like DAGs without modifiers. Given the moderate complexity, the description is adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already describes the 'dag' parameter as 'Canonical DAG with modifiers populated', and coverage is 100%. The description adds context by saying 'For each modifier annotation on the DAG', which clarifies that the input must contain annotations, but this is a minor addition beyond the schema. The description focuses more on output taxonomy than on parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's verb ('classify') and resource ('modifier annotation on the DAG'), and differentiates it from siblings by focusing on effect-modification structure. It specifies the classification framework (VanderWeele & Robins 2007, Weinberg 2007), making the purpose precise and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (when you have a DAG with modifier annotations), but it does not explicitly contrast it with sibling tools like analyze_dag or check_overadjustment. There is no 'use this when' or 'for other analyses see X' guidance, so the agent is left to infer the tool's niche.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_codeA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool returns a dagstudio_url, sanitizes identifiers via an identifier_map (though not in schema), and adheres to paper claims. This goes beyond basic behavior but leaves details like error handling unspecified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with its main purpose but includes a lengthy second paragraph about paper fidelity that, while relevant, could be more concise. The identifier_map reference adds verbosity without clear payoff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has two parameters and no output schema, so the description needs to explain return values. It mentions dagstudio_url but does not specify the overall output structure (e.g., whether code and URL are returned together). The identifier_map ambiguity also leaves gaps in understanding how to use the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides a description for 'dag' and an enum for 'language', but the description adds little clarity. It introduces an 'identifier_map' that is not part of the input schema, potentially confusing agents about tool usage. With 50% schema coverage, the description does not compensate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Emit idiomatic Python (networkx.DiGraph) or R (dagitty DSL) source representing the DAG.' It uses a specific verb and resource, distinguishing it from siblings like parse_dagitty or analyze_dag.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context ('Useful for handing the DAG to a downstream analysis pipeline') and includes a caution about preserving the paper's structural assumptions. However, it does not explicitly mention alternatives or when not to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compute_biasA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It thoroughly discloses the computation approach (analytic path product, two OLS regressions), the outputs (estimates, biases, bias reduction), the internal dependency on check_overadjustment, and the linear Gaussian SEM assumption. This goes beyond the schema and gives the agent a solid mental model, though exact output format is not specified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: it leads with the main method, then explains the use case and internal composition, and closes with assumptions. Every sentence carries essential information, and the length is appropriate for the tool's complexity. It avoids redundancy and remains highly scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with five parameters and no output schema, the description covers the core behavior, the product of the tool (estimates, biases, bias reduction), the link to check_overadjustment, and the underlying model assumptions. It could be more explicit about the exact return structure, but it lists the reported quantities, making it largely complete for an agent to anticipate results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 60%, and the description adds useful context for adjustment_set (e.g., 'adjust for {age, smoking}') and implicitly for coefficients via 'SEM edge coefficients'. However, it does not clarify the dag parameter beyond its role, leaving a gap for the 40% of parameters not covered by schema or description. This is the expected baseline for 60% coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: it numerically demonstrates the bias of a proposed adjustment set by computing the true total effect and comparing crude/adjusted OLS estimates. It distinguishes itself from siblings by mentioning the internal composition of check_overadjustment and focusing on bias quantification.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool (to translate a structural adjustment claim into a numerical demonstration) and highlights that it also surfaces overadjustment flags from check_overadjustment, combining both structural and numerical insights. However, it does not explicitly name alternatives or say when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• parse_dagittyA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does so well. It explicitly discloses that bidirected and undirected edges are skipped, and that the tool does not validate causal correctness. This goes beyond basic operation and informs the agent of significant behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The first paragraph is concise and front-loaded, but the third paragraph becomes a lengthy discourse on causal modeling instructions that is tangential to the tool's parsing function. This weakens overall conciseness; some sentences could be tightened or moved to a different context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a single parameter and no output schema. The description indicates the output is 'the canonical DAG shape' but does not detail the return structure or error behavior for invalid syntax. Given the simplicity of the tool, the description is adequate but leaves gaps around edge cases and output details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, providing a baseline of 3. The description adds value by enumerating accepted input forms: dagitty('dag { ... }'), dagitty("..."), and dagitty::dagitty(...). This clarifies parameter syntax beyond the schema's single example.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb and resource: 'Parse a dagitty DSL string into the canonical DAG shape used by other DAG Studio tools.' This distinguishes it from sibling tools like analyze_dag or validate_engine, which operate on the parsed DAG.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use the tool, noting it accepts various dagitty DSL forms and is a preliminary step for other tools. It also implies limitations ('does not validate') but does not explicitly name alternative tools or exclusion scenarios. This is clear context without formal exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_engineA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool runs specific validation suites (T01–T15, EM01–EM20) and returns pass/fail per case, which is the core behavioral expectation. It does not explicitly state whether it has side effects, but 'run... and return' implies a read-only verification operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main action and return type, then a clear usage sentence. Every clause provides useful information—case IDs, references, intended use—and no words are wasted.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a low-complexity tool with one optional parameter and no output schema, the description adequately specifies the return behavior (pass/fail per case) and the suite contents. It also ties the tool into the larger toolset, making the context complete for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of the single parameter 'suite' with enum values and descriptions. The tool description adds contextual references to the canonical cases but does not materially enhance parameter semantics beyond what the schema already provides, so the baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'Run the canonical validation suite... against the current engine and return pass/fail per case.' It clearly identifies what the tool does and distinguishes it from siblings by naming analyze_dag, check_overadjustment, and classify_effect_modification as tools to verify before relying on.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use it: 'Use this when an agent or reviewer wants to verify the engine is trustworthy before relying on...' It names the specific sibling tools as downstream consumers. It does not provide explicit when-not-to-use guidance, but the context is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_dagA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden and does so thoroughly. It discloses dagitty_string precedence, error fallback behavior, the static concordance attestation, and the epistemic limitation that outputs are 'conditional on the encoded structure.' This goes well beyond a bare action statement, effectively setting expectations for the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than typical but uses a clear structure: a direct result sentence, a clarification of accepted inputs, and a contextual paragraph on use with paper DAGs. It front-loads the core purpose, though the final paragraph is fairly lengthy and could be tighter.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is complex and the description covers input modes, precedence, output blocks, attestation, and limitations, which is robust given the lack of an output schema. The absence of per-parameter details is a minor gap, but overall the description provides a complete picture of what the tool does and when to apply it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers only dagitty_string (20% coverage), and the description compensates by explaining the two input modes and the precedence rule ('only the dagitty_string is used when present'). However, individual fields like edges, nodes, exposure, and outcome are not explained beyond their names, leaving a significant parameter-semantics gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Returns identifiability status and minimal adjustment sets given the DAG provided,' and enumerates concrete outputs (open backdoor paths, adjustment sets, directed paths). This clearly distinguishes the tool from siblings like check_overadjustment or simulate_data by focusing on causal identification from a DAG.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear when-not guidance: it states the tool 'does not verify that the DAG correctly encodes domain knowledge' and advises surfacing ambiguity when a paper lacks an explicit DAG. However, it never names alternative sibling tools or explicitly says 'use this instead of X', so it falls short of the top score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_canonical_exampleA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden and does well by stating the DAG is canonical, validated, and identical to the one used for engine validation, making downstream analyze_dag output predictable. It does not describe error behavior or the exact return format, but as a simple getter it provides meaningful behavioral context beyond just 'returns a DAG.'

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each earning its place: the first states what is returned and the ID families, the second gives use cases, and the third explains the useful property of matching validated expected values. No filler or redundancy; information density is high and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter getter with no output schema and no annotations, the description is largely sufficient: it covers scope, ID format, use cases, and the relationship to analyze_dag. It stops short of specifying the exact return representation (e.g., dagitty string vs. JSON object), which is a minor gap for an agent expecting to consume the DAG.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already describes the id parameter with 100% coverage, including the T/EM ranges, so baseline is 3. The description adds semantic depth by explaining that T IDs correspond to backdoor/adjustment-set structures and EM IDs to effect-modification structures from specific references, and by noting the validation relationship with analyze_dag.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Return') and clearly identifies the resource ('canonical validated DAGs from the engine's test suite') with explicit ID ranges (T01–T15, EM01–EM20). This distinguishes it from sibling tools like analyze_dag or parse_dagitty, which operate on rather than retrieve canonical examples.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear usage context: 'few-shot prompting, regression checks, and teaching.' It also notes that returned DAGs are the ones the engine is validated against, implying use for testing analyze_dag against expected values. No explicit exclusions or alternatives are given, but the intended use cases are clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• simulate_dataA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description burdens itself well by disclosing the full generative process: root variables N(0,1), non-roots with β=0.5 and N(0,0.25) noise, and determinism via seed. It also clearly states the limitation that real data won't match the distribution, which is critical context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tightly structured paragraphs: the first explains the model and parameters, the second gives use cases and limitations. No filler; every sentence earns its place and the core purpose is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers statistical model, determinism, use cases, and a critical validity limitation, making it quite complete for a simulation tool. However, it does not specify the output shape or the expected DAG input format, which an agent might need to invoke successfully.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 80% of parameters, leaving dag without a description. The description compensates by explaining that the DAG defines the structural model and that outputs are consistent with it. It also enriches coefficients and seed semantics beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Generate synthetic data from a Linear Gaussian Structural Equation Model consistent with the DAG,' clearly identifying the verb, output, and model type. This directly distinguishes it from analysis/validation siblings like analyze_dag and check_overadjustment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states it is useful for sensitivity analysis and demonstrating structural claims, with a concrete example. It also warns that outputs are only valid under the linear Gaussian model, implying when not to use it, though it does not explicitly name alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_overadjustmentA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavior by explaining the causal logic behind each failure mode and the assumption that the DAG encodes the true structure. The note that DAG Studio verifies analyses but not the DAG's domain correctness adds important limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with bullet points, each explaining a distinct failure mode with citations. Every sentence contributes to understanding the tool's behavior or limitations, with no unnecessary filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, so the description should logically hint at return values. It implies the output is the set of biased variables, but doesn't explicitly state the return format. However, the behavioral depth and input constraints make it largely complete for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds meaningful context beyond the schema by clarifying that dag and dagitty_string are alternatives and that adjustment_set is the target of evaluation, reinforcing the mutual exclusivity and purpose.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: given a DAG and adjustment set, identify variables whose inclusion biases the estimate. It lists three specific failure modes (descendant_of_exposure, collider, descendant_of_collider), which distinguishes it from sibling tools like compute_bias or analyze_dag.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear input requirements: either a canonical DAG object or dagitty_string, both paired with adjustment_set. It does not explicitly mention when not to use this tool or name alternative tools, but the context of evaluating adjustment sets is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.