Server Quality Checklist

Tool Scores

• compute_classical_statsA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral burden, and it does so thoroughly: it discloses that the tool REFUSES below 20 examinees and warns below 100, it explains that flags are review triggers not verdicts, and it warns that a negative corrected punit-biserial on ordinary difficulty usually means mis-keying and that this warning must be relayed. It also warns against interpreting raw options as scored data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than average but every sentence earns its place: a one-line purpose, exact output lists, hard operational thresholds, and a critical interpretation warning. It is front-loaded with the summary before the details and contains no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with eight parameters, an output schema, and sibling tools, this description is remarkably complete: it states inputs implicitly, outputs explicitly, sample-size constraints, warning behavior, and how to interpret edge cases like negative discrimination. The existence of an output schema covers return structure, and nothing needed for safe invocation is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all eight parameters in detail. The description adds useful interpretive context (e.g., corrected punit-biserial is the discrimination index to use; Cronbach's alpha equals KR-20 here), but it does not add new parameter-level meaning that the schema already lacks.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific domain and output class: 'Classical item analysis: difficulty, discrimination and form reliability.' It then enumerates exact per-item and per-form statistics, so an agent knows precisely what the tool computes and can distinguish it from siblings like flag_dif or compute_local_dependence.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description strongly implies when to use the tool – when classical item-quality statistics are needed – and even gives interpretive guidance about flags. However, it never names siblings or states explicit exclusion conditions ('use flag_dif for subgroup DIF', 'use describe_dataset first'), so the routing burden falls on the agent's inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• flag_difA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral disclosure burden and does so thoroughly. It explains that purification changes results relative to difR, that the tool refuses under n=50 and warns under n=200, and that DIF flags must not be described as bias. It also instructs use of Benjamini-Hochberg adjusted p-values because screening many items is a multiple-testing situation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but every section earns its place: method rationale, purification behavior, sample-size thresholds, and critical reporting constraints. The bulleted method list and clear warning blocks make the structure easy to scan, and there is no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity, the description is remarkably complete: it explains all three methods, their blind spots, matching and purification, operational constraints, and interpretation obligations. An output schema exists, so the description does not need to specify return fields, and the parameter schema covers the remaining operational details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the parameter semantics baseline is 3. The main description adds useful method-level context, such as purification and matching on number-correct score, but it does not need to explain individual parameters because the schema already documents them in depth.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Screen every item for differential item functioning between subgroups.' This clearly identifies the tool's function and its subgroup-comparison scope, which is enough to distinguish it from siblings like compute_classical_stats and compute_local_dependence without opening the schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives strong context about when and how the analysis runs, including purification, sample-size refusals, and reporting cautions. However, it never explicitly says when to choose this tool over sibling tools or mentions alternatives, so the usage guidance is implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_datasetA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden, and it delivers: it discloses that the tool reports counts and missing-data breakdown, reproduces none of the responses themselves, is safe to summarise in conversation, and that diagnostics warnings must be relayed verbatim because they affect every downstream statistic. This goes well beyond what the schema encodes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three short paragraphs front-load the purpose, then the usage guidance, then the diagnostics warning. Every sentence carries operational value, and the instruction to relay warnings verbatim earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the 100% schema coverage, an output schema, and a clear description of when and how to use the tool, nothing essential is missing. The description covers the tool's role in the workflow, its safe output characteristics, and the critical warning semantics.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3 and the description need not restate parameter details. It does add workflow context around layout, scoring, and column names, but it does not add parameter-level semantics beyond the rich schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource ('Load a response file and describe its shape, blanks and subgroups') and defines a clear diagnostic scope: counts, missing-data breakdown, and subgroup sizes. It also distinguishes itself from the analytical siblings by positioning itself as the first call before running analysis or asking for DIF.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit usage timing: 'Call this FIRST on any unfamiliar file', 'confirm ... before running an analysis', and 'check subgroup sizes before asking for DIF'. It does not explicitly name sibling tools as alternatives or state when not to use it, but the before/analysis framing provides clear context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compute_local_dependenceA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and delivers: refusal conditions ("REFUSES below 5 items or 100 examinees"), a distinct refusal code for an unfixable case ("no_estimable_pairs"), and the return contract including why the reference distribution must accompany the flag list. It also discloses non-behaviors ("there are no p-values here") that prevent misreading the output.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Front-loaded with the core action in the first sentence, then organized into distinct paragraphs on rationale, return semantics, refusals, and exclusions. It is long (three dense paragraphs), but every paragraph carries different information and the stylistic asides ("a pair to READ, not a pair to cut") compactly encode interpretation rules. A minor deduction for overall length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For an 11-parameter, zero-annotation tool with an output schema, the description covers everything structural fields cannot express: refusal thresholds, the purpose of returning the full Q3 distribution, hub-item aggregation, and the non-inferential reading of flags. Nothing an agent needs to select or invoke it correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline of 3 applies. The description adds only marginal parameter context, connecting the threshold to the flagging rule ("how many SDs from the form mean") and echoing threshold_reference='centred' via "centred Q3*", but the per-parameter schema descriptions are the real carrier of meaning (e.g., "20 means 20 -- pass 0.2, not 20").

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Opens with a specific verb, resource, and method: "Screen every item pair for local dependence (Yen's Q3) and flag entangled pairs." It distinguishes itself from siblings by name ("Answers a question compute_classical_stats structurally cannot") and by exclusion ("not an enemy-item check... not inference"). An agent can tell exactly what this tool computes and how it differs from its siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly names the alternative and the decision condition separating them: compute_classical_stats for per-item checks, this tool for the pairwise question. It gives the triggering scenario ("a fifth of its items are entangled; that is the case for running this at all") and states what not to do with the output ("Do not present a flagged pair as a significant finding"). No inference is left to the agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_standard_settingA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the full behavioral burden, and it is exceptionally transparent. It states the cut score is not the output, warns that differences within about two standard errors are sampling noise, and discloses refusal rules (fewer than 3 panelists, ratings outside [0, 1]) and the warning below 8 panelists. It also discloses that RP criterion is recorded but not applied and that the package does not fit IRT models.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but well organized and front-loaded, with the most important operational warning—'THE CUT SCORE IS NOT THE OUTPUT'—placed early. Some points are repeated from the schema descriptions, such as inline acceptability and the rationale for empirical_p_values, but for an 11-parameter tool with conditional requirements the length is largely justified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex tool with 11 parameters, an output schema, and method-dependent requirements, the description covers the important combinations: rating input options, bookmark prerequisites, the non-substitutability of classical item difficulty, refusal conditions, and the defensibility-record framing. The presence of an output schema means return fields do not need to be enumerated, and the description goes beyond that requirement anyway.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3; the description earns an extra point by adding cross-parameter meaning that individual schema descriptions cannot: the relationship between panel_ratings and ratings_file, the conditional requirement of ordered_item_thetas for bookmark, and panel-size rules tied to interpretability of the standard error. It does not need to re-explain every parameter because the schema already does that well.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The opening sentence gives a specific verb and resource: 'Aggregate standard-setting panel judgements into a cut score with its defensibility record.' The method list (angoff, yes_no_angoff, bookmark) further clarifies the tool's domain and clearly distinguishes it from the sibling psychometrics tools such as compute_classical_stats, flag_dif, and compute_local_dependence.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives strong contextual guidance: when ratings can be inline versus CSV, when bookmark requires IRT-based ordered_item_thetas, when empirical p-values should be supplied, and when the tool refuses to run. It does not explicitly name sibling tools as alternatives or state a 'do not use this for X' rule, so it stops just short of the strictest 5 standard.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.