Server Quality Checklist

Tool Scores

• ctt_item_analysisC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It does mention 'with warnings,' but gives no detail about what triggers warnings, how missing responses are handled, or what assumptions are checked. The description is too thin to fully describe the tool's runtime behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single efficient sentence with no wasted words, and the primary action is front-loaded. It is appropriately concise, though it sacrifices useful detail that other dimensions require.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no annotations and zero parameter description coverage, so the description must carry more weight. It omits input-format expectations, warning semantics, and any guidance for choosing this analysis over the sibling tools. With an output schema present, return values may be covered, but the input and decision context are not.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description provides no information about the input parameters. It does not explain the expected responses matrix, the optional item_names field, or the format required for correct invocation, so it adds no value beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Compute') and lists concrete outputs: item summaries, item-rest correlations, raw alpha, and SEM. This clearly identifies the tool as a classical test theory item analysis, though it does not explicitly differentiate it from sibling tools like correlation_matrix or rasch_model.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives such as descriptive_statistics, correlation_matrix, or rasch_model. The intended use is only implied by the tool name and the listed outputs, with no exclusions, prerequisites, or decision context provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• plan_psychometric_analysisC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It does not state whether the tool only returns a plan, whether it has side effects, or what the 'analysis sequence' output looks like. 'Create' is ambiguous and could imply persistence.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single focused sentence, front-loading the verb and object. There is no redundancy or filler, making it efficiently scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the rich request schema and the existence of sibling analysis tools, the description is too abstract to fully guide invocation. It does not clarify the nature of the output plan, the relationship to the sibling tools, or the role of each input. The presence of an output schema helps, but the description remains incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description should compensate by describing the request fields. The phrase 'intended use and design' loosely maps to purpose and design parameters but does not explain any specific field, enum, or default. It adds only minimal semantic value over the schema's property names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action and resource: 'Create a measurement-aware analysis sequence.' It also identifies the input basis ('intended use and design'). However, it does not explicitly contrast with sibling analysis tools, so an agent may not immediately know this is a planning tool distinct from ctt_item_analysis or rasch_model.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus the sibling analysis tools. The description implies planning from design inputs but never states that this should be used before executing analyses, nor mentions any prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_response_dataB
  3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden and does disclose meaningful behavioral scope: it audits shape, missingness, categories, ranges, and zero-variance items. However, it does not describe return behavior, whether it is read-only, or how it handles malformed or unexpected data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single front-loaded sentence communicates the tool's purpose and audit dimensions with no wasted words. It is appropriately concise and scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  An output schema exists, so return values need not be described, but the definition still lacks usage context relative to the sibling tools and leaves the optional item_names parameter unexplained. For a tool intended to guide when to run deeper psychometric analyses, more contextual guidance is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, but it does not explain the meaning or usage of the single required 'data' parameter beyond the schema's structural definition. It also omits the optional 'item_names' field, making the parameter semantics incomplete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Audit') and clearly identifies the resource (response data) and the dimensions it covers (shape, missingness, categories, ranges, zero-variance). This distinguishes it from the analysis-oriented siblings, though it does not explicitly name another tool or contrast itself with descriptive_statistics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is a data-quality inspection/pre-analysis step but provides no explicit guidance on when to use it versus descriptive_statistics, ctt_item_analysis, or the other sibling tools. There are no stated exclusions or routing conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• descriptive_statisticsB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It reveals that the tool computes summaries, tracks missingness, and supplies robust boundaries, implying a read-only computation. It does not explicitly state there are no side effects or clarify what 'sample flow' means, but the stated output topics provide some behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single compact sentence with no filler, and the core purpose is front-loaded. The phrasing is efficient, though the terms 'sample flow' and 'robust boundaries' are terse and could be clearer for the target audience.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, return values need no explanation. The description plus schema gives enough to call the tool with numeric data, but it lacks clarification on ambiguous terms and offers no usage context or example. It is minimally viable but leaves meaningful gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for the schema's silence, but it only says 'numeric variables,' which weakly maps to the required 'data' parameter. It does not explain the nested 'values' array shape or the optional 'variable_names' field, leaving the agent to infer parameter semantics from the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Summarize') and resource ('numeric variables'), and names the output topics: sample flow, missingness, and robust boundaries. This makes the tool's role clear and distinguishes it from siblings like correlation_matrix or ctt_item_analysis, though the domain jargon 'sample flow' is somewhat ambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for descriptively summarizing numeric data, but it gives no explicit when-to-use guidance and does not reference alternatives among the sibling tools. There is clear context, but no exclusions or direct comparison to other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rasch_modelA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It usefully discloses that eRm::RM is used and that arbitrary code is rejected, but it does not describe estimation behavior, missing-data handling, or any operational constraints beyond code rejection.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single front-loaded sentence with no filler. It clearly states the action, model type, implementation, and a critical constraint without wasting words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is narrow and an output schema exists, so some return-value details are handled elsewhere. Still, the description lacks usage context, parameter clarification, and deeper behavioral details, leaving the agent to infer several things needed for confident invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, but it does not explain the 'data' parameter, 'responses', or 'item_names'. The word 'dichotomous' hints that responses should be binary, but the meaning and format of parameters are left entirely to the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific action ('Fit'), a specific resource ('fixed dichotomous Rasch model'), and the implementation ('eRm::RM'). This clearly differentiates it from sibling tools like ctt_item_analysis or correlation_matrix.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The intended use is implied by the phrase 'Fit a fixed dichotomous Rasch model', and the explicit rule 'arbitrary code is never accepted' guides input type. However, it does not state when to prefer this tool over alternatives or mention prerequisites such as suitability only for binary response data.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• correlation_matrixA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses one important behavioral trait: missing-data handling is explicit, which alerts the agent that the tool will not silently drop missing values. However, it does not describe what the tool returns (covered by output schema), how it treats non-numeric data, or any limitations. For a read-only compute operation this is acceptable but sparse.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single ten-word sentence that front-loads the action and resource. Every word adds information: 'Compute' sets the operation, 'Pearson or Spearman' specifies method options, and 'explicit missing-data handling' flags a behavior. There is no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the rich input schema and the presence of an output schema, the description provides enough orientation to know the tool is for correlation computations. But it does not explain when to select pairwise vs listwise deletion, nor does it contrast with descriptive_statistics for univariate analysis. The lack of any annotation means the agent gets no safety or cost context, so a slightly more detailed description would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description maps to two nested parameters: method (Pearson/Spearman) and missing (handling). It adds no detail on the request structure, the expected data shape, or variable_names, leaving that to the schema, which is well-defined with $defs and titles. Since schema prose coverage is 0%, the description only partially compensates for the missing parameter explanation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with the specific verb 'Compute' and names the exact resource ('Pearson or Spearman correlations'), immediately distinguishing this from sibling tools like descriptive_statistics or ctt_item_analysis. It also highlights a distinctive feature ('explicit missing-data handling'), reinforcing scope. The only minor omission is not explicitly saying the result is a matrix, but the tool name and verb make that clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a use case: when Pearson or Spearman correlations are needed. However, it gives no explicit when-to-use versus alternatives, no exclusions, and no guidance on choosing between pairwise and listwise missing-data handling. It does not reference sibling tools or conditions under which a different analysis tool should be preferred.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_computation_capabilitiesA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the tool checks two distinct capabilities and that it reports availability, which implies a non-mutating read operation. However, it does not describe what happens when capabilities are missing, whether there are side effects, or how detailed the report is.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that conveys exactly what the tool checks. There is no redundant information, filler, or repetition of the tool name.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter capability check with an output schema available, the description fully answers what the tool does. It is simple enough that no further detail about return values or invocation is required to use it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the parameter schema is trivially complete and the description does not need to explain parameter meaning. The baseline for zero-parameter tools is 4, and nothing in the description detracts from that.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Report') and clearly identifies the two resources being checked: local Python and the fixed R/eRm Rasch engine. This distinguishes the tool from its analysis-focused siblings, which perform data inspection, statistics, or modeling rather than capability checks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no explicit guidance about when to call this tool versus the sibling tools. While it is implicitly a preflight check before using Python-dependent or R/eRm-dependent tools, the description does not state this or list any alternatives or exclusions, leaving the agent to infer usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.