Server Quality Checklist

Tool Scores

• search_publicationC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of disclosing behavior. It adds that the search occurs after a 'shared scope gate,' implying a precondition, but it does not clarify whether the operation is read-only, what the gate involves, or what happens if the gate is not passed. No information about output format, error behavior, or rate limits is provided.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that is front-loaded with the core action and resource. No words are wasted, but the cryptic 'shared scope gate' phrase could be clarified without adding much length. Overall, it is compact and to the point.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 4 parameters (3 required), no output schema, no annotations, and sibling tools with overlapping functions. The description does not explain the parameters, the expected inputs, the output, or when to use this tool. This is inadequate for an AI agent to invoke it correctly without additional information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the description must compensate by explaining the parameters. It does not mention 'url', 'question', 'search_term', or 'max_matches' at all, leaving their roles entirely unclear. This is a significant gap because the agent cannot determine what to pass as a 'question' versus a 'search_term' or how 'url' relates to the search.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the action (search) and the resource (publication text and linked data files), which distinguishes it from sibling getter tools like get_publication_text and get_data_file. However, the phrase 'after passing the shared scope gate' is vague and could confuse agents about the actual purpose or prerequisite.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions a prerequisite ('after passing the shared scope gate') but does not explicitly tell the agent when to use this tool versus alternatives such as get_publication_text or get_data_file. There is no guidance on when to prefer search_publication over the sibling getter tools, and the nature of the 'shared scope gate' is left undefined.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_publication_textC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of disclosure. It mentions a 'shared scope gate' as a behavior, but gives no detail on what happens if the gate is not passed, error conditions, permissions, or response format. The minimal context is insufficient for an unannotated tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is only one sentence and concise in length, but it is under-specified rather than appropriately concise. It omits critical operational details, making the brevity a liability rather than a strength.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 3 parameters, no output schema, and no annotations, the description needs to provide more context but does not. It covers the basic purpose and a vague prerequisite but leaves the tool's usage, parameter semantics, and expected results unexplained, making it incomplete for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description does not describe any parameter (url, question, section_keyword). The agent receives no guidance on what each parameter means or how to fill them, leaving the schema's bare names and types as the only information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches 'narrative publication text,' which is specific and distinguishes it from siblings like get_publication_overview (overview) and get_data_file (data file). However, it lacks explicit mention of how it differs from search_publication or check_query_scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'after passing the shared scope gate' implies a prerequisite step (likely using check_query_scope first) but does not explicitly name the gate or state when to use this tool versus alternatives. Usage guidance is implied rather than clearly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_data_fileB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It adds useful context about parsing CSV/XLSX and the scope gate prerequisite, but it does not disclose potential side effects, return structure, error behavior, or access requirements, so it remains thin.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that immediately states the verb and resource. There is no filler or redundant phrasing, making it highly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters, no annotations, and no output schema, this description is too minimal. It does not clarify how parameters interact, what 'shared scope gate' entails, what the tool returns, or failure behavior, leaving major gaps for a tool of this complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description provides no explanation of the five parameters (url, file_url, max_rows, question, sheet_name). The agent is left to guess parameter meaning from names alone, which is inadequate for a multi-parameter tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('fetch and parse') and resource ('linked CSV or XLSX file'), and the formats help distinguish it from sibling publication/query tools. However, it does not explicitly differentiate itself by naming alternatives or exclusions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'after passing the shared scope gate' implies a prerequisite and some contextual timing, likely related to check_query_scope, but it is vague and does not explicitly state when to use this tool versus alternatives or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_query_scopeB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It only states the intent, not what the tool returns (e.g., boolean), whether it is read-only, or what 'valid' means in context. This is insufficient for safe invocation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no wasted words. It is concise, though the brevity contributes to under-specification in other dimensions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and a 0% parameter description coverage, the description is too sparse. It does not explain what 'valid' means, what the output looks like, or how this tool fits into the workflow with the sibling tools, making the overall context incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not add meaning beyond the parameter names. It mentions 'publication URL' and 'question' but does not define expected formats, constraints, or examples, leaving the agent to guess at valid inputs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Determine' with a clear resource: 'whether a question is valid for the supplied publication URL.' This clearly distinguishes it from sibling tools like search_publication or get_publication_text, which retrieve or search content rather than validate scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a validation or pre-check purpose ('whether a question is valid'), but it does not explicitly state when to use this tool instead of siblings or when not to use it. No alternatives are named, leaving room for interpretation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_publication_overviewA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It correctly implies a read-only operation via 'Fetch,' but does not disclose potential quirks like response size, pagination, or error behavior. The description is minimal but not misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence with no filler or redundant information. Every word contributes to conveying what the tool does.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema and no annotations, so the description needs to clarify what the agent can expect. It identifies three components (metadata, sections, linked data files) but leaves details ambiguous—for instance, whether 'linked data files' returns actual content or references. The description is adequate for a simple overview tool but leaves gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The 'url' parameter has no schema description (0% coverage), so the tool description must compensate. It indirectly implies that 'url' refers to the publication URL, but it never explicitly states this. The meaning is inferable from the tool name and description, but the description adds little direct parameter guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Fetch') and the resource ('publication metadata, sections, and linked data files'). It distinguishes the tool from siblings like get_publication_text (which likely fetches full text) and get_data_file (which likely fetches a specific data file), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description does not mention criteria such as 'when you need an overview' or suggest using get_publication_text for full text. Despite being a simple tool, it lacks any contextual pointers.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.