Server Quality Checklist

Tool Scores

• get_dataset_download_planD
  1.3/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It says 'without writing files' implying read-only, but doesn't explain what a 'plan' entails, whether auth is required, or what happens if auth is missing. The phrase 'including local provider-auth guidance when required' is vague and doesn't describe output or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short but under-specifies. It uses jargon like 'trusted OpenConstruction metadata route' without explanation, so brevity is not conciseness; it's vague.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that likely returns a download plan, there is no information about output, preconditions, or why to use it over download_dataset. The description is inadequate for an agent to understand what it does or when to invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter is dataset_id, and the description does not mention it at all. Schema coverage is 0%, so the description must compensate but offers no explanation of how dataset_id is used.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses vague terminology like 'Resolve the trusted OpenConstruction metadata route' without specifying what a download plan is or what the tool concretely returns. It does mention 'without writing files' which hints at a non-destructive operation, but the purpose is not clearly differentiated from siblings like download_dataset or get_download_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus the siblings (e.g., download_dataset, get_download_status). It does not state prerequisites, conditions, or scenarios where this plan would be needed before an actual download.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• prepare_benchmark_submissionD
  1.5/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must convey behavior. It only says 'create a checklist,' which implies some output or side effect, but there is no mention of what happens, whether it returns data, whether it has side effects, or whether it requires specific inputs or permissions. Completely opaque.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, which is concise, but it lacks structure and specificity. It could be expanded to mention key parameters or expected behavior without becoming verbose. As is, it's minimal but does not fully utilize the format.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no annotations, no output schema, and zero parameter documentation, this description is severely under-specified. An agent cannot determine what the tool does, what inputs mean, what it returns, or how it relates to sibling tools. It is inadequate for reliable invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description mentions none of the four parameters (task, metric, model_id, dataset_id). The description adds zero information about what these parameters mean, how they relate to the checklist, or how to populate them. Given high parameter count and zero coverage in schema, this is a critical gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a verb ('Create a checklist') and a resource ('benchmark metadata and linked resources'), but the exact function is vague. It doesn't clarify what the tool actually does—does it generate a to-do list, return a template, or validate something? The phrase 'checklist' is ambiguous and doesn't distinguish from siblings like validate_metadata_record or find_datasets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus any sibling. There is no mention of prerequisites, context, or what scenario this tool is designed for. Users are left to guess whether this is a planning step, a validation step, or something else.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_resourcesC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It does not disclose any operational behavior such as whether the comparison is side-by-side, whether it ranks resources, whether it requires a goal to produce a meaningful result, or how it handles heterogeneous types (e.g., comparing a dataset and a model). Since there is no annotation coverage, this shallow description leaves the agent uninformed about side effects, output nature, or constraints like the max of 8 resources.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely short (one sentence) and front-loads the primary verb and object, which is technically efficient. However, it is under-specified to the point of being vague. It earns a middle score because it is concise but not effectively structured for an agent: it omits critical use-case details that would take a few extra words to include.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that the tool has two parameters, one required, and no output schema, and that it likely involves a comparison of up to 8 heterogeneous resource types, the description is inadequate. It doesn't inform about the purpose of 'goal', the expected format of resources (type/id lookups?), the result shape (e.g., a report?, a set of pros/cons?), or any limitations. For a tool that is a sibling to a rich set of exploration tools, this is incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description only mentions 'two or more catalog resources' and 'a user goal' without explaining the semantics of the 'goal' parameter or the resource structure (type+id). The schema itself is sparse: 'goal' is just a string, and 'resources' requires type and id, but the description does not elaborate on what each field means for the comparison or how the goal influences the comparison. The agent has to guess what 'goal' should contain and what the comparison output will be, so the description fails to compensate for the poor schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear action ('Compare') and a generic resource target ('two or more catalog resources'), and mentions it is 'for a user goal,' which suggests the comparison is goal-driven. However, it does not distinguish itself from sibling tools like search_resources, get_resource, or find_datasets – any of which could be involved in comparing resources. The title 'Compare OpenConstruction resources' mostly restates the name, so the description adds only marginal clarity about what the comparison actually does (e.g., side-by-side attributes? differences? ranking?), which is not specified.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives. The description does not mention that it can compare different types of resources, that it requires at least two resources, or how it relates to search_resources or get_resource. An agent reading this would not know if this is the right tool for comparing specific resources or if they should use other tools first to collect data. No exclusions or preferred-use context is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_dataset_discoveryC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description alone must reveal behavior. It does mention 'searches, ranks, and explains' and 'structured workflow', which gives a rough sense of the process, but it does not disclose important behavioral traits such as whether the skill orchestrates other tools, whether it writes files, whether it requires user confirmation, or what the final output format looks like.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and front-loaded with the main action, which is acceptable. However, it is a single clause that gives minimal information, and the opening 'Run the Dataset Discovery skill' largely echoes the tool name rather than adding new, substantive detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a required task parameter, four optional filter-like parameters, no output schema, no annotations, and many siblings; this description is far too thin to serve as adequate context. It does not explain what task should contain, what the workflow's output is, how the ranking works, or how it relates to other dataset discovery/skill tools. An agent cannot confidently invoke it correctly based on this definition.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description does not compensate. It does not explain parameters such as task, limit, modality, license, object_class, access_requirement, or annotation, nor how they affect the search/ranking workflow. The agent would have to guess the meaning and purpose of all seven parameters from their names alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Run the Dataset Discovery skill as a structured workflow') and the outcome ('searches, ranks, and explains dataset candidates'). It communicates the resource and core behavior, but it does not explicitly contrast itself with siblings like find_datasets or search_resources, so it does not fully maximize distinguishing power.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus sibling tools such as find_datasets, get_skill, or search_resources. The phrase 'skill as a structured workflow' only implies the user should choose it when they want a full workflow rather than a direct resource lookup, but no explicit alternatives or conditions are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_schemaC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing behavior. It states the tool 'explains' but does not mention whether it makes network calls, returns static information, or has side effects. It also doesn't specify the format or scope of the explanation (e.g., does it return text? a list?). The behavioral transparency is minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, which is concise)Skip, but it is also under-specified. It is not bloated, but it could be more informative without sacrificing brevity. The structure is acceptable for a one-liner, but it lacks prioritization of key information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no output schema Synd, no annotations, and minimal parameter documentation, the description is insufficiently complete. It doesn't explain what constitutes a 'common field', what the response looks like, or how parameters influence the output. Agents would be uncertain about the exact behavior and may misuse the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has two parameters (field, resource_type) with no descriptions, and schema description coverage is 0%. The description does not elaborate on what these parameters mean, how they filter the explanation, or their expected values beyond the enum for resource_type. It adds no semantic value over the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear purpose: explaining common OpenConstruction metadata fields and their locations in catalog records. It uses a specific verb ('explain') and resource ('metadata fields'). While it doesn't explicitly differentiate from siblings, its educational intent is distinct from the operational sibling tools (e.g., download, search, get), so the purpose is reasonably clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. It doesn't mention whether it should be consulted before calling other tools, or when to use it instead of, say, get_catalog_stats or search tools. The description implies a general use case but provides no context or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_datasetsC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears full responsibility for behavior. It implies a read-only search ("find") but doesn't state return format, pagination, side effects, or any constraints. Agents cannot predict what the tool returns or how filters combine.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence, front-loaded with the core action. It is concise, but its brevity sacrifices the clarity needed for a tool with six unlabeled parameters. No wasted words, yet it under-delivers on useful guidance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Missing critical context for a search tool: no mention of default behavior, result limits, output format, or interaction with sibling tools like `download_dataset`. The description leaves agents guessing about pagination, filtering semantics, and whether multiple criteria are ANDed or ORed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% coverageainerneither descriptions nor enums, so the description must compensate. It lists most parameters (task, modality, annotation, object_class, license) but omits `limit` and the phrase "access needs" doesn't map to any parameter. No value formats or examples are given, leaving the agent to guess valid inputs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a clear verb (find) and resource (datasets) and enumerates filtering dimensions (task, modality, object class, annotation type, license, access). It distinguishes itself from sibling tools like list_skills and get_dataset_download_plan, though it doesn't explicitly contrast with run_dataset_discovery, which might overlap.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. The description only lists filter criteria but does not mention when to prefer find_datasets over search_resources or run_dataset_discovery, nor any exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_modelsC
  2.7/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must carry the full burden of behavioral disclosure. It only states 'Find models by...' with no mention of return format, pagination, side effects, authorization requirements, or whether it is read-only. For a search tool, agents need to know if it returns a list, how results are sorted, etc. This is a significant omission.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, concise and to the point. It front-loads the primary action ('Find models') and lists key filters. However, it's extremely brief and doesn't provide structure or additional helpful details. Still, it's not verbose and gets the core purpose across.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 5 optional parameters, no output schema, and no annotations, the description is insufficient. An agent doesn't know what the result looks like, how to interpret the filters, whether multiple filters can be combined, or what the default behavior is when no filters are provided. Also, the mention of 'publication evidence' is not in the schema, which could confuse the agent. The description is too sparse for a discovery tool with this much parameter variety.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain the parameters. It lists task, dataset, modality, method, but omits 'limit' and introduces 'publication evidence' which is not a parameter in the schema. The description gives minimal meaning to the parameters but doesn't explain how they combine, value formats, or edge cases. It provides some value but does not fully compensate for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool finds models and lists the primary filter dimensions (task, limit, method, dataset, modality). It distinguishes from siblings like find_datasets by naming the resource type (models). However, it mentions 'publication evidence' which is not in the schema, introducing slight ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus siblings like find_datasets or search_resources. The description does not state when this tool is preferred, nor when it should not be used. The context implies model discovery, but no exclusions or alternatives are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_skillsC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries the full behavioral burden. It signals a non-mutating list operation and the word 'metadata' is helpful, but it does not disclose the output structure, whether all skills or only a filtered subset are returned, or any pagination/ordering behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one short, front-loaded sentence with no unnecessary details. It loses one point because much of the text simply restates the title rather than adding substantive behavioral or contextual value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema and no annotations, the description is too thin to be considered complete. It would be materially better with a sentence about returning all repo-defined skills by default, optionally filtering by lifecycle_stage, and using get_skill for full details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverageand the tool description never mentions lifecycle_stage. The enum and parameter name are self-descriptive, but the description does not compensate for the lack of schema documentation, leaving the effect of lifecycle_stage implicit.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear verb ('List') and a specific resource ('repo-defined OpenConstruction skill metadata'), so an agent can see this is a collection-returning operation and not a single-skill getter. It does not explicitly mention the sibling get_skill, but the phrase 'repo-defined skill metadata' provides enough distinction for basic identification.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance about when to call this tool versus get_skill, search_resources, or the other siblings. The description also does not say whether lifecycle_stage is an optional filter or make clear the point at which listing skills is the right step.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_resourcesC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of disclosing behavior, but it only names the action and scope. It does not explain whether results are ranked, how filtering works, whether pagination exists, or what the response looks like. While 'Search' implies a read-only operation, no behavioral depth is provided.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence with no filler words. The list of resource types is dense but relevant. It is efficient, though it could have used the space to add behavioral or usage context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with three parameters, no output schema, no annotations, and many sibling tools, this description is too thin. It does not cover return values, query semantics, or when to choose this over specialized search siblings, leaving an agent to infer important details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description should compensate for the undocumented query and limit parameters. It lists the resource categories, which map to the types enum, but it does not explain the query parameter, limit constraints, or how multiple types are combined.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('Search') and the resource scope ('OpenConstruction resources'), enumerating nine concrete resource types. It is distinct from an abstract placeholder, though it does not explicitly differentiate itself from sibling tools like find_datasets or find_models.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided about when to use this tool instead of alternatives such as find_datasets, find_models, or get_resource. The description only states what it searches, not the context that should trigger selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_metadata_recordB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the full burden of behavioral disclosure. It says 'Check' which hints at a read-only operation, but it does not explicitly state that the tool is non-destructive, whether authentication is required, or any side effects. It also does not disclose what the output is or whether the record is modified, leaving behavioral uncertainty.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no fluff and is front-loaded with the core purpose. It is concise, but the brevity contributes to the lack of detail in other dimensions. Still, as a standalone statement, it is well-structured and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has two parameters (one a nested object) and no output schema, the description fails to describe the return value (e.g., list of missing fields, boolean, etc.) or how resource_type influences validation. This leaves agents without essential information to correctly interpret results or parameter usage. The description is incomplete for a validation tool with these schema characteristics.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate by explaining the parameters. It does not explain what 'record' should contain or how 'resource_type' affects the validation. The phrase 'draft metadata record' is generic and does not tie to the specific properties. The description adds no meaning beyond the schema, which is a significant gap given zero coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('Check one draft metadata record') and its purpose ('for important missing fields'). It identifies the resource (metadata record) and the operation (validation), and it is distinct from any sibling tool, none of which perform validation. This is a clear and specific statement of purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: when you have a draft metadata record and want to check missing fields. However, it does not explicitly state when to use this tool over alternatives, nor does it provide exclusions or mention any alternative. The context of 'draft' hints at a pre-publication step, but the guidance is implicit rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ask_openconstructionB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It does reveal that the tool grounds answers in the catalog (a useful anti-hallucination signal) and that it returns 'matching resources,' but it does not disclose failure behavior (e.g., no matching resources), the nature of the answer format, whether it executes a retrieval/LLM pipeline, or any auth/permission requirements. For a question-answering tool this is thin on behavioral detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single efficient sentence of eight words with no filler. The action verb 'Answer' is front-loaded and the key qualifier 'catalog-grounded' appears early. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with no output schema and no annotations, the description covers the purpose and the nature of the query adequately. Yet it omits response characteristics (no output schema exists to fill this gap), no-match behavior, and any distinction from similarly-purposed siblings. These gaps mean the description alone is not fully sufficient for confident, correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate fully for the single 'query' parameter. It does add some meaning by characterizing the query as a 'catalog-grounded question,' which helps an agent understand the expected input is a natural-language question about the catalog. However, it does not elaborate on acceptable question scope, expected phrasing, or how the query is interpreted against the catalog, leaving the parameter semantics only partially enriched.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Answer') with a qualified resource ('catalog-grounded question') and a concrete outcome ('matching resources'). It communicates that the tool performs natural-language question answering against a catalog, which is a discernible purpose. However, it does not explicitly differentiate itself from sibling tools like search_resources or find_datasets, which could plausibly overlap in returning matching resources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives such as search_resources, find_datasets, or get_resource. The phrase 'catalog-grounded' only implicitly suggests a use case (questions grounded in catalog data), but there is no explicit condition for selection, no exclusions, and no mention of when the structured search tools would be preferable. An agent has to guess at the routing decision.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_skillB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, and the description provides minimal behavioral information. It implies a read-only operation by using 'get', but does not specify what is returned, potential errors, or any side effects. The description carries the full burden and fails to disclose anything beyond the basic action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, consisting of a single sentence with no superfluous words. It is well-structured and front-loads the action and resource.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is minimal but sufficient for a simple get-by-id operation. It does not explain what a skill definition contains or how it relates to sibling tools, but given the straightforward nature of the operation and the presence of a list_skills sibling, the context is reasonably inferable. However, it lacks details about output or error handling.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter 'id' is described as 'by id' in the description, but this adds little beyond the schema's type and required flag. It does not explain what constitutes a valid id, its format, or any constraints. Given the schema provides no parameter description, the description should compensate but does not.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: to retrieve a single OpenConstruction skill definition by its id. This is unambiguous and distinct from the sibling 'list_skills' which presumably retrieves multiple skills.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide guidance on when to use this tool versus alternatives. It does not mention that one should know the id beforehand or that list_skills should be used to discover ids. Users must infer usage from the verb 'get'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_catalog_gapsB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The verbs 'inspect' and 'return' signal a read-oriented tool that produces output, which is useful given that no annotations are present. However, it does not disclose behavior such as whether the analysis is expensive, whether the catalog is ever modified, or what happens when no topic or limit is supplied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler or redundant restatement of the tool name. Every phrase adds meaning about scope or output.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is adequate for a fairly simple two-parameter tool, and the return concept is at least named. But because there is no output schema, the agent is left guessing about the shape of the gap report and about how limit affects results, making the description only partially complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage and no parameter explanations, so the description must compensate. It only clarifies that the analysis is 'for a topic'; the limit parameter is completely unexplained, and the effect of providing or omitting parameters is unclear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a concrete action (inspect catalog coverage by type) and an intended result (return possible metadata gaps), which maps directly to the tool's name. It is clear enough to guide an agent, though it does not explicitly differentiate itself from close siblings like get_catalog_stats or search_resources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for analyzing coverage and metadata gaps for a topic, but it gives no explicit guidance about when to choose it over sibling tools. No alternatives, exclusions, or fallback conditions are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_resourceB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility for behavioral disclosure. It only says 'Get one resource' but does not mention what happens for missing/unknown IDs, whether an error or null is returned, or any access prerequisites. This is a notable gap for a tool with no structured behavioral hints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. Every word contributes to identifying the tool's purpose and parameters.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter getter, the description is mostly adequate, but because there is no output schema or annotations, it would benefit from stating what is returned or how failures are represented. Still, the tool is straightforward enough that this is a moderate gap rather than a critical one.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does confirm that 'type' and 'id' are the lookup identifiers, but it adds little beyond the schema itself. The enum values in the schema already communicate the allowed resource types, and the description does not explain id format or semantics further.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and the resource ('one resource by type and id'), which is specific and distinguishable from list/search tools. However, it doesn't explicitly differentiate from sibling tool get_skill, leaving some ambiguity about scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'by type and id' implies this should be used when the agent already knows the exact resource type and identifier. No explicit when-not-to-use guidance or alternative tool routing is provided, so the usage context is only implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cancel_downloadB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It reveals that cancellation is only 'requested' and applies to a running local job, but it does not state whether the request is asynchronous, whether the job is actually stopped, what happens to partial data, or what errors may occur.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single compact sentence with no filler or redundant restatement of the title. Every word earns its place and the key constraint ('running', 'local') is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one parameter, so a minimal description is partially adequate. However, without annotations or an output schema, the description should have clarified the cancellation effect and expected outcome to be fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description does not elaborate on download_id beyond what the parameter name already implies. The ID is plausibly a download job identifier, but the description does not confirm its exact referent or format, leaving some ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific action ('request cancellation') and a specific resource ('running local OC dataset download job'), making the tool's purpose unambiguous. It also distinguishes itself from siblings like download_dataset and get_download_status by focusing on the cancellation workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'running ... download job' implies the tool should be used while a download is active, but there is no explicit guidance about when not to use it or which sibling tool to prefer. Usage context is present only by implication, not direct instruction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_download_statusA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description provides no information about side effects, permissions, or read-only nature. It only mentions the output, so transparency is low.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is brief and does not elaborate on the output structure or any conditions, leaving ambiguity about the response format.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter, download_id, is not explained in the description. The schema only provides type and minLength, which is insufficient for an agent to understand its purpose.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool returns progress, files, and errors for a download job, distinguishing it from planning, starting, or canceling downloads.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for checking the status of a download, but does not explicitly mention when to use it over alternatives like cancel_download or get_dataset_download_plan. However, the name and context make it clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_catalog_statsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral burden. It does state that the tool returns counts, implying a read-only operation, and references the 'loaded catalog snapshot' as a precondition. However, it does not disclose side effects, error behavior, or what happens if the snapshot is not loaded. This is acceptable for a simple getter but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no filler. The core action and target are front-loaded, making it immediately scannable. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with no output schema, the description covers the essential purpose and a key precondition (loaded catalog snapshot). It does not mention the exact format of the counts or any additional context, but for such a simple operation this is largely sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and the schema is an empty object, so the schema fully covers all parameters (100% coverage). The baseline for zero parameters is 4, and the description does not need to add anything about parameters. It correctly remains silent.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Return' and identifies the resource ('counts by resource type') and the context ('loaded catalog snapshot'). It unambiguously states what the tool does and is clearly distinct from any sibling tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no guidance on when to use this tool versus alternatives, such as analyze_catalog_gaps or compare_resources. It does not mention any exclusions or preferable conditions. For a simple read-only stats tool, some context about typical use cases would be useful.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• download_datasetA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full transparency burden and does well: it reveals background execution, the local root directory, the auth_required retry flow, the no-credentials-in-chat rule, and license prerequisites. It stops short of explaining how to poll or interpret the success status beyond the auth_required case.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two dense sentences with no filler. The first sentence front-loads the core operation and location; the second packs the auth handling, license requirement, and security constraints without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description covers the important behavioral context: background execution, destination root, auth handling, and security guardrails. The main gap is the lack of an explicit success response/polling note, such as telling the agent to use get_download_status, though this can be inferred from the background-download wording and sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 67%, but the description adds meaningful semantics: destination is understood as relative to OC_DOWNLOAD_ROOT, accept_license must reflect explicit user review, and dataset_id is an identifier rather than an arbitrary URL. This goes beyond the schema's basic property descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The opening phrase 'Start a background download inside OC_DOWNLOAD_ROOT' expresses a specific verb, resource, and scoping location. It clearly distinguishes the tool from sibling status/plan/cancel tools by emphasizing that this is the action that initiates the download rather than checking or managing it.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear when-to-use and when-not-to-use signals: it requires explicit license acceptance, and it explicitly rejects arbitrary source URLs and shell commands. It does not name sibling alternatives such as get_dataset_download_plan or get_download_status, so the routing is slightly less explicit than a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.