Server Quality Checklist

Tool Scores

• berlin_get_datasetB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint=true, destructiveHint=false, idempotentHint=true, and openWorldHint=true, covering safety and idempotency. The description adds value by specifying the return format ('detaillierte Markdown-Ansicht' - detailed Markdown view) and listing what metadata is included (title, description, author, license, file formats, download URLs). However, it doesn't disclose rate limits, authentication needs, or error behaviors beyond what annotations imply.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core purpose. The two sentences efficiently convey what the tool does and what it returns. There's minimal waste, though the bilingual structure (German then English) is slightly redundant but not excessive.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (simple read operation), rich annotations (readOnlyHint, idempotentHint, etc.), and the presence of an output schema, the description is reasonably complete. It explains the return content in detail, which complements the output schema. However, the lack of parameter discussion and usage guidelines leaves minor gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description carries the burden. It doesn't mention the dataset_id parameter at all, leaving it undocumented. The schema provides a description and example for dataset_id, but the description adds no semantic context about parameters. With one parameter and no coverage, the baseline is 3 since the schema does provide some documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Ruft vollstaendige Metadaten und Ressourcen eines Datensatzes ab' (retrieves complete metadata and resources of a dataset). It specifies the verb ('abrufen' - retrieve) and resource ('Datensatz' - dataset), distinguishing it from siblings like berlin_list_categories or berlin_search_datasets. However, it doesn't explicitly differentiate from berlin_analyze_datasets, which might have overlapping functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when to choose this over berlin_search_datasets (which might return multiple datasets) or berlin_analyze_datasets (which might process dataset contents). There are no explicit usage contexts, exclusions, or prerequisites stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• berlin_catalog_statsA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds valuable context beyond annotations: it specifies what statistical information is returned (total datasets, category distribution, formats, tags). Annotations already declare readOnlyHint=true, openWorldHint=true, idempotentHint=true, and destructiveHint=false, so the agent knows it's a safe, cacheable read operation. The description doesn't contradict annotations and provides useful output details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with three sentences (German overview, specific metrics, return statement). Each sentence adds value: the first establishes scope, the second lists specific statistics, the third clarifies the return. The bilingual nature (German/English) is slightly inefficient but not wasteful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 0 parameters, rich annotations (readOnly, openWorld, idempotent, destructive hints), and an output schema exists, the description provides adequate context. It explains what statistical overview is returned, which complements the structured data. For a simple statistical retrieval tool, this is reasonably complete, though usage guidance is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0 parameters and 100% schema description coverage, the baseline would be 4. The description appropriately doesn't discuss parameters since none exist. It focuses instead on what the tool returns, which is appropriate given the parameterless nature.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Gibt einen Ueberblick ueber den gesamten Open-Data-Katalog des Landes Berlin' (provides an overview of Berlin's open data catalog). It specifies what information is returned (total datasets, category distribution, common formats and tags). However, it doesn't explicitly differentiate from siblings like 'berlin_analyze_datasets' or 'berlin_list_categories' beyond the statistical focus.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'berlin_analyze_datasets' (which might provide deeper analysis) or 'berlin_list_categories' (which might list categories without statistics). There's no indication of prerequisites, timing, or context for choosing this statistical overview tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• berlin_analyze_datasetsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide key behavioral hints: readOnlyHint=true, destructiveHint=false, openWorldHint=true, and idempotentHint=true. The description adds useful context beyond this: it notes that Berlin's CKAN has no DataStore, so data must be downloaded via resource URLs, and it specifies the tool returns a comprehensive analysis report. This adds value but doesn't detail aspects like rate limits or authentication needs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core purpose and key features. It uses three concise paragraphs with no wasted sentences, each adding value: analysis scope, utility, and a technical note. However, the mix of German and English might slightly hinder clarity for non-German speakers.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (analysis with multiple parameters), rich annotations (covering safety and behavior), and the presence of an output schema (implied by 'Returns' in description), the description is complete enough. It covers purpose, usage context, key behavioral notes, and output format, without needing to repeat structured data from annotations or schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage (all parameters are well-documented with titles and descriptions), so the baseline is 3. The description doesn't add specific parameter semantics beyond what the schema provides; it mentions analyzing relevance, freshness, and formats, which aligns with parameters like 'include_freshness' and 'include_structure', but doesn't explain syntax or usage details for parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: analyzing datasets for relevance, freshness, and available formats, combining search with metadata and resource format analysis. It distinguishes itself from siblings like 'berlin_search_datasets' by emphasizing comprehensive analysis rather than just searching. However, it doesn't explicitly contrast with all siblings (e.g., 'berlin_get_dataset' might also retrieve metadata).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: to find out what data is available and how current it is, particularly useful for understanding dataset availability and freshness. It mentions Berlin's CKAN lacks a DataStore, implying data must be downloaded via resource URLs, which guides usage. However, it doesn't explicitly state when not to use it or name alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• berlin_list_categoriesA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint=true, destructiveHint=false, openWorldHint=true, and idempotentHint=true, covering safety and idempotency. The description adds valuable context beyond this: it explains the organizational structure (25 thematic categories like education, transport, environment), specifies the return format (Markdown list with dataset counts), and clarifies the dual functionality (list all or show details). This enhances behavioral understanding without contradicting annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core functionality. It uses three sentences: the first states the purpose, the second adds Berlin-specific context, and the third specifies the return format. Each sentence adds value without redundancy, though minor improvements in flow could elevate it to a 5.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter), rich annotations (covering read-only, open-world, idempotent, non-destructive), and the presence of an output schema (implied by 'Has output schema: true'), the description is complete enough. It explains the purpose, organizational context, and return format, addressing key aspects without needing to detail parameters or output values explicitly, as these are handled by structured fields.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the input schema's properties include a detailed description for group_id, listing all 25 available values and explaining that an empty value lists all categories. The description itself doesn't add parameter semantics beyond what's in the schema (e.g., it doesn't clarify param syntax or constraints). With one parameter well-documented in the schema, the baseline score of 3 is appropriate as the description doesn't compensate but the schema carries the burden.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Listet alle Datenkategorien (Gruppen) im Katalog auf oder zeigt Details einer Kategorie' (Lists all data categories/groups in the catalog or shows details of a category). It specifies the verb (list/show) and resource (data categories/groups), and mentions the Berlin context with 25 thematic categories. However, it doesn't explicitly differentiate from sibling tools like berlin_list_tags or berlin_search_datasets, which prevents a score of 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by mentioning that it lists all categories or shows details for a specific one, with the input schema clarifying that an empty group_id returns all categories. However, it doesn't provide explicit guidance on when to use this tool versus alternatives like berlin_list_tags or berlin_search_datasets, nor does it mention prerequisites or exclusions. This leaves usage context somewhat implied rather than clearly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• berlin_list_tagsA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint=true, destructiveHint=false, openWorldHint=true, and idempotentHint=true, covering safety and idempotency. The description adds value by specifying that it returns a list of matching tags ('Returns: Liste passender Tags'), which clarifies the output behavior. It doesn't contradict annotations, and the context about tags helping find datasets is useful beyond the structured data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized with three sentences: purpose statement, explanation of tags, and return value. It's front-loaded with the core functionality. The example tags are relevant but slightly redundant with the schema. Overall, it's efficient with minimal waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one parameter object with two fields), rich annotations (covering safety and behavior), and the presence of an output schema (which handles return value documentation), the description is complete enough. It provides purpose, context, and output clarification without needing to detail parameters or behavioral traits already covered elsewhere.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage (both 'query' and 'limit' parameters are well-documented in the schema with descriptions, titles, and constraints). The description mentions examples like 'einwohner', 'bezirk', 'kita' which align with the 'query' parameter, but doesn't add significant meaning beyond what's in the schema. With high schema coverage, the baseline is 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Durchsucht verfuegbare Tags im Open-Data-Katalog' (searches available tags in the open data catalog). It specifies the resource (tags) and verb (search), though it doesn't explicitly differentiate from sibling tools like 'berlin_list_categories' or 'berlin_search_datasets' which might handle different resources or search types.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides implied usage context by explaining that 'Tags helfen, thematisch verwandte Datensaetze zu finden' (tags help find thematically related datasets) and gives examples like 'einwohner', 'bezirk', 'kita'. However, it doesn't explicitly state when to use this tool versus alternatives like 'berlin_search_datasets' or 'berlin_list_categories', nor does it mention any exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• berlin_search_datasetsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, destructiveHint=false, openWorldHint=true, and idempotentHint=true. The description adds valuable context beyond annotations: it specifies the search engine (CKAN-Solr), the scope (full-text across multiple fields), and the approximate dataset count (2500+). It also mentions the return format (Markdown-formatted summaries), which is helpful though output schema exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three tightly focused sentences with zero waste: first states purpose and implementation, second details search scope, third specifies output format. Every sentence adds distinct value. The description is appropriately sized and front-loaded with the core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (search with multiple parameters), excellent schema documentation (100% coverage in practice despite 0% metric), comprehensive annotations, and existing output schema, the description provides complete contextual understanding. It covers what the tool does, how it works, what it returns, and the data scope.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the input schema itself has excellent descriptions for all parameters (query, rows, offset, sort, filter_group) with examples and constraints. The tool description doesn't add any parameter-specific information beyond what's in the schema. With rich schema documentation, the baseline 3 is appropriate as the description doesn't compensate but doesn't need to.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Durchsucht' - searches), the target resource ('Open-Data-Katalog des Landes Berlin nach Datensaetzen'), and distinguishes it from siblings by specifying it's a search tool (vs. analyze, get, list, or stats tools). It provides both the German and technical implementation detail (CKAN-Solr).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by mentioning it searches across 'Titel, Beschreibung, Tags und Metadaten aller 2500+ Datensaetze,' which helps differentiate from more specific retrieval tools like berlin_get_dataset. However, it doesn't explicitly state when to use this vs. berlin_analyze_datasets or berlin_list_categories/tags, nor does it provide exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.