Server Quality Checklist

Tool Scores

• list_mapsA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations present, the description carries the transparency burden and explicitly notes the operation is read-only, which is valuable. It also discloses the result surface (metadata only) rather than implying full map contents are returned.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tightly organized with a front-loaded summary of the tool's behavior followed by a compact Args list. Every sentence adds direct value with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple listing operation, the description covers purpose, parameter behavior, pagination, and return scope (metadata fields). While no output schema exists and exact response shape isn't detailed, the metadata field list gives an agent enough to understand what to expect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description provides meaningful semantics for all three parameters: search as a text filter on name/description, limit with range and default, and offset as paging. This fully compensates for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb and object: 'List saved maps', and enumerates the metadata fields (id, name, visibility, layer count). It is distinguishable from sibling tools like get_map, though it doesn't explicitly call out the difference.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is the tool for listing saved maps, which conveys a general usage context. However, it does not state when not to use it or which sibling tool to choose instead for map detail, dataset search, or feature queries.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_mapA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It describes what the tool returns but does not explicitly state whether it is read-only, whether it has side effects, or if there are any limitations, errors, or permission requirements. The verb 'get' implies a read operation, but this is not made explicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded. The first sentence clearly states the tool's purpose, and the second sentence explains the parameter. No unnecessary words or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of an output schema, the description provides a reasonable outline of the return value (full metadata with layers, view state, basemap, terrain). It does not cover error cases or edge conditions, but for a single-id fetch operation, the information is largely sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only the parameter name and type (string). The description adds useful context by giving an example source for the value ('e.g. from list_maps'), which helps an agent know how to obtain a valid map_id. It does not specify format or constraints, but the parameter is a simple identifier.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('Get') and resource ('one saved map's full metadata') with a clear scope (layers, view state, basemap, terrain configuration). It is distinct from sibling tools like list_maps (which lists maps) and query (which queries features).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving details of a single map by ID, with an example source ('e.g. from list_maps'). However, it does not explicitly state when to use this tool over alternatives or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_dataset_schemaA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It does an excellent job explaining the semantics of tricky fields like source_health (inaccessible vs unknown), source_health_detail (fixed reason code), and freshness (overdue meaning). It also discloses that raw provider URLs and credentials are excluded, which is important for agent expectations. It mentions the read-only nature implicitly by describing data retrieval, but does not explicitly state 'this is a read-only operation'. Given the lack of annotations, this is a strong effort, but could be a 5 if it explicitly stated it does not modify data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is thorough and well-structured. It starts with a clear summary sentence, then details the return fields and their semantics, and ends with usage guidance. It is moderately long but every sentence adds critical information about field meanings and exclusions. The main body is front-loaded with the tool's purpose. It could be slightly trimmed, but the richness justifies the length. A 5 would require even more efficiency, but this is clean.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's single parameter, no output schema, and no annotations, the description provides comprehensive information: what is returned (columns, geometry, etc.), the meaning of all status fields, and exclusions. It also gives usage context ('before writing spatial questions'), which is crucial for correct invocation. The agent has everything needed to call the tool correctly and interpret results. This is complete for its complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage for the single parameter, dataset_id. The tool description provides context: 'Dataset id (e.g. from search_datasets).' This is minimal but adds value by indicating the source of the ID. However, it doesn't describe format or constraints, but the parameter is simple. Given the low schema coverage, the description partially compensates, but could be improved by explicitly saying 'A string identifier of the dataset as returned by search_datasets'.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool's purpose: retrieving a dataset's schema and source trust metadata. It enumerates the specific elements returned (columns, geometry, CRS, etc.), distinguishing it clearly from siblings like search_datasets (which finds datasets) and get_features (which retrieves data). The verb 'get' is precise and the resource is identifiable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage guidance: 'Call this before writing spatial questions so you know both the available columns and whether the source may be stale.' This tells the agent when to use it in a workflow. It does not explicitly mention when not to use it or name alternatives, but the context is strong enough to differentiate from search_datasets and get_features. It could be improved by explicitly stating 'use search_datasets to find datasets' but that's implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_featuresA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It explicitly mentions the return format, the error on raster datasets, and the limit/offset paging behavior. It does not cover authentication, invalid dataset_id handling, or the meaning of 'bounded', but it discloses the key traits an agent needs to use it safely.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and logically ordered: a one-line summary, then key behavior, then parameter details. No filler or repetition. Every sentence contributes useful information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity and the absence of an output schema or annotations, the description covers everything essential: return type, error condition, pagination, and parameter semantics. It lacks only minor details like how to obtain a dataset_id or exact behavior for invalid IDs, but the provided information is sufficient to call the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description goes beyond the structured schema. The Args section adds meaning: limit constraints (1-200, default 10), offset purpose ('paging'), bbox format ('minx,miny,maxx,maxy' in WGS84), and dataset_id as the target. This fully compensates for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific action and resource: 'Get GeoJSON features for a dataset'. The return type (OGC-API FeatureCollection) and the error condition for raster datasets further clarify what the tool does. This clearly distinguishes it from siblings like get_dataset_schema (metadata) and search_datasets (dataset discovery).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides concrete usage context: it returns features only for vector datasets, errors on raster datasets, and supports pagination via limit/offset. It does not explicitly name alternatives or state when not to use it beyond the raster error, so it misses the 'use X instead' guidance, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_datasetsA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral disclosure burden, and it does so thoroughly. It explains the GeoJSON FeatureCollection return format, semantics of source_origin and source_freshness, null behavior for health fields, absence of per-result detail requests, and exclusion of credentials. This is far beyond a generic search-tool description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The purpose is front-loaded in the first sentence, and subsequent paragraphs are organized by result format, trust metadata caveats, and parameter details. The description is longer than strictly necessary for three parameters, but the extra detail on freshness and null fields is substantive and earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no annotations and no output schema, the description fully covers inputs, output shape, property enums, null semantics, and even the follow-up tool for richer trust metadata. An agent can invoke and interpret results correctly without guessing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the Args block is the sole parameter documentation. It provides search text semantics, a range and default for limit (1-200, default 10), and offset's basic purpose. The offset explanation is terse and could clarify paging interaction with limit, but the description does compensate for the absent schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence states a specific verb and resource: 'Search the GeoLens catalog for datasets by free text.' This unambiguously distinguishes it from sibling tools like get_features, list_maps, and get_map. The additional note about using the feature id as dataset_id for other tools reinforces its role in the catalog workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for when to use this tool: free-text search across catalog metadata, with semantic ranking mentioned. It also directs the agent to call get_dataset_schema when trust metadata is needed. However, it does not explicitly contrast search_datasets with alternatives like query or list_maps, so it lacks full when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• queryA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries the full burden. It discloses the read-only nature, sandbox constraints (allowlisted functions, server-side budget), and explicitly states what is rejected (writes, other schemas, unlisted functions), as well as the return format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-organized but somewhat repetitive, especially regarding the table reference format ('data.<table_name>') and the restriction on restrict_tables. It could be tightened without losing clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is comprehensive: it covers purpose, constraints, workflow, parameter details, return format, and credential requirements. It provides enough context for an agent to use the tool correctly, leaving no significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description explains each parameter beyond the schema: sql is a single SELECT, restrict_tables must list all accessed tables (with no 'data.' prefix) and is required, and row_limit has a default. It clarifies the scope restriction and adds meaning to the schema's bare types.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Run one read-only SQL SELECT against accessible datasets.' It specifies the verb, resource, and scope, and distinguishes it from sibling tools by being the primary query tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage context: it is for running SQL SELECT queries, and it even outlines a workflow (search_datasets, get_dataset_schema) and mentions required credentials. It implicitly indicates when not to use it (for non-SELECT operations) by emphasizing read-only.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.