Server Quality Checklist

Tool Scores

• care_membrane_policyC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears full responsibility for disclosing behavioral traits. It does not state that the call is read-only, safe, or has no side effects. It only says 'Return', implying idempotence, but fails to explicitly confirm safety or mention authentication needs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no wasted words, but it lacks structure or supplementary information. It meets the minimum standard but provides no additional value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite low complexity and the presence of an output schema, the description fails to cover the parameter or explain the tool's role within the tool ecosystem. It is too sparse to be considered complete for effective agent usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%; the description completely ignores the single optional parameter 'api_key'. The description adds no meaning beyond the schema, despite the need to compensate for low coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Return'), the resource ('Care Membrane policy'), and the scope ('what Gods Eye will and will not do'). It is distinct from sibling tools which focus on data provenance and queries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. There is no mention of prerequisites, context, or exclusions, leaving the agent to infer usage from the name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_data_sourcesC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must convey behavioral traits. It implies a read-only operation ('List') but does not explicitly state that the tool is non-destructive, requires no authentication beyond an optional API key, or has rate limits. No side effects or constraints are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no fluff. It is appropriately sized for a simple list tool and front-loads the action verb 'List'.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (1 optional param) and the presence of an output schema (not shown), the description is minimally adequate. However, it could add context about what the returned list contains (e.g., IDs, names, metadata) to improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one undocumented parameter 'api_key' (0% schema description coverage). The description does not mention this parameter at all, failing to explain its purpose, format, or when it is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists 'open-licence geospatial data sources wrapped by Gods Eye,' providing a specific verb and resource. However, it does not explicitly distinguish from sibling tools like 'check_data_provenance' or 'situational_query,' though the name implies a listing function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description offers no guidance on when to use this tool versus alternatives. No exclusions, prerequisites, or context are provided, leaving the agent to infer usage solely from the name.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• situational_queryA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must fully disclose behavior. It states the tool returns a structured plan and does not perform actual data fetching (Pro-tier). However, it does not mention whether the tool is read-only, has side effects, or requires specific authentication (api_key parameter is listed but not explained). The disclosure is not complete, but it provides some useful behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (three sentences) and effectively front-loaded: first sentence states purpose, second lists input variations, third describes output and a limitation. There is no redundancy or wasted wording.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (so return values are defined elsewhere) but schema coverage is 0% and no annotations exist, the description partially compensates by explaining the output conceptually. However, it fails to specify that bbox and aoi_name are mutually exclusive, does not detail the exact format of the bound box, and does not explain the meaning of all parameters. The description is adequate but leaves notable gaps for an agent to fully use the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, so the description must add meaning. It explains that 'query' is natural language and describes the optionally alternatives of 'bbox' (with format minLon,minLat,maxLon,maxLat) and 'aoi_name'. However, it does not clarify 'time_window', 'preferred_sources', or 'api_key', leaving gaps in understanding those parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to plan a geospatial situational-awareness query. It specifies the inputs (natural-language query plus bbox or aoi_name) and outputs (structured plan). This distinguishes it from sibling tools like care_membrane_policy or list_data_sources, which have different functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains how to use the tool (provide query and either bbox or aoi_name) and notes that tile fetching is a Pro-tier feature. However, it does not explicitly state when to use this tool versus alternatives or provide any context about prerequisites or exclusions, leaving the agent to infer usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• sign_data_provenance_attestationA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description partially explains behavior: it produces HMAC-SHA256 signed JSON, a public verify URL, and optional PDF. It mentions offline validation but does not disclose auth requirements (api_key), mutation semantics, or error handling. Key parameters like entity_name and stack_description are not described.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with a clear overview sentence and bullet points for key parameters. It is front-loaded with purpose, but the bullets could be more consistently structured. No wasted sentences.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 6 parameters, no annotations, and an output schema, the description is incomplete. It does not explain the output structure, the required input parameters (entity_name, stack_description, api_key), or the prerequisite step (check_data_provenance). Adequate for basic understanding but lacks detail.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning for 3 out of 6 parameters (open_licence_score, findings_csv, include_pdf_base64) by explaining their source and format. However, it omits semantics for entity_name, stack_description, and api_key, and schema coverage is 50% (3 of 6 parameters mentioned).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool generates a cryptographically signed data-provenance attestation, specifying the verb 'Generate', the resource 'attestation', and the context 'geospatial stack'. It distinguishes from siblings like check_data_provenance and list_data_sources by focusing on signing, not checking or listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly mentions use cases such as CSRD E3/E4 evidence packs, GDPR Article 30 records, and municipal mandates, indicating when to use. It also hints at external validation by auditors, but does not directly contrast with siblings or state when not to use this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_data_provenanceA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description implies a read-only analysis tool. It describes the action and target but lacks details on side effects, required permissions, or whether the api_key parameter affects behavior. This is adequate but not thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences: the first clearly states the action, the second lists use cases. No unnecessary words, well-structured and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains input and use cases, and an output schema exists so return details are covered. It lacks mention of potential error conditions or internet access requirements, but is sufficient for most scenarios.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage. The tool description adds meaning for the 'stack_description' parameter ('geospatial stack description') but does not mention the optional 'api_key' parameter. Partial compensation for a low-coverage schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool flags proprietary dependencies and suggests open-licence alternatives for a geospatial stack. This distinguishes it from siblings like list_data_sources or sign_data_provenance_attestation, which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description specifies when to use the tool: civilian projects with GDPR, data-residency, or open-data mandates. It provides clear context but does not explicitly exclude scenarios or mention alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.