Server Quality Checklist

Tool Scores

• profile_columnA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It states the core behavior (shows frequent values with counts) and implies a read-only operation. However, it does not disclose edge-case behavior such as handling of nulls, the exact meaning of the 'top' parameter, or any limitations on output size. This is adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the action and followed by a purposeful rationale. Every word earns its place, with no fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple and has an output schema, so return values are covered. However, the lack of parameter explanations and the omission of usage context around dataset_id/domain leaves gaps. The description is complete enough for a basic understanding but not for confident invocation without further inference.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain parameter meaning. It only hints at 'one column' and 'most frequent values' (implying top), but does not explain 'dataset_id', 'domain', or 'top' explicitly. The description fails to compensate for the complete lack of schema descriptions for any parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Show the most frequent values of one column with counts' — a specific verb, resource, and output. It also distinguishes itself from siblings by focusing on column vocabulary rather than dataset search, schema retrieval, or arbitrary querying.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a clear use case: 'a quick way to learn a column's vocabulary before writing a where clause.' This implies when to use it relative to querying but does not explicitly name alternatives or exclusions. It gives enough context to know when it is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_datasetsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It communicates that this is a read-only search operation and outlines the return fields, which is useful. However, it does not disclose potential behaviors like default domain scope, result limits, or rate limiting, leaving some ambiguity about execution expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loads the core action, and has no wasted words. The second sentence adds essential workflow guidance without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's purpose, return fields, and next steps, but falls short on parameter semantics and lacks any caveats about pagination, default domain, or result size. While an output schema exists, the absence of parameter descriptions and annotations makes the tool incomplete for an agent to invoke correctly without external knowledge.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It only mentions 'keyword' which maps to the 'query' parameter, but omits 'limit' and 'domain' entirely. The agent cannot infer the meaning or defaults of these parameters from the description, creating a significant gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('Search') and resource ('Socrata portal's catalog'), and explains what it returns (dataset ids, names, descriptions). It also distinguishes itself from siblings by explicitly linking the returned id to get_schema/query_dataset, establishing its role as the entry point.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides workflow context by stating 'Use the returned id with get_schema/query_dataset', which implies when to use this tool and what to do next. However, it does not explicitly mention when not to use it or alternative search strategies, leaving a small gap in exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_schemaA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses what is returned (name, description, column names/types) and prefacing querying implies a read-only operation. However, it doesn't mention return format, pagination, or any other behavioral traits beyond the basics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main purpose and followed by a practical usage hint. Every sentence earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity and the presence of an output schema, the description covers the essential context: what the tool does and when to use it. The only gap is the meaning of the domain parameter, which is more of a parameter semantics issue than overall completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description should compensate for parameter meaning. It implies context for dataset_id via 'dataset's' but leaves domain completely unexplained. The schema default for domain is helpful but the description adds no value in clarifying its purpose.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Get' and clearly states the resource: a dataset's name, description, and column names/types. This distinguishes it from siblings like search_datasets, query_dataset, and profile_column, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs to call the tool before querying, explaining that this ensures column names in SoQL clauses are exact. This is a clear usage context, though it doesn't explicitly list exclusions or alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_datasetA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full behavioral burden. It discloses the 1000-row cap per call, which is valuable, but it does not mention pagination via offset, error behavior, or authentication requirements. The transparency is moderate but incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with purpose, and includes practical examples without any waste. Every word contributes to understanding the tool's function and syntax.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description effectively covers the core query capabilities and the row cap, which are the most critical aspects. The existence of an output schema covers return values, so the description doesn't need to explain them. Minor gaps like offset/limit semantics are not explicitly addressed, but they are intuitive from parameter names.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description compensates by providing concrete examples for select, where, group, and order. Other parameters like dataset_id, domain, limit, and offset are self-evident from their names and defaults, so the overall parameter understanding is good.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs a SoQL query against a dataset, which is specific and distinct from sibling tools like search_datasets, get_schema, and profile_column. The examples reinforce this unique purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for querying data with SQL-like clauses, and the examples demonstrate syntax. It does not explicitly state when not to use it or mention alternatives, but the context is clear enough for a knowledgeable agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.