Server Quality Checklist

Tool Scores

• get_jobA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnly, idempotent, and non-destructive behavior, so the description does not need to repeat those. It adds useful context about what is read (locations, description, pay ranges) and how addresses work, but does not disclose deeper behavioral details like response size or optional parameter effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the primary purpose, and contains no fluff. It efficiently communicates the essential behavior and the source of required identifiers.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema and rich annotations, the description is largely complete. It explains the core read intent and required identifiers, though it could more explicitly connect the optional include_compensation and description parameters to the overall behavior, but the schema covers these.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers most parameter descriptions (75% coverage), and the description adds the meaningful context that both board and job_id are returned by search_jobs. However, it does not explain the include_compensation parameter or the description format options beyond what the schema already states, so it adds limited value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: reading one Ashby posting in full, including locations, description, and pay ranges. It distinguishes itself from the sibling search_jobs tool by emphasizing it reads a single posting rather than searching, and references how Ashby addresses postings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states that both the board and identifier should come from search_jobs, giving clear prerequisite context. However, it does not explicitly mention when to avoid this tool or compare it with alternatives like compare_compensation, so it lacks explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_filter_valuesA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint=false, covering the safety profile. The description adds that output includes counts and a 'nothing' bucket, and stresses that values are board-specific—useful behavioral context beyond the annotations. It does not mention rate limits or auth, but that is not a major gap here.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two tight sentences: the first states the action and output, the second provides a crucial context warning. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present, the description need not detail return values. It covers the action, output composition, and a board-specific caveat. However, the unclarified facet parameter is a small gap, keeping it from a perfect score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50%. The 'board' parameter is already described in the schema, while the 'facet' parameter has only an enum without explanatory text. The description does not clarify the facet parameter's meaning or how it relates to the tool's purpose, so it adds no value beyond the schema for parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists the actual filter words used by an Ashby board, along with counts and a 'nothing' bucket. It uses a specific verb 'list' and identifies the resource (board's filter wordings), distinguishing it from siblings that search, get jobs, or compare compensation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains a key use case: since each board has its own vocabulary, writing filters with another board's terms yields empty results. This implies using the tool to obtain board-specific terms before filtering, offering clear context. It does not explicitly name alternatives or exclusions, so it does not earn a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_jobsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint='false', so the safety profile is covered. The description adds useful behavioral context about client-side filtering, the reporting of undeclared field counts, and the absence of descriptions in rows, which is valuable beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences and front-loaded with the core purpose. Every sentence conveys essential information: the action, the constraint (companies required), and a behavioral note about filtering and output, plus a pointer to get_job. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 18 parameters and an output schema, the description doesn't need to explain every parameter or return value. It covers the critical context: company requirement, client-side filtering, undeclared counts, and no descriptions in rows. With an output schema present and rich annotations, this is sufficient, though it omits details about pagination that the schema does provide.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 67% description coverage with detailed parameter descriptions, so the description doesn't need to add much. The description does mention that companies are required, but that's already in the schema. It provides no additional parameter-level semantics, so a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches open postings at named companies on Ashby, with a specific verb and resource. It distinguishes itself from siblings like get_job (read one posting) and list_filter_values by indicating its scope and the requirement to specify companies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains that companies are required, which is a key usage constraint, and explicitly points to get_job for reading full descriptions. It also notes that filters run client-side, implying this tool is used for cross-company searches rather than source-level filtering. However, it doesn't contrast with all siblings (e.g., list_filter_values) explicitly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_compensationA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite having readOnlyHint, idempotentHint, and openWorldHint annotations, the description adds non-obvious behavior: 'Nothing is converted between currencies, summed across components or averaged' and 'a third of the postings publish no range at all, and they are named rather than dropped.' This is valuable context beyond what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tight sentences, front-loaded with the core action, then crucial caveats. Every clause earns its place with zero waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present and strong annotations, the description sufficiently covers the tool's behavioral quirks. It doesn't mention filtering options like team or department, but those are documented in the schema, so for a comparison tool this is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema descriptions already cover the parameters well (~75% coverage), including explaining component and interval behavior. The description reinforces the 'one component/period at a time' rule but doesn't add new parameter-level details beyond what the schema already states.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource ('Put the pay ranges of one Ashby board side by side') and clearly distinguishes this from sibling tools like search_jobs or get_job. It also provides scope constraints (one component, one period), making the tool's function unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool: to compare pay ranges across postings on a board, with explicit constraints on components and periods. It doesn't explicitly name alternatives or state when not to use, but the context is clear enough given the sibling tools are distinct.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• resolve_boardA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare read-only, idempotent, non-destructive behavior. The description adds the behavioral detail that the tool 'reports every form that answered' and shows 'spellings that were tried', which goes beyond the safety profile and explains its observational nature.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exactly two sentences, front-loaded with the core action, and the second sentence provides essential usage guidance. Every word earns its place with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With a single parameter, an output schema, and comprehensive annotations, the description covers purpose, usage conditions, and key behavior. The tool is simple, and the description fully equips an agent to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a description for 'name' that already mentions it accepts a company name or an existing token. The tool description does not add new parameter information beyond the schema, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Turn') and resource ('company name into the token'), clearly stating the tool's function. It distinguishes itself from sibling search_jobs by noting that search_jobs already does this, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'call this when a name returns nothing' and names the alternative (search_jobs). This provides clear context for deciding between tools, which is exactly what usage guidelines should do.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.