Server Quality Checklist

Tool Scores

• remove_companyB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It does add one useful behavioral detail: the company slug is matched case-insensitively. However, it does not disclose what happens if the company is not tracked, whether removal is reversible, or what the output or error behavior will be.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief and front-loaded, with no filler. The core action appears first, and the parameter explanation is compact and directly useful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter removal tool, this is minimally adequate. The agent can identify the argument and matching behavior, but the description omits edge-case behavior like removing a non-existent company and any prerequisites for removal, leaving some ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only the string type for 'company,' with 0% schema description coverage. The description compensates by explaining that company is the slug and is matched case-insensitively, adding meaningful semantics beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action and resource: 'Remove a company from the tracked company list.' This clearly identifies the tool's function and is distinct from siblings like list_companies or add_company, though it does not explicitly name an alternative.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided about when to use this tool versus alternatives like add_company or list_companies. The usage context is only implied by the tool name and the phrase 'tracked company list,' with no exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• iterate_listA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility for behavioral disclosure. It explains keyword matching semantics and the hours window, but omits side effects, what happens when no companies are tracked, behavior with no matches, and potential execution cost of checking every company.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a clear one-sentence summary followed by compact, information-dense parameter documentation. Every sentence contributes meaningful detail with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-optional-parameter tool with no output schema or annotations, the description covers the main behavior and parameters. However, it leaves edge cases unspecified (no tracked companies, no saved department, no matches) and does not describe the return format, which is more burden because no output schema exists.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only types and defaults with 0% coverage, so the description must compensate. It fully explains keyword semantics: case-insensitive matching against job title/department, comma-separated OR behavior, blank meaning set_department, and 'all' meaning every department. It also clarifies the hours parameter's meaning and default.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the operation: iterate over tracked companies and return jobs posted in the last 24 hours. It is specific and uses a verb+resource pattern, but it does not explicitly differentiate itself from siblings like list_jobs or get_latest_jobs, relying instead on the 'tracked company' and time-window phrasing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains how to use parameters (keyword, hours) but never states when to prefer iterate_list over sibling tools, nor does it give exclusions or alternatives. There is no direct guidance on tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_companiesA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It adds useful context that the list is the target of iterate_list sweeps and is populated by add_company or any list_jobs lookup, revealing the list's provenance and mutability. It does not disclose return format, ordering, or pagination, but for a read-only 0-param listing the core behavior is clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The first sentence is front-loaded and precise. The second sentence adds valuable context but is grammatically awkward ('These are the boards iterate_list sweeps'), which costs it a top conciseness score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple, zero-parameter, read-only listing tool with no output schema, the description conveys the core behavior and the list's relationship to sibling tools. The main omission is the return value shape (e.g., IDs vs names), which is not specified anywhere and could require inference by the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and schema coverage is vacuously 100%, so the baseline is 4. The description therefore needs no parameter details and does not add misleading parameter information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence, 'Show every company currently saved in the tracked company list,' uses a specific verb and resource and clearly scopes the result to the tracked-list with no filtering. This distinguishes it from job-focused siblings like list_jobs and mutation tools like add_company/remove_company.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clarifies that this is the board list used by iterate_list and how companies enter it, which implies when to use it (to inspect the tracked working set). However, it never explicitly tells the agent when not to use it or names an alternative for other needs, so guidance is implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_latest_jobsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden. It explains that results are sorted by recency, top_k controls count, and department is an exact filter, but it is silent on return format, error behavior, and whether the company must already exist in the system.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a single purpose sentence, followed by a compact Args list. Every sentence contributes either behavioral meaning or parameter semantics; there is no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has an output schema, so return-value documentation is not required, and all parameters are covered. However, it doesn't clarify how this relates to sibling company-management tools (e.g., whether add_company is a prerequisite), nor does it state what happens for invalid slugs or missing departments.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the parameter explanations in the Args block are essential. Each parameter gets a meaningful explanation beyond the bare schema: the slug format, the 'exact department name' filter, and the default top_k behavior.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The opening sentence names the operation precisely: get the most recently posted jobs for a company, optionally filtered by department. This distinguishes it from siblings like get_job or list_jobs, but it never explicitly contrasts those alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context on what the tool does and how to use the optional filters ('Leave blank to include all departments'), but it doesn't say when to prefer this over list_jobs or get_job, nor mention any prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• add_companyA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It adds useful traits: no job lookup occurs, names match case-insensitively, and tracking can be done ahead of time. However, it does not disclose duplicate behavior, persistence effects, or what happens for an invalid or unknown slug.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three tight sentences plus an Args line with no filler. The main purpose is front-loaded, the timing context follows, and each sentence adds a distinct piece of needed information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter tool with an output schema, the core purpose, parameter source, and matching behavior are covered, so the definition is largely complete. It falls slightly short by not saying whether duplicate companies are ignored/error and whether the slug must already exist in Ashby.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% description coverage and one generic string parameter, so this description must compensate. It does so by defining company as 'the company's slug from jobs.ashbyhq.com/{slug}', giving the agent an exact format and source. This is decisive for choosing the right value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific action and object: 'Track a company by name' and explicitly scopes it as 'without looking up its jobs yet,' which distinguishes it from list_jobs and job-lookup siblings. It names the exact resource and the intent.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives a concrete usage condition: companies are normally tracked automatically when list_jobs runs, but this tool is for adding one ahead of time. This is clear when-to-use guidance, though it does not explicitly list exclusions or point to alternatives like check_company or remove_company.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_jobA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. The verb 'Get' implies a read-only retrieval, and the description states that results include the full description. However, it does not explicitly confirm non-mutation, auth requirements, or any rate limiting; for a simple getter this is adequate but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the primary purpose, followed by a clean two-argument breakdown. Every line provides useful information with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a straightforward read tool with no output schema and no annotations, the description covers the core purpose and parameter semantics well. It does not describe the exact return shape beyond 'full details including its description,' but this is acceptable given the tool's low complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, and it does. It explains job_board_name as the company slug from the URL pattern jobs.ashbyhq.com/{slug} and job_id as returned by list_jobs, adding essential provenance and format details absent from the bare schema titles.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Get') and resource ('full details for one specific job posting') and explicitly notes the inclusion of the job description. This distinguishes it from sibling tools like list_jobs and get_latest_jobs, which target collections or recent postings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The parameter note that job_id is 'the job's id, as returned by list_jobs' establishes a clear workflow: call list_jobs first, then get_job for details. This provides clear context for when to use the tool, though it does not explicitly name alternatives or exclusion conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_companyA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations present, the description carries the full behavioral burden and does so thoroughly. It discloses three non-obvious behaviors: the answer reflects board existence rather than posting availability (yes with open_jobs of 0), on_ashby is null—never false—when Ashby is unreachable, and plain names are retried as normalized slugs. These edge cases are exactly what an agent needs to interpret results correctly.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Approximately four sentences, front-loaded with the core purpose, followed by two high-value semantic clarifications and a clean Args block. Every sentence earns its place; there is no filler, and the structure makes the behavioral caveats easy to absorb.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one required parameter, no annotations, no output schema), the description is nearly complete. It documents input formats, the key output fields (on_ashby, open_jobs, job_board), and failure semantics. The only residual gap is that the exact return shape/type is left implicit rather than stated, and explicit sibling selection guidance is absent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the schema merely types company as a string. The description fully compensates by explaining that the input accepts either a jobs.ashbyhq.com/{slug} slug or a plain name, that names are retried across normalized slugs, and that the resolved slug is surfaced as job_board. This is rich meaning the schema alone would never convey.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence states a specific verb and resource: 'Check whether a company hosts its job board on Ashby.' This is immediately distinguishable from the sibling set — it is a presence verification tool, not a job-listing tool (list_jobs, get_job) or a company-management tool (add_company, remove_company).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage context is implied rather than stated. The line 'comes back as job_board for use with the other tools' suggests this is a prerequisite step before chaining other Ashby tools, and the existence-vs-postings clarification tells the caller how to interpret the result. However, there is no explicit when-to-use versus when-not-to-use guidance, alternative routing, or exclusions relative to siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_jobsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral disclosure burden. It does well by explicitly describing case-insensitive matching, comma-separated keyword handling, the 'any keyword' match semantics, the blank-keyword fallback to set_department, and the 'all' escape hatch. It also mentions the returned count. Minor omissions like error behavior or pagination are acceptable given the tool's simplicity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: two introductory sentences establish the core action and return value, followed by a concise argument list. Every sentence adds meaningful information, and the most important information about what the tool does is front-loaded. No filler or redundant restatement of the tool name.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no annotations and no output schema, the description provides solid coverage of its inputs, behavior, and a key return value (total count). It lacks an explicit description of the posting object's shape, but the phrase 'list open job postings' implies a collection of job records, and the sibling set provides enough context for an agent to use the tool correctly. Overall, the description is nearly complete for a simple list operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides only types and a default, with 0% schema description coverage. The description fully compensates by defining job_board_name as the company's slug from jobs.ashbyhq.com/{slug} and by thoroughly explaining keyword semantics, including filtering target, case-insensitivity, multiple keywords, and special reserved values. This gives an agent everything needed to fill both parameters correctly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'List open job postings for a company's public Ashby job board.' It also clearly states it returns the total number of open postings, and the keyword filter behavior distinguishes it from sibling tools like get_job or get_latest_jobs. This is unambiguous and not a tautology.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use the tool: for any public Ashby job board, optionally filtered by keyword. It also gives concrete usage guidance for the keyword parameter, including the special values 'blank' and 'all', and references set_department as a way to persist a department filter. It does not explicitly name alternative tools or say when not to use it, but the context strongly implies the intended use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• set_departmentA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden, and it delivers: it explains comma-separated keyword matching, case-insensitive word-start matching, OR semantics across keywords, and that an empty string clears the default. It also uses 'save' and 'go back to searching every department' to disclose the persistent state change.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose and then expands into matching semantics, examples, and parameter details. Each sentence adds necessary information for correct use, and the structure is easy to scan despite the length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, parameter syntax, matching behavior, edge cases, and clearing behavior. Since an output schema is present and the tool has only one simple parameter, nothing essential is missing for an agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description fully compensates by explaining the single department parameter in depth: how to pass multiple keywords, how matching works, examples, and the special empty-string clearing behavior. This far exceeds the bare schema definition.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific action: 'Set the default department or keyword used when searching and iterating.' This clearly identifies the resource (the saved filter) and the verb (set), and the tool is readily distinguishable from sibling tools like list_jobs and get_job.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states the tool affects searching and iterating, giving clear context for when it applies. It does not explicitly name alternative tools or conditionally exclude them, but the usage context is strong enough for an agent to infer when to call this setter versus listing or fetching jobs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.