Server Quality Checklist

Tool Scores

• building_complaintsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses key behavioral details like substring matching and exact match for house number, and states 'Keyless', but no annotations exist to supplement. Could mention data freshness or pagination.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficient sentences, front-loaded with purpose and output, no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose and output structure, but lacks guidance on sibling tools, error cases, or detailed output field descriptions. Adequate for a simple lookup.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so description adds limited value. It reiterates required parameters and matching details, but doesn't enrich semantics beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as retrieving HPD complaints for a building, specifies the dataset, and distinguishes it from siblings like building_violations by focusing on complaints.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to provide house number, street, and borough, and describes the output. Lacks explicit when-not-to-use or comparisons with siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• eviction_lookupA
  4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It lacks explicit statements about read-only nature, rate limits, or auth requirements. While the function is implied non-destructive, these traits are not disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is succinct, front-loads the key purpose and distinction, then explains parameters and output. Every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers the main purpose, required parameter combinations, matching behavior, and output fields. Missing pagination details and the meaning of 'Keyless' is unclear, but overall sufficient for a lookup tool with no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% but the description adds valuable constraints: at least one of court_index_number or address/borough is required, and address matching is case-insensitive substring. It also lists the return fields, which is not in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool looks up marshal-executed evictions from a specific dataset, distinguishing it from filings or pending cases. It specifies the lookup methods and return fields, making it distinct from sibling tools which cover other property records.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly warns against using this for filings, warrants, or pending cases, and states that at least one of court index number or address/borough is required. It provides clear context for when to use, though it doesn't compare directly with siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• landlord_portfolioA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description discloses case-insensitive substring matching, resolution to buildings with details, current registration only, and the 'Keyless' access. It implies a read-only query.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured but slightly verbose. It efficiently covers purpose, process, usage, and limitations in a few sentences. Could be trimmed slightly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description fully explains the return values (address, borough, zip, BIN, registration dates, matched contact) and data source. Context is thorough for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds meaningful guidance: name is case-insensitive substring, pass fullest name to avoid over-matching, limit default 50, borough accepts multiple formats.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it is the reverse of who_owns, listing buildings under a landlord/corporation/officer/agent name. It specifies the data source (HPD Registration Contacts) and distinguishes from sibling tools like who_owns and landlord_litigation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It advises starting from names surfaced by who_owns or landlord_litigation, explaining when to use this tool to uncover hidden connections. It does not explicitly state when not to use, but provides clear context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• landlord_litigationA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral disclosure. It details return fields (case type, open date, status, etc.), matching behavior (case-insensitive substrings for respondent and street), and notes the tool is 'keyless'. No side effects or rate limits are mentioned, but for a read-only lookup this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (about 5 sentences), front-loaded with the dataset ID and purpose, and contains no redundant information. Every sentence serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema, the description adequately explains return values and includes a summary by status. It covers the key usage scenarios and matching behavior, making it complete for this lookup tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for all 6 parameters. The description adds value by explaining required parameter combinations (building vs respondent), substring matching details, and borough shorthand (MN/BX/BK/QN/SI).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: looking up HPD housing litigations by building or respondent. It specifies the data source (dataset 59kj-x8nc) and distinguishes from sibling tools like building_violations which handle different types of records.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains how to use the tool: look up by building (house_number + street + borough) or by respondent name, with at least one required. It does not explicitly state when not to use it or name alternative tools, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• building_violationsA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses: returns a server-side per-class count summary plus most recent violations, street matched as uppercase substring, house number matched exactly, keyless, optional token raises rate limit. This is excellent behavioral detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is relatively long but every sentence adds value. It front-loads the purpose and dataset ID, then explains parameters and behavior. Could be slightly more concise, but no extraneous information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description compensates by explaining the two-part return (class count summary + recent violations). It also covers matching behavior, required parameters, and optional token. For a tool with 7 parameters, this is complete and leaves no major gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, providing a baseline of 3. However, the description adds significant semantic value beyond schema: explains matching logic for street and house_number, clarifies the class summary, and notes that the optional token only affects rate limits. This greatly aids correct invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns HPD Housing Maintenance Code violations for one building, with a specific dataset reference. It distinguishes from siblings by focusing on violations and mentioning the class count summary and recent violations. The verb 'get' is implied, and the resource is explicitly 'violations for one building'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description tells users to provide house number, street, and borough, and mentions an optional token. It does not explicitly state when to use this tool versus siblings like building_complaints or who_owns, but the purpose is clear enough that an agent can infer it is for violations only.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• who_ownsA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses behavioral traits: it joins specific datasets, returns contact names and addresses, reflects HPD filings (which may lag reality), and notes it requires no API key. No contradictions exist.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with three sentences that front-load the purpose and provide essential details without fluff. Every sentence adds value: purpose, data join, input/output, and limitations.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with three parameters and no output schema, the description sufficiently explains inputs, outputs, data source, and limitations. It covers all necessary context for an AI agent to decide when and how to use it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema fully covers all three required parameters with descriptions (100% coverage). The description adds value by framing them as user instructions ('Give the house number...') and mentions matching behavior indirectly (substring matching for street is detailed in the schema).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states exactly what the tool does: it identifies who is on file with HPD, acts as the 'who do I actually serve' tool, and explains the data join process. It clearly distinguishes from sibling tools like building_violations by focusing on ownership contacts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly lists required inputs (house number, street, borough) and mentions the output (registrations and contacts). It provides a caveat about data lag, but lacks explicit guidance on when to use this tool versus sibling tools like landlord_portfolio.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.