Server Quality Checklist

Tool Scores

• search_propertiesC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the behavioral burden, but it only reveals that results display on a map. It does not explain matching semantics, result limits, pagination, or side effects, leaving significant ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with no filler, and both sentences contribute useful information. While the brevity is arguably insufficient for the tool's complexity, conciseness itself is well handled.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and ten parameters, the description is far from complete. It fails to describe result format, map data details, defaults, or required constraints beyond the schema, making it inadequate for an agent to fully understand the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema descriptions cover only 40% of the parameters, and the tool description names no parameters or criteria details. With low coverage, the description was expected to compensate but does not, leaving six parameters without any explanation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches a property database for matching criteria, with results displayed on an interactive map. It does not explicitly differentiate itself from sibling tools like search_sales or search_comps, so it stops short of a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description does not mention exclusions, prerequisites, or recommend other tools for different search needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_sale_historyB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavioral traits. It only states 'get all recorded sales', which essentially restates the purpose without explaining return format, sorting, pagination, or matching semantics. This is insufficient for a tool with no other behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence with no unnecessary words. It is concise and front-loaded, effectively communicating the core action without noise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple get-by-address tool, the description is minimally sufficient, but the absence of an output schema and annotations means it could explain what 'recorded sales' includes (e.g., dates, prices, parties). This is a notable gap, so the description is adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers 100% of parameters with descriptions (city, address, borough) and enums for city and borough. The tool description adds no additional parameter semantics, but the high schema coverage means the parameters are already documented well enough, warranting the baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves all recorded sales for a specific property, using a specific verb and resource. It does not explicitly distinguish from the sibling tool 'search_sales', but the phrase 'for a specific property' gives scope that hints at the difference.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'search_sales' or 'search_properties'. There is no mention of scenarios, prerequisites, or exclusions, leaving the agent to infer when this tool is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_tax_benefitsB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It only states that the tool returns certain benefit types, which is essentially a restatement of output. It does not disclose any side effects, prerequisites, or behaviors such as how property identification works or what happens if no benefits exist. The mismatch between 'NYC' and the city enum further obscures behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, front-loading the core action and then listing the returned benefit types. It contains no redundant wording or unnecessary details, achieving high efficiency for its limited scope.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description does not adequately explain how to construct a query (e.g., address vs. block/lot) or how to handle the Philadelphia option, and it lacks output schema to clarify the response structure. This makes it insufficiently complete for a tool with six parameters and no structured output schema, especially given the NYC/Philadelphia ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides descriptions for all six parameters (100% coverage), so the baseline is 3. The description adds no parameter-specific meaning and instead introduces a potential conflict by insisting on 'NYC' while the schema permits both nyc and philadelphia. This could mislead an agent on the valid values for the 'city' parameter, reducing the value of the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('Get'), a resource ('tax exemptions and abatements'), and lists specific benefit programs (421a, J-51, ICAP, STAR). This distinguishes it from sibling tools like get_property and search_sales. However, the description specifies 'NYC property' while the schema allows city=philadelphia, creating a minor inconsistency that is not addressed.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for retrieving tax benefits but provides no explicit guidance on when to use it over alternatives such as search_properties or get_property. There are no exclusions or alternative recommendations, relying on the tool name and description to imply context. This meets the 'implied usage' level but lacks clear context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_compsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses that results 'display on an interactive map' and mentions 'similarity scoring,' which are useful behaviors. However, it does not clarify read-only nature, data sources, or any limitations, which is a moderate gap given the lack of annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary purpose, and includes a notable behavior (interactive map). Every word earns its place with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 6 parameters and no output schema, so the description should compensate for missing return-value details. It mentions map display and similarity scoring but does not describe the returned data structure (e.g., property details, score interpretation). The description is adequate for a straightforward search tool but leaves room for clarifying output semantics.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds no additional parameter semantics beyond what the schema already provides (e.g., city, address, borough, limit). It does not name or explain any parameters, but this is acceptable given full schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Find comparable sales for a subject property with similarity scoring.' This uses a specific verb ('find') and resource ('comparable sales') and distinguishes it from sibling tools like search_sales by emphasizing similarity scoring and subject-property focus.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for property valuation or comps analysis ('for a subject property'), but it does not explicitly state when to use this tool instead of alternatives like search_sales or provide exclusions. Usage context is inferable but not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_propertyA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations to rely on, the description discloses key behaviors: it lists the types of data returned and states the result 'displays on an interactive map.' This goes beyond a minimal 'get' statement, though it stops short of explicitly noting read-only status or potential edge cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the verb and resource, and every word adds value. No filler or redundant repetition of the title 'Get Property.'

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple get tool with 3 parameters and no output schema, the description covers the core purpose and output presentation. It lacks explicit differentiation from sibling tools but is otherwise complete for typical use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% description coverage for all three parameters, so a baseline of 3 applies. The description mentions fields like owner and zoning but does not add meaningful parameter-level syntax or relationships beyond the schema's one-line labels.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the tool's function: 'Get detailed property information including owner, units, zoning, coordinates, and rent stabilization analysis.' This distinguishes it from sibling search tools by emphasizing a single property's comprehensive detail, though it does not explicitly name siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage—call this when you need detailed data for a specific property—but provides no explicit guidance on when to choose it over search_properties, search_sales, or other siblings. No exclusions or alternative scenarios are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_deed_partiesA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the full burden. It does disclose what the tool returns (buyer/grantee and seller/grantor) and the data source (ACRIS deed records). However, it implies a read-only lookup without explicitly stating it, and it fails to address the apparent discrepancy between 'NYC property' and the schema's 'philadelphia' enum value, leaving room for confusion about scope.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with the primary action front-loaded. No filler or repetition. Every clause contributes, and the 'Use when...' clause is a practical addition. This is appropriately concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 3 parameters, no output schema, and no annotations. The description addresses purpose and when-to-use, but does not explain return format, error behavior, or the city/borough relationship. Given the missing structured metadata, the description should be more explicit about edge cases and the mismatch in the city enum.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds context that the tool works with deed records, but it does not elaborate on how to format the address or when the optional borough parameter should be included. The city enum discrepancy is not clarified, so the description adds minimal value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'Look up buyer (grantee) and seller (grantor) for a NYC property from ACRIS deed records.' This clearly distinguishes it from sibling tools like get_property or search_sales, which are broader. The only minor confusion is the schema allows 'philadelphia' while the description says 'NYC,' but the core purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states an explicit trigger: 'Use when the user specifically asks who bought or sold a property.' This gives clear context for when to select the tool. However, it does not mention alternatives, exclusions, or cases where it should not be used (e.g., when a broader property search is needed), so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_salesA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals that results include price_per_unit and price_per_sqft and display on an interactive map, providing useful output context. However, it does not explicitly confirm that the operation is read-only or mention any rate limits or permissions, though this is largely implied by 'Search'.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loads the purpose, and each sentence earns its place. It is concise, informative, and avoids unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core purpose, filter capabilities, return fields, and display behavior, which is strong for a search tool. However, it claims 'in NYC' while the schema also supports Philadelphia, and it does not mention the required 'city' parameter or the NYC-only 'borough' field. These gaps are partially addressed by the schema, so the overall completeness is strong but not perfect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers 100% of parameters with descriptive text, so the description adds little beyond what the schema already provides. Mentioning 'neighborhood, price range, building type' simply restates existing schema fields, and 'and more' is vague. The description does not clarify parameter syntax or add new semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as a search for recent property sales in NYC, which distinguishes it from sibling tools like search_properties (broader property search) and get_sale_history (historical sales for a specific property). It also mentions filtering options and return fields, making the purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for finding recent sales with filters, but it does not explicitly state when to use this tool over alternatives such as search_comps or search_properties. No exclusions or alternative tool names are mentioned, so the guidance is only implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.