Server Quality Checklist

Tool Scores

• list-organizationsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses the tool retrieves organizations with changes to tax-exempt status or filing information, which adds behavioral context. However, it doesn't mention important behavioral aspects like pagination, rate limits, authentication requirements, or what constitutes 'updated' (metadata changes vs. substantive changes).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections: purpose, parameters, return values, and usage guidance. It's appropriately sized for a single-parameter tool. Minor improvement could be made by front-loading the most critical information more aggressively, but overall it's efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter read operation with no output schema, the description provides adequate context about what data is returned. However, without annotations and with no output schema, it should ideally provide more detail about response format, pagination, or limitations. The description compensates somewhat by listing return data categories but doesn't fully address the lack of structured output information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents the single 'since' parameter. The description repeats the parameter information but doesn't add meaningful semantic context beyond what's in the schema. The baseline of 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and resource 'nonprofit organizations from the IRS database' with specific scope 'updated since a specified date'. It distinguishes from siblings by specifying this is for bulk organization data or tracking recent changes, unlike lookup/search/check tools that likely focus on individual organizations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: 'to get bulk organization data or track recent changes to nonprofit status'. However, it doesn't explicitly state when NOT to use it or name specific alternatives among the sibling tools (charity-lookup, charity-search, public-charity-check).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• charity-lookupA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the data source (IRS database) and lists the types of information retrieved, which adds useful context. However, it does not mention behavioral aspects like rate limits, authentication needs, error handling, or whether the lookup is cached, leaving gaps in transparency for a tool with no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: the first sentence states the core purpose, followed by a bulleted list of retrieved information for clarity, and ends with usage guidance. Every sentence earns its place without redundancy, making it efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (single parameter, no output schema, no annotations), the description is moderately complete. It explains what the tool does and what information it retrieves, but lacks details on output format, error cases, or integration with sibling tools. This is adequate but has clear gaps for a tool without annotations or output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, clearly documenting the 'ein' parameter with format and examples. The description adds no additional parameter semantics beyond what the schema provides, such as explaining EIN significance or validation rules. This meets the baseline of 3 since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Look up detailed information') and resource ('charity or nonprofit organization'), distinguishing it from siblings like 'charity-search' (likely broader search) and 'list-organizations' (likely listing multiple). It specifies using EIN as the identifier, making the purpose explicit and differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use it ('when you need complete details about a specific charity'), but does not explicitly state when not to use it or name alternatives among the sibling tools. This gives adequate guidance but lacks explicit exclusions or comparisons to other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• public-charity-checkA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does disclose key behavioral traits: it's a verification/checking tool (not a mutation), returns specific information (public charity status, tax deductibility, EIN confirmation), and mentions the IRS context. However, it doesn't cover potential limitations like rate limits, error conditions, or data freshness.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized with four sentences that each serve a purpose: defines the tool's purpose, provides IRS context, lists return values, and gives usage guidance. It's front-loaded with the core purpose, though the structure could be slightly more streamlined.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter verification tool with no output schema, the description provides good contextual completeness: it explains what the tool does, what it returns, and when to use it. The main gap is the lack of output format details, but given the tool's relative simplicity, this is acceptable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single 'ein' parameter, so the schema already fully documents the parameter format and requirements. The description doesn't add any meaningful parameter semantics beyond what's in the schema, maintaining the baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('verify', 'qualifies') and resource ('nonprofit organization'), and distinguishes it from siblings by focusing on IRS public charity status verification rather than general lookup/search/list operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('to quickly verify if donations to an organization are tax-deductible'), but doesn't explicitly mention when NOT to use it or name specific alternatives among the sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• charity-searchA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses the tool's purpose (search) and return format (list with basic information), but lacks details about behavioral traits like rate limits, authentication requirements, pagination, or error conditions. The description doesn't contradict any annotations since none exist.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured with clear sections: purpose statement, search parameters explanation, return format description, and usage guidance. Every sentence adds value without redundancy, and the information is front-loaded appropriately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (search with optional filters) and 100% schema coverage but no output schema, the description provides good context about what the tool does and returns. However, without an output schema, the description should ideally provide more detail about the return structure beyond just listing field names.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters thoroughly. The description adds minimal value by listing parameters in bullet form and clarifying that city and state are filters, but doesn't provide additional semantic context beyond what's in the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Search for charities and nonprofit organizations'), resource ('IRS database'), and scope ('by organization name, location, or combine multiple criteria'). It distinguishes from sibling tools by focusing on search functionality rather than lookup, listing, or checking operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('when you don't have their exact EIN'), which implicitly suggests charity-lookup might be for exact EIN searches. However, it doesn't explicitly state when NOT to use it or name specific alternatives among the siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.