Server Quality Checklist

Tool Scores

• search_nsf_awardsC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the full burden of behavioral disclosure. It only notes 'public API' but does not mention rate limits, authentication needs, pagination, return format, or any side effects. The phrase 'historical and active awards' is a scope note, not behavior detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single short sentence, which is efficient in length but sacrifices critical information. It is not verbose, but under-specification makes it less useful than a slightly longer, more informative description would be.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has two simple parameters and an output schema, but the description provides no usage context, parameter semantics, or behavior details. Given the sparse input schema (no parameter descriptions) and lack of annotations, the description is inadequate for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description mentions no parameters. It does not explain what 'keyword' should contain, how 'max_results' affects output, or the meaning of the default value. The agent must rely solely on parameter names, which is insufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Search NSF Awards (public API) for historical and active awards' clearly identifies the verb (Search), resource (NSF Awards), and scope (historical/active). It distinguishes from the sibling 'search_grants' by specifying NSF and public API, though it does not explicitly contrast with alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'search_grants' or 'list_open_source_relevant_grants.' There is no mention of context, exclusions, or preferred use cases; usage is only implicit from the tool's name and generic search verb.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refresh_corpusC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It indicates a mutation (refreshing the corpus) but does not explain side effects, duration, idempotency, or consequences for existing data. 'Continuous-ingestion style' hints at behavior but leaves important details unstated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no filler words. It is appropriately front-loaded with the action and target, though the brevity leaves out useful context that would not hurt structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema, the tool lacks annotations and provides minimal context about its effect on the system. The description does not explain prerequisites, parameter usage, or what the refresh does to the corpus, making it incomplete for a mutation tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description entirely omits the 'keywords' parameter. While the schema shows a default and type, the meaning of keywords in the context of a corpus refresh is not explained, leaving the agent to guess how to set it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Trigger') and resource ('refresh of the local grant corpus'), clearly distinguishing this from sibling search/list/run tools. The phrase 'continuous-ingestion style' adds some ambiguity about what exactly the refresh does, but the overall purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no explicit guidance on when to use this tool versus alternatives. It only implies usage through the word 'refresh,' without stating prerequisites, when not to use it, or that it should precede search operations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_evaluationC
  2.6/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden but discloses no behavioral traits, side effects, or expected outcomes. It doesn't say what 'evaluation harness' does or what happens when run.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one short, front-loaded sentence with no redundancy. It's concise, though not particularly informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and an extremely sparse description, the tool's operation and purpose remain unclear. The output schema exists but is not helpful in the description. For a no-parameter tool, more could be said about the evaluation process or expected behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema lists none. Per guidelines, this is baseline 4; the description is not required to explain parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear action ('Run') and resource ('OpenGrants evaluation harness'), but leaves the tool's precise function and scope undefined. It distinguishes from siblings at a high level (no other sibling is about evaluation), but 'basic' is vague.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description offers no guidance on when to run this tool, prerequisites, or alternatives. It doesn't mention if this is for testing, validating, or something else.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_open_source_relevant_grantsC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It only implies a read-like action ('Discover') but does not explain any behavioral traits such as filtering behavior, data sources, rate limits, or whether it covers both open-source AI and public goods as separate categories. This is a minimal hint rather than a transparent description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with zero filler. Every word contributes meaning, and it is appropriately concise for the tool's simplicity, even though other dimensions suffer from the brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While an output schema exists (so return values need no description), the tool has two parameters and no annotations, and the description lacks any context about the default focus value, how relevance is determined, or how this tool differs from sibling grant-listing tools. The description is too sparse to be considered complete for effective tool selection.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate by explaining parameter meanings or usage, but it does not mention 'focus' or 'max_results' at all. It adds no value beyond the bare names and defaults in the schema, leaving the agent to guess what 'focus' means or how 'max_results' affects results.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description "Discover grants particularly relevant to open-source AI and public goods" clearly states the tool's function (discover/list grants) and its scope (open-source AI and public goods). It does not explicitly distinguish from sibling tools like list_public_goods_funding, but the name and description provide a specific, actionable purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives such as search_grants or list_public_goods_funding. It implies a discovery use case but does not state any conditions, exclusions, or context that would help an agent choose this tool over its siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_grantsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility for behavioral disclosure. 'With provenance' is vague and does not explain how the search behaves (e.g., live Grants.gov fetch, caching, return format, or rate limits), and no other behavioral traits are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and front-loaded, conveying the core purpose in a single sentence. However, the phrase 'with provenance' is somewhat vague and could be expanded, though it does not add significant length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 3-parameter tool with multiple siblings and no annotations, the description is too sparse. It omits parameter semantics, usage context, and behavioral details, leaving the agent without adequate information for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description mentions 'keyword', which maps to the required parameter, but offers no explanation of 'status' or 'max_results'. With 0% schema_description_coverage, the description does not compensate for these undocumented parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches open U.S. federal grant opportunities on Grants.gov by keyword. This distinguishes it from siblings like search_nsf_awards, which targets NSF awards, and get_grant_details, which is for retrieving details of a specific grant.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool over alternatives. It does not mention exclusions, prerequisites, or the difference between this and related search tools like search_nsf_awards or list_open_source_relevant_grants.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• upsert_profileB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It reveals the upsert nature ('Create or update') and a parameter format (focus_areas comma-separated), but does not state whether existing data is overwritten, any auth requirements, or side effects. This is insufficient for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and front-loaded with the core action. It wastes no words, though it is under-specified. As a concise statement, it is effective, but it sacrifices necessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a mutation tool with no annotations and low schema coverage, the description is too sparse. It doesn't explain the relationship to matching/drafting workflows, or provide context on required fields or expected outcomes. An output schema exists, but the description still needs more context to be complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must add meaning for the 5 parameters. Only 'focus_areas' is explained (comma-separated). No guidance is given for profile_id, name, description, or github_url, leaving the agent to guess their roles.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear action ('Create or update') on a specific resource ('project/researcher profile') and explains its purpose ('for matching and drafting'). This clearly distinguishes it from sibling tools that focus on grants and funding.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The purpose implies when to use the tool (managing profiles), but there is no explicit guidance on when to use it instead of alternatives, nor any exclusions or prerequisites. The context 'for matching and drafting' gives some direction but is not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_public_goods_fundingB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the read-only nature is implied by the verb 'List' and the description conveys the category scope. However, it does not disclose behavioral traits like whether results are filtered by default focus, pagination, or external source behavior; the output schema covers return shape, but no further behavioral context is added.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, starts with the action verb, and packs the category scope plus examples efficiently with no filler. Minor redundancy between 'public-goods funding' and 'funding opportunities' is negligible.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Even though an output schema exists, the description lacks parameter semantics and usage guidance relative to sibling tools. For a two-parameter tool with no annotations, this one-sentence description leaves significant decision context unanswered, especially around 'type_filter' and tool selection.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% property description coverage, and the description never mentions the 'focus' or 'type_filter' parameters. The platform examples loosely suggest possible categories but do not explain accepted values, defaults, or how the filters combine, so the agent cannot infer parameter semantics from the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'List' and clearly identifies the resource as donation, sponsorship, quadratic funding, and open-source public-goods funding opportunities, with concrete platform examples. This distinguishes it from sibling tools like search_grants, which focus on narrower grant search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives such as list_open_source_relevant_grants or search_grants. The description only states what it does, leaving the selection decision to the agent without exclusion or preference context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• draft_proposal_outlineA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It only says 'grounded' and states the applicable domain, but it doesn't clarify whether the tool modifies state, requires authentication, has side effects, or is safe to call. This is a significant gap for a generation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the action and deliverable. The parenthetical adds useful scope clarification without unnecessary verbosity. Every word contributes to the core meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  An output schema exists (which may define the return value), but the description itself is sparse. It doesn't mention required parameters, where 'profile_id' comes from, or how the 'opportunity' fields are used. While the tool's role as a proposal-drafting companion to the sibling search tools is implied, an agent would need more context to use it confidently.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain the parameters. It only vaguely refers to 'profile + opportunity' without naming profile_id, opportunity_type, opportunity_title, or opportunity_description, and gives no details about their meaning or format. This does not compensate for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Generate') and identifies the deliverable as 'a grounded proposal outline + short pitch'. It clearly differs from sibling tools, which are mostly search/list/retrieve operations or profile-management utilities, so there is no ambiguity about what this tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description specifies that it works for 'grants and donation/public-goods vehicles,' providing clear context for when to use it. However, it doesn't explicitly mention alternatives or when not to use this tool, so it falls short of a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_grant_detailsA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavior. It only states the basic read action, with no mention of error handling, return format, or authentication. This adds minimal behavioral transparency beyond what the tool name implies.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one sentence, directly states the action, and contains no filler. It is appropriately sized for the tool's simplicity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (single ID parameter) and the presence of an output schema that details return structure, the description is adequate for the agent to understand the tool's purpose. It could mention error cases but is otherwise complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% description coverage. The description does mention 'by ID', linking the parameter to the concept, but it doesn't elaborate on the format or provenance of the ID. For a single string parameter, this is marginal compensation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Retrieve' with the resource 'grant opportunity details' and specifies the scope 'by ID', clearly distinguishing it from sibling tools like search_grants which search rather than fetch a single record.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. The context implies use when an opportunity ID is available, but it doesn't mention that search_grants should be used to find IDs, nor does it state exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.