Server Quality Checklist

Tool Scores

• list_required_verificationsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the transparency burden. It adds meaningful behavioral context beyond the schema: the tool derives verifications from household facts (e.g., earned income implies pay stubs, shelter costs imply a lease) and states the authorization requirement ('Requires the caseworker scope'). It does not explicitly mention read-only status or error behavior, but the 'List' verb and output schema mitigate this.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose, and zero redundant or filler content. Examples are concise and illustrate behavior effectively. Every clause earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a complex nested household schema and an output schema, which reduces the need for the description to explain return values. The description covers the core purpose, derivation logic, and a permission constraint, making it complete enough for an agent to select and invoke the tool. The only notable gap is parameter guidance for 'program', but overall context is solid.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds some semantic value by linking household facts (earned income, shelter costs) to expected verifications, which helps an agent understand how the household parameter is used. However, it does not describe the 'program' parameter's allowed values or format, and the schema description coverage for top-level parameters is 0%, leaving the program field ambiguous. The household structure is well-documented in the schema, so the description's contribution is moderate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and a clear resource ('the documents a caseworker must verify to confirm SNAP eligibility for a household'). It also distinguishes from sibling tools by focusing on documents/verifications rather than program screening, eligibility checking, or policy lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool (when needing required verifications for a SNAP household based on presented facts) and includes an exclusion ('Requires the caseworker scope'). It does not name alternative tools explicitly, but the purpose is distinct enough that an agent can infer when to choose it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lookup_policyA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It transparently states the tool is currently a stub and that it returns an explicit placeholder rather than an invented answer, which is critical for an AI agent to avoid hallucination. It also mentions delegation to a RAG index, adding useful mechanism context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with three sentences, each providing distinct value: purpose, mechanism, and current stub behavior. It is front-loaded with the primary purpose and contains no filler or redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, output schema present), the description covers the essential aspects: purpose, delegation, and the stub limitation. It sufficiently sets expectations for an agent, though it could optionally clarify the difference from eligibility-checking tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter is 'question' with schema coverage at 0%, so the description must compensate. It adds that the question should be a SNAP policy question, which gives some context, but does not provide examples, formatting, or guidance on question complexity. This is adequate but not highly informative.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool answers SNAP policy questions with citations to the eligibility manual, using a specific verb ('answer') and a specific resource. This distinguishes it from sibling tools like screen_programs and check_program_eligibility, which focus on screening and eligibility determination.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: for SNAP policy questions, via the policy-manual-rag retrieval index. It also importantly discloses that the tool is currently a stub returning a placeholder, which sets expectations. It does not explicitly name alternative tools, but the context is strong enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• screen_programsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does well: it discloses that the tool is anonymous-safe, stores no data, and is not a determination. This goes beyond a simple read/write hint. It doesn't mention rate limits or detailed side effects, but for a screening tool these are the key behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: the first is a punchy core definition, the second add essential caveats and the alternative tool. Every sentence earns its place, and the key info is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the moderate complexity of screening two programs, the description covers purpose, scope, safety, and alternatives. An output schema likely covers the return value details. It doesn't explain what 'citation' means or how to interpret the 'likely eligible' signal, but this is reasonable given the output schema exists.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool description makes no mention of the single 'household' parameter, and schema description coverage is 0%, so it fails to compensate. The input schema itself has detailed definitions for Household and its fields, but the description adds no guidance on how to construct the input or what is expected beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool screens a household across SNAP and simplified Medicaid, returning a 'likely eligible' signal per program. It distinguishes itself from check_program_eligibility by explicitly labeling this as a non-determination screening, which sets it apart from the sibling tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'This is NOT a determination' and directs users to check_program_eligibility for real determinations. It also mentions 'Available to the screening scope', which clarifies when this tool should be used (initial screening without data storage).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_program_eligibilityA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses that the operation is deterministic, returns a decision plus rule trace, policy citation, and ruleset version, and explicitly states it never gives a bare yes/no. It also states the required scope and program constraint. This is strong transparency, though it could mention potential error behavior or side effects (e.g., whether it modifies anything), but for a test/calculation tool, the lack of side effects is implied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, with the central action in the first sentence, return behavior in the second, and constraints in the third. Every sentence adds value with no redundancy or fluff. It's front-loaded and efficiently structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a full eligibility test with many household fields, the description is complete enough: it explains what the tool does, what it returns, the required scope, and the program constraint. The detailed household schema covers the input structure, and the presence of an output schema means return format need not be explained. It could be more explicit about how this relates to sibling tools, but the core context for using the tool correctly is present.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The top-level schema has no descriptions (0% coverage), so the description must compensate. It adds critical semantics for the 'program' parameter by stating it must be 'SNAP'. For 'household', the description implies it's the household to test, and the rich nested schema already provides detailed per-field descriptions. The description adds essential constraint and context that the schema lacks, making it more than a simple baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs a full deterministic SNAP financial eligibility test for a household, which is a specific verb+resource. It also distinguishes itself from siblings by emphasizing it returns a rule trace and policy citations, not a bare yes/no, and is SNAP-specific. This is a precise, well-defined purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: it's for a full SNAP eligibility test, requires the caseworker scope, and mandates that 'program' be 'SNAP'. It implicitly differentiates from siblings like screen_programs (by being 'full' and 'deterministic') and explain_determination (by producing its own trace). However, it does not explicitly state when NOT to use it or direct users to alternatives for simpler screening, so it falls just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_determinationA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses a permission requirement (caseworker scope) and describes the output style (plain, step-by-step). It doesn't mention side effects, but the nature of 'explain' implies read-only behavior, which is acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no wasted words. The purpose is front-loaded, and each sentence adds essential information (what it does, prerequisite, source of ID).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, output schema present), the description covers purpose, usage, permissions, and integration guidance. The existence of an output schema means return values need not be described.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has no description for determination_id, and schema coverage is 0%. The description compensates by explaining the parameter is a prior determination id returned by check_program_eligibility, giving practical provenance and usage context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Return' with a clear resource 'a plain, step-by-step trace of a prior determination' and lists what the trace contains (rules fired, inputs, citations). This clearly distinguishes it from siblings like check_program_eligibility.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states the prerequisite (caseworker scope) and instructs the user to use the determination_id from check_program_eligibility, which implies when to use this tool. It doesn't explicitly contrast with alternatives but provides strong contextual guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.