Server Quality Checklist

Tool Scores

• describe_reportA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must convey behavioral traits. It states the tool returns a specification, implying a read-only operation, and details the contents (filters, types, enums, defaults, etc.). However, it does not discuss permissions, error behavior, or side effects, which is a moderate gap given the lack of annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose, and includes a helpful usage tip. Every word adds value—no fluff, no redundancy—making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one parameter and no output schema. The description adequately explains the purpose and usage but fails to explicitly identify the input parameter or discuss potential errors/auth. Given the simplicity, it is minimally sufficient but not comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description does not mention the actual parameter 'report_name' or its type/format, leaving the agent to infer it from the schema. The schema does provide a description and example, but the tool description fails to compensate for the 0% schema description coverage in the text, making the semantics unclear without relying on structured data.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get the full parameter specification for a single AppFolio report.' The verb 'Get' plus the resource 'parameter specification' make it evident, and it distinguishes itself from siblings like list_reports (which lists reports) and run_report (which executes reports).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Call this before run_report to know exactly which parameters to pass,' which provides a clear usage context. It does not explicitly mention alternatives or exclusions, but the guidance to call before run_report is actionable and sufficient for choosing this tool over siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_endpointA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. While it accurately describes the tool's behavior (retrieving endpoint specs, not executing them), it doesn't disclose what happens on invalid operation_id, whether the full response fields might be extensive, or potential size/rate implications. However, since this is a fundamentally read-only, low-risk metadata tool, the description is adequate but could note error behavior for invalid operationIds.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense sentence plus one directive sentence. It front-loads the purpose and packs a substantial amount of concrete detail (method, path, params, filters, request body, response fields, sample). The only minor inefficiency is the length of the enumeration, but every element is informative. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a metadata/read tool with just one parameter and no output schema, the description covers the essentials: what it returns (full spec contents), when to use it (before call_endpoint), and how to identify the target (operation_id examples). It could improve by noting the handle for invalid/unknown operation_ids, but overall it's quite complete for a tool of this simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the schema provides no description for the operation_id parameter. However, the description compensates by explaining operation_id is an 'endpoint operationId' with examples ('get_tenants', 'post_bills') and references list_endpoints as the source. While it doesn't fully explain what an operationId is or its format constraints, the examples plus context are reasonable. The 0% coverage means the description must compensate, and it does partially.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Get' with a clear resource: 'the full specification for a single AppFolio Database API endpoint'. It enumerates exactly what the specification contains (method, path, query filters, request body, response fields, request sample). This clearly distinguishes it from siblings like list_endpoints (which lists endpoints) and call_endpoint (which executes them).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is provided: 'Call this before call_endpoint to know exactly which parameters to pass.' This gives a clear when-to-use directive and names the specific alternative it pairs with, establishing a workflow. The contrast with call_endpoint makes the usage context unambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_endpointsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. The description discloses the filter capabilities (category, resource, method, reads-only) and the API version (v0), which is useful, but does not describe pagination behavior, output volume, or whether results are ordered. It does convey listing/discovery semantics effectively, though the scale of results is not addressed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact (three sentences) and front-loaded with the core purpose, followed by the optional filter capabilities, then workflow guidance. Every sentence adds value with zero waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a discovery/listing tool with no output schema, the description covers purpose, filters, workflow linkage to sibling tools, and API/version differentiation. It lacks explicit note on return volume/pagination, which is a minor gap for a listing tool that could return many endpoints. Overall it is well-rounded and sufficient for an agent to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the parameters are wrapped in a 'params' object with individual property descriptions in the schema that are reasonably self-explanatory (method, category, resource, reads_only each have filtering descriptions with examples). The description adds workflow context but the parameter meanings are largely conveyed through the schema's property-level descriptions rather than the tool description itself.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List the available AppFolio Database API endpoints' with specific output fields (operationId, method, path, category, description). It distinguishes this discovery tool from siblings by naming describe_endpoint (for params) and call_endpoint (for execution), and explicitly contrasts the Database API with the Reporting API (list_reports/run_report).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit workflow guidance: 'Use this to discover which endpoint to use, then call describe_endpoint for its parameters and call_endpoint to execute it.' It also names the alternative toolset for Reporting API, giving clear when-to-use vs when-not-to-use context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_reportsA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses that filtering is optional and that it returns a listing rather than data. However, it doesn't disclose pagination, response volume, or whether 'required filters' are full or abbreviated. For a read-only listing tool, the disclosure is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences that are dense with information - return fields, filtering behavior, workflow guidance, and sibling-tool routing. No filler. Slightly longer than strictly minimal, but each clause earns its place by aiding agent selection and next steps.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a discovery/list tool with one optional parameter and no output schema, the description is quite complete. It covers purpose, output contents, filtering, and next-step routing. The main omission is lacking detail on the structure of the returned list (e.g., whether required filters come as a list or string), but this is modest for a discovery tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% for the single category parameter, but the description compensates well by explaining what category does, providing examples ('Financial', 'Leasing', 'Tenant'), noting it's optional, and advising to omit it initially to see all categories. This adds meaningful usage insight beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb+resource ('List the available AppFolio reports') and enumerates the return fields (slug, name, category, required filters), which clearly distinguishes it from siblings like run_report, describe_report, and call_endpoint. It also explains the tool's role in the discovery workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use ('to discover which report to run') and how it fits with alternatives ('then call describe_report for its parameters and run_report to execute it'), naming sibling tools directly. It also explains the category filtering use case and suggests 'call without args first to see all categories'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• call_endpointA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden - and it does well: it discloses that GET reads are auto-paginated, that write operations execute live against the database (a significant side-effect disclosure), and that it validates against a catalog before calling. It falls just short of 5 because it doesn't explicitly note result size limits, rate limits, or what happens on validation failure, though the schema's pagination default of 1000 partially fills this gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, zero filler. Front-loaded with the primary purpose, then the payload structure. Every clause earns its place, covering validation, execution mode, pagination, and the prerequisite workflow without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex, generic dispatch tool (no output schema, nested objects, many free-form params), the description explains the overall execution model, safety difference between reads and writes, and the prerequisite (describe_endpoint). It's fairly complete given the inherent genericness of the tool. Slight gap: it doesn't describe error/response format or note limits, but given the tool is inherently generic across many endpoints, the description realistically cannot enumerate every behavior, so 4 is appropriate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate - and it partially does by explaining the flow of validation and pagination. However, the parameter-level semantics are mostly carried by the schema's own per-field descriptions (filters, paginate_results, body, etc.) which are detailed. The description adds value by explaining the GET-write behavioral distinction and the describe_endpoint prerequisite, but doesn't explain the filters wrapper or pagination fields - those are already in the schema, which does the heavy lifting for parameter meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states a specific verb+resource ('Call an AppFolio Database API endpoint and return its JSON response') and clearly outlines the execution flow (validates operationId, auto-paginates GET reads, executes live writes). It distinguishes from siblings by explicitly referencing describe_endpoint and list_endpoints as the setup tools, establishing a clear division of labor within the tool family.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance on when to use this tool: 'Use describe_endpoint first to learn the exact params and body.' It also clarifies behavioral differences between GET (auto-paginated) and POST/PATCH/PUT/DELETE (live writes), helping the agent understand when live writes occur versus safe reads. This functional distinction is effectively an implied when-to-use/alternatives guideline given sibling tools are list/describe functions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_reportA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool validates against a report catalog before calling AppFolio, implying pre-flight checks. It mentions required filters must be included or the request is rejected, which is a useful behavioral constraint. However, it doesn't disclose potential rate limits, error behaviors beyond validation, or what happens on partial failures; still, the validation disclosure is substantive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences that each earn their place: the first states the primary action and validates, the second gives critical usage guidance (use describe_report first). No filler, no redundancy with the schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has nested objects (params > report_name required, filters as arbitrary JSON object), no output schema, and no annotations. The description handles this well by pointing to describe_report for filter keys and stating the validation behavior, which collectively inform the agent how to construct a valid call. It covers the critical unknowns (required filters, parameter naming), though paginate_results behavior is left to the schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It references describe_report for filter keys, which directs the agent to the authoritative source for parameter semantics. It explains that filters must match parameter names from describe_report. However, the description itself doesn't detail individual parameter formats beyond pointing to describe_report, so the agent must do a follow-up lookup; still, this is a reasonable delegation given the report-specific nature of filters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool's function: 'Run an AppFolio report and return its rows.' It names the specific verb (run), resource (AppFolio report), and outcome (return rows). It distinguishes from siblings: list_reports lists available reports while this executes them, and call_endpoint/describe_endpoint target endpoints rather than reports.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance provided: 'Use describe_report first to learn the exact filter keys for the report.' This clearly instructs the agent on the prerequisite step before invocation. The description also clarifies validation behavior (validates report name, required filters, parameter names against the catalog), setting expectations that malformed calls will be rejected.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.