Server Quality Checklist

Tool Scores

• get_collection_statusD
  1.7/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations and no description of side effects, read-only nature, or data access, the tool's behavior is completely opaque. The description does not disclose whether it is safe, mutating, or resource-intensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very short but cryptic and poorly structured. It reads like a sentence fragment rather than a clear, concise explanation, and the wording 'how far to trust the analytics yet' is ambiguous and not easily parsed.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description provides no information about the output schema, return values, or how the status relates to the overall workflow. An agent cannot infer what to expect from calling this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters, so schema coverage is trivially complete. The description adds no parameter information, but none is needed; the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description is phrased as a vague question rather than a clear statement of functionality. It mentions data collection and analytics trust but does not explicitly say what the tool does or returns, making it difficult for an agent to understand the tool's purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus the sibling tools. No context, conditions, or alternative suggestions are provided, leaving the agent without direction on selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_watchlistB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of explaining behavior. It adds that results are limited to currently-listed vehicles and include days_on_market, but it does not explicitly state read-only behavior, authentication needs, or rate limits. For a get/list operation, the risk is low, so this is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences with the factual behavior front-loaded and a strategic usage note following. There is no redundant wording or restatement of the tool name.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter list tool with an output schema, the description is mostly sufficient, but it leaves the meaning of 'best-first' undefined and does not explain the limit parameter or how this tool relates to sibling watchlist/search tools. The strategic note adds context but does not fill those gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one parameter, limit, with no description and 0% schema description coverage, and the description does not mention limit at all. The agent must infer from the parameter name and default value that it caps the number of results, so the description does not compensate for the missing schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns currently-listed vehicles ranked best-first, with days-on-market information. This identifies the resource and behavior, but it does not explicitly contrast it with sibling tools like search_listings or get_listing_history.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The second paragraph implies the output is useful for spotting negotiating positions ('seller has had no luck and knows it'), but it does not explicitly state when to choose this tool over alternatives such as get_listing_history or analyze_sale_triggers. The usage context is implied rather than direct.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_listing_historyB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It does not disclose that this is a read-only operation, nor any limitations such as pagination or authentication requirements. The phrase 'every observation, every price change' hints at comprehensiveness but lacks explicit behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise clauses with zero waste, front-loading the core purpose. Every word contributes to defining the tool's function and scope.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema and a simple single-parameter tool, the lack of annotations means the description should explicitly state that this is a read-only operation and any constraints. It does not, leaving behavioral expectations unclear for an agent deciding whether to invoke it safely.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, fully documenting the listing_id parameter as the digits in the marketplace URL. The description adds no additional parameter information, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Full timeline for one listing' with a specific scope ('every observation, every price change'), which clearly identifies the verb (get) and resource (listing history). It distinguishes itself from siblings like get_daily_digest or get_watchlist by focusing on a single listing's timeline.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for a single listing's history but does not explicitly state when to use it over alternatives or when not to. No sibling tools are referenced, so the agent must infer from the name and description that it is for a specific listing rather than a digest or watchlist.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_daily_scanA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It discloses that the tool is deliberately slow ('Slow (minutes, deliberately)'), which is valuable. However, it doesn't describe the output, side effects, or failure modes. It implies it updates data via 'force a refresh' but doesn't clarify whether it blocks or returns results. For a trigger tool, this is above minimal but incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured. The first sentence states the core action and purpose. The second paragraph provides context and a usage warning without unnecessary fluff. All information is front-loaded, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one optional parameter and an output schema, the description is incomplete. It fails to explain the use_llm parameter, which is a significant gap. It also doesn't describe the return value or what happens after the scan runs (e.g., whether it blocks or returns immediately). While the output schema might cover return structure, the description doesn't connect the tool's behavior to its effect. Given the simplicity of the tool, more could be done.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter, use_llm, has 0% schema description coverage, and the description does not mention it at all. The agent is left to guess what 'use_llm' controls (likely whether to use LLM processing during the scan). Since the description adds no meaning beyond the schema, and the schema itself has no description, the parameter is effectively undocumented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: run the daily crawl now instead of waiting for the scheduled job. It uses a specific verb ('run'), names the resource ('daily crawl'), and distinguishes itself from the scheduled job by emphasizing manual initiation. It also explicitly mentions 'force a refresh', which helps differentiate from sibling tools that read data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context: 'Normally Task Scheduler calls this via `fbmarket-scan`; use this tool when you want to force a refresh.' This tells the agent when to use it (forced refresh) and implies when not (normal scheduled runs). It also warns about slowness, which is a practical usage guideline. However, it doesn't explicitly mention alternative tools, but the context is sufficient for a unique trigger action.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_daily_digestA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of behavioral disclosure. It discloses defaults ('Defaults to today and to the min_rating in your config') and the nature of results (listings and price drops with scores). It does not mention potential side effects, authentication, rate limits, or read-only guarantees, but for a simple retrieval tool this is acceptable. The description adds value beyond the schema by mentioning config-based defaults.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise: two short sentences. It front-loads the core function (what it returns) and immediately follows with the main usage instruction. Every word earns its place; no fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity, the description covers purpose, defaults, and usage context. An output schema exists, so return structure is handled elsewhere. The only missing element is explicit guidance on when this tool is NOT appropriate (e.g., for detailed per-listing history), but the core calling contract is well covered. This is nearly complete for a read-only digest tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters with defaults and ranges. The description adds the context that min_rating defaults to the configured value, which is not in the schema. This extra context helps the agent understand the fallback behavior, so it earns a 4 above the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states what the tool returns: 'New listings and price drops from a given day, with their scores.' It uses a specific verb ('get') and resource (daily digest) and adds detail about content. It does not explicitly differentiate from siblings like get_watchlist or search_listings, but the phrase 'main entry point' implies a distinct role, so it earns a 4 rather than a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear usage context: 'ask for it each morning' and identifies it as the 'main entry point.' However, it does not mention when NOT to use this tool or suggest alternatives (e.g., for specific searches or watchlist details). This is partial guidance, not explicit routing, so a 3 is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_searchesA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility for behavioral transparency. It indicates a read-only operation (reading config.toml and producing counts), but does not explicitly mention that no modifications occur or that there are no side effects. It also does not clarify whether it might access external listing data or just the config file.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no redundancy or filler. It conveys the essential information about what the tool does without unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a no-parameter tool, the description provides sufficient context: it lists saved searches and their listing counts. The output schema is not provided, but the description gives a general idea of the output structure. It could be slightly more explicit about the format or types of data returned, but it is adequate for basic usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters, as shown in the input schema. The description does not need to explain parameter semantics because there are none, and the schema coverage is 100% with zero properties. This is perfectly adequate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns saved searches from config.toml and their listing counts. It implies a 'list' operation, and the name 'list_searches' is self-explanatory. However, it lacks explicit distinction from sibling tools like 'get_watchlist' or 'get_daily_digest', though the function is reasonably clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not explicitly state when to use this tool versus alternatives such as 'search_listings' or 'run_daily_scan'. It provides no usage context or prerequisites, leaving the agent to infer based on the name and description alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_compsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It transparently notes the data collection requirement and that the response indicates thin samples, making the behavior reasonably predictable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three short, focused sentences with no redundant wording or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  It provides useful context about output caveats and data requirements, but lacks explicit parameter definitions. The output schema exists, so not describing return values is acceptable, yet the parameter ambiguity leaves some gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has no descriptions for parameters. The text mentions widening by model years and mileage, but it does not clarify how year and mileage_km parameters affect the comparison or what happens when they are omitted.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns price data for comparable vehicles based on collected history, distinguishing it from listing/search tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explains the tolerance for comparable matches and the need for at least three weeks of daily scans, giving practical guidance on when results are meaningful. It does not explicitly contrast with sibling tools but implies the data-sufficiency requirement.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_listingsA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses key behaviors beyond the schema: it makes live network calls to Facebook, is rate-limited, takes a few seconds, and does NOT write to the tracking database. These are critical side effects and constraints that an agent must know.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, direct, and free of fluff. It front-loads the primary action and then provides important behavioral notes. Excellent structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, usage timing, and side effects, which is sufficient for a simple search tool. It does not describe parameters (counted in parameter_semantics) or error behavior, but since an output schema exists (not shown), lack of return details is acceptable. Overall it provides enough context to call the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 4 parameters, none of which are described in the schema. The tool description only implies the query is the search term but never explains limit, max_price, or min_price. With 0% schema coverage, the description should compensate, but it does not address any parameter meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Search Facebook Marketplace live, right now.' It also differentiates from the daily scan tool by noting it's for one-off questions, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly advises when to use this tool ('use this for one-off questions') and when not to rely on it for collection ('let the daily scan do the collecting'). It also warns about rate-limiting and latency, setting expectations for usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_sale_triggersA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses an important behavior: it reports its own sample size first and explicitly warns when results may be noise due to insufficient data. It does not mention side effects, but the tool is clearly analytical and non-mutating in nature.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loaded with the core purpose, and uses a second sentence to add important methodological detail and caveats without unnecessary fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter analysis tool, the description is complete: it states what is analyzed, which factors are considered, and how reliability is handled. Since an output schema exists, detailed return-value documentation is not required.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters and an empty input schema, so there is no parameter information to add. Baseline of 4 applies for zero-parameter tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool analyzes what correlates with fast vehicle sales in the user's own data, and distinguishes itself by listing the specific cross-referenced factors (days-on-market, price-versus-comps, price-drop history, vehicle type, price band, seller phrasing).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context for when to use it—when investigating sale-speed correlations in collected data—and includes a concrete reliability caveat about needing a few weeks of collection. It does not explicitly compare against sibling tools, but the usage context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.