Server Quality Checklist

Tool Scores

• checkA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It mentions returning 'passed' and severity counts, and the fail_on effect. However, it does not explicitly state if the tool is read-only or if it has any side effects. The name 'check' implies non-destructive, but not confirmed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise, using one short paragraph with two key pieces of information (purpose and parameter). Every sentence is functional, front-loaded with the main action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has only one parameter and a simple output (with an output schema), the description is fairly complete. It covers the return value and parameter options. However, it doesn't elaborate on what 'watch' entails (e.g., one-time vs ongoing) or any prerequisites.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, so the description must compensate. It explains the two values of 'fail_on' ('critical' and 'warning') and their meaning, adding value beyond the schema's bare type and default.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs a watch over all tables and returns a pass/fail gate. The verb 'run' and resource 'all tables' are specific. It differentiates from siblings like 'watch' indirectly via the CI-style gate concept, but does not explicitly compare.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains the fail_on parameter and that it mirrors `scherlok ci`, but it does not specify when to use this tool versus siblings like 'watch' or 'status'. There is no explicit when-not-to-use or alternative context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• historyA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses it reads from local profile store and does not re-profile the warehouse, indicating a non-destructive read operation, but lacks details on capping behavior, authorization, or data freshness.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences: first states purpose, second adds behavioral context. Every word is necessary with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With only one simple parameter and an output schema present, the description covers the core idea but omits specifics about the cap, anomaly types, or result structure, leaving gaps for a complete understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Adds meaning by explaining the 'days' parameter specifies the time window, but does not elaborate on allowed values, range, or cap limit beyond what the schema indicates (default 30).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly specifies the tool returns anomalies within a time range, using a specific verb ('Return') and resource ('anomalies recorded'), and distinguishes from siblings like 'check' and 'investigate' by focusing on historical data with a time constraint.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies usage for quick reads from local profile store without warehouse re-profiling, but no explicit when-to-use or contrast with siblings like 'investigate' which may also deal with anomalies.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• statusA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry full behavioral context. It describes the output (connection target redacted, table count, anomalies) but does not explicitly state side effects (likely read-only) or any hidden behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: the first states the purpose, the second lists key outputs. Every sentence adds value, and no redundant information is present. Front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters and presence of an output schema, the description explains the returned values (connection target, tables, anomalies) and the tool's purpose as a health glance. It is fully complete for a simple status tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, so schema coverage is 100%. The description adds value by explaining what the tool returns, providing meaning beyond the empty schema and helping the agent understand the tool's output.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it reports the current monitoring state and lists specific returned values (connection target, table count, anomaly count). It implies a quick health glance, distinguishing from siblings like 'investigate' or 'watch', but does not explicitly differentiate.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description says it's a quick health glance without profiling, suggesting use for lightweight checks. No explicit when-not or alternatives are given, relying on implicit contrast with sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• watchA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses that tables without a prior baseline are profiled silently and their baseline is set for next time, indicating a write operation. However, it does not clarify whether the tool is read-only or destructive, nor does it specify the cap limit or potential side effects beyond setting baselines.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise at four sentences. It front-loads the main purpose in the first sentence and provides necessary details without extraneous information. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one optional parameter, output schema exists), the description covers the core functionality, parameter usage, and edge case (no baseline). It lacks details like the cap limit and what 'profile' entails, but these are minor given the output schema. A 4 is suitable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, but the tool description explains the parameter's purpose: 'Pass `tables` to limit scope, or omit to watch everything.' This compensates well, adding meaning beyond the schema's type definition. A 4 reflects strong compensation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool profiles tables and detects anomalies against a stored baseline. It specifies the return value (type, severity, message per table) and that results are capped. While the purpose is specific and distinct from siblings like 'check' or 'history', it does not explicitly differentiate itself, so a 4 is appropriate.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on the parameter: 'Pass `tables` to limit scope, or omit to watch everything.' It also explains behavior for tables without a baseline. However, it does not mention when to use this tool over siblings like 'check' or 'investigate', missing some usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• investigateA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It mentions storing baselines (write operation) and that no anomalies are reported, but does not disclose permission requirements, side effects, or whether it overrides existing baselines.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two paragraphs that front-load the core purpose and smoothly explain the parameter and workflow. Every sentence provides value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, a single optional parameter, and an output schema, the description covers the core workflow: profiling, baselines, and subsequent drift detection. Could be more complete about permissions or if the operation is reversible, but adequate for typical use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds crucial meaning: 'tables' limits profiling to specific tables, and omitting profiles everything visible. This goes beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool profiles tables and stores baselines for anomaly checks. Distinguishes from sibling 'watch' by noting that investigate establishes baseline and does not report anomalies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly describes when to pass 'tables' vs omit, and directs to use 'watch' later for drift detection. Provides clear context and alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_tablesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but description adequately explains behavior: lists tables, returns names and count. For a read-only listing, this is sufficient. Could mention if any authentication requirements exist, but not critical.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, each essential. First states action, second describes output and use case. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no parameters and an output schema present, the description fully covers what the tool does, what it returns, and how it relates to siblings. Complete for a simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters, so baseline is 4. Description adds value by explaining return format and usage context, which complements the empty input schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool lists tables visible to the warehouse connection, specifies return values (capped names and count), and distinguishes from siblings by recommending use before investigate or watch.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to use this before investigate or watch, providing clear context. However, no mention of when not to use or alternatives for other siblings like check or history.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.