Server Quality Checklist

Tool Scores

• scan_piiA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description provides thorough behavioral details: it returns only metadata (not values), skips unreadable files, caps very large sheets, and reads the entities file locally without returning the names. It also explains the automatic fallback for the entities file. This gives a clear picture of side effects and limitations, exceeding the typical level of transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear first paragraph on functionality and a second on the entities parameter. It is concise enough to convey necessary details without fluff, though the entities explanation could be slightly tighter. The information is relevant and not redundant, earning a high score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description provides a high-level overview of the return value (types, locations, counts) without specifying the exact output format, which is acceptable given no output schema. It covers main behaviors (recursion, skipping, capping) and the entities file. It lacks explicit error handling or return structure details, but for a scan tool, the description sufficiently completes the context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds significant meaning for the 'entities' parameter by explaining its purpose, format, and default behavior. It indirectly touches on 'max_cells' by mentioning capping large sheets, but does not explicitly link it to the parameter. The 'path' parameter is self-explanatory given the context. Overall, it compensates for the lack of schema descriptions, though not perfectly for max_cells.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool scans files or folders for PII and returns only types, locations, and counts, never the values. It also mentions recursion, skipping unreadable files, and capping large sheets, which fully specifies the tool's function. This distinguishes it from sibling tools like redact_text or make_synthetic_twin.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not explicitly explain when to use this tool over its siblings. It implies usage for scanning and reporting PII metadata, and the safety note ('Safe to surface to an LLM') hints at a use case, but there is no direct comparison or guidance on choosing between tools. The behavior details (recursion, skipping) could inform usage, but explicit 'use when' instructions are absent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• redact_textA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It does well by disclosing determinism, checksum-validated identifiers, over-masking avoidance, and the entities-file auto-use behavior. Minor gaps remain around error handling or what happens when no PII is detected, but transparency is strong overall.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: the first sentence states the core operation, followed by key behavioral constraints and then the optional parameter explanation. Every sentence earns its place without unnecessary verbiage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simple 2-parameter shape and the presence of an output schema, the description is highly complete. It covers the main transformation, important edge-case prevention (bare numbers), and the optional entities file behavior. The description is sufficient for an agent to invoke the tool correctly without needing further clarification.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, and it does. It explains that `entities` is a path to a local `.obsify.entities` file, that matched names are masked as `<KNOWN_ENTITY>`, and that a nearby file is auto-used if omitted. This adds substantial meaning beyond the bare schema fields.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence clearly states the verb (redact), the resource (text), and the output format (PII replaced by placeholders), making the purpose immediately obvious. It also distinguishes itself from sibling tools like scan_pii and verify_value_free by explicitly conveying the redaction operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about how the tool behaves and when the optional entities file applies, but it does not explicitly state when to prefer redact_text over sibling tools or when not to use it. There are no alternative tool comparisons or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_value_freeA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral clarity. It discloses the fail-closed behavior, the variant-matching behavior, and the deliberately detail-poor return shape. It does not explicitly state there are no side effects, but the read-only check nature is strongly implied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded, stating the check first, then the return contract, then the intended target scenario. Every sentence contributes useful information without repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter boolean verification tool, the description is nearly complete: it names inputs, behavior, return value, and intended boundary context. It leaves minor edge-case behavior unspecified, but this does not materially hamper selection or invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has no parameter descriptions, but the description defines the core semantics: `text` is the artifact being verified and `terms` are the prohibited strings matched directly or through normalized variants. It adds meaningful algorithmic context beyond the bare schema, though it omits edge cases like empty terms behavior.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a fail-closed verification check that `text` contains none of `terms` or their variants, giving a specific verb, resource, and scope. It distinguishes itself from sibling tools by being a boolean verification gate rather than a scanning or redaction operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly identifies the intended use case: verifying an artifact before it leaves the perimeter. It implies this is a pre-release/compliance gate rather than a diagnostic tool, and the zero-detail return further signals it is not for troubleshooting that needs matched context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• make_synthetic_twinA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does so well. It discloses key behavioral traits: schema is preserved, every data value is freshly FAKED, no real value is copied, and it returns a safe schema summary. This gives the agent essential safety and data-handling context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three dense sentences, front-loaded with the core action and then efficient supplementary detail. There is no fluff or redundancy; every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core function, the workflow pairing with run_on_real, and the return value (schema summary). It lacks any explanation of cap_rows and its potential effect on 'true row counts,' which would be a notable gap for a tool of moderate complexity. Overall, it is quite complete but not flawless.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does explain path and out (': path', 'written to out'), but it does not mention cap_rows at all. This leaves one of three parameters semantically opaque, so the compensation is partial.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb+resource: 'Generate a SYNTHETIC TWIN of a real Excel workbook at `path`, written to `out`.' It clearly differentiates from siblings by framing this as the twin-creation step and explicitly mentions run_on_real as the subsequent step for real-file execution.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit workflow guidance: 'Reason and write your analysis code against the twin; then run it on the real file with run_on_real.' This tells exactly when to use this tool and names the alternative (run_on_real) for the next phase, satisfying the 'when/when-not/alternatives' criterion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_on_realA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and handles it well: output is size-capped, PII masking is best-effort and explicitly not a guarantee, network is disabled, a timeout applies, and execution is local. It also warns that raw records/identifiers should never be printed, adding important safety context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but well organized: concept label, action, safety constraints, and usage guidance. Bolded callouts ('Return AGGREGATES...', 'best-effort') make key instructions easy to parse, and no sentence is filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a code-execution tool with no output schema and no annotations, the description covers the essential operational surface: local execution, data binding, output size, masking caveat, aggregate printing, network isolation, and timeout. It is sufficient for an agent to invoke the tool safely and interpret the result.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds strong semantics for code (Python executed locally) and data_path (bound to DATA_PATH), but timeout is only indirectly covered by 'a timeout applies' and the schema's default, not explained as a configurable parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'COMPUTE-TO-DATA' and clearly states the tool executes Python code locally against a real file at data_path, binding it to DATA_PATH. This is a specific verb+resource pairing and is distinct from siblings like make_synthetic_twin, which implies synthetic data operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description strongly implies when to use it: when you need to compute over real data without pulling raw data into context ('data never enters your context'). It gives actionable guidance to print aggregates and avoid raw records, but it does not explicitly name alternative tools or state when-not-to-use conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.