Server Quality Checklist

Tool Scores

• praxis_get_runA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint and idempotentHint, so the description doesn't need to re-state safety. It adds value by noting that operator filesystem paths are removed, which is a behavioral trait not obvious from annotations. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that efficiently lists all key return fields without extraneous words. It is front-loaded with the purpose and provides immediate clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description provides a good overview of the returned data. It covers all major components (evidence, artifacts, verification, ledger) and mentions removal of paths. It is sufficiently complete for an agent to understand the tool's scope, though some structural details are omitted.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Both parameters are fully described in the input schema (run_id and include_events), so schema coverage is 100%. The description does not add additional meaning beyond what the schema provides, meeting the baseline expectation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns 'the full record for a run' and enumerates specific data fields (URN evidence, blast radius, artifacts, verification result, ledger). This distinguishes it from sibling tools like praxis_get_artifact which is for single artifacts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not explicitly state when to use this tool versus alternatives. While the intent is clear (get comprehensive run data), there are no when-to-use or when-not-to-use guidelines, leaving the agent to infer context from the tool name and sibling list.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• praxis_integration_statusA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint. The description adds behavioral context: that the tool checks configuration and that the 'probe' parameter triggers actual contact with servers. This goes beyond the annotations without contradicting them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, clearly structured. The first sentence defines the tool's output, the second gives a usage directive. No unnecessary words; every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains what the tool reports (configuration status) and why it matters (which modes/outputs are available). It lacks a detailed return structure, but since there is no output schema, the description provides enough high-level context for a simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already has 100% description coverage for the single parameter. The description adds meaning by explaining that 'probe' means actually contacting the services, which enhances understanding. The tool description itself does not repeat parameter details but adds context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool checks configuration of DataHub, GitHub, target repository, and dbt, and indicates which evidence modes and outputs are available. It uses specific verbs and resources, distinguishing it from sibling tools like praxis_prepare_remediation or praxis_get_run.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Call this before choosing 'live' mode,' providing clear when-to-use guidance. It does not explicitly mention when not to use or alternatives, but the context is sufficient for most agents.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• praxis_get_artifactA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint, indicating safe read operation. Description adds that the tool returns content, diff, and SHA, and notes omitting path lists existing paths. Does not mention rate limits or authentication, but annotations reduce burden.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first states what the tool returns (content, diff, SHA), second gives explicit usage guidance. No redundant words, front-loaded with essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given only 2 parameters (one optional), read-only, idempotent, and no output schema, the description is complete. Covers purpose, what is returned, and usage instructions. No missing critical details for the tool's simple function.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. Description adds value by explaining the path parameter's format (e.g., 'models/staging/stg_encounters.sql') and the optional behavior (omit to list paths). This clarifies usage beyond schema definitions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it returns exact content, unified diff, and SHA-256 of a generated candidate file. Specifies use case: showing a reviewer the change. Distinguishes from sibling tools like praxis_get_run (lists artifacts) and praxis_prepare_remediation (generates).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises using this tool 'to show a reviewer the change rather than describing it.' Implicitly contrasts with other tools but does not explicitly list when not to use or alternative tools. Guidance is clear for primary use case.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• praxis_prepare_remediationA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description explicitly states that the tool mutates nothing (aligned with readOnlyHint=true), waits for a human gate, and runs fail-closed verification. It adds valuable behavioral context beyond annotations, such as waiting behavior and the fact that it blocks on nothing external.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences front-load key actions and return values, with no wasted words. Every sentence adds essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description covers return values, behavioral traits (non-mutation, waiting for gate, fail-closed), and required inputs. It provides a complete picture for a complex tool with 6 parameters.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so parameters are well-defined in the schema. The description adds minor guidance (e.g., 'resolve the URN first') but does not significantly expand on parameter meaning beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it resolves DataHub evidence, generates a dbt compatibility patch, runs verification, and returns specific outputs (run id, risk score, verification tier, checks). It distinguishes itself from sibling tools by describing a unique preparation step that blocks on a human gate.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly sets context as the preparation step for schema-drift incidents, with no explicit when-not or alternatives. However, it clearly indicates the tool's role in the workflow, and sibling tools like praxis_approve_remediation are naturally distinct. A slight lack of explicit guidance keeps it from a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• praxis_approve_remediationA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint=false and destructiveHint=false. The description goes beyond by detailing what happens on approval: re-reads repository, re-verifies, compares candidate hashes, and publishes only outputs confirmed by an external system. This adds significant behavioral context beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is somewhat lengthy but every sentence provides necessary context. It front-loads the purpose and then adds behavioral details. Minimal redundancy; could be slightly trimmed but overall effective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (human decision recording, multiple required parameters, no output schema), the description is thorough. It explains the full workflow, constraints, and the role of each parameter, leaving no gaps for an AI agent to misunderstand the tool's operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline 3. The description adds meaning beyond schema descriptions: it explains that 'human_approved' must be true and only set by a human, and that 'approver' is the identity recorded in the ledger. This adds value over the basic parameter names and schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool records an approval or rejection for a run that passed verification, using the verb 'record' and the resource 'decision'. It distinguishes itself from siblings like praxis_prepare_remediation and praxis_get_run by emphasizing it is a human decision action, not an automated operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly says 'you must not call this on your own judgement' and 'only call it when a person has seen the diff and the verification result and told you their decision'. It provides clear context for when to use the tool, but does not explicitly mention when not to use it relative to other tools, though the sibling list helps infer that.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.