Post-tx outcome verification via LLM-as-judge. Zero Core Verify.
Scored across 5 tools
The only plausible confusion is between report_outcome and verify_outcome, since both record verification data; however, the descriptions explicitly contrast simple pass/fail logging with code-plus-judge verification. get_service_quality, health, and list_tools are clearly distinct in purpose.
Four of five names follow a snake_case verb_noun pattern: get_service_quality, list_tools, report_outcome, verify_outcome. The bare noun health deviates slightly and would be more consistent as check_health.
Five tools cover discovery, health, simple reporting, full verification, and quality aggregation without obvious redundancy. This is well within the typical 3-15 range and each tool has a clear role.
The core lifecycle of verify, report, and aggregate quality is present, along with discovery and health. Minor gaps exist around retrieving individual verification records or listing services, but agents can largely work through the aggregate and reporting tools.
Get aggregated quality scores for a service based on past verifications and reports, counting each paying wallet once (its latest). Returns the pass rate (overall, and over the rows where code checked the format or a measured time), average completeness and accuracy over the scored rows with their count, the count of inconclusive rows, format and SLA compliance rates, and a trend by pass rate. The scores are self-reported by paying wallets and not attested: nothing proves a reporter used the service.
| Name | Required | Description | Default |
|---|---|---|---|
| service_id | Yes | Service identifier to query (e.g. 'harvey-tools/scrape_url') |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations present, the description carries the full behavioral burden and does so well. It discloses the deduplication rule ('counting each paying wallet once (its latest)'), enumerates the returned metrics, and adds a crucial trust caveat that scores are self-reported and not attested.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose and then expands into return details and a caveat. It is somewhat long because it enumerates metrics in a single dense sentence, but every element is relevant given the absence of an output schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read tool with one parameter and no output schema, the description is complete: it explains what is aggregated, how wallets are counted, which metrics are returned, and the limitation of self-reported data. An agent has enough information to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the sole parameter service_id is already fully documented with an example. The description only refers to querying 'a service' generally and adds no syntax or format details beyond what the schema provides, which matches the baseline for high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Get aggregated quality scores for a service.' It also clarifies the data source ('past verifications and reports') and aggregation rule, making the tool's purpose clear, though it does not explicitly differentiate itself from siblings like health or verify_outcome.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by the phrase 'based on past verifications and reports' and the focus on aggregated service quality, but there is no explicit guidance on when to choose this tool over siblings such as health or verify_outcome, nor any stated prerequisites or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Check Harvey Verify server status, uptime, and payment network configuration.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavior. It implies a read-only check via the verb 'Check', but does not explicitly state safety, permissions, or lack of side effects. It is minimally transparent but lacks explicit behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, focused sentence that conveys the essential purpose without fluff. It is front-loaded with the action and resource, earning a top score for efficiency.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple health-check tool with no parameters and no output schema, the description covers the main aspects. However, it does not explain what the response contains (e.g., status object, error behavior), leaving a gap in the agent's understanding of what to expect.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, and the schema confirms this with an empty object. The baseline is 4 because no parameter explanation is needed; the description adds no parameter info, which is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'Check' with a clear resource ('Harvey Verify server') and details what is checked (status, uptime, payment network configuration). This clearly distinguishes it from sibling tools like get_service_quality or report_outcome.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It does not mention explicit use cases, prerequisites, or exclusions, leaving the agent without context for tool selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List all available Harvey Verify tools with pricing and input requirements. Use this for discovery.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It states that the tool lists all tools with pricing and input requirements, which is a clear, read-only operation. It doesn't disclose whether auth is needed or if there is any cost, but for a simple list tool this is adequate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that is concise, front-loaded, and contains no filler. Every word contributes meaning, covering what the tool does and when to use it.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple, parameterless discovery tool, the description is fully sufficient. It explains the return contents (pricing and input requirements) and the use case, and no output schema is present, so the description does not need to explain return values beyond what it already states.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and the schema description coverage is 100%. Per the rubric, a baseline of 4 is appropriate when there are no parameters, as the schema already provides complete information.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('List'), names the resource ('all available Harvey Verify tools'), and specifies the content ('pricing and input requirements'). This clearly distinguishes it from sibling tools like 'get_service_quality' or 'verify_outcome', which are domain operations rather than a discovery/meta tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'Use this for discovery' provides clear context for when the tool should be invoked. However, it does not mention when not to use it or name alternatives, so it lacks explicit exclusions or alternative references.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Record a simple pass/fail outcome report for a service call. No LLM analysis - just logs the result to the quality database. Cheaper alternative to verify_outcome when you only need to record success/failure.
| Name | Required | Description | Default |
|---|---|---|---|
| notes | No | Optional notes about the outcome | |
| service_id | Yes | Service identifier (e.g. 'harvey-tools/scrape_url') | |
| was_successful | Yes | Whether the service call succeeded | |
| response_time_ms | No | Response time in milliseconds |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations are empty, so the description carries the full burden. It usefully discloses that no LLM analysis happens, that the result is logged to a quality database, and the relative cost. It omits meaningful write semantics: whether repeat reports for the same service_id overwrite or accumulate, permission requirements, and what confirmation is returned.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, each doing distinct work: what it does, what it does not do, and when to prefer it. The key differentiator is front-loaded after the purpose statement, with zero filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 4-parameter, 2-required write tool with no output schema, the description covers purpose, cost profile, and routing adequately. The remaining gap is post-call behavior — whether the report is idempotent and what the tool returns — which an agent would want before calling it repeatedly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, including an example format for service_id and units/bounds for response_time_ms, so the schema does the heavy lifting and baseline 3 applies. The description adds no syntax, format, or validation nuance beyond it (e.g., nothing about when notes or response_time_ms are worth supplying).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ("Record a simple pass/fail outcome report for a service call") and immediately distinguishes the scope from the sibling verify_outcome. An agent can tell what this does and what it does not do without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly names the alternative (verify_outcome) and the selection condition: use this when you only need to record success/failure and don't want LLM analysis. It also frames the trade-off (cheaper), which is the practical deciding factor between the two.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Post-transaction verification. Code checks what it can: the response against expected_schema (a JSON Schema or a list of required field names), and a measured response time against a stated limit. A response that fails code's checks fails without the AI judge; otherwise the judge scores accuracy and completeness as advice. Without a schema or a time limit only the judge decides, and decided_by says so. Returns a verdict (pass, fail or inconclusive), decided_by, scores, format compliance, SLA result, issues, and what code checked.
| Name | Required | Description | Default |
|---|---|---|---|
| service_id | No | Service identifier for quality tracking (e.g. 'harvey-tools/scrape_url') | |
| response_data | Yes | The actual response/output received from the service | |
| expected_schema | No | A JSON Schema, or two or more required top-level field names (letters, digits, _ and -) separated by commas, e.g. 'name, price, url'. Anything else is passed to the AI judge only. | |
| sla_requirements | No | SLA wording for the AI judge (e.g. 'must include all fields'). For a time limit code can check, use the two fields below. | |
| request_description | Yes | What was requested from the service | |
| sla_max_response_time_ms | No | The response time limit, in milliseconds | |
| observed_response_time_ms | No | The response time you measured, in milliseconds |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden and does well: it discloses that code-check failures bypass the judge, that judge scores are advisory, and that decided_by reflects which path was taken. It stops short of stating side effects, persistence, or whether verification results are recorded.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose and remains dense with relevant information. The final sentence enumerates return values, which is justified by the absence of an output schema, though the overall text is somewhat packed.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a seven-parameter verification tool with no output schema, the description covers the evaluation logic, judge fallback, and returned fields. It does not explain whether results are stored or how this relates to quality tracking implied by service_id, leaving a minor contextual gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all seven parameters in detail. The description reinforces expected_schema as either a JSON Schema or a field list and mentions response-time limits, but adds little syntax or meaning beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: post-transaction verification of a response against expected_schema and response-time limits, with judge fallback. It is clear what the tool does, but it does not explicitly distinguish itself from sibling tools like report_outcome or get_service_quality.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains the conditional logic of when code checks apply versus when only the judge decides, which is useful context. However, it gives no explicit guidance on when to call this tool versus alternatives, and does not mention siblings or exclusion conditions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
get_service_quality1 field changedInput schema / properties / service_id / maxLengthAdded value: +200report_outcome5 fields changedInput schema / properties / notes / maxLengthAdded value: +2000Input schema / properties / response_time_ms / maximumAdded value: +3600000Input schema / properties / response_time_ms / minimumAdded value: +0Input schema / properties / response_time_ms / typePrevious value: -"number"New value: +"integer"Input schema / properties / service_id / maxLengthAdded value: +200verify_outcome9 fields changedInput schema / properties / expected_schema / descriptionPrevious value: -"Expected output format or JSON schema"New value: +"A JSON Schema, or two or more required top-level field names (letters, digits, _ and -) separated by commas, e.g. 'name, price, url'. Anything else is passed to the AI judge only."Input schema / properties / expected_schema / maxLengthAdded value: +20000Input schema / properties / observed_response_time_msAdded value: +{
+ "description": "The response time you measured, in milliseconds",
+ "maximum": 3600000,
+ "minimum": 0,
+ "type": "integer"
+}Input schema / properties / request_description / maxLengthAdded value: +8000Input schema / properties / response_data / maxLengthAdded value: +100000Input schema / properties / service_id / maxLengthAdded value: +200Input schema / properties / sla_max_response_time_msAdded value: +{
+ "description": "The response time limit, in milliseconds",
+ "maximum": 3600000,
+ "minimum": 0,
+ "type": "integer"
+}Input schema / properties / sla_requirements / descriptionPrevious value: -"SLA requirements to check against (e.g. 'response under 5s, must include all fields')"New value: +"SLA wording for the AI judge (e.g. 'must include all fields'). For a time limit code can check, use the two fields below."Input schema / properties / sla_requirements / maxLengthAdded value: +2000get_service_qualityhealthlist_toolsreport_outcomeverify_outcomeDeterministic verifier: LLM proposes, graph verifies. Sub-ms, offline, 0 tokens.
Verifies AI agent work end to end: real artifacts and outcomes checked, not self-reported success.
AI/LLM agent output audit MCP: policy eval, tamper-evident chain, AI safety, x402 USDC on Base.
Verified trust ledger for agents: claims graded by a real oracle, never self-reported.
Add one secure layer between your agents and this server.