Free agent cost, latency and quality analysis with a key. Evidence purchases are sandbox-only.
Scored across 9 tools
Most tools target a distinct action or product, and the catalog helps clarify the latency/cost/quality analysis APIs. However, analyze_agent_latency and check_agent_quality both describe generic 'analyze supplied agent traces' inputs, so an agent could initially confuse them.
All nine tool names follow the same snake_case verb_noun pattern (analyze_, audit_, check_, get_, purchase_, save_). The repeated purchase_ prefix for related simulated purchases is predictable and clear.
Nine tools is well within the ideal range and each one maps to a distinct part of the agent performance workflow: analysis, discovery, sampling, purchasing, and reporting. The count feels appropriately scoped without redundant or excessive entries.
The surface covers the core performance-analysis lifecycle: latency, cost, quality, catalog discovery, sample data, simulated purchases, and report generation. Minor gaps exist, such as no way to read saved reports or list receipts, but these appear intentional sandbox constraints rather than dead ends.
Analyze supplied agent traces. Free, read-only calculation. Requires an active agent key; does not change agents or initiate payments.
| Name | Required | Description | Default |
|---|---|---|---|
| runs | Yes | ||
| config | No |
| Name | Required | Description |
|---|---|---|
| groups | Yes | |
| measurement | Yes | |
| schema_version | Yes | |
| percentile_method | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, and the description aligns with them. It adds value beyond annotations by revealing the authentication requirement ('requires an active agent key') and explicitly ruling out payment initiation, which is relevant given purchase_* siblings. No contradiction exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences deliver the core action, cost, read-only nature, auth requirement, and absence of side effects. Every clause earns its place and the most important constraint ('read-only calculation') is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the rich input schema (nested runs structure, config constraints, cross-field rules) and presence of an output schema, the description does not need to restate return formats or field semantics. It supplies the missing contextual essentials: free, read-only, requires agent key, no payment initiation. The main gap is that it does not position itself relative to audit_agent_costs and check_agent_quality.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool description does not explain the runs or config parameters, nor does it mention latency_ms optionality, minSamples defaults, or the single-workflow constraint tied to monthlyTasks. The input schema carries detailed descriptions, but with schema description coverage reported as 0%, the description itself fails to compensate for parameter-level meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a clear verb and resource ('Analyze supplied agent traces') and the title 'Latency Lab' reinforces the focus. It is not a tautology. However, it does not explicitly distinguish itself from sibling analysis tools such as audit_agent_costs or check_agent_quality, so it stops short of full differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives operational context ('requires an active agent key', 'does not change agents or initiate payments') but gives no guidance on when to choose this tool over the other analysis siblings. There is no when/when-not statement, no mention of latency-specific use cases, and no alternative tool named.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Free comparison of cost per successful task on supplied paired traces. Requires a sandbox agent key. No purchase, storage, external model call or automatic deployment. Success labels are supplied by the client.
| Name | Required | Description | Default |
|---|---|---|---|
| runs | Yes | ||
| config | No |
| Name | Required | Description |
|---|---|---|
| config | Yes | |
| purpose | Yes | |
| workflows | Yes | |
| methodology | Yes | |
| schema_version | Yes | |
| data_provenance | Yes | |
| input_attempt_records | Yes | |
| experimental_bias_control | Yes | |
| automatic_deployment_authorized | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already provide readOnlyHint=true, and the description adds concrete behavioral specifics: no purchase, storage, external model call, or automatic deployment. It also discloses the authentication requirement (sandbox agent key) and that success labels are client-supplied. These are valuable details beyond the annotation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, each serving a distinct purpose: core function, prerequisite, and behavioral constraints. No filler, and the primary purpose is front-loaded. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
It covers operational prerequisites and side effects (free, no storage, requires key). However, it does not explicitly mention the baseline/candidate pairing or the config options, though the rich schema and output schema reduce the burden. It is mostly complete but could be clearer about the input structure.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, but it only vaguely mentions 'paired traces' without explaining the runs, config, variant, or cost fields. The schema itself contains detailed field descriptions, but the tool description adds no parameter-level guidance, leaving an agent to rely solely on the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action ('comparison of cost per successful task') and its input ('supplied paired traces'). This distinguishes it from sibling tools like analyze_agent_latency (latency analysis) and check_agent_quality (quality checks), making it unambiguous what this tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly lists prerequisites: a sandbox agent key and client-supplied success labels. The 'Free' and 'No purchase' phrasing implies this is a no-cost evaluation tool compared to the purchase_* siblings, though it does not explicitly name alternatives or state when to choose this over them. That is a minor gap.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Analyze supplied agent traces. Free, read-only calculation. Requires an active agent key; does not change agents or initiate payments.
| Name | Required | Description | Default |
|---|---|---|---|
| runs | Yes | ||
| config | No |
| Name | Required | Description |
|---|---|---|
| config | Yes | |
| workflows | Yes | |
| schema_version | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds concrete behavioral context beyond the annotations: the authentication requirement ('active agent key') and financial neutrality ('does not initiate payments'). It also explicitly states 'does not change agents,' reinforcing the readOnlyHint and destructiveHint annotations. No contradictions with the annotation set.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single dense sentence conveys the action, scope, cost, and side-effect profile. Every clause earns its place, and the most important facts are front-loaded. There is no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description, together with a detailed input schema and an output schema, covers the essential call context: what input to provide, what operation is performed, the key requirement, and the absence of side effects. Minor gap: it does not explain what 'quality' means or how thresholds in config affect the result, but the schema and output schema largely cover those details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description provides no parameter-level information at all; schema_description_coverage is 0%. It does not mention runs or config, so it fails to compensate for the low coverage. The input schema itself is well-documented, but the tool description adds zero parameter meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Analyze supplied agent traces.' The title 'Quality Gate' reinforces the tool's purpose. It distinguishes itself from the purchase_* siblings via 'Free, read-only calculation' and 'does not initiate payments,' though it does not explicitly differentiate from the other analysis sibling tools like analyze_agent_latency.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Clear context is provided: free, read-only, requires an active agent key, and no side effects. This implies use when a no-cost quality assessment is needed without triggering payments. However, it does not explicitly name alternatives or state when not to use this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Discover cost, latency and quality analysis APIs, their inputs, outputs, free reproducible example, authentication and current payment availability. No payment or budget debit.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark the tool as read-only, idempotent, and non-destructive. The description adds the meaningful behavioral guarantee of 'No payment or budget debit' and notes the catalog includes authentication and payment availability, which helps an agent avoid expecting a paid or side-effecting operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences with no filler; the catalog contents are front-loaded and the clarifying no-debit statement is placed after the main payload. Every clause adds information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple no-parameter catalog lookup with no output schema, the description adequately lists what will be returned (API metadata, inputs/outputs, example, auth, payment availability) and the non-side-effect nature. It does not state the exact response format, but that is not critical for a zero-input introspection tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so the input schema fully describes the call shape and the description adds no parameter burden. Baseline 4 applies because there is nothing for the description to explain about inputs.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Discover') and identifies the resource (cost, latency, and quality analysis APIs) plus the exact metadata returned (inputs, outputs, example, authentication, payment availability). 'No payment or budget debit' clearly separates this catalog/introspection tool from the purchase_* siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context: use this tool to explore available analysis APIs and their metadata, not to perform actions. 'No payment or budget debit' functions as an exclusion, but it does not explicitly name alternative tools such as purchase_changes or get_free_sample.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Read a dated primary-source sample about OpenAI. Not real-time or exhaustive.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds behavioral context beyond the readOnlyHint annotation by noting the data is 'dated,' a 'sample,' and 'not real-time or exhaustive.' This helps an agent set expectations about freshness and completeness. It does not contradict the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that front-loads the core function and caveats. Every word earns its place; there is no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, read-only tool with no output schema, the description provides sufficient context about the nature and limitations of the sample. It does not detail the output structure, but the name and description are enough for an agent to invoke correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the baseline is 4. There is no parameter semantics burden on the description, and no parameter information is needed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb and resource: 'Read a dated primary-source sample about OpenAI.' It clearly identifies the tool's scope as a sample and distinguishes it from the sibling analysis/purchase tools, though it does not explicitly name an alternative for broader or real-time data.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'Not real-time or exhaustive' gives explicit negative guidance about when not to use the tool, and 'dated primary-source sample' implies the appropriate use case. However, it does not name sibling tools like get_catalog as alternatives for broader coverage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Simulate a Change Set purchase for 0.05 fictitious USDC. Creates a persistent sandbox receipt and debits the authorized agent test budget. Never transfers real funds.
| Name | Required | Description | Default |
|---|---|---|---|
| since | No | ||
| idempotency_key | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations, the description discloses consequential behaviors: creates a persistent sandbox receipt, debits the authorized agent test budget, and never transfers real funds. This is exactly the kind of side-effect context an agent needs and goes well beyond readOnlyHint/idempotentHint/destructiveHint.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two focused sentences, with the primary purpose first and side effects immediately after. Every sentence earns its place; there is no filler or repetition of schema details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a low-complexity tool with annotations and a required idempotency key, the description covers the main behavioral contract: simulation, receipt persistence, budget debit, and no real funds. It could be more complete by explaining the since parameter or the returned receipt shape, but the core calling context is present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for the undocumented parameters. It does not mention idempotency_key or since at all, leaving the agent to rely solely on parameter names and regex patterns. This is a significant gap for a tool with an obvious retry/idempotency parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Simulate a Change Set purchase' for 0.05 fictitious USDC. It clearly marks the operation as a sandbox simulation, distinguishing it from real purchase tools without needing to read the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The sandbox and 'fictitious USDC' language clearly establish the intended testing context particular. It also states 'Never transfers real funds,' which serves as an implicit exclusion for any real-money purchase. It does not explicitly name sibling alternatives like purchase_evidence, but the context is clear enough.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Simulate a Evidence Pack purchase for 0.25 fictitious USDC. Creates a persistent sandbox receipt and debits the authorized agent test budget. Never transfers real funds.
| Name | Required | Description | Default |
|---|---|---|---|
| since | No | ||
| idempotency_key | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare readOnlyHint=false, destructiveHint=false, idempotentHint=true. The description adds important behavior beyond that: creates a persistent sandbox receipt, debits an authorized agent test budget, and never transfers real funds. These side effects are not inferable from the annotations, and the description does not contradict them.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the action and key limitation ('Never transfers real funds'). No waste; every clause adds crucial information. The structure is efficient and scannable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description discloses main side effects but omits both parameter semantics (see above) and return value details (no output schema). It also does not address usage relative to siblings. Given the tool's simplicity (2 params, no nested objects) and that the safety profile is clear, it is partially complete but not fully self-sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description does not mention either 'since' or 'idempotency_key'. The schema provides only format patterns (date format, key pattern) without semantic meaning. The description fails entirely to compensate for the lack of parameter documentation, leaving the agent without guidance on what these parameters represent or how to use them.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb (simulate purchase) on a specific resource (Evidence Pack) and adds concrete consequences (creates receipt, debits budget). It does not explicitly name sibling alternatives, but the resource name 'Evidence Pack' distinguishes it from purchase_changes and purchase_snapshot, so purpose is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus siblings like purchase_changes, purchase_snapshot, or get_free_sample. The term 'sandbox' implies a testing context, but the description does not state conditions for selection or exclude any alternatives, leaving the agent to infer usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Simulate a Snapshot purchase for 0.01 fictitious USDC. Creates a persistent sandbox receipt and debits the authorized agent test budget. Never transfers real funds.
| Name | Required | Description | Default |
|---|---|---|---|
| since | No | ||
| idempotency_key | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate non-readonly and idempotent, but the description adds critical context: no real funds transferred, and it debits an agent test budget. However, it doesn't disclose the eventual effects of repeated sandbox receipts or budget depletion.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three concise sentences, front-loaded with the core purpose. Every word earns its place, and the safety note is clear.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simulation tool with no output schema, it explains the key side effects (debiting test budget, no real funds). However, lack of parameter explanation is a gap, and it doesn't mention the idempotency key's role in preventing duplicate sandbox receipts.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must explain parameters. It doesn't mention the purpose of 'since' or 'idempotency_key' beyond what the schema provides (names and patterns). The idempotency_key is required and critical for retry safety, but the description is silent on it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool simulates a Snapshot purchase using fictitious USDC, creating a sandbox receipt and debiting a test budget. This is specific and distinguishes it from real purchase tools like purchase_changes and purchase_evidence.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies safe usage for testing without real funds, but it does not explicitly state when to choose this tool over alternatives like purchase_changes or purchase_evidence. It lacks explicit 'when not to use' guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Compute and save an aggregate performance report in the fixed Projects destination authorized by the account owner. Uses the existing free or paid report allowance. Requires an active agent key and explicit owner write permission. Reuse request_id with identical content to retry within the same permission grant. No report reading, deletion, subscription or payment authorization. Send measurements without secrets.
| Name | Required | Description | Default |
|---|---|---|---|
| input | Yes | Recorded attempts, not prompts or secrets. Identical rows count as separate attempts. All three analysis tools accept this same input. The runtime validator additionally enforces six-decimal micro-USD precision, the same task_id belonging to one workflow, one workflow when monthlyTasks is given, and UTF-16 length limits. HTTP body limit: 512000 UTF-8 bytes. | |
| title | Yes | ||
| request_id | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide idempotentHint=true, readOnlyHint=false, and destructiveHint=false. The description adds valuable behavior beyond those: permission requirements, allowance consumption, fixed destination, retry semantics within a permission grant, and a security directive ('Send measurements without secrets'). There is no contradiction with the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two dense sentences front-load the core purpose and destination, then pack the necessary permissions, idempotency, exclusions, and data-sensitivity warning. Every clause adds useful guidance and there is no filler or duplication of schema details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the nested input schema, the description supplies essential invocation context: authorization, destination, allowance, retry, and 'no reading/deletion/payment' boundaries. The main gap is that there is no output schema and the description does not state what the tool returns or how failure manifests, which would make the calling contract fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 33%, and the description compensates somewhat by explaining request_id retry semantics and warning that input should contain measurements, not secrets. However, it does not explain title's purpose, config semantics, or how the input object maps to report computation. The schema itself covers some parameters, but the description leaves meaning gaps for the remaining ones.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource: 'Compute and save an aggregate performance report in the fixed Projects destination.' This clearly states the action, object, and destination, and the explicit exclusions ('No report reading, deletion, subscription or payment authorization') distinguish it from sibling analysis and purchase tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear operational context: it requires an active agent key and explicit owner write permission, uses the report allowance, and supports idempotent retry via request_id. It states exclusions ('No report reading, deletion, subscription or payment authorization') but does not explicitly name sibling alternatives or state 'use X when Y' conditions, so it stops short of a full 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
analyze_agent_latency9 fields changedInput schema / descriptionAdded value: +"Recorded attempts, not prompts or secrets. Identical rows count as separate attempts. All three analysis tools accept this same input. The runtime validator additionally enforces six-decimal micro-USD precision, the same task_id belonging to one workflow, one workflow when monthlyTasks is given, and UTF-16 length limits. HTTP body limit: 512000 UTF-8 bytes."Input schema / properties / config / additionalPropertiesAdded value: +falseInput schema / properties / config / propertiesAdded value: +{
+ "maxP95LatencyMs": {
+ "maximum": 86400000000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "maxSuccessRateDrop": {
+ "default": 0.02,
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "minSamples": {
+ "default": 30,
+ "maximum": 500,
+ "minimum": 2,
+ "type": "integer"
+ },
+ "minSuccessRate": {
+ "default": 0.95,
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "monthlyTasks": {
+ "description": "Baseline logical tasks launched per month; the engine requires exactly one workflow if supplied.",
+ "maximum": 1000000,
+ "minimum": 1,
+ "type": "integer"
+ }
+}Input schema / properties / config / requiredAdded value: +[]Input schema / properties / runs / items / additionalPropertiesAdded value: +falseInput schema / properties / runs / items / propertiesAdded value: +{
+ "cost_usd": {
+ "description": "USD cost of this attempt. At most six decimal places; the engine verifies whole micro-USD using floating-point tolerance. No multipleOf keyword is used, to avoid rejecting valid JSON decimals.",
+ "maximum": 10000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "latency_ms": {
+ "description": "Recorded attempt duration in milliseconds; decimals are accepted. Omit if not measured.",
+ "maximum": 86400000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "success": {
+ "type": "boolean"
+ },
+ "task_id": {
+ "description": "Nonempty, no control characters or surrounding whitespace. The engine additionally enforces at most 128 UTF-16 code units.",
+ "maxLength": 128,
+ "minLength": 1,
+ "pattern": "^(?!\\s)(?![\\s\\S]*\\s$)[^\\u0000-\\u001F\\u007F]+$",
+ "type": "string"
+ },
+ "variant": {
+ "enum": [
+ "baseline",
+ "candidate"
+ ],
+ "type": "string"
+ },
+ "workflow": {
+ "description": "Nonempty, no control characters or surrounding whitespace. The engine additionally enforces at most 80 UTF-16 code units.",
+ "maxLength": 80,
+ "minLength": 1,
+ "pattern": "^(?!\\s)(?![\\s\\S]*\\s$)[^\\u0000-\\u001F\\u007F]+$",
+ "type": "string"
+ }
+}Input schema / properties / runs / items / requiredAdded value: +[
+ "task_id",
+ "workflow",
+ "variant",
+ "cost_usd",
+ "success"
+]Input schema / titleAdded value: +"ALPNAI recorded agent attempts"Output schema / (root)Previous value: -nullNew value: +{
+ "additionalProperties": true,
+ "properties": {
+ "groups": {
+ "items": {
+ "additionalProperties": true,
+ "properties": {
+ "duration_coverage": {
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "max_ms": {
+ "anyOf": [
+ {
+ "maximum": 86400000000,
+ "minimum": 0,
+ "type": "number"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "p50_ms": {
+ "anyOf": [
+ {
+ "maximum": 86400000000,
+ "minimum": 0,
+ "type": "number"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "p95_ms": {
+ "anyOf": [
+ {
+ "maximum": 86400000000,
+ "minimum": 0,
+ "type": "number"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "recorded_attempts": {
+ "maximum": 1000,
+ "minimum": 1,
+ "type": "integer"
+ },
+ "retry_attempts": {
+ "maximum": 999,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "tasks": {
+ "maximum": 1000,
+ "minimum": 1,
+ "type": "integer"
+ },
+ "threshold_ms": {
+ "anyOf": [
+ {
+ "maximum": 86400000000,
+ "minimum": 0,
+ "type": "number"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "variant": {
+ "enum": [
+ "baseline",
+ "candidate"
+ ],
+ "type": "string"
+ },
+ "within_p95_threshold": {
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "workflow": {
+ "description": "Nonempty, no control characters or surrounding whitespace. The engine additionally enforces at most 80 UTF-16 code units.",
+ "maxLength": 80,
+ "minLength": 1,
+ "pattern": "^(?!\\s)(?![\\s\\S]*\\s$)[^\\u0000-\\u001F\\u007F]+$",
+ "type": "string"
+ }
+ },
+ "required": [
+ "workflow",
+ "variant",
+ "tasks",
+ "recorded_attempts",
+ "retry_attempts",
+ "duration_coverage",
+ "p50_ms",
+ "p95_ms",
+ "max_ms",
+ "threshold_ms",
+ "within_p95_threshold"
+ ],
+ "type": "object"
+ },
+ "maxItems": 1000,
+ "minItems": 1,
+ "type": "array"
+ },
+ "measurement": {
+ "const": "summed_recorded_attempt_durations_per_task",
+ "type": "string"
+ },
+ "percentile_method": {
+ "const": "nearest_rank",
+ "type": "string"
+ },
+ "schema_version": {
+ "const": "1.0.0",
+ "type": "string"
+ }
+ },
+ "required": [
+ "schema_version",
+ "measurement",
+ "percentile_method",
+ "groups"
+ ],
+ "type": "object"
+}audit_agent_costs9 fields changedInput schema / descriptionAdded value: +"Recorded attempts, not prompts or secrets. Identical rows count as separate attempts. All three analysis tools accept this same input. The runtime validator additionally enforces six-decimal micro-USD precision, the same task_id belonging to one workflow, one workflow when monthlyTasks is given, and UTF-16 length limits. HTTP body limit: 512000 UTF-8 bytes."Input schema / properties / config / additionalPropertiesAdded value: +falseInput schema / properties / config / propertiesAdded value: +{
+ "maxP95LatencyMs": {
+ "maximum": 86400000000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "maxSuccessRateDrop": {
+ "default": 0.02,
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "minSamples": {
+ "default": 30,
+ "maximum": 500,
+ "minimum": 2,
+ "type": "integer"
+ },
+ "minSuccessRate": {
+ "default": 0.95,
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "monthlyTasks": {
+ "description": "Baseline logical tasks launched per month; the engine requires exactly one workflow if supplied.",
+ "maximum": 1000000,
+ "minimum": 1,
+ "type": "integer"
+ }
+}Input schema / properties / config / requiredAdded value: +[]Input schema / properties / runs / items / additionalPropertiesAdded value: +falseInput schema / properties / runs / items / propertiesAdded value: +{
+ "cost_usd": {
+ "description": "USD cost of this attempt. At most six decimal places; the engine verifies whole micro-USD using floating-point tolerance. No multipleOf keyword is used, to avoid rejecting valid JSON decimals.",
+ "maximum": 10000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "latency_ms": {
+ "description": "Recorded attempt duration in milliseconds; decimals are accepted. Omit if not measured.",
+ "maximum": 86400000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "success": {
+ "type": "boolean"
+ },
+ "task_id": {
+ "description": "Nonempty, no control characters or surrounding whitespace. The engine additionally enforces at most 128 UTF-16 code units.",
+ "maxLength": 128,
+ "minLength": 1,
+ "pattern": "^(?!\\s)(?![\\s\\S]*\\s$)[^\\u0000-\\u001F\\u007F]+$",
+ "type": "string"
+ },
+ "variant": {
+ "enum": [
+ "baseline",
+ "candidate"
+ ],
+ "type": "string"
+ },
+ "workflow": {
+ "description": "Nonempty, no control characters or surrounding whitespace. The engine additionally enforces at most 80 UTF-16 code units.",
+ "maxLength": 80,
+ "minLength": 1,
+ "pattern": "^(?!\\s)(?![\\s\\S]*\\s$)[^\\u0000-\\u001F\\u007F]+$",
+ "type": "string"
+ }
+}Input schema / properties / runs / items / requiredAdded value: +[
+ "task_id",
+ "workflow",
+ "variant",
+ "cost_usd",
+ "success"
+]Input schema / titleAdded value: +"ALPNAI recorded agent attempts"Output schema / (root)Previous value: -nullNew value: +{
+ "additionalProperties": true,
+ "properties": {
+ "automatic_deployment_authorized": {
+ "const": false,
+ "type": "boolean"
+ },
+ "config": {
+ "additionalProperties": true,
+ "properties": {
+ "maxP95LatencyMs": {
+ "maximum": 86400000000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "maxSuccessRateDrop": {
+ "default": 0.02,
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "minSamples": {
+ "default": 30,
+ "maximum": 500,
+ "minimum": 2,
+ "type": "integer"
+ },
+ "minSuccessRate": {
+ "default": 0.95,
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "monthlyTasks": {
+ "description": "Baseline logical tasks launched per month; the engine requires exactly one workflow if supplied.",
+ "maximum": 1000000,
+ "minimum": 1,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "minSamples",
+ "minSuccessRate",
+ "maxSuccessRateDrop"
+ ],
+ "type": "object"
+ },
+ "data_provenance": {
+ "const": "user_supplied_not_verified",
+ "type": "string"
+ },
+ "experimental_bias_control": {
+ "const": "unknown",
+ "type": "string"
+ },
+ "input_attempt_records": {
+ "maximum": 1000,
+ "minimum": 1,
+ "type": "integer"
+ },
+ "methodology": {
+ "additionalProperties": true,
+ "properties": {
+ "duplicates": {
+ "type": "string"
+ },
+ "latency": {
+ "type": "string"
+ },
+ "logical_task_key": {
+ "items": {
+ "type": "string"
+ },
+ "maxItems": 3,
+ "minItems": 3,
+ "type": "array"
+ },
+ "missing_costs": {
+ "type": "string"
+ },
+ "quality": {
+ "type": "string"
+ },
+ "success": {
+ "type": "string"
+ },
+ "uncertainty": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "logical_task_key",
+ "success",
+ "duplicates",
+ "latency",
+ "quality",
+ "uncertainty",
+ "missing_costs"
+ ],
+ "type": "object"
+ },
+ "purpose": {
+ "const": "cost_per_successful_agent_task_audit",
+ "type": "string"
+ },
+ "schema_version": {
+ "const": "1.0.0",
+ "type": "string"
+ },
+ "workflows": {
+ "items": {
+ "additionalProperties": true,
+ "properties": {
+ "automatic_deployment_authorized": {
+ "const": false,
+ "type": "boolean"
+ },
+ "baseline": {
+ "anyOf": [
+ {
+ "additionalProperties": true,
+ "properties": {
+ "attempts": {
+ "maximum": 1000,
+ "minimum": 1,
+ "type": "integer"
+ },
+ "cost_per_successful_task_usd": {
+ "anyOf": [
+ {
+ "minimum": 0,
+ "type": "number"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "cost_per_task_usd": {
+ "minimum": 0,
+ "type": "number"
+ },
+ "latency_complete": {
+ "type": "boolean"
+ },
+ "p95_recorded_attempt_latency_per_task_ms": {
+ "anyOf": [
+ {
+ "maximum": 86400000000,
+ "minimum": 0,
+ "type": "number"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "retry_attempts": {
+ "maximum": 999,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "success_rate": {
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "success_rate_interval_95_wilson": {
+ "items": {
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "maxItems": 2,
+ "minItems": 2,
+ "type": "array"
+ },
+ "successful_tasks": {
+ "maximum": 1000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "tasks": {
+ "maximum": 1000,
+ "minimum": 1,
+ "type": "integer"
+ },
+ "total_cost_usd": {
+ "minimum": 0,
+ "type": "number"
+ }
+ },
+ "required": [
+ "tasks",
+ "attempts",
+ "retry_attempts",
+ "successful_tasks",
+ "total_cost_usd",
+ "cost_per_task_usd",
+ "cost_per_successful_task_usd",
+ "success_rate",
+ "success_rate_interval_95_wilson",
+ "p95_recorded_attempt_latency_per_task_ms",
+ "latency_complete"
+ ],
+ "type": "object"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "candidate": {
+ "anyOf": [
+ {
+ "additionalProperties": true,
+ "properties": {
+ "attempts": {
+ "maximum": 1000,
+ "minimum": 1,
+ "type": "integer"
+ },
+ "cost_per_successful_task_usd": {
+ "anyOf": [
+ {
+ "minimum": 0,
+ "type": "number"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "cost_per_task_usd": {
+ "minimum": 0,
+ "type": "number"
+ },
+ "latency_complete": {
+ "type": "boolean"
+ },
+ "p95_recorded_attempt_latency_per_task_ms": {
+ "anyOf": [
+ {
+ "maximum": 86400000000,
+ "minimum": 0,
+ "type": "number"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "retry_attempts": {
+ "maximum": 999,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "success_rate": {
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "success_rate_interval_95_wilson": {
+ "items": {
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "maxItems": 2,
+ "minItems": 2,
+ "type": "array"
+ },
+ "successful_tasks": {
+ "maximum": 1000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "tasks": {
+ "maximum": 1000,
+ "minimum": 1,
+ "type": "integer"
+ },
+ "total_cost_usd": {
+ "minimum": 0,
+ "type": "number"
+ }
+ },
+ "required": [
+ "tasks",
+ "attempts",
+ "retry_attempts",
+ "successful_tasks",
+ "total_cost_usd",
+ "cost_per_task_usd",
+ "cost_per_successful_task_usd",
+ "success_rate",
+ "success_rate_interval_95_wilson",
+ "p95_recorded_attempt_latency_per_task_ms",
+ "latency_complete"
+ ],
+ "type": "object"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "comparison": {
+ "additionalProperties": true,
+ "properties": {
+ "baseline_only_tasks": {
+ "maximum": 1000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "candidate_only_tasks": {
+ "maximum": 1000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "matched_task_ids": {
+ "maximum": 1000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "same_task_set": {
+ "type": "boolean"
+ }
+ },
+ "required": [
+ "matched_task_ids",
+ "baseline_only_tasks",
+ "candidate_only_tasks",
+ "same_task_set"
+ ],
+ "type": "object"
+ },
+ "decision": {
+ "enum": [
+ "missing_comparison",
+ "collect_more_data",
+ "quality_regression",
+ "latency_data_required",
+ "latency_regression",
+ "no_economic_advantage",
+ "candidate_for_controlled_trial"
+ ],
+ "type": "string"
+ },
+ "gates": {
+ "additionalProperties": true,
+ "properties": {
+ "both_variants": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown",
+ "not_requested"
+ ],
+ "type": "string"
+ },
+ "experimental_bias_control": {
+ "const": "unknown",
+ "type": "string"
+ },
+ "lower_cost_per_successful_task": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown",
+ "not_requested"
+ ],
+ "type": "string"
+ },
+ "minimum_distinct_tasks_per_variant": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown",
+ "not_requested"
+ ],
+ "type": "string"
+ },
+ "observed_success_rate": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown",
+ "not_requested"
+ ],
+ "type": "string"
+ },
+ "recorded_latency": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown",
+ "not_requested"
+ ],
+ "type": "string"
+ },
+ "same_task_set": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown",
+ "not_requested"
+ ],
+ "type": "string"
+ }
+ },
+ "required": [
+ "both_variants",
+ "minimum_distinct_tasks_per_variant",
+ "same_task_set",
+ "observed_success_rate",
+ "recorded_latency",
+ "lower_cost_per_successful_task",
+ "experimental_bias_control"
+ ],
+ "type": "object"
+ },
+ "opportunity_monthly": {
+ "anyOf": [
+ {
+ "additionalProperties": true,
+ "properties": {
+ "baseline_expected_cost_usd": {
+ "minimum": 0,
+ "type": "number"
+ },
+ "baseline_launched_tasks_per_month": {
+ "maximum": 1000000,
+ "minimum": 1,
+ "type": "integer"
+ },
+ "basis": {
+ "const": "same_expected_successful_task_volume",
+ "type": "string"
+ },
+ "candidate_expected_cost_usd": {
+ "minimum": 0,
+ "type": "number"
+ },
+ "candidate_expected_launched_tasks_per_month": {
+ "minimum": 0,
+ "type": "number"
+ },
+ "conditions": {
+ "items": {
+ "type": "string"
+ },
+ "minItems": 1,
+ "type": "array"
+ },
+ "currency": {
+ "const": "USD",
+ "type": "string"
+ },
+ "excludes": {
+ "items": {
+ "type": "string"
+ },
+ "minItems": 1,
+ "type": "array"
+ },
+ "expected_successful_tasks_per_month": {
+ "minimum": 0,
+ "type": "number"
+ },
+ "period": {
+ "const": "month",
+ "type": "string"
+ },
+ "potential_cost_difference_usd": {
+ "minimum": 0,
+ "type": "number"
+ },
+ "status": {
+ "const": "conditional_projection_not_realized",
+ "type": "string"
+ }
+ },
+ "required": [
+ "status",
+ "basis",
+ "currency",
+ "period",
+ "baseline_launched_tasks_per_month",
+ "expected_successful_tasks_per_month",
+ "candidate_expected_launched_tasks_per_month",
+ "baseline_expected_cost_usd",
+ "candidate_expected_cost_usd",
+ "potential_cost_difference_usd",
+ "excludes",
+ "conditions"
+ ],
+ "type": "object"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "realized_savings_usd": {
+ "type": "null"
+ },
+ "workflow": {
+ "description": "Nonempty, no control characters or surrounding whitespace. The engine additionally enforces at most 80 UTF-16 code units.",
+ "maxLength": 80,
+ "minLength": 1,
+ "pattern": "^(?!\\s)(?![\\s\\S]*\\s$)[^\\u0000-\\u001F\\u007F]+$",
+ "type": "string"
+ }
+ },
+ "required": [
+ "workflow",
+ "baseline",
+ "candidate",
+ "comparison",
+ "gates",
+ "decision",
+ "automatic_deployment_authorized",
+ "realized_savings_usd",
+ "opportunity_monthly"
+ ],
+ "type": "object"
+ },
+ "maxItems": 1000,
+ "minItems": 1,
+ "type": "array"
+ }
+ },
+ "required": [
+ "schema_version",
+ "purpose",
+ "data_provenance",
+ "config",
+ "input_attempt_records",
+ "experimental_bias_control",
+ "automatic_deployment_authorized",
+ "methodology",
+ "workflows"
+ ],
+ "type": "object"
+}check_agent_quality9 fields changedInput schema / descriptionAdded value: +"Recorded attempts, not prompts or secrets. Identical rows count as separate attempts. All three analysis tools accept this same input. The runtime validator additionally enforces six-decimal micro-USD precision, the same task_id belonging to one workflow, one workflow when monthlyTasks is given, and UTF-16 length limits. HTTP body limit: 512000 UTF-8 bytes."Input schema / properties / config / additionalPropertiesAdded value: +falseInput schema / properties / config / propertiesAdded value: +{
+ "maxP95LatencyMs": {
+ "maximum": 86400000000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "maxSuccessRateDrop": {
+ "default": 0.02,
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "minSamples": {
+ "default": 30,
+ "maximum": 500,
+ "minimum": 2,
+ "type": "integer"
+ },
+ "minSuccessRate": {
+ "default": 0.95,
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "monthlyTasks": {
+ "description": "Baseline logical tasks launched per month; the engine requires exactly one workflow if supplied.",
+ "maximum": 1000000,
+ "minimum": 1,
+ "type": "integer"
+ }
+}Input schema / properties / config / requiredAdded value: +[]Input schema / properties / runs / items / additionalPropertiesAdded value: +falseInput schema / properties / runs / items / propertiesAdded value: +{
+ "cost_usd": {
+ "description": "USD cost of this attempt. At most six decimal places; the engine verifies whole micro-USD using floating-point tolerance. No multipleOf keyword is used, to avoid rejecting valid JSON decimals.",
+ "maximum": 10000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "latency_ms": {
+ "description": "Recorded attempt duration in milliseconds; decimals are accepted. Omit if not measured.",
+ "maximum": 86400000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "success": {
+ "type": "boolean"
+ },
+ "task_id": {
+ "description": "Nonempty, no control characters or surrounding whitespace. The engine additionally enforces at most 128 UTF-16 code units.",
+ "maxLength": 128,
+ "minLength": 1,
+ "pattern": "^(?!\\s)(?![\\s\\S]*\\s$)[^\\u0000-\\u001F\\u007F]+$",
+ "type": "string"
+ },
+ "variant": {
+ "enum": [
+ "baseline",
+ "candidate"
+ ],
+ "type": "string"
+ },
+ "workflow": {
+ "description": "Nonempty, no control characters or surrounding whitespace. The engine additionally enforces at most 80 UTF-16 code units.",
+ "maxLength": 80,
+ "minLength": 1,
+ "pattern": "^(?!\\s)(?![\\s\\S]*\\s$)[^\\u0000-\\u001F\\u007F]+$",
+ "type": "string"
+ }
+}Input schema / properties / runs / items / requiredAdded value: +[
+ "task_id",
+ "workflow",
+ "variant",
+ "cost_usd",
+ "success"
+]Input schema / titleAdded value: +"ALPNAI recorded agent attempts"Output schema / (root)Previous value: -nullNew value: +{
+ "additionalProperties": true,
+ "properties": {
+ "config": {
+ "additionalProperties": true,
+ "properties": {
+ "maxP95LatencyMs": {
+ "maximum": 86400000000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "maxSuccessRateDrop": {
+ "default": 0.02,
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "minSamples": {
+ "default": 30,
+ "maximum": 500,
+ "minimum": 2,
+ "type": "integer"
+ },
+ "minSuccessRate": {
+ "default": 0.95,
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "monthlyTasks": {
+ "description": "Baseline logical tasks launched per month; the engine requires exactly one workflow if supplied.",
+ "maximum": 1000000,
+ "minimum": 1,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "minSamples",
+ "minSuccessRate",
+ "maxSuccessRateDrop"
+ ],
+ "type": "object"
+ },
+ "schema_version": {
+ "const": "1.0.0",
+ "type": "string"
+ },
+ "workflows": {
+ "items": {
+ "additionalProperties": true,
+ "properties": {
+ "automatic_deployment_authorized": {
+ "const": false,
+ "type": "boolean"
+ },
+ "baseline_success": {
+ "anyOf": [
+ {
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "candidate_success": {
+ "anyOf": [
+ {
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "comparison": {
+ "additionalProperties": true,
+ "properties": {
+ "baseline_only_tasks": {
+ "maximum": 1000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "candidate_only_tasks": {
+ "maximum": 1000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "matched_task_ids": {
+ "maximum": 1000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "same_task_set": {
+ "type": "boolean"
+ }
+ },
+ "required": [
+ "matched_task_ids",
+ "baseline_only_tasks",
+ "candidate_only_tasks",
+ "same_task_set"
+ ],
+ "type": "object"
+ },
+ "decision": {
+ "enum": [
+ "missing_comparison",
+ "collect_more_data",
+ "quality_regression",
+ "latency_data_required",
+ "latency_regression",
+ "no_economic_advantage",
+ "candidate_for_controlled_trial"
+ ],
+ "type": "string"
+ },
+ "gates": {
+ "additionalProperties": true,
+ "properties": {
+ "both_variants": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown",
+ "not_requested"
+ ],
+ "type": "string"
+ },
+ "experimental_bias_control": {
+ "const": "unknown",
+ "type": "string"
+ },
+ "lower_cost_per_successful_task": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown",
+ "not_requested"
+ ],
+ "type": "string"
+ },
+ "minimum_distinct_tasks_per_variant": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown",
+ "not_requested"
+ ],
+ "type": "string"
+ },
+ "observed_success_rate": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown",
+ "not_requested"
+ ],
+ "type": "string"
+ },
+ "recorded_latency": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown",
+ "not_requested"
+ ],
+ "type": "string"
+ },
+ "same_task_set": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown",
+ "not_requested"
+ ],
+ "type": "string"
+ }
+ },
+ "required": [
+ "both_variants",
+ "minimum_distinct_tasks_per_variant",
+ "same_task_set",
+ "observed_success_rate",
+ "recorded_latency",
+ "lower_cost_per_successful_task",
+ "experimental_bias_control"
+ ],
+ "type": "object"
+ },
+ "workflow": {
+ "description": "Nonempty, no control characters or surrounding whitespace. The engine additionally enforces at most 80 UTF-16 code units.",
+ "maxLength": 80,
+ "minLength": 1,
+ "pattern": "^(?!\\s)(?![\\s\\S]*\\s$)[^\\u0000-\\u001F\\u007F]+$",
+ "type": "string"
+ }
+ },
+ "required": [
+ "workflow",
+ "comparison",
+ "gates",
+ "decision",
+ "baseline_success",
+ "candidate_success",
+ "automatic_deployment_authorized"
+ ],
+ "type": "object"
+ },
+ "maxItems": 1000,
+ "minItems": 1,
+ "type": "array"
+ }
+ },
+ "required": [
+ "schema_version",
+ "config",
+ "workflows"
+ ],
+ "type": "object"
+}save_project_report8 fields changedInput schema / properties / input / descriptionAdded value: +"Recorded attempts, not prompts or secrets. Identical rows count as separate attempts. All three analysis tools accept this same input. The runtime validator additionally enforces six-decimal micro-USD precision, the same task_id belonging to one workflow, one workflow when monthlyTasks is given, and UTF-16 length limits. HTTP body limit: 512000 UTF-8 bytes."Input schema / properties / input / properties / config / additionalPropertiesAdded value: +falseInput schema / properties / input / properties / config / propertiesAdded value: +{
+ "maxP95LatencyMs": {
+ "maximum": 86400000000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "maxSuccessRateDrop": {
+ "default": 0.02,
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "minSamples": {
+ "default": 30,
+ "maximum": 500,
+ "minimum": 2,
+ "type": "integer"
+ },
+ "minSuccessRate": {
+ "default": 0.95,
+ "maximum": 1,
+ "minimum": 0,
+ "type": "number"
+ },
+ "monthlyTasks": {
+ "description": "Baseline logical tasks launched per month; the engine requires exactly one workflow if supplied.",
+ "maximum": 1000000,
+ "minimum": 1,
+ "type": "integer"
+ }
+}Input schema / properties / input / properties / config / requiredAdded value: +[]Input schema / properties / input / properties / runs / items / additionalPropertiesAdded value: +falseInput schema / properties / input / properties / runs / items / propertiesAdded value: +{
+ "cost_usd": {
+ "description": "USD cost of this attempt. At most six decimal places; the engine verifies whole micro-USD using floating-point tolerance. No multipleOf keyword is used, to avoid rejecting valid JSON decimals.",
+ "maximum": 10000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "latency_ms": {
+ "description": "Recorded attempt duration in milliseconds; decimals are accepted. Omit if not measured.",
+ "maximum": 86400000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "success": {
+ "type": "boolean"
+ },
+ "task_id": {
+ "description": "Nonempty, no control characters or surrounding whitespace. The engine additionally enforces at most 128 UTF-16 code units.",
+ "maxLength": 128,
+ "minLength": 1,
+ "pattern": "^(?!\\s)(?![\\s\\S]*\\s$)[^\\u0000-\\u001F\\u007F]+$",
+ "type": "string"
+ },
+ "variant": {
+ "enum": [
+ "baseline",
+ "candidate"
+ ],
+ "type": "string"
+ },
+ "workflow": {
+ "description": "Nonempty, no control characters or surrounding whitespace. The engine additionally enforces at most 80 UTF-16 code units.",
+ "maxLength": 80,
+ "minLength": 1,
+ "pattern": "^(?!\\s)(?![\\s\\S]*\\s$)[^\\u0000-\\u001F\\u007F]+$",
+ "type": "string"
+ }
+}Input schema / properties / input / properties / runs / items / requiredAdded value: +[
+ "task_id",
+ "workflow",
+ "variant",
+ "cost_usd",
+ "success"
+]Input schema / properties / input / titleAdded value: +"ALPNAI recorded agent attempts"save_project_reportanalyze_agent_latencyaudit_agent_costscheck_agent_qualityget_catalogget_free_samplepurchase_changespurchase_evidencepurchase_snapshotFree agent tests (no account or payment); paid ping.
Free spend report from an agent log, no key. Hosted proxies: caps, audit export, buy calls.
Free buyer-side outcome verification and provider comparison for supplied agent receipts.
Free agent-service discovery, OpenAPI document checks, and receipt verification. No API key needed.
Add one secure layer between your agents and this server.