agentflight

Overview Schema Source Related Servers Discussions Admin

agentflight_diff_cassettes

Read-onlyIdempotent

Compares a baseline golden cassette against a candidate to detect agent behavioral regressions, tool sequence divergence, new tool failures, loops, and token or cost deltas after model or prompt upgrades.

Instructions

Compare a baseline golden AgentFlight cassette against a candidate cassette to detect behavioral regressions.

Purpose

Performs semantic trajectory diffing across model, prompt, or tool definition upgrades. Evaluates tool sequence divergence using Levenshtein distance and Longest Common Subsequence (LCS), identifies newly introduced tool failures, flags recursive loop regressions, and calculates token spend and USD cost deltas.

When to use

After upgrading LLM foundation models, prompt templates, or system instructions to verify agent behavior parity.
In CI/CD pipelines to block regressions when agent trajectory similarity drops below threshold or costs spike.

When NOT to use

Do NOT use to inspect single cassette integrity; use agentflight_verify_cassette instead.
Do NOT use to convert cassettes to telemetry traces; use agentflight_export_otel instead.

Behavior & Side Effects

Read-only: Reads both cassette files from disk without altering them.
Deterministic & Idempotent: Repeated invocations with identical cassettes yield exact identical diff metrics.
Returns structured diff metrics including similarity score (0.0 to 1.0), regression alerts, and cost deltas.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "baseline_path": "<baseline_path>",
      "candidate_path": "<candidate_path>"
    },
    "name": "agentflight_diff_cassettes"
  }
}

Input Schema

Name	Required	Description
`baseline_path`	Yes	Filesystem path to the golden reference baseline cassette (.flight.json or .flight.jsonl).
`candidate_path`	Yes	Filesystem path to the candidate cassette to evaluate against the baseline (.flight.json or .flight.jsonl).
`similarity_threshold`	No	Minimum acceptable trajectory similarity coefficient in the range [0.0, 1.0]. Defaults to 0.80.
`max_cost_increase_pct`	No	Maximum tolerated percentage increase in USD cost before triggering a cost regression alert. Defaults to 50.0.
`max_token_increase_pct`	No	Maximum tolerated percentage increase in total tokens before triggering a token blowout alert. Defaults to 50.0.

Output Schema

Name	Required	Description
`passed`	Yes	True if candidate satisfies similarity and cost/token thresholds without critical regressions.
`similarity`	Yes	Trajectory similarity coefficient between 0.0 and 1.0 based on normalized sequence alignment.
`baseline_id`	No	Identifier or filename of baseline cassette.
`regressions`	No	Detailed list of identified behavioral regressions (new tool errors, loop states, missing tools).
`token_delta`	No	Net difference in total tokens consumed.
`candidate_id`	No	Identifier or filename of candidate cassette.
`cost_delta_usd`	No	USD cost variance between candidate and baseline trajectories.
`cost_increase_pct`	No	Percentage increase in USD cost.
`token_increase_pct`	No	Percentage increase in total token consumption.

Schema Changelog

Changes observed during successful MCP inspections.

Changed13 schema fields changedv1.1.1Nov 10, 2026

changedInput schema / properties / baseline_path / description

Previous value: -"Path to the golden baseline .flight.json or .flight.jsonl cassette."New value: +"Filesystem path to the golden reference baseline cassette (.flight.json or .flight.jsonl)."

changedInput schema / properties / candidate_path / description

Previous value: -"Path to the candidate .flight.json or .flight.jsonl cassette."New value: +"Filesystem path to the candidate cassette to evaluate against the baseline (.flight.json or .flight.jsonl)."

addedInput schema / properties / max_cost_increase_pct / default
```
Added value: +50
```

changedInput schema / properties / max_cost_increase_pct / description

Previous value: -"Maximum allowed percentage increase in USD cost (default 50.0)."New value: +"Maximum tolerated percentage increase in USD cost before triggering a cost regression alert. Defaults to 50.0."

addedInput schema / properties / max_cost_increase_pct / minimum
```
Added value: +0
```
addedInput schema / properties / max_token_increase_pct / default
```
Added value: +50
```

changedInput schema / properties / max_token_increase_pct / description

Previous value: -"Maximum allowed percentage increase in total tokens (default 50.0)."New value: +"Maximum tolerated percentage increase in total tokens before triggering a token blowout alert. Defaults to 50.0."

addedInput schema / properties / max_token_increase_pct / minimum
```
Added value: +0
```
addedInput schema / properties / similarity_threshold / default
```
Added value: +0.8
```

changedInput schema / properties / similarity_threshold / description

Previous value: -"Minimum trajectory similarity threshold in [0.0, 1.0] (default 0.80)."New value: +"Minimum acceptable trajectory similarity coefficient in the range [0.0, 1.0]. Defaults to 0.80."

addedInput schema / properties / similarity_threshold / maximum
```
Added value: +1
```
addedInput schema / properties / similarity_threshold / minimum
```
Added value: +0
```

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "properties": {
+    "baseline_id": {
+      "description": "Identifier or filename of baseline cassette.",
+      "type": "string"
+    },
+    "candidate_id": {
+      "description": "Identifier or filename of candidate cassette.",
+      "type": "string"
+    },
+    "cost_delta_usd": {
+      "description": "USD cost variance between candidate and baseline trajectories.",
+      "type": "number"
+    },
+    "cost_increase_pct": {
+      "description": "Percentage increase in USD cost.",
+      "type": "number"
+    },
+    "passed": {
+      "description": "True if candidate satisfies similarity and cost/token thresholds without critical regressions.",
+      "type": "boolean"
+    },
+    "regressions": {
+      "description": "Detailed list of identified behavioral regressions (new tool errors, loop states, missing tools).",
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "similarity": {
+      "description": "Trajectory similarity coefficient between 0.0 and 1.0 based on normalized sequence alignment.",
+      "type": "number"
+    },
+    "token_delta": {
+      "description": "Net difference in total tokens consumed.",
+      "type": "integer"
+    },
+    "token_increase_pct": {
+      "description": "Percentage increase in total token consumption.",
+      "type": "number"
+    }
+  },
+  "required": [
+    "passed",
+    "similarity"
+  ],
+  "type": "object"
+}

First observedv1.1.0Nov 10, 2026

TDQS

A4.8/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, yet the description still adds substance: it discloses that both files are read without alteration, that repeated invocations are deterministic and produce identical metrics, and what the diff output contains (similarity score 0.0-1.0, regression alerts, cost deltas). This is behavioral context an agent cannot derive from the annotation flags alone.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Header-delimited sections make it easy to scan and the purpose sentence is front-loaded. Minor waste exists where 'Read-only' and 'Deterministic & Idempotent' largely restate the supplied annotation hints, but the remaining content earns its space.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given an output schema exists, the description need not detail return values, yet it summarizes the key metrics anyway. All five parameters, both file formats, the CI-gating use case, and the sibling routing are covered, leaving nothing an agent needs in order to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so parameter documentation is largely handled structurally, but the description adds semantic framing by tying the similarity threshold to regression blocking and the cost/token tolerances to 'cost spike' alerts. It does not restate defaults or units, so it stays additive rather than duplicative.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Compare a baseline golden AgentFlight cassette against a candidate cassette to detect behavioral regressions') and then enumerates the concrete computations: Levenshtein/LCS trajectory divergence, new tool failures, loop regressions, token and USD cost deltas. It is clearly distinguishable from verify_cassette (integrity check) and export_otel (trace conversion).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit when-to-use triggers (model/prompt/system-instruction upgrades, CI/CD gating) and an explicit when-NOT-to-use list naming the correct sibling for each excluded case. The conditions that select this tool versus alternatives are fully spelled out.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server