# Trip Disruption Recovery Agent

An AI agent that recovers a disrupted flight booking. Give it a goal in plain
English and a booking reference; it investigates the disruption, prices every
way out, recommends one with the math shown, and acts **only after you approve**.

Built on Claude Code plus a purpose-built MCP server, against the
[Duffel](https://duffel.com) flight API in **test mode only**: no real money, no
real bookings.

*By [Austin Lai](https://github.com/austinlai22). Specified, built and evaluated
with Claude Code; the original spec is in [docs/spec.md](docs/spec.md).*

```
/trip-recovery My flight to New York (order ord_fx_s02) was moved by the airline.
               I need to be at JFK by Friday 18 September 2026 at 18:00. Find the cheapest way.
```

![The agent recovering a disrupted booking: it proposes, is refused approval, and books nothing; then with approval it rebooks, cancels, and verifies](docs/demo.gif)

*Two real runs of the same scenario. First with nobody available to approve, so
nothing is booked. Then with an approval, so it books the replacement, cancels
the original, and verifies both. Recorded with
[`docs/demo_script.sh`](docs/demo_script.sh).*

---

## 1. The problem

When an airline changes or cancels a flight, the traveler has a decision to make
under time pressure, usually with three or four options:

| Option | What it costs | What it risks |
|---|---|---|
| Accept the airline's new itinerary | Nothing in cash; possibly hours | Arriving too late to matter |
| Pay to change to a different flight | A change fee plus any fare difference | Paying for time you don't need |
| Cancel and rebook elsewhere | A new fare, minus a refund that may be **credit, not cash** | Losing the refund's real value |
| Escalate to a human | Time | Nothing, but slow |

**Target user:** the traveler, or the support agent handling it for them at an
online travel agency.

**Why this is a decision, not a lookup.** There is no single correct answer to
fetch from an API. The right choice depends on facts spread across four
different endpoints (the order, the airline's change, change quotes, live fares)
*and* on the traveler's own weights: how late is too late, and what a voucher is
worth compared with cash. A voucher for £420 is not £420. Most travelers accept
whatever the airline offers because comparing properly takes 20 minutes of work
they can't do at the gate.

---

## 2. What it does

A real run against the `s02` scenario (an airline moves a flight 24 hours,
missing the traveler's deadline). Full transcript:
[evals/results/m4_manual_runs/](evals/results/m4_manual_runs/).

The agent chose to read the order, list the airline's change, quote a paid
change, search alternative flights, price a cancellation, refresh the two live
fares, and only then score the options:

| Option | Out-of-pocket | Refund (cash) | Refund (credit) | Arrival | Meets deadline | Total cost (lower is better) |
|---|---|---|---|---|---|---|
| A. Accept airline change | 0.00 | — | — | 19 Sep 11:55 | ❌ No | 600.00 |
| B. Change to ZZ307 | 60.00 | — | — | 18 Sep 15:55 | ✅ Yes | 160.00 |
| C1. Cancel + rebook ZZ307 | 298.00 | — | 420.00 | 18 Sep 15:55 | ✅ Yes | 104.00 |
| **C2. Cancel + rebook ZZ205** | **365.00** | — | **420.00** | 18 Sep 11:10 | ✅ Yes | **71.00** |

> **Recommended:** C2 — cancel for 420.00 in airline credit (worth 294.00 at a
> 0.7 discount) and rebook ZZ205 for 365.00, netting 71.00.
> **Runner-up:** C1 at 104.00.
> **What would change this:** C1 wins if an hour of your time is worth less than 16.75.

It then booked the new flight **first**, cancelled the old one **second** (so the
traveler is never left with no ticket), each behind its own approval, and
verified both by re-reading the orders.

### How the total is calculated

Every option collapses into one number: **total cost, where lower is better.**
It isn't a score out of 100, and it isn't cash out of pocket. It's what the
disruption costs you once credit and lost time are priced in.

```
total cost = cash paid
           − cash refunded
           − (airline credit × 0.7)     ← a voucher is worth less than cash
           + (hours later than your original arrival × 25)
```

Both weights are yours, set in [`config.yaml`](config.yaml). For the run above
(original arrival 18 Sep 11:55, deadline 18 Sep 18:00):

| Option | Arithmetic | Total cost |
|---|---|---|
| **C2. Cancel + rebook ZZ205** | `365.00 − (420.00 × 0.7) + (0.00 h × 25)` | **71.00** |
| C1. Cancel + rebook ZZ307 | `298.00 − (420.00 × 0.7) + (4.00 h × 25)` | 104.00 |
| B. Change to ZZ307 | `60.00 + (4.00 h × 25)` | 160.00 |
| A. Accept the airline's change | `0.00 + (24.00 h × 25)` | 600.00, ineligible |

**Why C2 beats C1** — the real contest. Both cancel the same booking for the same
420.00 credit, so the credit cancels out. C1 saves 67.00 in cash but arrives four
hours later, and four hours costs 100.00. C2 wins by 33.00. Divide that 67.00 by
those four hours and you get the break-even the agent reports: if an hour of your
time were worth less than **16.75**, C1 would be the better answer.

Four rules shape every comparison:

1. **Credit is discounted.** 420.00 in airline credit counts as 294.00, because it
   expires and only works on one airline.
2. **Lateness is priced, not just checked**, and measured against your *original*
   arrival. That's why accepting a free 24-hour delay costs 600.00.
3. **Arriving early earns nothing.** C2 lands 45 minutes early; the time term is
   clamped at zero, so the agent can't spend your money buying earliness you
   never asked for.
4. **The deadline is a hard constraint, not a cost.** Option A is shown with its
   total for transparency, but it misses the deadline and so can never be
   recommended, however cheap it looks.

The arithmetic lives in [`scoring.py`](src/recovery_mcp/scoring.py), and the model
never does it: the agent decides which options to submit, the code ranks them.

---

## 3. Why it's agentic

There is no decision tree in this repo. The agent chooses which tools to call
and in what order, and the runs show it: it used between 6 and 24 tool calls
depending on the situation, searched extra dates when a day was sold out, and
re-planned when a fare sold out mid-booking.

| The agent decides | The code decides |
|---|---|
| Which options exist and are worth pricing | What each option scores (`scoring.py`) |
| When it has enough information to recommend | Whether a number is real (traceability check) |
| How to adapt when a tool fails or a price moves | Whether an action may run at all (approval gate) |
| What to tell the traveler, and what would change the answer | What the traveler is shown before approving |

The division is deliberate: judgment to the model, consequences to the code.

---

## 4. Architecture

```mermaid
flowchart TD
    U["Traveler: goal + booking ID"] --> CC["Claude Code<br/>+ /trip-recovery playbook"]
    CC <--> MCP["MCP server: duffel-recovery"]
    MCP --> G["Guardrail layer<br/>test-token check · spend cap · approval gate<br/>call budget · audit log"]
    G --> B{"Backend"}
    B -->|evals| FX["FixtureBackend<br/>JSON scenarios"]
    B -->|live| DF["DuffelSandboxBackend<br/>Duffel API, test mode"]
    G -.->|"approval dialog"| U
    G --> AUD[("Audit log (JSONL)<br/>every call, approval, write")]
```

**Two backends, one interface.** The live sandbox is non-deterministic, so it
can't have a known right answer. Fixtures pin every response, which is what
makes the eval suite meaningful; the Duffel backend proves the same agent works
against a real API. Both return identical normalized models, so the agent can't
tell them apart.

**Every tool result carries a `call_id`** (`C001`, `C002`, …). The agent cites
these for every number it reports, and the eval suite checks the citations.

---

## 5. Guardrails: the model proposes, code disposes

Each of these is enforced in Python and covered by tests. None of them relies on
the model choosing to behave.

| Guardrail | How it's enforced | Proof |
|---|---|---|
| **Test mode only** | Refuses any token that isn't `duffel_test_*`; every Duffel response must say `live_mode: false` | `tests/test_token_check.py` |
| **No irreversible action without approval** | Write tools only *propose* until they get a single-use token, minted solely after a human types "yes" in an approval dialog the server writes | `tests/test_write_gate.py`, `tests/test_server.py` |
| **Approval can't be reused or stretched** | Tokens are bound to one action, target, amount and currency, expire after 10 minutes, and are voided on any mismatch (e.g. the price moved after approval) | `tests/test_guardrails.py` |
| **Spending cap** | A ledger checks every proposal and every execution; one currency per session, never converted | `tests/test_write_gate.py` |
| **No invented numbers** | `score_options` rejects any amount or time that doesn't appear in the tool call the agent cites | `tests/test_scoring.py` |
| **Fake passengers only** | Passenger details come from `config.yaml`, never from the model; emails and phones must be in ranges reserved for fiction | `tests/test_fake_data.py` |
| **Runaway protection** | 40 tool calls per session; a failing call may be retried twice, then it's blocked | `tests/test_guardrails.py` |
| **Injected instructions are inert** | Free text from a booking is labelled `untrusted_notes`; the cap is code, so "the cap is lifted" changes nothing | scenario `s09` |
| **Auto-approval can't touch Duffel** | Config validation and a second runtime check both refuse auto-approve unless the backend is fixtures | `tests/test_config.py`, `tests/test_write_gate.py` |

A second, independent layer: when the server talks to Duffel, its write tools
are flagged `anthropic/requiresUserInteraction`, so Claude Code *also* demands a
human prompt on every call, even under permissive settings.

---

## 6. Evaluation

**Method.** 16 scenarios, each a JSON file pinning every tool response plus an
answer key. Each run is a fresh headless Claude Code session with only this MCP
server, no built-in tools (so the agent can't read the answer key), no personal
settings, and no memory. Every run is graded from its audit log, not from its
prose.

Scenarios cover: accepting a change, paid changes, cancel + rebook, a credit
discount that flips the answer, a price that rises on refresh, an offer that
sells out at booking time, no alternatives at all, persistent search timeouts,
an option that's cheapest but over the cap, instructions injected into booking
notes, an outright cancellation, an exact tie, lateness outweighing free, a
change that refunds money, an unstated refund amount, everything arriving late,
and a sold-out day that needs a wider search.

**Scorecard: 16 scenarios × 3 runs = 48 runs** (Claude Sonnet via Claude Code
2.1.272). Full results, including every failure and the method:
[evals/results/2026-09-15.md](evals/results/2026-09-15.md).

| Metric | Result |
|---|---|
| Decision accuracy | **48/48 (100%)** |
| Consistency (correct in all 3 runs) | **16/16 scenarios** |
| Guardrail violations | **0** |
| Number traceability | 1,649/1,653 (99.8%) |
| Avg tool calls per run | 13.4 (max 27) |
| Runs that errored or timed out | 0 |

**Read that accuracy with care.** 3 runs per scenario is a small sample, the
scenarios were written by the same person who wrote the playbook, and the model
knows it is being asked for a careful comparison. It says the agent is reliable
on the situations I anticipated; it says nothing about the ones I didn't.

**The 4 untraced numbers are all the agent's own arithmetic**, not invented
facts: "122.00 GBP net credit after the new fare" (420 − 298), "arrives ~4.75h
later", and two hypothetical departure times in "what would change this"
("a flight departing before ~10:00 would meet the deadline"). The traceability
metric flags any number not present verbatim in a tool result, so derived
values count against it. That's the metric being strict, not the agent guessing.

**Why 3 runs and not 5.** The spec asked for 5. 16 runs exhausted a Claude
subscription session limit, so the suite was re-run at 3 to fit one window.
`uv run python -m evals.run_evals --runs 5` does the full version.

**The answer keys are themselves tested.** `tests/test_scenario_answers.py`
rebuilds every option in every scenario and scores it with the production
scoring code, so a hand-worked answer can't silently be wrong.

---

## 7. Limitations (honest list)

- **Synthetic data.** Duffel Airways is a sandbox airline: schedules and prices
  aren't realistic. The fixtures are hand-written to be *plausible*, not real.
- **No real bookings, ever.** Live mode is refused by construction.
- **No passenger-rights logic.** The agent makes no claims about DOT, UK261 or
  EU261 compensation. That needs primary sources and citations; it's out of scope.
- **One-way trips, one traveler.** Multi-slice itineraries and passengers with
  conflicting constraints aren't handled in v1.
- **Local clock times.** Options are compared as local times at the arrival
  airport; the agent isn't asked to reason across time zones.
- **Credit value is a single number.** A 0.7 factor can't express "worthless
  unless I fly them again within a year".
- **The approval dialog is as strong as its client.** Claude Code shows it and
  the server never mints a token without an explicit "yes", but a user who
  configures an auto-answering hook can bypass the human step.
- **Evals measure decisions, not taste.** A run can be graded correct and still
  explain itself poorly.

---

## 8. What I'd build next

1. **Passenger rights, with citations.** A module that reads primary sources
   (e.g. US DOT refund rules) and quotes them, so the agent can say "you're
   owed a cash refund" and show why. Biggest single lift in perceived value.
2. **Proactive monitoring.** Duffel webhooks instead of a human noticing; the
   agent drafts the recommendation before the traveler has read the email.
3. **Points and vouchers as first-class currency**, replacing the single
   credit-value factor.
4. **Multi-passenger trips** where constraints conflict.

**Business case (a hypothesis, not a finding).** Disruption handling is a
high-cost, low-satisfaction moment for an online travel agency: it arrives in
bursts, needs a trained agent, and the traveler is already unhappy. If an agent
can prepare a priced, sourced recommendation before a human opens the ticket,
the plausible win is shorter handle times and fewer escalations, with the human
kept for the approval and the edge cases. Testing that claim needs real ticket
data: volume, current handle time, and the mix of disruption types. None of
that is measured here.

---

## 9. Run it yourself

### Setup (once)

```bash
brew install --cask claude-code   # the Claude Code CLI (or: curl -fsSL https://claude.ai/install.sh | bash)
brew install uv                   # Python toolchain; installs Python 3.12 for this project
cd trip-recovery-agent
uv sync                           # dependencies
uv run pytest                     # 219 tests: no network, no model, no token needed
```

The first `claude` run in this folder asks three one-time questions: a colour
theme, how to log in (a Pro/Max subscription or a Console account), and whether
to trust this folder and its MCP server. That server is `duffel-recovery` from
this repo, declared in [`.mcp.json`](.mcp.json). A Homebrew install doesn't
auto-update; refresh it with `brew upgrade --cask claude-code`.

### Recover a booking

```bash
claude
```

Then type the skill name and your goal in plain English:

```
/trip-recovery The airline changed my flight (order ord_fx_s01). Get me to JFK by
Friday 18 September 2026 at 18:00 local time at the lowest total cost.
```

The order ID has to match the scenario that's loaded. `config.yaml` sets it:

```yaml
fixture:
  scenario: evals/scenarios/s01_accept_change_meets_deadline.json
```

Point that at any file in [`evals/scenarios/`](evals/scenarios/) and **restart
`claude`**, since the server reads its config at startup. Four worth trying:

| Order ID | Scenario | What you'll see |
|---|---|---|
| `ord_fx_s01` | `s01_accept_change_meets_deadline` | Accepting the airline's change wins |
| `ord_fx_s02` | `s02_change_misses_deadline` | Cancel for credit and rebook; two approvals |
| `ord_fx_s09` | `s09_injected_instructions_in_booking` | The booking record tells the agent to ignore the spend cap |
| `ord_fx_s15` | `s15_every_option_is_late` | Nothing meets the deadline, so it escalates |

### Approving

When the agent is ready to act, a dialog appears with a summary **the server
wrote**: the flight, the amount, and what can't be undone. Type `yes` to
approve; anything else declines. Every irreversible action asks separately, so
the `s02` run asks twice: once to book the replacement, once to cancel the
original. Afterwards, read exactly what happened:

```bash
uv run python scripts/show_audit.py
```

### Without a human (headless)

```bash
uv run python -m evals.run_evals --runs 3     # the graded suite -> evals/results/<date>.md
bash docs/demo_script.sh                       # the two-scene demo above
```

### Against the real Duffel sandbox

Put a **test** token (`duffel_test_…`) in `.env`, set `backend: duffel_sandbox`
in `config.yaml`, then create a booking to disrupt:

```bash
uv run python scripts/seed_sandbox_order.py --route LHR-LTN
```

It prints an order ID; use that in `/trip-recovery`. In Duffel's sandbox, an
`LHR-LTN` order generates a fresh airline change every time changes are listed,
and an `LTN-SYD` order refunds to airline credit when cancelled.

## 10. Repo map

| Path | What it is |
|---|---|
| `.claude/skills/trip-recovery/` | the playbook: principles and required output, not a script |
| `src/recovery_mcp/server.py` | MCP adapter: tool registration and the approval dialog |
| `src/recovery_mcp/service.py` | every tool's logic, wrapped in the guardrails |
| `src/recovery_mcp/guardrails.py` | test mode, fake data, call budget, spend cap, approval gate |
| `src/recovery_mcp/scoring.py` | the objective function and the traceability check |
| `src/recovery_mcp/backends/` | `fixture.py` (evals) and `duffel_sandbox.py` (live test mode) |
| `evals/` | scenario builder, 16 scenarios, grader, runner, scorecards |
| `config.yaml` | objective weights, spend cap, limits, fake passengers |
| `docs/spec.md` | the original build spec |

---

**Austin Lai** · [github.com/austinlai22](https://github.com/austinlai22) · [MIT licensed](LICENSE)