Trip Disruption Recovery Agent

An AI agent that recovers a disrupted flight booking. Give it a goal in plain English and a booking reference; it investigates the disruption, prices every way out, recommends one with the math shown, and acts only after you approve.

Built on Claude Code plus a purpose-built MCP server, against the Duffel flight API in test mode only: no real money, no real bookings.

By Austin Lai. Specified, built and evaluated with Claude Code; the original spec is in docs/spec.md.

/trip-recovery My flight to New York (order ord_fx_s02) was moved by the airline.
               I need to be at JFK by Friday 18 September 2026 at 18:00. Find the cheapest way.

Two real runs of the same scenario. First with nobody available to approve, so nothing is booked. Then with an approval, so it books the replacement, cancels the original, and verifies both. Recorded with docs/demo_script.sh.

1. The problem

When an airline changes or cancels a flight, the traveler has a decision to make under time pressure, usually with three or four options:

Target user: the traveler, or the support agent handling it for them at an online travel agency.

Why this is a decision, not a lookup. There is no single correct answer to fetch from an API. The right choice depends on facts spread across four different endpoints (the order, the airline's change, change quotes, live fares) and on the traveler's own weights: how late is too late, and what a voucher is worth compared with cash. A voucher for £420 is not £420. Most travelers accept whatever the airline offers because comparing properly takes 20 minutes of work they can't do at the gate.

Related MCP server: exchange-ai-connector

2. What it does

A real run against the s02 scenario (an airline moves a flight 24 hours, missing the traveler's deadline). Full transcript: evals/results/m4_manual_runs/.

The agent chose to read the order, list the airline's change, quote a paid change, search alternative flights, price a cancellation, refresh the two live fares, and only then score the options:

Recommended: C2 — cancel for 420.00 in airline credit (worth 294.00 at a 0.7 discount) and rebook ZZ205 for 365.00, netting 71.00. Runner-up: C1 at 104.00. What would change this: C1 wins if an hour of your time is worth less than 16.75.

It then booked the new flight first, cancelled the old one second (so the traveler is never left with no ticket), each behind its own approval, and verified both by re-reading the orders.

How the total is calculated

Every option collapses into one number: total cost, where lower is better. It isn't a score out of 100, and it isn't cash out of pocket. It's what the disruption costs you once credit and lost time are priced in.

total cost = cash paid
           − cash refunded
           − (airline credit × 0.7)     ← a voucher is worth less than cash
           + (hours later than your original arrival × 25)

Both weights are yours, set in config.yaml. For the run above (original arrival 18 Sep 11:55, deadline 18 Sep 18:00):

Why C2 beats C1 — the real contest. Both cancel the same booking for the same 420.00 credit, so the credit cancels out. C1 saves 67.00 in cash but arrives four hours later, and four hours costs 100.00. C2 wins by 33.00. Divide that 67.00 by those four hours and you get the break-even the agent reports: if an hour of your time were worth less than 16.75, C1 would be the better answer.

Four rules shape every comparison:

1. Credit is discounted. 420.00 in airline credit counts as 294.00, because it expires and only works on one airline.

2. Lateness is priced, not just checked, and measured against your original arrival. That's why accepting a free 24-hour delay costs 600.00.

3. Arriving early earns nothing. C2 lands 45 minutes early; the time term is clamped at zero, so the agent can't spend your money buying earliness you never asked for.

4. The deadline is a hard constraint, not a cost. Option A is shown with its total for transparency, but it misses the deadline and so can never be recommended, however cheap it looks.

The arithmetic lives in scoring.py, and the model never does it: the agent decides which options to submit, the code ranks them.

3. Why it's agentic

There is no decision tree in this repo. The agent chooses which tools to call and in what order, and the runs show it: it used between 6 and 24 tool calls depending on the situation, searched extra dates when a day was sold out, and re-planned when a fare sold out mid-booking.

The division is deliberate: judgment to the model, consequences to the code.

4. Architecture

flowchart TD
    U["Traveler: goal + booking ID"] --> CC["Claude Code<br/>+ /trip-recovery playbook"]
    CC <--> MCP["MCP server: duffel-recovery"]
    MCP --> G["Guardrail layer<br/>test-token check · spend cap · approval gate<br/>call budget · audit log"]
    G --> B{"Backend"}
    B -->|evals| FX["FixtureBackend<br/>JSON scenarios"]
    B -->|live| DF["DuffelSandboxBackend<br/>Duffel API, test mode"]
    G -.->|"approval dialog"| U
    G --> AUD[("Audit log (JSONL)<br/>every call, approval, write")]

Two backends, one interface. The live sandbox is non-deterministic, so it can't have a known right answer. Fixtures pin every response, which is what makes the eval suite meaningful; the Duffel backend proves the same agent works against a real API. Both return identical normalized models, so the agent can't tell them apart.

Every tool result carries a call_id (C001, C002, …). The agent cites these for every number it reports, and the eval suite checks the citations.

5. Guardrails: the model proposes, code disposes

Each of these is enforced in Python and covered by tests. None of them relies on the model choosing to behave.

A second, independent layer: when the server talks to Duffel, its write tools are flagged anthropic/requiresUserInteraction, so Claude Code also demands a human prompt on every call, even under permissive settings.

6. Evaluation

Method. 16 scenarios, each a JSON file pinning every tool response plus an answer key. Each run is a fresh headless Claude Code session with only this MCP server, no built-in tools (so the agent can't read the answer key), no personal settings, and no memory. Every run is graded from its audit log, not from its prose.

Scenarios cover: accepting a change, paid changes, cancel + rebook, a credit discount that flips the answer, a price that rises on refresh, an offer that sells out at booking time, no alternatives at all, persistent search timeouts, an option that's cheapest but over the cap, instructions injected into booking notes, an outright cancellation, an exact tie, lateness outweighing free, a change that refunds money, an unstated refund amount, everything arriving late, and a sold-out day that needs a wider search.

Scorecard: 16 scenarios × 3 runs = 48 runs (Claude Sonnet via Claude Code 2.1.272). Full results, including every failure and the method: evals/results/2026-09-15.md.

Read that accuracy with care. 3 runs per scenario is a small sample, the scenarios were written by the same person who wrote the playbook, and the model knows it is being asked for a careful comparison. It says the agent is reliable on the situations I anticipated; it says nothing about the ones I didn't.

The 4 untraced numbers are all the agent's own arithmetic, not invented facts: "122.00 GBP net credit after the new fare" (420 − 298), "arrives ~4.75h later", and two hypothetical departure times in "what would change this" ("a flight departing before ~10:00 would meet the deadline"). The traceability metric flags any number not present verbatim in a tool result, so derived values count against it. That's the metric being strict, not the agent guessing.

Why 3 runs and not 5. The spec asked for 5. 16 runs exhausted a Claude subscription session limit, so the suite was re-run at 3 to fit one window. uv run python -m evals.run_evals --runs 5 does the full version.

The answer keys are themselves tested. tests/test_scenario_answers.py rebuilds every option in every scenario and scores it with the production scoring code, so a hand-worked answer can't silently be wrong.

7. Limitations (honest list)

• Synthetic data. Duffel Airways is a sandbox airline: schedules and prices aren't realistic. The fixtures are hand-written to be plausible, not real.

• No real bookings, ever. Live mode is refused by construction.

• No passenger-rights logic. The agent makes no claims about DOT, UK261 or EU261 compensation. That needs primary sources and citations; it's out of scope.

• One-way trips, one traveler. Multi-slice itineraries and passengers with conflicting constraints aren't handled in v1.

• Local clock times. Options are compared as local times at the arrival airport; the agent isn't asked to reason across time zones.

• Credit value is a single number. A 0.7 factor can't express "worthless unless I fly them again within a year".

• The approval dialog is as strong as its client. Claude Code shows it and the server never mints a token without an explicit "yes", but a user who configures an auto-answering hook can bypass the human step.

• Evals measure decisions, not taste. A run can be graded correct and still explain itself poorly.

8. What I'd build next

1. Passenger rights, with citations. A module that reads primary sources (e.g. US DOT refund rules) and quotes them, so the agent can say "you're owed a cash refund" and show why. Biggest single lift in perceived value.

2. Proactive monitoring. Duffel webhooks instead of a human noticing; the agent drafts the recommendation before the traveler has read the email.

3. Points and vouchers as first-class currency, replacing the single credit-value factor.

4. Multi-passenger trips where constraints conflict.

Business case (a hypothesis, not a finding). Disruption handling is a high-cost, low-satisfaction moment for an online travel agency: it arrives in bursts, needs a trained agent, and the traveler is already unhappy. If an agent can prepare a priced, sourced recommendation before a human opens the ticket, the plausible win is shorter handle times and fewer escalations, with the human kept for the approval and the edge cases. Testing that claim needs real ticket data: volume, current handle time, and the mix of disruption types. None of that is measured here.

9. Run it yourself

Setup (once)

brew install --cask claude-code   # the Claude Code CLI (or: curl -fsSL https://claude.ai/install.sh | bash)
brew install uv                   # Python toolchain; installs Python 3.12 for this project
cd trip-recovery-agent
uv sync                           # dependencies
uv run pytest                     # 219 tests: no network, no model, no token needed

The first claude run in this folder asks three one-time questions: a colour theme, how to log in (a Pro/Max subscription or a Console account), and whether to trust this folder and its MCP server. That server is duffel-recovery from this repo, declared in .mcp.json. A Homebrew install doesn't auto-update; refresh it with brew upgrade --cask claude-code.

Recover a booking

claude

Then type the skill name and your goal in plain English:

/trip-recovery The airline changed my flight (order ord_fx_s01). Get me to JFK by
Friday 18 September 2026 at 18:00 local time at the lowest total cost.

The order ID has to match the scenario that's loaded. config.yaml sets it:

fixture:
  scenario: evals/scenarios/s01_accept_change_meets_deadline.json

Point that at any file in evals/scenarios/ and restart claude, since the server reads its config at startup. Four worth trying:

Approving

When the agent is ready to act, a dialog appears with a summary the server wrote: the flight, the amount, and what can't be undone. Type yes to approve; anything else declines. Every irreversible action asks separately, so the s02 run asks twice: once to book the replacement, once to cancel the original. Afterwards, read exactly what happened:

uv run python scripts/show_audit.py

Without a human (headless)

uv run python -m evals.run_evals --runs 3     # the graded suite -> evals/results/<date>.md
bash docs/demo_script.sh                       # the two-scene demo above

Against the real Duffel sandbox

Put a test token (duffel_test_…) in .env, set backend: duffel_sandbox in config.yaml, then create a booking to disrupt:

uv run python scripts/seed_sandbox_order.py --route LHR-LTN

It prints an order ID; use that in /trip-recovery. In Duffel's sandbox, an LHR-LTN order generates a fresh airline change every time changes are listed, and an LTN-SYD order refunds to airline credit when cancelled.

10. Repo map

Austin Lai · github.com/austinlai22 · MIT licensed