# aviation-mcp

An MCP tool server that exposes live aviation weather to an LLM — and, more to the point, **knows when it shouldn't be trusted to answer.**

Built in TypeScript against [aviationweather.gov](https://aviationweather.gov). No API key, no mock data.

```
  aviation-mcp — eval suite
  15 golden cases, frozen fixtures, no network

  PASS  VFR  auto   conf 1.00  clear VFR
  PASS  IFR  auto   conf 1.00  solid IFR — low ceiling
  PASS  VFR  gated  conf 0.75  thunderstorm with technically-VFR numbers
  PASS  IFR  gated  conf 0.65  sources disagree
  ...

  ── scores ──
  category accuracy    100%  (15/15)
  gate precision       100%  (5 correct escalations, 0 spurious)
  gate recall          100%  (0 missed escalations)
  confidence separation  gated 0.68 vs ungated 1.00
```

## The idea

Most tool servers are a thin wrapper over an API: the model asks, the tool answers, everyone assumes the answer is good. That's fine until the tool is asked something the underlying data can't actually support — at which point a language model will cheerfully produce a confident answer anyway, because that is what language models do.

So the interesting tool here isn't `get_metar`. It's **`assess_conditions`**, which returns three things instead of one:

```jsonc
{
  "flightCategory": "VFR",
  "confidence": 0.75,
  "requiresHumanReview": true,          // ← the point
  "reviewReasons": [
    "thunderstorm reported — category alone understates the hazard."
  ],
  "reasoning": [
    "No broken or overcast layer reported; ceiling is not a limiting factor.",
    "Visibility 10 sm.",
    "Category is the worse of the two components: VFR."
  ]
}
```

Ceiling and visibility both say VFR. There is a thunderstorm overhead. The number is right and the answer is dangerous — so the tool refuses to present it as settled and hands off to a person.

## The escalation gates

`requiresHumanReview` fires on four conditions, each of which is a way for a technically-correct answer to be wrong:

| Gate | Why |
|---|---|
| **Boundary conditions** | Visibility of exactly 3.0 sm sits on the IFR/MVFR line. The honest answer to "which side?" is *"I can't tell you from this data."* |
| **Missing inputs** | Upstream omitted visibility. A naive implementation defaults to VFR and reports it confidently. **Absence of data is not evidence of good weather.** |
| **Significant weather** | Thunderstorms, freezing rain, hail. The category is right; the category is also not the whole story. |
| **Source disagreement** | Our computed category differs from upstream's own. One of us is wrong, and we don't get to assume it's them. |

Confidence degrades per gate and the eval suite asserts that gated cases actually score lower than ungated ones — otherwise the number is decorative.

## What the eval harness caught

This is the part worth reading.

The first version used **fixed** boundary margins (±0.5 sm, ±200 ft). Every unit test passed. The implementation was, by any reasonable reading, correct.

The eval suite failed it at **56% gate precision**. It was escalating half-mile fog as "near the 1 sm boundary" — half-mile fog is unambiguously the worst category there is — and a 700 ft overcast as "near 500 ft", when 700 ft is squarely IFR. It was crying wolf on a third of the cases, and a gate that fires on obvious cases is a gate humans learn to ignore.

The bug: fixed margins assume measurement precision is constant across magnitudes. It isn't. The gap between 0.5 and 1.0 sm is enormous; the gap between 700 and 500 ft is not. Switching to a proportional margin (5% of the threshold) took precision to **100% with no loss of recall**.

No unit test would have caught that, because nothing was *broken*. It took a suite that graded the system's judgment rather than its behaviour. That's the argument for building evals before you think you need them.

## Production hygiene

The boring parts, which are the parts that decide whether this survives contact with production:

- **Structured contracts.** Every upstream response crosses a zod boundary. A malformed payload fails loudly here, rather than silently becoming a plausible hallucination three tool calls later. Contract violations are reported as contract violations, not disguised as network blips.
- **Retries.** Exponential backoff with jitter, on transient failures only (5xx, 429, timeouts). A 400 is not retried — retrying a 400 is just being wrong twice. Jitter matters: without it, a fleet of retrying clients wakes up in lockstep and stampedes an upstream that's already struggling.
- **Timeouts.** An agent blocked forever on a hung socket is worse than one that fails fast, because nothing upstream can distinguish "thinking" from "dead".
- **Idempotent reads, cached.** METARs update hourly. A chatty agent asking about CYOW eight times in one turn costs one upstream call, not eight.
- **Tracing.** One JSON line per tool call to stderr — trace id, tool, duration, cache hit, outcome, and for assessments the category, confidence, and gate decision. Stderr, never stdout: stdout is the MCP transport, and writing to it corrupts the protocol. After the fact you can ask *"how often did we gate?"* and *"did confidence track correctness?"* without re-running anything.
- **Tool errors, not crashes.** A tool that throws into the transport takes down every other tool with it. Failures come back as `isError` with a reason the model can act on.
- **Pure core.** Categorization, confidence, and the gates have no network, no clock, and no model in them. That's why they can be evaluated exhaustively — all the nondeterminism lives at the edges.

## Tools

| Tool | Purpose |
|---|---|
| `assess_conditions` | Flight category **+ confidence + escalation gate**. The one that matters. |
| `get_metar` | Current observations, normalized. |
| `get_taf` | Terminal aerodrome forecasts. |
| `search_stations` | Resolve a place name to an ICAO identifier. |

## Running it

```bash
npm install
npm test        # 43 unit tests
npm run eval    # 15 golden cases, graded — fails the build on a recall miss
npm run build
npm start       # MCP server over stdio
```

Register with any MCP client:

```jsonc
{
  "mcpServers": {
    "aviation": {
      "command": "node",
      "args": ["/absolute/path/to/aviation-mcp/dist/index.js"]
    }
  }
}
```

Then ask it *"what are conditions at CYOW, and should I trust the answer?"*

## Notes on the upstream

`aviationweather.gov` is a free public service with an undocumented contract, which is to say it is like every other integration. Visibility arrives as a number (`15`), a string with a plus (`"10+"`), or a fraction (`"1 1/2"`). Wind direction is `"VRB"` when it's variable. None of this is in any spec; all of it is discovered by hitting the API and watching things break. It's normalized once, in `normalize.ts`, where it's tested — not in six places downstream, and not by hoping the model figures it out.

## Layout

```
src/
  index.ts      MCP transport wiring. Thin on purpose.
  tools.ts      Tool definitions + handlers. Uniform shape: validate, trace, call, normalize.
  assess.ts     Categorization, confidence, and the escalation gates. Pure functions.
  client.ts     HTTP: retries, backoff, timeouts, TTL cache.
  normalize.ts  Upstream's bad habits → our contract.
  schemas.ts    zod contracts.
  logger.ts     Structured traces.
evals/
  cases.ts      15 golden cases, frozen fixtures.
  run.ts        Graded harness: accuracy, gate precision, gate recall.
test/           43 unit tests.
```

## License

MIT.