aviation-mcp

An MCP tool server that exposes live aviation weather to an LLM — and, more to the point, knows when it shouldn't be trusted to answer.

Built in TypeScript against aviationweather.gov. No API key, no mock data.

  aviation-mcp — eval suite
  15 golden cases, frozen fixtures, no network

  PASS  VFR  auto   conf 1.00  clear VFR
  PASS  IFR  auto   conf 1.00  solid IFR — low ceiling
  PASS  VFR  gated  conf 0.75  thunderstorm with technically-VFR numbers
  PASS  IFR  gated  conf 0.65  sources disagree
  ...

  ── scores ──
  category accuracy    100%  (15/15)
  gate precision       100%  (5 correct escalations, 0 spurious)
  gate recall          100%  (0 missed escalations)
  confidence separation  gated 0.68 vs ungated 1.00

The idea

Most tool servers are a thin wrapper over an API: the model asks, the tool answers, everyone assumes the answer is good. That's fine until the tool is asked something the underlying data can't actually support — at which point a language model will cheerfully produce a confident answer anyway, because that is what language models do.

So the interesting tool here isn't get_metar. It's assess_conditions, which returns three things instead of one:

{
  "flightCategory": "VFR",
  "confidence": 0.75,
  "requiresHumanReview": true,          // ← the point
  "reviewReasons": [
    "thunderstorm reported — category alone understates the hazard."
  ],
  "reasoning": [
    "No broken or overcast layer reported; ceiling is not a limiting factor.",
    "Visibility 10 sm.",
    "Category is the worse of the two components: VFR."
  ]
}

Ceiling and visibility both say VFR. There is a thunderstorm overhead. The number is right and the answer is dangerous — so the tool refuses to present it as settled and hands off to a person.

Related MCP server: stormscope

The escalation gates

requiresHumanReview fires on four conditions, each of which is a way for a technically-correct answer to be wrong:

Confidence degrades per gate and the eval suite asserts that gated cases actually score lower than ungated ones — otherwise the number is decorative.

What the eval harness caught

This is the part worth reading.

The first version used fixed boundary margins (±0.5 sm, ±200 ft). Every unit test passed. The implementation was, by any reasonable reading, correct.

The eval suite failed it at 56% gate precision. It was escalating half-mile fog as "near the 1 sm boundary" — half-mile fog is unambiguously the worst category there is — and a 700 ft overcast as "near 500 ft", when 700 ft is squarely IFR. It was crying wolf on a third of the cases, and a gate that fires on obvious cases is a gate humans learn to ignore.

The bug: fixed margins assume measurement precision is constant across magnitudes. It isn't. The gap between 0.5 and 1.0 sm is enormous; the gap between 700 and 500 ft is not. Switching to a proportional margin (5% of the threshold) took precision to 100% with no loss of recall.

No unit test would have caught that, because nothing was broken. It took a suite that graded the system's judgment rather than its behaviour. That's the argument for building evals before you think you need them.

Production hygiene

The boring parts, which are the parts that decide whether this survives contact with production:

• Structured contracts. Every upstream response crosses a zod boundary. A malformed payload fails loudly here, rather than silently becoming a plausible hallucination three tool calls later. Contract violations are reported as contract violations, not disguised as network blips.

• Retries. Exponential backoff with jitter, on transient failures only (5xx, 429, timeouts). A 400 is not retried — retrying a 400 is just being wrong twice. Jitter matters: without it, a fleet of retrying clients wakes up in lockstep and stampedes an upstream that's already struggling.

• Timeouts. An agent blocked forever on a hung socket is worse than one that fails fast, because nothing upstream can distinguish "thinking" from "dead".

• Idempotent reads, cached. METARs update hourly. A chatty agent asking about CYOW eight times in one turn costs one upstream call, not eight.

• Tracing. One JSON line per tool call to stderr — trace id, tool, duration, cache hit, outcome, and for assessments the category, confidence, and gate decision. Stderr, never stdout: stdout is the MCP transport, and writing to it corrupts the protocol. After the fact you can ask "how often did we gate?" and "did confidence track correctness?" without re-running anything.

• Tool errors, not crashes. A tool that throws into the transport takes down every other tool with it. Failures come back as isError with a reason the model can act on.

• Pure core. Categorization, confidence, and the gates have no network, no clock, and no model in them. That's why they can be evaluated exhaustively — all the nondeterminism lives at the edges.

Tools

Running it

npm install
npm test        # 43 unit tests
npm run eval    # 15 golden cases, graded — fails the build on a recall miss
npm run build
npm start       # MCP server over stdio

Register with any MCP client:

{
  "mcpServers": {
    "aviation": {
      "command": "node",
      "args": ["/absolute/path/to/aviation-mcp/dist/index.js"]
    }
  }
}

Then ask it "what are conditions at CYOW, and should I trust the answer?"

Notes on the upstream

aviationweather.gov is a free public service with an undocumented contract, which is to say it is like every other integration. Visibility arrives as a number (15), a string with a plus ("10+"), or a fraction ("1 1/2"). Wind direction is "VRB" when it's variable. None of this is in any spec; all of it is discovered by hitting the API and watching things break. It's normalized once, in normalize.ts, where it's tested — not in six places downstream, and not by hoping the model figures it out.

Layout

src/
  index.ts      MCP transport wiring. Thin on purpose.
  tools.ts      Tool definitions + handlers. Uniform shape: validate, trace, call, normalize.
  assess.ts     Categorization, confidence, and the escalation gates. Pure functions.
  client.ts     HTTP: retries, backoff, timeouts, TTL cache.
  normalize.ts  Upstream's bad habits → our contract.
  schemas.ts    zod contracts.
  logger.ts     Structured traces.
evals/
  cases.ts      15 golden cases, frozen fixtures.
  run.ts        Graded harness: accuracy, gate precision, gate recall.
test/           43 unit tests.

License

MIT.