WeatherXM FACT — Forecast Accuracy Comparison Testbed (Methodology, Verification Pipeline & MCP Server)

WeatherXM FACT (Forecast Accuracy Comparison Testbed) is an empirical, station-based weather forecast verification benchmark scoring 45 numerical weather prediction (NWP), AI foundation, regional, and post-processed forecast models against 8,927 quality-controlled physical WeatherXM ground stations (1.83 billion+ hourly forecast–observation evaluation records across the 2025 archive).

This repository serves two purposes:

1. Scientific & Engineering Transparency (docs/, pipeline/, examples/): Documents the exact verification methodology, quality-control gates (QoD >= 0.80), sample-weighted pooling math, categorical rain-event skill scores (HSS, ETS, POD, FAR, CSI), station-commissioned matched-cohort rules (>= 90% station-day overlap), and the DuckDB/Python/TypeScript aggregation and serving code behind fact.weatherxm.com.

2. Model Context Protocol (MCP) Server & REST Reference (src/, functions/, Dockerfile): Provides both the remote Streamable-HTTP edge MCP server (https://fact.weatherxm.com/mcp) and a zero-dependency local stdio / Docker MCP runner so AI agents (Claude, ChatGPT, Gemini, Grok, Cursor, OpenClaw) and researchers can query empirical forecast accuracy programmatically.

• Full Scientific Methodology: docs/FACT_METHODOLOGY.md

• Interactive Global Map & Leaderboards: https://fact.weatherxm.com

• Public Streamable-HTTP MCP Server (No Auth): https://fact.weatherxm.com/mcp

• LLM & Agent Reference (llms.txt): https://fact.weatherxm.com/llms.txt

• OpenAPI 3.1 REST Specification: https://fact.weatherxm.com/openapi.json

1. Why FACT Exists: Point-Truth Last-Mile Verification vs. Gridded Reanalysis

Most atmospheric research benchmarks (such as WeatherBench 2) answer a model-builder's question: "How well does this model represent the global state of the atmosphere on a coarse grid?" They regrid forecasts to a common resolution (0.25°–1.5°), verify against gridded reanalysis fields (e.g., ERA5), and weight scores by surface area.

FACT answers the operational buyer's and practitioner's question:

"If a solar operator, farmer, port master, or logistics planner relied on forecast model $X$ at this exact location, for this variable, $k$ days ahead, how far off was it against an independent physical sensor on the ground?"

Related MCP server: weather-for-grown-ups

2. Verification Methodology & Mathematical Formulation

The complete 10-section specification is in docs/FACT_METHODOLOGY.md. Below is the core mathematical pipeline with reference code from examples/verification_formulas.py and pipeline/.

2.1 Observation Construction & Hourly UTC Pairing

Raw sub-hourly station telemetry is aggregated in UTC to match what each forecast variable represents physically:

Lead-day definition: Lead time is measured from when the model run started (initialization_date in UTC)—never from when the forecast was fetched—so all 45 models are evaluated on a single, fair synoptic clock:

$$\text{lead_day} = \text{datediff}\left(\text{'day'},\ \text{ initialization_date}{\text{UTC}},\ \text{target_date}{\text{UTC}}\right) \in {0, 1, 2, 3, 4, 5, 6, 7}$$

2.2 Three-Stage Quality of Data (QoD) Gate

Community-hosted weather stations require strict automated screening before any verification metric is computed. A station-day record $(s, m, r, v, d)$ is admitted into FACT if and only if it passes all three gates:

1. Daily WeatherXM QoD >= 0.80 (Quality of Data):

   • Out-of-Bounds Check (OBC): Rejects readings outside physical sensor hardware limits.

   • Self Quality Check (SQC): Flags stuck sensors (constancy > 4–6h), unphysical minute-to-minute spikes (> 2 °C raw or > 3 °C minute jump),frozen anemometers at sub-zero temperatures, and packet dropouts.

   • Indoors Station Detector (ISD): Rolling 7-day solar clear-sky irradiance envelope + thermal/wind damping check to reject indoor, balcony, or tilted deployments.

   • Station Photo Verification (SPV): Four-direction installation audit against WMO obstacle-distance rules.

2. Minimum Hourly Completeness (n >= 18 of 24 hours): At least 75% of the UTC day must have valid paired forecast and observation values.

3. Physical Plausibility Screen: Rejects broken records where daily_MAE exceeds the gross-error ceiling (40 °C, 30 m/s, 50 mm, 100%).

2.3 Continuous Error Scoring & Sample-Weighted Pooling (1.83B+ Records)

For a single station $s$, model $m$, initialization run $r$, variable $v$, and UTC target day $d$ with $n \in [18, 24]$ valid hourly pairs $(f_h, o_h)$:

$$\text{MAE}{s,m,r,d} = \frac{1}{n} \sum{h=1}^{n} |f_h - o_h|, \qquad \text{RMSE}{s,m,r,d} = \sqrt{\frac{1}{n} \sum{h=1}^{n} (f_h - o_h)^2}$$

When pooling across $K$ station-days (e.g., an H3 cell, a country, or a global lead-day curve), FACT never takes an unweighted average of daily errors or a linear average of daily RMSEs (which mathematically understates large errors). Every pooled metric weights by the exact hourly pair count $n_i$, and RMSE is pooled through mean squared error:

$$\text{Pooled MAE} = \frac{\sum_{i=1}^{K} n_i \cdot \text{MAE}i}{\sum{i=1}^{K} n_i}, \qquad \text{Pooled RMSE} = \sqrt{\frac{\sum_{i=1}^{K} n_i \cdot \text{RMSE}i^2}{\sum{i=1}^{K} n_i}}$$

Reference Implementation (DuckDB SQL & Python — from pipeline/bake_fact_summary.py)

-- Exact DuckDB aggregation used to bake global lead-day curves (pipeline/bake_fact_summary.py)
SELECT 
    model,
    datediff('day', cast(initialization_date as timestamp), cast(target_date as timestamp)) AS lead_day,
    count(*) AS station_days,
    round(sum(daily_MAE * n_samples_MAE_RMSE) / sum(n_samples_MAE_RMSE), 3) AS pooled_mae,
    round(sqrt(sum(daily_RMSE * daily_RMSE * n_samples_MAE_RMSE) / sum(n_samples_MAE_RMSE)), 3) AS pooled_rmse,
    sum(n_samples_MAE_RMSE) AS total_hourly_samples
FROM read_parquet('data/2025_full/*/*.parquet')
WHERE variable = 'temperature'
  AND daily_MAE BETWEEN 0 AND 40.0
  AND n_samples_MAE_RMSE >= 18
  AND datediff('day', cast(initialization_date as timestamp), cast(target_date as timestamp)) BETWEEN 0 AND 7
GROUP BY model, lead_day
HAVING count(*) > 50
ORDER BY lead_day ASC, pooled_mae ASC;

2.4 Categorical Rain-Event Verification (HSS, ETS, POD, FAR, CSI) & Station Bootstrap

Millimeter precipitation MAE is heavily distorted by dry days: a model that forecasts "0 mm" every day achieves a deceptively low millimeter MAE in dry climates while missing 100% of rain events.

To evaluate operational rain-detection skill, FACT classifies each UTC station-day using a >= 0.25 mm/day wet-day threshold into a $2 \times 2$ contingency table:

• $a$ (Hits): Forecast $\ge 0.25\text{ mm}$ and Observed $\ge 0.25\text{ mm}$

• $b$ (False Alarms): Forecast $\ge 0.25\text{ mm}$ and Observed $< 0.25\text{ mm}$

• $c$ (Misses): Forecast $< 0.25\text{ mm}$ and Observed $\ge 0.25\text{ mm}$

• $d$ (Correct Negatives): Forecast $< 0.25\text{ mm}$ and Observed $< 0.25\text{ mm}$

Counts $(a, b, c, d)$ are summed across all station-days first (daily skill scores are never averaged), yielding:

$$\text{HSS} = \frac{2(ad - bc)}{(a+c)(c+d) + (a+b)(b+d)}, \qquad \text{POD} = \frac{a}{a+c}, \qquad \text{FAR} = \frac{b}{a+b}, \qquad \text{CSI} = \frac{a}{a+b+c}$$

Reference Implementation (from pipeline/bake_precip_skill.py)

import numpy as np

def contingency_scores(a, b, c, d):
    """Scores from summed contingency counts: a hits, b false alarms, c misses, d correct negatives."""
    a, b, c, d = (np.asarray(x, dtype=float) for x in (a, b, c, d))
    n = a + b + c + d
    with np.errstate(divide='ignore', invalid='ignore'):
        hss = 2 * (a * d - b * c) / ((a + c) * (c + d) + (a + b) * (b + d))
        a_random = (a + b) * (a + c) / n
        ets = (a - a_random) / (a + b + c - a_random)
        return {
            'hss': hss,          # Heidke Skill Score (1 = perfect, 0 = random chance)
            'ets': ets,          # Equitable Threat Score (Gilbert Skill Score)
            'pod': a / (a + c),  # Probability of Detection (Hit Rate)
            'far': b / (a + b),  # False Alarm Ratio
            'csi': a / (a + b + c),  # Critical Success Index (Threat Score)
            'fbi': (a + b) / (a + c),  # Frequency Bias Index (>1 wet bias, <1 dry bias)
            'base_rate': (a + c) / n,
        }

def bootstrap_hss(station_counts, n_boot=400, rng=None, level=0.90):
    """Station-level Poisson block bootstrap for 90% HSS confidence intervals.
    Resamples whole stations (shape: [n_stations, 4]) to preserve intra-station temporal correlation."""
    rng = rng or np.random.default_rng(42)
    weights = rng.poisson(1.0, size=(n_boot, len(station_counts)))
    totals = weights @ station_counts  # (n_boot, 4) -> summed [a, b, c, d] per bootstrap draw
    hss = contingency_scores(totals[:, 0], totals[:, 1], totals[:, 2], totals[:, 3])['hss']
    hss = hss[np.isfinite(hss)]
    tail = (1 - level) / 2 * 100
    return float(np.percentile(hss, tail)), float(np.percentile(hss, 100 - tail))

2.5 Conflict-of-Interest Safeguards & Matched-Cohort Verification (>= 90% Overlap)

WeatherXM operates FACT and also produces or collaborates on station-informed forecasts (MOSAIC and mLM). To prevent apples-to-oranges comparisons against uncalibrated global NWP models:

1. Explicit is_commissioned Flag & exclude_commissioned Toggle:

   • mLM (meteoblue station-calibrated ML) is trained on specific commissioned stations (~50 stations / ~10k station-days per lead day vs. ~2.4M for global NWP). Every tool accepts exclude_commissioned: true to strip mLM and return pure non-commissioned rankings (top_non_commissioned_models).

2. >= 90% Station-Day Overlap Matched Cohort (commissioned_evaluation.matched_cohort_top_models):

   • In any H3 cell where mLM is active on a subset of stations/days, FACT computes a restricted matched-cohort evaluation (pipeline/enrich_cells_top5.py) scoring all competing models strictly on the exact (device_id, target_date) pairs where mLM was active, and excluding any model that covers < 90% of those station-days (excluded_low_overlap).

3. The 45 Evaluated Forecast Models

4. Connecting Your AI Agent (MCP & REST)

Option A: Remote Streamable-HTTP MCP Server (Recommended — Zero Install, No Auth)

Point any MCP-compatible client directly to https://fact.weatherxm.com/mcp:

• Claude Code CLI:

  claude mcp add --transport http weatherxm-fact https://fact.weatherxm.com/mcp

• Cursor / Windsurf / VS Code / Claude Desktop (mcp.json):

  {
    "mcpServers": {
      "weatherxm-fact": {
        "url": "https://fact.weatherxm.com/mcp"
      }
    }
  }

Option B: Local stdio or Docker MCP Server (This Repository)

# Run directly with Node.js (>= 18)
git clone https://github.com/WeatherXM/weatherxm-fact-mcp.git
cd weatherxm-fact-mcp
npm test
node src/index.mjs

Or via Docker:

docker build -t weatherxm-fact-mcp .
docker run -i --rm weatherxm-fact-mcp

5. Exposed MCP Tools & REST Endpoints

Quick Python Example (examples/query_fact_mcp.py)

python3 examples/query_fact_mcp.py
python3 examples/verification_formulas.py

6. Repository Structure

weatherxm-fact-mcp/
├── docs/
│   └── FACT_METHODOLOGY.md          # Complete 10-section scientific methodology & limitations
├── examples/
│   ├── verification_formulas.py     # Standalone runnable Python implementation of FACT pooling & HSS math
│   ├── query_fact_mcp.py            # Zero-dependency Python client querying https://fact.weatherxm.com/mcp
│   └── test_stdio_and_scoring.mjs   # Automated test for stdio MCP handshake, tools/list & scoring formulas
├── functions/
│   └── mcp.ts                       # Production Cloudflare Pages Worker MCP & edge handler implementation
├── pipeline/
│   ├── bake_fact_summary.py         # DuckDB continuous MAE/RMSE sample-weighted pooling pipeline
│   ├── bake_precip_skill.py         # 2x2 rain contingency table (HSS, ETS, POD, FAR, CSI) & Poisson bootstrap
│   ├── enrich_cells_top5.py         # Per-cell top-5 non-commissioned & >=90% mLM matched-cohort builder
│   └── bake_mcp_index.mjs           # H3 Res-7 spatial shard & country leaderboard compiler
├── src/
│   └── index.mjs                    # Local stdio MCP server entrypoint (Node >= 18 / Docker)
├── Dockerfile                       # Container image for local stdio / Glama introspection
├── glama.json                       # Glama MCP server manifest
├── server.json                      # Official MCP Registry manifest (com.weatherxm.fact/mcp)
└── LICENSE                          # MIT License (WeatherXM AG)

License

Released under the MIT License by WeatherXM AG (weatherxm.com · fact.weatherxm.com).