# MCP Public Data

A generalized MCP server over **free public data sources** (EU, Global, CH, AT, DE, US). One uniform tool surface across all sources — no bespoke per-API clients.

## Design

The core idea is **generalization over specialization**:

- One parser per on-the-wire format (`public_data/parsers.py`): CSV, JSON-stat, SDMX-JSON, World-Bank tabular, OData.
- One **mother class** (`public_data/base.py: DataSource`) providing all shared plumbing (HTTP client, fail-soft catalog, uniform capability surface, export helpers).
- **Archetypes** (`public_data/archetypes.py`) — reusable middle layer: `JsonStatSource`, `ODataSource`, `TabularJsonSource`, `SdmxSource`, `DirectCsvSource`.
- **One file per concrete source** (`public_data/sources/<source>.py`), each a small subclass that only declares metadata + a fetch strategy. No bespoke parsing.

```
Schema
  └── DataSource (mother)
        ├── JsonStatSource      → Eurostat, (OECD)
        ├── TabularJsonSource   → World Bank
        ├── ODataSource         → WHO GHO, OpenFEMA
        ├── SdmxSource          → Bundesbank / OECD SDMX
        ├── DirectCsvSource     → OWID, + direct-download sources
        └── (custom subclass)   → BFS STAT-TAB (PxWeb POST), geo.admin.ch,
                                  Swiss transport — each overrides only the
                                  one step that differs and inherits the rest
```

BFS is the one source that breaks the GET pattern — STAT-TAB answers table
queries with a POST and (on this host) only serves the `de`/`fr` language
slices — so it subclasses `DataSource` directly and overrides only `fetch`.

## Tools (same for every source)

| Tool | Purpose |
|---|---|
| `list_databases()` | List sources + the tables each exposes |
| `get_query_guide(source)` | Query syntax / docs for one source |
| `get_schema(source, table)` | Column names + types |
| `get_sample_data(source, table, limit)` | Small sample (default 5 rows) |
| `execute_query(source, table, query)` | Run a query → rows |
| `export_data(source, table, query, format)` | CSV (default) or JSON export |

## Enabled sources (keyless, verified live)

| Source | Slug | Scope | Category | Format |
|---|---|---|---|---|
| Eurostat | `eurostat` | EU | multi | JSON-stat |
| World Bank WDI | `worldbank` | Global | multi | tabular JSON |
| WHO Global Health Observatory | `who_gho` | Global | health | OData |
| OpenFEMA | `openfema` | US | other (disasters) | OData |
| US Bureau of Labor Statistics | `bls` | US | labor | JSON (POST) |
| Our World in Data | `owid` | Global | multi | direct CSV |
| swisstopo geo.admin.ch | `geo_admin` | CH | infra/logistics | REST |
| Swiss public transport | `swiss_transport` | CH | infra/logistics | REST |
| BFS STAT-TAB (CH federal stats) | `bfs` | CH | multi | JSON-stat (POST) |
| Deutsche Bundesbank | `bundesbank` | DE/EU | gov finance | SDMX-JSON |
| DWD Climate Data Center | `dwd` | DE | other (climate) | direct download (dir) |
| European Central Bank | `ecb` | EU/euro | finance | SDMX-JSON 2.1 |
| OpenAlex | `openalex` | Global | research | REST JSON |
| Crossref | `crossref` | Global | research | REST JSON |
| UK Office for National Statistics | `ons` | UK | multi | REST JSON |
| US SEC EDGAR | `sec_edgar` | US | finance | REST JSON |
| Open-Meteo | `open_meteo` | Global | climate & weather | columnar JSON |
| US National Weather Service | `nws` | US | climate & weather | GeoJSON |
| USGS Earthquakes | `usgs` | Global | geohazards | GeoJSON |
| openFDA | `openfda` | US | health | REST JSON |
| NOAA NCEI Climate Data Online | `noaa` | US/Global | climate & weather | REST JSON |
| OpenStreetMap Nominatim | `nominatim` | Global | geospatial | REST JSON |
| Socrata (SODA) portals | `socrata` | Global/US | multi | SODA |

## Running

```bash
python3 -m venv .venv
.venv/bin/pip install -e .
.venv/bin/python server.py        # starts MCP on stdio
```

### Test

```bash
.venv/bin/python smoke_test.py    # hits live endpoints; 23 sources
PYTHONPATH=src python3 tests/adversarial_test.py   # fail-soft + edge-case suite
```

## Free-key APIs (documented, NOT wired yet)

These need free registration and are intentionally left out of the live surface
for now (no-key-first). Add them as subclasses behind optional credentials when
desired.

| Source | Slug suggestion | Cat | Key |
|---|---|---|---|
| UN Comtrade | `comtrade` | trade | free key |
| WTO Statistics | `wto` | trade/tariffs | free key |
| FAOSTAT (FAO) | `faostat` | agriculture | free key (bulk download keyless) |
| ILOSTAT (ILO) | `ilostat` | labor | free key (bulk CSV keyless) |
| NASA FIRMS (wildfires) | `firms` | other | free MAP_KEY (5000 req/10min) |
| US Census Bureau | `census` | other | free key |
| NOAA CDO v2 | `noaa` | other | free token |

## Direct-download targets worth adding (next)

- Bureau of Justice Statistics (US prisons) — CSV/XLSX downloads
- ReliefWeb v2 (disasters) — `/v2/disasters`
- OECD SDMX — `https://sdmx.oecd.org/public/rest` (dataflow verified 200; dataset-specific keys need tuning)
- UN/LOCODE (global ports/logistics), ECDC, UCDP (conflict), data.gv.at OGD / Statistik Austria STATcube / GeoSphere (AT) — keyless when enabled, not yet wired

## Notes / gotchas learned

- **Pagination** is implemented for OpenAlex (per-page+page), ONS (limit+offset), and NOAA (limit+offset). All cap at `MAX_QUERY_ROWS` (20,000). SEC EDGAR returns only the ~1000 most recent filings (server-capped; older filings require separate archive requests).
- **geo.admin.ch STAC v0.9** returns 500 intermittently (server-side flake / limit param). The `api3.geo.admin.ch` REST endpoint is reliable — the adapter uses that.
- **FAOSTAT** old `fenixservices.fao.org` endpoint is dead; current base is `faostatservices.fao.org/api/v1`.
- **ReliefWeb v1** is decommissioned; use v2.
- **data.gv.at** (AT) CKAN API is deprecated (404); portal moved to Piveau `/api/hub/`.
- **Bundesbank** serializes **SDMX-JSON 1.0** (`{"meta","data"}` envelope), not 2.1 — the parser unwraps it. Series keys are flow-specific (use e.g. `BBEX3/A.AED.DEM.CA.AA.A04`); the `M.USD.EUR...` form is not accepted.
- **BFS STAT-TAB** pxweb REST v1 serves only the `de`/`fr` language slices on `pxweb.bfs.admin.ch` (`en`/`it` on this route → 400); the adapter defaults to German.
- **data.europa.eu** hub search (`/api/hub/search/datasets`) returns 400/401 on keyless requests — not integrated.
- **Zefix** (CH company register) returns 401 without credentials despite "PublicREST".
- **BLS** allows ~5k requests/day keyless (v2) but raises limits with a free key.
- **openFDA** query filters need the `search=` prefix (e.g. `search=receivedate:[20240101+TO+20240131]`), not a bare field.
- **NOAA CDO v1** rejects `dataTypes=TMAX,TMIN` on some endpoints (400) — drop `dataTypes` for the default daily-summaries listing.
- **Nominatim** requires a browser-style `User-Agent` (`Mozilla/5.0 (compatible; ...)`) — a plain `app/1.0` UA gets 403 — and enforces 1 req/s fair use.
- **SEC EDGAR** and **NWS** mandate a descriptive `User-Agent`; both are set in their adapters.
- **ECB** serves SDMX-JSON **2.1** (`dataSets` + `structure` dimensions) — the existing `parse_sdmx_json` already handles it; no new parser needed.
- **IMF** only serves SDMX over plain HTTP — excluded on security grounds.


## License

Licensed under the Apache License, Version 2.0 — see [LICENSE](LICENSE).