# CKAN MCP Server (Python)
A Model Context Protocol (MCP) server that exposes the 600 plus[^1][^2] global CKAN open-data portals to AI assistants, CLI tools, and other MCP-aware clients. It bundles curated portal presets, strict Pydantic models, and a batteries-included tool suite so analysts, application developers, and operators can explore public datasets without writing bespoke CKAN integrations.
## Persona-centric Guidance
| Persona | Primary questions | Suggested section |
| --- | --- | --- |
| Curious evaluators | "What can this server do?" "Which CKAN actions are covered?" | [Potential users](#potential-users-what-this-server-does) |
| Data analysts / MCP end-users | "How do I set it up locally?" "How do I connect to a remote MCP server?" | [Data analysts](#data-analysts-getting-insights-fast) |
| Contributors / maintainers | "How is the code organized?" "How do I run tests?" | [Developers](#developers-extend-and-contribute) |
| Platform / infra teams | "Can I deploy this to Cloud Run?" | [Production deployment](#production-deployment-google-cloud-run-example) |
---
## Potential Users: What This Server Does
### Why it exists
- **Purpose-built CKAN interface**: Wraps the CKAN Action & Datastore APIs behind MCP tools that AI agents and CLI clients understand.
- **Consistent insights**: Presents dataset summaries, freshness analysis, schemas, and download helpers so exploratory conversations stay grounded in real CKAN metadata.
- **Portal-aware behavior**: Curated overrides (transport method, dataset URL templates, helper prompts) keep the experience consistent across CKAN portals that deviate from defaults.
### Tool catalog (14 tools)
| Category | Tool | What it returns |
| --- | --- | --- |
| Session configuration | `ckan_api_initialise` | Selects a portal (country/location + overrides) and stores API keys/session metadata. |
| | `ckan_api_availability` | Lists the configured CKAN portals and reports the current session's selection (when set). |
| | `audit_ckan_api` | Probes GET/POST behavior, datastore aliases, and helper metadata; emits recommended overrides for future sessions. |
| Dataset retrieval | `get_package` | Full CKAN dataset metadata (resources, organization, extras). |
| | `list_datasets` | Paginated package list with optional total counts. |
| | `search_datasets` | Action API `package_search` wrapper with passthrough Solr parameters. |
| | `get_data_categories` | Organizations and groups for navigation. |
| Datastore access | `get_first_datastore_resource_records` | Pulls preview rows from the first active datastore resource. |
| | `get_resource_records` | Targeted datastore search with filters, sorts, distinct, etc. |
| | `download_dataset_locally` | Metadata-rich archive/download helper with MIME detection, extraction, and how-to snippets. |
| Analysis | `find_relevant_datasets` | Weighted scoring across title/description/tags/org/resource metadata. |
| | `analyze_dataset_updates` | Frequency heuristics plus CKAN update timestamps. |
| | `analyze_dataset_structure` | Schema summaries, record counts, sample fields. |
| | `get_dataset_insights` | Combines discovery, updates, structure, and helper prompts into one rich response. |
### Architecture at a glance
- `src/ckan_mcp/main.py` – MCP server entry point supporting stdio and HTTP (SSE) transports.
- `src/ckan_mcp/ckan_tools.py` – Tool implementations, transport probes, download helpers, and archive extraction.
- `src/ckan_mcp/helpers.py` – Relevance scoring, update frequency analysis, summary builders.
- `src/ckan_mcp/types.py` – Strict Pydantic models with `extra="allow"` for portal-specific metadata.
- `src/ckan_mcp/config_selection.py` & `src/ckan_mcp/data/ckan_config_selection.json` – Curated CKAN portal catalog and overrides consumed by `ckan_api_initialise`.
- `tests/` & `test_runner.py` – Pytest suite plus quick smoke runner mirroring production behaviors.
Tip: start with `ckan_api_initialise` to choose a portal, then call the analysis tools to see the depth of insights returned.
---
## Data Analysts: Getting Insights Fast
### Shared prerequisites
- CKAN portal URL (or use the curated list during initialization).
- Python 3.11+ and [`uv`](https://github.com/astral-sh/uv) or `pip` for installing dependencies.
- [curl](https://curl.se/) and the POSIX [`file`](https://man7.org/linux/man-pages/man1/file.1.html) command on your `PATH` (the `download_dataset_locally` tool shells out to both binaries).
- An MCP-compatible client (Claude CLI, Gemini CLI, etc.).
### Option A – Run the MCP server locally
1. **Clone & create a virtual environment**
```bash
git clone https://github.com/<org>/ckan-mcp.git
cd ckan-mcp
uv venv venv
source venv/bin/activate
```
2. **Install runtime dependencies**
```bash
uv pip install -e .
```
(Add `".[dev]"` for development tooling and `".[examples]"` if you want to run the sample scripts that load `.env` files.)
3. **Optional defaults** – export CKAN env vars if you always talk to the same portal:
```bash
export CKAN_BASE_URL="https://ckan0.cf.opendata.inter.prod-toronto.ca/api/3/action"
export CKAN_SITE_URL="https://ckan0.cf.opendata.inter.prod-toronto.ca"
export CKAN_DATASET_URL_TEMPLATE="https://ckan0.cf.opendata.inter.prod-toronto.ca/dataset/{name}"
```
These are fallback values; interactive sessions normally rely on `ckan_api_initialise` to pick a portal.
4. **Launch in stdio mode** (best for desktop MCP clients):
```bash
python -m ckan_mcp.main
```
5. **Connect your MCP client** – example Claude CLI snippet (see [core environment variables](#core-environment-variables) for transport overrides):
```json
{
"mcpServers": {
"ckan-mcp": {
"command": "python",
"args": ["-m", "ckan_mcp.main"],
"env": {
"CKAN_MCP_LOCAL_DATASTORE": "~/dataset-store/",
}
}
}
}
```
6. **Start a session** – ask your assistant to "Initialize a CKAN connection"; it will call `ckan_api_initialise` and then the discovery tools.
### Option B – Use a remote MCP server with a local client
This flow is perfect when someone else operates the MCP server on shared infrastructure and you only need local MCP tooling.
1. **Server operator** sets up HTTP transport:
```bash
export CKAN_MCP_MODE=http
export CKAN_MCP_HOST=0.0.0.0
export CKAN_MCP_PORT=8000
python -m ckan_mcp.main
```
or run `docker compose up --build` to expose `http://localhost:8000/mcp` and front it with your preferred reverse proxy.
2. **Expose the `/mcp` endpoint** via HTTPS (Cloud Run, Fly.io, Tailscale, etc.) and share the URL with analysts.
3. **Analyst registers the remote MCP server** (Claude CLI example):
```bash
claude mcp add --transport http ckan-mcp https://mcp.example.com/mcp
claude mcp list
```
Gemini CLI uses `gemini mcp add --transport http ckan-mcp https://mcp.example.com/mcp` with the same URL.
4. **Use normally** – all CLI/desktop prompts now tunnel through the remote MCP server. The analyst still decides which CKAN portal to inspect via `ckan_api_initialise`.
### Day-to-day usage tips
- `ckan_api_availability` lists every CKAN portal packaged with this MCP build and reiterates which portal is currently selected (if any) before issuing expensive searches.
- `find_relevant_datasets` quickly surfaces top matches for natural-language prompts; follow up with `get_dataset_insights` for a detailed brief.
- `download_dataset_locally` writes metadata, datastore previews, and shell instructions to `~/.cache/ckan-mcp/...` so you can pivot to pandas immediately.
---
## Developers: Extend and Contribute
### Repository map
```
src/ckan_mcp/
├── main.py # MCP entry point + HTTP transport
├── ckan_tools.py # Tool implementations & download helpers
├── helpers.py # Scoring, frequency, and summary helpers
├── types.py # Pydantic models
├── config_selection.py # Catalog loader & helper utilities
├── data/
│ └── ckan_config_selection.json # Curated CKAN catalog & overrides
└── __init__.py
```
Supporting files: `pyproject.toml` (uv/poetry style metadata), `tests/`, `test_runner.py`, `examples/` for fixtures, and Docker/Make targets for container workflows.
### Local development workflow
1. **Activate the virtualenv** and install dev dependencies:
```bash
source venv/bin/activate
uv pip install -e ".[dev]"
```
2. **Run formatters and linters** (Black first, then Ruff as required by the project guidelines):
```bash
black src/ tests/
ruff check src/ tests/ --fix
```
3. **Type checking**:
```bash
mypy src/
```
4. **Tests**:
```bash
pytest tests/ -v
python test_runner.py # lightweight smoke run
# Live integration tests against the curated CKAN portals
CKAN_RUN_INTEGRATION_TESTS=1 pytest tests/ -m integration -v
```
Integration tests talk to the public CKAN portal configured via `CKAN_TEST_COUNTRY`/`CKAN_TEST_LOCATION` (defaults to Canada/Toronto) and accept overrides such as `CKAN_TEST_BASE_URL`, `CKAN_TEST_SITE_URL`, `CKAN_TEST_DATASET_URL_TEMPLATE`, or `CKAN_TEST_SEARCH_TERMS` for custom portals.
5. **GitHub Actions verification** (optional, requires the [GitHub CLI](https://cli.github.com/) authenticated against `openascot/ckan-mcp-private`):
```bash
# trigger the full workflow (lint/unit + integration jobs) for your current branch
gh workflow run ci.yml --ref "$(git rev-parse --abbrev-ref HEAD)"
# tail the logs for the most recent run
gh run watch
gh run view --log-failed
```
The workflow only runs when triggered manually; the `quality-checks` job runs Black/Ruff/mypy, and the dependent `pytest-suite` job reuses `.github/workflows/pytest.yml` to execute the standard pytest run plus the integration suite (with `CKAN_RUN_INTEGRATION_TESTS=1`). Trigger the standalone `Pytest` workflow directly if you only need the testing jobs.
5. **Docker-based workflow** (optional, HTTP transport exposed at `http://localhost:8000/mcp`):
```bash
make dev # foreground dev stack with reload
make quick-start # background stack, uses docker-compose.dev.yml
make dev-tools # helper container via the tools profile
make shell # attach to the running dev app container
make test-production # builds docker-compose.yml and curls /mcp
```
Follow the AGENTS.md guidance for naming, docstrings, and how to place fixtures under `examples/`. Always update or add pytest coverage alongside new tools or helper behaviors.
See [CHANGELOG.md](CHANGELOG.md) for release history and public milestone notes.
### Contributing checklist
- Create a feature branch and keep commits focused.
- Add or update tests under `tests/` mirroring the target module name (e.g., `tests/test_ckan_tools.py`).
- Run `pytest`, `black`, `ruff`, and `mypy` locally (or via the docker helpers) before opening a PR.
- Document new environment variables or tool behaviors in this README or `EVALUATION_GUIDE.md` as appropriate.
---
## Production Deployment (Google Cloud Run Example)
Cloud Run pairs nicely with the built-in HTTP transport. The following example assumes you have the Google Cloud CLI configured and Artifact Registry enabled.
1. **Build and push the container** (uses the included multi-stage `Dockerfile`):
```bash
export PROJECT_ID="my-gcp-project"
gcloud auth configure-docker
gcloud builds submit --tag gcr.io/$PROJECT_ID/ckan-mcp
```
2. **Deploy to Cloud Run**:
```bash
gcloud run deploy ckan-mcp \
--image gcr.io/$PROJECT_ID/ckan-mcp \
--region us-central1 \
--platform managed \
--allow-unauthenticated \
--port 8000 \
--set-env-vars CKAN_MCP_MODE=http,CKAN_MCP_HTTP_PATH=/mcp,CKAN_MCP_HTTP_ALLOW_ORIGINS=* \
--set-env-vars CKAN_BASE_URL=https://ckan0.cf.opendata.inter.prod-toronto.ca/api/3/action,CKAN_SITE_URL=https://ckan0.cf.opendata.inter.prod-toronto.ca
```
Adjust env vars for your preferred portal or omit them so analysts always call `ckan_api_initialise`.
3. **Share the endpoint** – Cloud Run will emit a URL such as `https://ckan-mcp-12345-uc.a.run.app`. Provide the `/mcp` path to clients (`https://ckan-mcp-12345-uc.a.run.app/mcp`).
4. **Register with MCP clients** – same `claude mcp add --transport http ...` flow as in the analyst section.
5. **Operational tips**:
- Set `CKAN_MCP_HTTP_JSON_RESPONSE=true` if your proxy expects JSON instead of SSE.
- Use Secret Manager to supply `CKAN_API_KEY` for locked-down portals.
- Monitor Cloud Run metrics; the server makes outbound HTTPS calls to CKAN only when tools are invoked.
---
## Configuration Reference
### Core environment variables
| Variable | Default | Purpose |
| --- | --- | --- |
| `CKAN_BASE_URL` | none | Optional default Action API base; sessions can override via `ckan_api_initialise`. |
| `CKAN_SITE_URL` | none | Root site URL used for dataset links. |
| `CKAN_DATASET_URL_TEMPLATE` | none | Overrides dataset page URL format (`{name}` and `{id}` supported). |
| `CKAN_API_KEY` | none | API key used when the selected portal requires authentication. |
| `CKAN_MCP_MODE` | `stdio` | `stdio` for CLI integrations, `http` for streamable HTTP transport. |
| `CKAN_MCP_HOST` | `0.0.0.0` (HTTP mode) | Bind host when `CKAN_MCP_MODE=http`. |
| `CKAN_MCP_PORT` | `8000` | Bind port for HTTP mode. |
| `CKAN_MCP_HTTP_PATH` | `/mcp` | Mount path for HTTP transport (used both by builtin HTTP server and Cloud Run deployments). |
| `CKAN_MCP_HTTP_ALLOW_ORIGINS` | `*` | CORS allowlist for HTTP mode. |
| `CKAN_MCP_HTTP_JSON_RESPONSE` | `false` | Emit JSON responses instead of SSE when `true`. |
| `CKAN_MCP_HTTP_LOG_LEVEL` | `info` | Log verbosity for HTTP transport. |
| `CKAN_MCP_LOCAL_DATASTORE` | `./` (current directory) | Local directory path where downloaded datasets are stored. Defaults to
current working directory if not set. |
### CKAN portal overrides
The curated catalog in `src/ckan_mcp/data/ckan_config_selection.json` contains entries such as Toronto, NYC, etc. Each location can provide overrides like:
- `action_transport`: force GET vs POST for `/api/3/action` calls.
- `datastore_id_alias`: whether `datastore_search` accepts `id` instead of `resource_id`.
- `requires_api_key`: block initialization until an API key is supplied.
- `helper_prompt`: user-facing reminder echoed in tool responses.
- Pagination settings (`default_search_rows`, `max_search_rows`, `default_preview_limit`).
Call `audit_ckan_api` after selecting a portal to get automatically generated override recommendations, helper prompt text, and config snippets that can be pasted back into the catalog or used ad hoc via `ckan_api_initialise(overrides={...})`.
## Sources / References
[^1]: DataShades, *CKAN Instances*, accessed November 30, 2025, https://datashades.info/.
[^2]: commondataio/dataportals-registry, accessed November 30, 2025, https://raw.githubusercontent.com/commondataio/dataportals-registry/refs/heads/main/data/datasets/bysoftware/ckan.jsonl.
