# datagovin-mcp

Natural-language access to **India's Open Government Data platform**,
[data.gov.in](https://data.gov.in) — **235,000+ public datasets** covering air
quality, agriculture, health, fuel prices, census, education, rainfall, railways,
crime, budgets and more.

Two faces, one codebase:

- **An MCP server** — over stdio for local clients, or over Streamable HTTP so
  *any* MCP client connects by URL: Claude Desktop, Claude Code, Cursor, VS Code,
  Windsurf, ChatGPT connectors, or anything built on an MCP SDK.
- **A website** — instant catalog search plus natural-language answers, backed by
  the exact same tools.

The server ships **no data of its own**. Discovery runs against a local full-text
index built from data.gov.in's own catalog endpoint; every row you actually read
is a live call to data.gov.in using *your own* free API key.

## Why this exists

data.gov.in has an enormous catalog but no full-text search API a program can
call — the normal workflow is to browse the website and copy a dataset's resource
ID off its "API" button. That's a poor fit for a language model.

This project closes the gap. It harvests the platform's `/lists` endpoint into a
local SQLite FTS5 index — **235,241 datasets in about 150 seconds, no API key
required** — so a model can go from *"what's the AQI in Delhi right now?"* to real
rows without anyone hunting for a UUID. It also absorbs the upstream API's rough
edges (case-sensitive filters, occasional CSV responses, a last-page pagination
quirk) so the model doesn't have to.

## Tools

| Tool | What it does |
|------|--------------|
| `search_datasets(query, limit, sector)` | BM25-ranked full-text search across the whole catalog. |
| `list_sectors(limit)` | Sectors present in the catalog, with dataset counts. |
| `get_dataset_info(resource_id)` | Live schema: title, description, row count, exact field names + types. |
| `query_dataset(resource_id, filters, fields, sort, limit, offset)` | Pull actual filtered rows, live. |
| `catalog_status()` | How many datasets are indexed — distinguishes "no matches" from "not harvested yet". |

## Setup

**1. Install.**

```bash
git clone https://github.com/<your-username>/datagovin-mcp.git
cd datagovin-mcp
python -m venv .venv && source .venv/bin/activate
pip install -e .            # MCP server only
pip install -e ".[web]"     # + the website
```

**2. Build the search index.** No API key needed for this step.

```bash
datagovin-harvest
```

```
Harvesting the data.gov.in catalog (no API key required)...
  indexed 235,000/235,241 datasets (100%, 1,566/s)

Indexed 235,241 datasets in 150.2s -> ~/Library/Caches/datagovin-mcp/catalog.sqlite3 (346.2 MB)
```

Until you run this, search falls back to a small bundled seed catalog — the
server still works, it just knows about three datasets. Re-run it any time to
refresh; hand-curated entries are preserved.

**3. Get a free API key** — needed to read rows, not to search. Register at
[data.gov.in](https://data.gov.in) and generate one from your profile page.

```bash
cp .env.example .env    # then paste your key into it
```

The `.env` file is read automatically. Exporting `DATA_GOV_IN_API_KEY` works too,
and a real environment variable always wins over the file.

## Connecting a client

### Local (stdio)

Add to your MCP client config (`claude_desktop_config.json` or equivalent):

```json
{
  "mcpServers": {
    "datagovin": {
      "command": "/absolute/path/to/datagovin-mcp/.venv/bin/python",
      "args": ["/absolute/path/to/datagovin-mcp/server.py"],
      "env": { "DATA_GOV_IN_API_KEY": "your_key_here" }
    }
  }
}
```

### Remote (Streamable HTTP) — connects from anywhere

```bash
python server.py --transport http --host 0.0.0.0 --port 8000
# MCP endpoint: http://<host>:8000/mcp
```

Then point any MCP client at the URL:

```json
{
  "mcpServers": {
    "datagovin": { "url": "https://your-host.example.com/mcp" }
  }
}
```

Add `--stateless` to run several replicas behind a load balancer.

> **Before exposing this publicly**, put it behind TLS and authentication. The
> server has no auth of its own, and it spends *your* data.gov.in API key on
> every request it serves.

## The website

```bash
export ANTHROPIC_API_KEY=sk-ant-...   # optional — enables the "Ask" button
datagovin-web                          # http://127.0.0.1:8000
```

One process serves everything:

| Route | |
|---|---|
| `/` | search UI — type-ahead catalog search, click a dataset for its live schema and sample rows |
| `/api/search?q=` | BM25 search as JSON, no LLM involved |
| `/api/dataset/{id}` | live schema |
| `/api/dataset/{id}/rows` | live rows; any extra query param becomes an upstream filter |
| `/api/ask` | streaming natural-language answer (Server-Sent Events) |
| `/mcp` | the MCP endpoint — so the same deployment serves browsers *and* MCP clients |

Search works with no keys at all. `DATA_GOV_IN_API_KEY` unlocks rows;
`ANTHROPIC_API_KEY` unlocks answers. The UI tells you which are missing.

**How answers work.** `/api/ask` runs a streaming Claude tool-use loop over the
same five tools, narrating each step ("Searching the catalog for…", "Fetching 100
rows where city=Delhi") before the answer streams in. Claude is instructed to
answer only from rows it actually fetched, to name the dataset it used, and to say
so plainly when the data doesn't answer the question rather than filling the gap
from memory.

The tool definitions the website gives Claude are read directly off the MCP
server via `list_tools()` — there is exactly one description and one schema per
tool in this project, so the two surfaces cannot drift apart.

## Curating a dataset

Harvesting brings in every dataset automatically. Use this to *improve* one —
attach search keywords, a worked example filter, or a corrected sector, and pin
it above harvested results:

```bash
python scripts/add_dataset.py <resource_id> \
    --sector Agriculture \
    --keywords "wheat,crop,production" \
    --example-filters '{"State":"Punjab"}'
```

Curated fields survive later harvests.

## Notes on the upstream API

Behaviours this server handles for you:

- **Filter field names are case-sensitive** (`filters[State]` ≠ `filters[state]`)
  and this is undocumented. Always use the exact field `id` from `get_dataset_info`.
- **Some legacy datasets return CSV** regardless of `format=json`; the client
  detects this by Content-Type and parses it anyway. CSV carries no row total, so
  `total_records` comes back `null` rather than a misleading page count.
- **Last-page pagination** can return an empty `records` array with `status: ok`;
  `returned: 0` means you're done.
- **Max ~100 rows per request** on `/resource` — page with `offset`.
- **`/lists` needs no API key** and pages up to 1000 records at a time. It is
  slow and occasionally times out, so the harvester retries every page with
  backoff.
- **The API key travels in the query string** (upstream's design). Every error
  this package raises is passed through a redactor first, so a key can never
  reach a log line, a tool result, or the model's context.

## Project layout

```
datagovin-mcp/
├── server.py                    # entry point (kept for existing client configs)
├── datagovin/
│   ├── config.py                # .env loading, cache paths
│   ├── client.py                # async data.gov.in API wrapper (quirk handling)
│   ├── catalog.py               # SQLite FTS5 index: search, sectors, stats
│   ├── harvest.py               # builds the index from /lists
│   ├── mcp_server.py            # the five tools; stdio + Streamable HTTP
│   ├── data/seed_catalog.json   # bundled fallback, works before a harvest
│   └── web/
│       ├── app.py               # FastAPI: search API, /api/ask, mounts /mcp
│       ├── agent.py             # streaming Claude tool-use loop
│       └── static/index.html    # the UI (no build step, no CDN)
├── scripts/add_dataset.py       # curate/pin one dataset
└── tests/                       # 87 tests, no network required
```

The index lives in your platform cache directory, not in the package — it is
generated data, it is ~350 MB, and an installed package directory is often
read-only. Override with `DATAGOVIN_INDEX_PATH`.

## Development

```bash
pip install -e ".[web,dev]"
pytest                      # 87 tests, all offline
```

## License

MIT