# datasniff

<!-- mcp-name: io.github.Srikarmk/datasniff -->

<a href="https://github.com/Srikarmk/datasniff/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/Srikarmk/datasniff/actions/workflows/ci.yml/badge.svg" /></a> <a href="https://pypi.org/project/datasniff/"><img alt="PyPI" src="https://img.shields.io/pypi/v/datasniff?color=5b4fc9" /></a> <a href="https://pypi.org/project/datasniff/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/datasniff" /></a> <a href="https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.Srikarmk/datasniff"><img alt="MCP registry" src="https://img.shields.io/badge/MCP%20registry-io.github.Srikarmk%2Fdatasniff-5b4fc9" /></a> <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-MIT-blue" /></a>

**Is this dataset what I need, and am I allowed to use it?**

datasniff searches Hugging Face, Kaggle and data.gov at once and tells you, for each dataset,
what columns it has and their types, what licence it is under and what that licence lets you
do, and how big it is in bytes and rows. Every value says exactly where it came from: which
API, which field, which URL. When a value can't be found, datasniff says so and why, instead
of guessing.

It only reads public metadata. It never downloads a whole dataset: when the only way to learn
a file's columns is to look at it, it reads at most the first 64 KB.

There are two ways to use it, built on the same engine:

- **On the command line**: `datasniff search "air quality"`.
- **As an MCP server**, so an assistant such as Claude Code, Claude Desktop or Cursor can check
  a dataset's schema and licence before suggesting it.

## Install

datasniff needs Python 3.10 or newer. Pick one:

```
uv tool install datasniff     # with uv (https://docs.astral.sh/uv/)
pipx install datasniff        # with pipx
uvx datasniff search titanic  # run once without installing
```

This works the same on macOS, Windows and Linux. No account or API key is needed.

## Use it from the terminal

```
datasniff search "sentiment reviews"              # all three catalogs, 5 results each
datasniff search titanic --source kaggle -n 10    # one catalog (repeat --source for several)
datasniff details huggingface stanfordnlp/imdb    # schema, licence, size, files
datasniff details https://www.kaggle.com/datasets/heptapod/titanic
datasniff compare kaggle:heptapod/titanic huggingface:stanfordnlp/imdb
```

Sources are `huggingface` (or `hf`), `kaggle` and `datagov` (or `data.gov`). Ids are the ones
search prints: `owner/name` on Hugging Face, `owner/slug` on Kaggle, the slug on data.gov. A
dataset page URL works too.

Options: `--json` for machine-readable output, `--no-cache` to skip the cache for one run,
`--cursor SOURCE=CURSOR` for the next page (search prints the exact flag).

Exit codes: `0` success, `1` the dataset was not found or every source failed, `2` bad usage.

### Example

```
$ datasniff details hf stanfordnlp/imdb
IMDB  (huggingface: stanfordnlp/imdb)
https://huggingface.co/datasets/stanfordnlp/imdb

Licence
     other: Custom or unspecified terms; read the dataset's page.
     commercial use: unknown, attribution: unknown, share-alike: unknown
       from: cardData.license in https://huggingface.co/api/datasets/stanfordnlp/imdb?blobs=true
Size
     bytes: 79.6 MB (83,446,840 bytes)
       from: size.dataset.num_bytes_original_files in
       https://datasets-server.huggingface.co/size?dataset=stanfordnlp%2Fimdb
     rows: 100,000
       from: size.dataset.num_rows in
       https://datasets-server.huggingface.co/size?dataset=stanfordnlp%2Fimdb
Updated
     last modified: 2024-01-04T12:09:45.000Z
       from: lastModified in https://huggingface.co/api/datasets/stanfordnlp/imdb?blobs=true
Schema (1 table)
  plain_text: 2 columns; train 25,000, test 25,000, unsupervised 50,000
       from: dataset_info.plain_text.features in
       https://datasets-server.huggingface.co/info?dataset=stanfordnlp%2Fimdb
     text   string
     label  class_label[neg, pos]
Files (5)
     ...
```

In JSON, every value is an object with a `source`, and a missing one has a `reason`:

```json
"size": {
  "bytes": {
    "value": 29981,
    "source": "totalBytes (sum of the dataset's files, uncompressed) in https://www.kaggle.com/api/v1/datasets/view/yasserh/housing-prices-dataset"
  },
  "rows": {
    "value": null,
    "reason": "Kaggle does not publish row counts in its API or Croissant metadata",
    "source": "https://www.kaggle.com/api/v1/datasets/view/yasserh/housing-prices-dataset; ..."
  }
}
```

## Use it with an AI assistant (MCP)

`datasniff mcp` runs a Model Context Protocol server over stdio with three read-only tools:

| Tool | Use it when | Returns |
|---|---|---|
| `search_datasets(query, sources=all, limit=5, cursors=None)` | looking for data on a topic | a short list per catalog with licence, size and last update; a catalog that fails is reported without failing the call |
| `dataset_details(source, id)` | checking one dataset before relying on it | columns and types per file or config, licence and what it allows, bytes and rows, files, last update |
| `compare_licenses(datasets)` | combining several datasets | each licence and whether, together, they allow commercial use, need credit, or are share-alike |

### Claude Code

```
claude mcp add --scope user datasniff -- uvx datasniff mcp
```

`--scope user` makes it available in every project; leave it out to add it to the current
project only. Check it with `claude mcp list`.

### Claude Desktop

Open Settings > Developer > Edit Config, and add datasniff to
`claude_desktop_config.json` (on macOS `~/Library/Application Support/Claude/`, on Windows
`%APPDATA%\Claude\`):

```json
{
  "mcpServers": {
    "datasniff": {
      "command": "uvx",
      "args": ["datasniff", "mcp"]
    }
  }
}
```

Restart Claude Desktop. If it can't find `uvx`, use the full path that `which uvx` (macOS)
or `where uvx` (Windows) prints as the `command`.

### Cursor

Add the same block to `~/.cursor/mcp.json` (all projects) or `.cursor/mcp.json` in a project.

If you installed datasniff with `uv tool install` or `pipx`, you can use `"command": "datasniff"`
with `"args": ["mcp"]` instead. To pass credentials (below), add an `"env"` block.

## Where each value comes from

| | Hugging Face | Kaggle | data.gov |
|---|---|---|---|
| Search | Hub API `/api/datasets?search=` (most downloaded first) | `/api/v1/datasets/list` | catalog.data.gov search (see below) |
| Columns and types | dataset viewer `/info` (per config, with split sizes); the dataset card's `dataset_info` if the viewer has none | the dataset's Croissant metadata (`recordSet` fields and `dataType`) | the distribution's `describedBy` data dictionary (Socrata or CKAN), else sniffed from the first 64 KB of a CSV, TSV or JSON file |
| Licence | `cardData.license`, else `license:` tags | `licenseName` (plus the Croissant licence URL) | the DCAT-US `license` field |
| Bytes | viewer `/size` (original files), else the sum of the repository's file sizes | `totalBytes` (uncompressed); search shows the compressed download size | each file's `Content-Length` from a HEAD request (or a 1-byte ranged GET) |
| Rows | viewer `/size` | not published | the data dictionary's row count when the portal reports one |

Types are reported in the vocabulary of their source (`int64` and `class_label[...]` from
Hugging Face, `sc:Integer` from Croissant, `number` or `calendar_date` from a Socrata data
dictionary, `integer` / `number` / `date` / `string` when sniffed), so nothing is lost in
translation.

data.gov has no documented public JSON API any more (the old CKAN API was retired, and the
api.gsa.gov one needs a key). datasniff reads two undocumented endpoints of the current catalog
site: `catalog.data.gov/search?q=...` with `Accept: application/json`, and
`catalog.data.gov/api/dataset/<slug>`. They are isolated in one adapter, and if they change,
data.gov reports an error while the other catalogs keep working.

data.gov ranks results by relevance score, and those scores can shift slightly between requests, so
paging through results can occasionally repeat one from the previous page.

## Licences

datasniff turns a licence string or URL into an SPDX id, but only when that is unambiguous
(`cc-by-4.0`, `CC BY-NC-SA 4.0`, `Attribution 4.0 International (CC BY 4.0)` and
`https://creativecommons.org/licenses/by/4.0` all become `CC-BY-4.0`). Then it says, in one
line, whether commercial use is allowed, whether you must credit the source, and whether
adaptations must be shared under the same terms. Anything it doesn't recognise, including
Hugging Face's `other` and most custom terms, is reported as `unknown` with the original text,
never guessed. `compare_licenses` flags combinations such as two different share-alike
licences.

These meanings come from a small built-in table and are a reading aid, not legal advice. Read
the actual terms before you rely on them.

## Credentials (optional)

Everything works anonymously. If you have them:

- `HF_TOKEN`: a Hugging Face token, for gated or private datasets you have access to.
- `KAGGLE_USERNAME` and `KAGGLE_KEY`, or `~/.kaggle/kaggle.json` (Kaggle > Settings > API >
  Create New Token): used if Kaggle stops answering anonymous requests.

Tokens are sent only to their own service (never to another host, including after a
redirect), and are never printed: error messages are scrubbed of anything that looks like a
token, key or signed URL.

## Being a polite client

- Only GET and HEAD requests, enforced in code; one module does all network access.
- A clear `User-Agent`: `datasniff/<version> (+https://github.com/Srikarmk/datasniff)`.
- At most four requests at a time, timeouts on every request, and retries with exponential
  backoff (honouring `Retry-After`) on 429 and 5xx responses.
- Responses are cached on disk for 24 hours, in your platform's cache folder
  (`~/Library/Caches/datasniff` on macOS, `%LOCALAPPDATA%\datasniff\Cache` on Windows,
  `~/.cache/datasniff` on Linux). Set `DATASNIFF_CACHE_TTL` (seconds), `DATASNIFF_CACHE_DIR`, or
  `DATASNIFF_NO_CACHE=1` to change that.
- File sniffing asks for the first 64 KB with an HTTP Range request, and stops reading at
  64 KB even when a server ignores the Range header and starts sending the whole file.

## Privacy

datasniff collects nothing. It sends your search terms to the catalogs you search, and nothing
anywhere else.

## Licence

MIT. Copyright (c) 2026 Kameswara Sai Srikar Manda.