datamining-skill
README.md
# DataMining Skill
DataMining Skill pulls data out of very large text files (CSV, TSV, JSON, JSON Lines, logs and
plain text) on an ordinary computer. It is a command-line tool, a Python library and an MCP server
in one.
The file is read in pieces, so memory use follows the settings and not the size of the file.
Progress is saved as it goes: if a run is interrupted, the same command carries on from the last
checkpoint.
It was written for one job: getting a clean list of e-mail addresses out of a big export or log.
You can also give it a regular expression of your own. Use it from a terminal, call it from Python,
or point an assistant that speaks the [Model Context Protocol](https://modelcontextprotocol.io)
(Claude Code, Claude Desktop, Cursor and others) at the MCP server and the skill that come with it.
[](https://github.com/coreworkstr-art/datamining-skill/actions/workflows/ci.yml)






A few things worth knowing before you start:
- Everything stays on your machine. There is no network code, no telemetry and no third-party
package, and your data is never sent anywhere.
- Memory use is set by the configuration, not by the file. The tests check it, and the
[Performance](#performance) figures include a 10 GiB file.
- If a run is killed or the power goes, running the same command again gives the same records an
uninterrupted run would have. [Limitations](#limitations) has the details, such as the order of
records after a retry.
- gzip, bzip2, xz and single-file zip archives are read directly, and so are Excel's "Unicode Text"
(UTF-16) files. One command turns a messy export into a list of distinct, lower-case addresses.
- For assistants there are six MCP tools, a Claude Code plugin and a skill that says when to use
them. File access is limited to the directories you list.
## Contents
[Install](#install) | [Quick start](#quick-start) | [Use it from an AI assistant](#use-it-from-an-ai-assistant) |
[Command line](#command-line) | [Recipes](#recipes) | [Python API](#python-api) |
[How it works](#how-it-works) | [Performance](#performance) | [Guarantees](#guarantees) |
[Limitations](#limitations) | [Troubleshooting](#troubleshooting) | [Development](#development) |
[Contributing](#contributing)
## Install
Python 3.11 or newer; nothing else is needed at runtime.
```bash
# with uv: the command gets its own isolated environment
uv tool install git+https://github.com/coreworkstr-art/datamining-skill
# or with pip
pip install git+https://github.com/coreworkstr-art/datamining-skill
# or from a clone, which also gives you the tests and the example data
git clone https://github.com/coreworkstr-art/datamining-skill && cd datamining-skill
pip install .
```
Check it works: `datamining-skill --version`. If the command is not found, see
[Troubleshooting](docs/troubleshooting.md#installing-and-connecting) or run it as
`python -m datamining_skill`.
The examples below use a small sample file, `examples/contacts.csv`. A clone has it; otherwise
[download it](https://raw.githubusercontent.com/coreworkstr-art/datamining-skill/main/examples/contacts.csv)
or use any CSV, JSON, log or text file of your own.
## Quick start
Inspect a file without reading it into memory, extract a clean address list, check the result:
```bash
# What is in this file? (milliseconds, constant memory)
datamining-skill profile examples/contacts.csv
# Is the file in good shape? Malformed rows, empty values, type changes (counts and line numbers only)
datamining-skill quality examples/contacts.csv
# Every distinct e-mail address, lower-cased, one per line (resumable, crash-safe)
datamining-skill mine examples/contacts.csv emails.csv --unique --lowercase
# Look at the first records of the result
datamining-skill preview emails.csv --rows 5
```
`profile` prints a JSON description: size, encoding, format, columns or keys, and the record
count with its unit (a quoted CSV value can span lines, so a CSV is counted in lines). `mine`
writes `emails.csv` and prints a JSON summary:
```json
{
"output_name": "emails.csv", "resumed": false, "chunks_total": 1, "chunks_processed": 1,
"chunks_failed": 0, "records_written": 15, "duplicates_skipped": 3, "already_complete": false,
"oversized_lines": 0, "source_transform": null, "first_error": null
}
```
`quality` reads a CSV, TSV or JSON Lines file once and reports what is wrong with it: records that
are malformed (a row with too few or too many columns, a broken quote, a line that is not JSON)
with the first line numbers, how many values of each field are empty or missing, the types found in
each field, and fields whose values change type (a column of numbers with a few words in it). Each
field also gets an approximate count of its distinct values and, if it is numeric with at least ten
values, its mean, spread and estimated quartiles. It keeps counts, line numbers and small summaries
of fixed size, never values, so its memory does not grow with the file. It checks up to 1,000,000
records by default (`--max-records`, and `--max-seconds` if you want a time limit) and says whether
it read the whole file. On a laptop it takes about 30 seconds for a million CSV rows with six
columns, and about 20 seconds for half a million JSON Lines records. The file's format is detected
first: text with no table in it (a list of addresses, a web page, or a table too broken to
recognise) has no columns to check, so `quality` says so; it can still be mined.
If `mine` is interrupted, run the same command again: it resumes where it stopped. Run it again
after it finished and it says `already_complete` instead of doing the work twice. Changing any
setting (the pattern, `--unique`, ...) starts a new job.
## Use it from an AI assistant
`datamining-skill mcp` runs a [Model Context Protocol](https://modelcontextprotocol.io) server on
standard input/output, so an assistant can inspect and mine local files on your behalf.
| Tool | What it does |
| --- | --- |
| `profile_dataset` | Format, encoding, columns or keys, size and estimated record count of a CSV, TSV, JSON, JSONL, log or plain text file; also gzip, bzip2, xz and zip files. Read-only. |
| `check_quality` | Reads a CSV, TSV or JSON Lines file once and reports malformed records (by kind, with line numbers), empty or missing values per field, the value types found and schema drift, plus an approximate distinct count per field and, for numeric fields, the mean, spread and quartiles. Counts, line numbers and aggregates only, never values. Stops after `max_records` records or `max_seconds` seconds (default 20), whichever comes first, so a client that gives up after a minute still gets an answer. Read-only. |
| `mine_dataset` | Extracts data (e-mail addresses by default) into a `.csv`, `.jsonl` or `.ndjson` result in memory-bounded chunks with checkpoints. `unique`, `lowercase`, `csv_formula_guard` and `overwrite` options. With `wait=false` it runs in the background and returns a job id. |
| `mining_status` | Progress and result of a background run, or a list of the runs of this session. |
| `cancel_mining` | Stops a background run after its current chunk; calling `mine_dataset` again resumes it. |
| `preview_result` | The first records of a result file, or with `sample` a uniform sample from the whole file (repeatable with `seed`), so the assistant can verify the run. |
Every argument is described in [docs/mcp-tools.md](docs/mcp-tools.md).
The assistant can only reach the directories you list. Every file argument is resolved (symlinks
and `..` included) and must lie inside a directory given with `--allow-dir`; anything else is
refused. The first `--allow-dir` is the workspace for relative paths.
### Claude Code
The repository is a Claude Code plugin: it starts the server for you through
[uv](https://docs.astral.sh/uv/) (install it first; without it, register the server yourself, below),
restricted to the project directory, and adds the
[`datamining` skill](skills/datamining/SKILL.md), which tells Claude when to use the tools and how
to choose their arguments.
```bash
claude plugin marketplace add coreworkstr-art/datamining-skill
claude plugin install datamining-skill@datamining-skill
```
Then ask, for example: *"Profile `events.csv`, then give me a clean list of the distinct e-mail
addresses in it, lower-cased."* Run `/mcp` to see that the server is connected.
Prefer to register the server yourself? After installing the command (see [Install](#install)):
```bash
claude mcp add --scope project datamining-skill -- \
datamining-skill mcp --allow-dir /absolute/path/to/your/data
```
or add the equivalent entry to `.mcp.json` at the project root:
```json
{
"mcpServers": {
"datamining-skill": {
"command": "datamining-skill",
"args": ["mcp", "--allow-dir", "/absolute/path/to/your/data"]
}
}
}
```
### Claude Desktop
Edit `claude_desktop_config.json` (macOS: `~/Library/Application Support/Claude/`, Windows:
`%APPDATA%\Claude\`) and restart the app:
```json
{
"mcpServers": {
"datamining-skill": {
"command": "datamining-skill",
"args": ["mcp", "--allow-dir", "/absolute/path/to/your/data"]
}
}
}
```
### Cursor
Create `.cursor/mcp.json` in your project (or `~/.cursor/mcp.json` for all projects):
```json
{
"mcpServers": {
"datamining-skill": {
"command": "datamining-skill",
"args": ["mcp", "--allow-dir", "${workspaceFolder}"]
}
}
}
```
### Configuration notes
- If the client can't find the command, give it the full path. This happens when you installed into
a virtual environment. Point at the console script (`/path/to/.venv/bin/datamining-skill`, or
`...\.venv\Scripts\datamining-skill.exe` on Windows) or at the interpreter:
`"command": "/path/to/.venv/bin/python", "args": ["-m", "datamining_skill", "mcp", "--allow-dir", "..."]`.
- In a JSON file, Windows paths need doubled backslashes (`"C:\\Users\\you\\data"`). Forward slashes
work too.
- If Windows says an Application Control policy blocked the program (`os error 4551`), start the
server through Python instead of the command: `"command": "python", "args": ["-m",
"datamining_skill", "mcp", "--allow-dir", "..."]`. The plugin already does.
- You can repeat `--allow-dir`, or set `DATAMINING_SKILL_ALLOWED_DIRS` (paths separated by `;` on
Windows, `:` elsewhere). With neither, the server uses its working directory, and refuses to if
that is the filesystem root.
- Results, state and scratch files go to `<first allowed dir>/.scratch/` and to the result path you
name. Add `.scratch/` to your `.gitignore`.
- Custom patterns (the `pattern` and `fields` arguments) are off by default. A regular expression
written by a model can be made to backtrack until your machine stalls, so start the server with
`--allow-custom-patterns` only if you accept that. The built-in e-mail extraction is not
affected: it uses bounded patterns, and the test suite runs it against hostile lines a megabyte
long under a time limit.
- What the assistant sees is masked. In `preview_result` rows and in the column names from
`profile_dataset`, e-mail addresses, IPv4 and IPv6 addresses, card numbers, T.C. kimlik numbers,
IBANs and secrets (tokens, API keys) become tags such as `[EMAIL]`. It is a best-effort safety
net, not anonymisation (see [docs/mcp-tools.md](docs/mcp-tools.md#masking-and-the-audit-log)),
and `--unmasked-preview` turns it off.
- To keep a record of what the assistant did, pass `--audit-log /path/outside/your/data/audit.jsonl`.
It gets one line per tool call with the time, the tool, paths and counts, and never any contents.
- To test an installation without a client, run `python scripts/mcp_smoke_test.py`. It does a full
handshake, profile and mine against the installed command and exits non-zero on any problem.
The server is written against the standard library only and speaks both generations of MCP: the
stateful `initialize` handshake (revisions 2024-11-05 to 2025-11-25) and the stateless
2026-07-28 revision (`server/discover`, per-request metadata). Nothing but protocol messages is
ever written to stdout. Because tool arguments come from a language model they are treated as
untrusted; see [docs/privacy-and-security.md](docs/privacy-and-security.md) for the threat model.
## Command line
```text
datamining-skill profile PATH [--config FILE]
datamining-skill quality PATH [--max-records N] [--max-seconds N] [--workspace DIR] [--mask]
datamining-skill mine SOURCE OUTPUT [--workspace DIR] [--unique] [--lowercase]
[--pattern REGEX [--fields a,b]] [--overwrite]
[--csv-formula-guard] [--json-escapes {auto,on,off}]
[--max-expanded-gib GIB] [--no-progress]
datamining-skill preview PATH [--rows N] [--mask] [--sample [--seed N]]
datamining-skill clean [--workspace DIR] [--all] [--dry-run]
datamining-skill mcp [--allow-dir DIR ...] [--allow-custom-patterns]
[--unmasked-preview] [--audit-log FILE]
```
| Option | Meaning |
| --- | --- |
| `--unique` | Write each record once, across the whole file and across resumed runs. |
| `--lowercase` | Lower-case every value. With `--unique`, addresses differing only in case count once (the Turkish dotted capital I counts as a plain i). |
| `--pattern`, `--fields` | Extract your own regular expression; each capture group becomes a column named by `--fields`. Repeat a group with `(?:...)`, not `(...)`. |
| `--overwrite` | Discard earlier progress and replace an existing output file. |
| `--csv-formula-guard` | Prefix CSV cells that start with `= + - @` so a spreadsheet does not run them. |
| `--json-escapes` | Decode JSON string escapes such as `\u0040` before searching: `auto` (JSON and JSON Lines sources with the built-in extraction), `on` or `off`. |
| `--max-expanded-gib` | Refuse a compressed source that expands beyond this size (default 64). |
| `--no-progress` | Hide the progress line, which appears only when stderr is a terminal. |
| `--workspace` | Where the `.scratch/` folder with state and scratch files lives (default: the current directory). |
| `--rows` | How many records `preview` shows (1 to 200, default 20). |
| `--max-records` | `quality` stops after this many records (default 1,000,000) and says whether it read the whole file. |
| `--max-seconds` | `quality` also stops after this many seconds and says so (default: no limit; the MCP tool stops after 20). |
| `--sample` | `preview` draws its records from across the whole file (the first 64 MiB at most) instead of taking the first ones. |
| `--seed` | Makes `preview --sample` repeatable. |
| `--mask` | `preview` and `quality` replace e-mail, IP, card, T.C. kimlik and IBAN numbers and secrets by tags such as `[EMAIL]`. |
| `--unmasked-preview` | `mcp` returns preview rows and profile column names as they are. By default they are masked. |
| `--audit-log` | `mcp` appends one JSON line per tool call (never contents) to this file, which must lie outside the allowed directories. |
| `--all`, `--dry-run` | `clean` also removes unfinished jobs / only reports what it would remove. |
Every command also accepts `--log-level {DEBUG,INFO,WARNING,ERROR}` (structured JSON logs on
stderr, `WARNING` by default), `--no-logs` and `--debug`. Results go to stdout. A failure prints a
single `error:` line, never a Python traceback; add `--debug` to see the traceback as well.
| Exit code | Meaning |
| --- | --- |
| 0 | Success |
| 1 | Unexpected internal error (rerun with `--debug` for the traceback) |
| 2 | Unsupported or unrecognised format (empty, binary, an archive of several files, ...) |
| 3 | Unavailable source, invalid configuration, file-system error, low memory or disk, job already running, or another library error |
| 4 | `mine` finished but some chunks failed (the first reason is printed); run again to retry them |
| 130 | Interrupted (run the same command again to resume) |
| 141 | The reader of stdout closed the pipe (for example `\| head`) |
A fresh run refuses to overwrite an existing output file that it did not create. State and
scratch files live in `./.scratch/`; `datamining-skill clean` removes those of finished jobs and
never touches a result file or a running job.
## Recipes
```bash
# A clean list of the distinct addresses in a big export
datamining-skill mine export.csv addresses.csv --unique --lowercase
# Straight from a compressed log, as JSON Lines
datamining-skill mine app.log.gz addresses.jsonl --unique --lowercase
# An Excel "Unicode Text" export (UTF-16) works like any other file
datamining-skill mine customers.txt addresses.csv --unique --lowercase
# A JSON export, including addresses spelled with \u0040 escapes
datamining-skill mine users.json addresses.csv --unique --lowercase
# Your own pattern: IPv4 addresses (note the non-capturing group), one column
datamining-skill mine access.log hosts.csv --pattern "(?:\d{1,3}\.){3}\d{1,3}" --fields ip
# A pattern with capture groups: each group becomes a column
datamining-skill mine app.log errors.csv --pattern "(\d{4}-\d\d-\d\d) ERROR (\w+)" --fields date,code
# Opening the result in a spreadsheet? Neutralise formula injection
datamining-skill mine export.csv addresses.csv --csv-formula-guard
# Start over, ignoring earlier progress
datamining-skill mine export.csv addresses.csv --overwrite
```
Typical gotchas are collected in [docs/troubleshooting.md](docs/troubleshooting.md).
## Python API
```python
from datamining_skill import profile_source, run_mining
# Profile: size, encoding, format, structure, estimated record count (a plain dictionary)
profile = profile_source("events.jsonl")
print(profile["format"], profile["records"]["count"], profile["structure"]["fields"])
# Mine with managed state: resumes automatically if interrupted
summary = run_mining("events.csv", "emails.csv", unique=True, lowercase=True,
on_progress=lambda p: print(p))
print(summary.to_dict())
```
Each stage is also available on its own:
```python
from datamining_skill import StateManager, create_chunking_engine, create_orchestrator, create_profiler
profile = create_profiler().profile("events.csv")
# 1. Plan: memory-aware, newline-aligned byte ranges (a lazy generator; nothing is read)
for chunk in create_chunking_engine().plan("events.csv", profile):
print(chunk.chunk_id, chunk.start_byte, chunk.end_byte)
# 2. Mine with your own state database
with StateManager("data/run-001.sqlite3") as state:
orchestrator = create_orchestrator(output_path="data/results.csv", state=state)
summary = orchestrator.run("events.csv") # safe to re-run after a crash
```
### Write your own mining logic
An `ExtractionStrategy` turns one line into zero or more records. It performs no I/O; reading,
writing and checkpointing are handled for you.
```python
from collections.abc import Iterator, Sequence
from datamining_skill import StateManager, create_orchestrator
class StatusCodes:
fields = ("status",)
def extract(self, line: str) -> Iterator[Sequence[str]]:
if " 500 " in line:
yield ("server-error",)
with StateManager("data/run-002.sqlite3") as state:
create_orchestrator(
output_path="data/errors.jsonl", state=state, strategy=StatusCodes()
).run("access.log")
```
A strategy may also take `start` and `stop` (`def extract(self, line, start=0, stop=None)`) and
report only the matches that begin in `[start, stop)`. It is then searched exactly even in lines
longer than 1 MiB, which arrive in overlapping windows; one that takes only the line sees each
window's own span, so a match that straddles two windows can be missed.
## How it works
```mermaid
flowchart LR
F[("Huge file<br/>or .gz / .zip / UTF-16")] -->|"converted to a private<br/>UTF-8 copy if needed"| P["Data Profiler<br/>size, encoding, format,<br/>structure, record estimate"]
P --> C["Chunking Engine<br/>newline-aligned byte ranges<br/>sized from free RAM"]
C --> S[("State Manager<br/>SQLite, WAL")]
S -->|"claim next PENDING chunk"| W["Miner Worker<br/>read blocks of lines,<br/>apply strategy, drop repeats"]
W --> T["chunk_N.tmp"]
T --> A["Result Aggregator<br/>truncate to last commit,<br/>append, fsync"]
A --> O[("results.csv / .jsonl")]
A -->|"COMPLETED + output length<br/>in one transaction"| S
```
| Stage | What it guarantees |
| --- | --- |
| **Data Profiler** | Reports size, encoding, format and structure from bounded samples; the record count is extrapolated from windows spread across the file. Memory is O(1). |
| **Chunking Engine** | Chunk size is `min(15% of the RAM free now, 512 MiB)`. Boundaries are found by seeking, never by loading data, and always fall right after a newline. Refuses to run if free RAM is critically low. |
| **State Manager** | Every chunk is PENDING, IN_PROGRESS, COMPLETED or FAILED in a SQLite database (WAL mode). Chunks left IN_PROGRESS by a crash return to PENDING automatically. It also keeps the 16-byte keyed hashes behind `--unique`. |
| **Miner Worker** | Reads exactly `[start_byte, end_byte)` in constant memory and writes records to a private scratch file. A line longer than 1 MiB is searched in overlapping windows, never skipped or cut. |
| **Result Aggregator** | Appends a finished chunk to the output idempotently: the output's committed length is recorded in the same transaction as `COMPLETED`, and every append first truncates back to it. A crash at any point is repaired on restart. |
The code follows Clean Architecture; dependencies point inward only:
```mermaid
flowchart TB
CLI["cli.py and bootstrap.py<br/>composition root and delivery"] --> I
I["infrastructure (adapters)<br/>file streaming, SQLite state, source conversion,<br/>scratch and output stores, MCP server, JSON logging"] --> A
A["application (use cases)<br/>profiler, chunking engine, worker and strategies,<br/>aggregator, orchestrator"] --> D
D["domain<br/>models, ports (Protocols), exceptions"]
```
Design notes, diagrams and the crash-recovery analysis are in
[docs/architecture.md](docs/architecture.md).
## Performance
Measured on a Windows laptop with Python 3.14; speeds vary a lot between machines and moments,
so treat them as orders of magnitude.
| Operation | Result |
| --- | --- |
| Profile a 10 GiB CSV | about 50 ms, under 1 MiB of memory growth |
| Plan chunks for 2 GiB of free RAM (34 chunks of 307 MiB) | about 26 ms |
| Mine a 512 MiB log with an address every few kilobytes | 2 to 3 seconds (200+ MiB/s) |
| Mine a 517 MiB CSV with an address on every row | roughly 20 to 40 seconds (15 to 25 MiB/s) |
The built-in extraction looks for each `@` with a plain string search and runs the pattern only
around it, so sparse text is read at close to disk speed; text with an address on every line is
bound by the interpreter. Mining runs on one thread by default ([Limitations](#limitations));
[docs/troubleshooting.md](docs/troubleshooting.md#speed) says what helps.
## Guarantees
- The runtime imports no networking modules and has no dependencies.
- Source files are only read. They are never modified, indexed or kept. A compressed or UTF-16
source is converted to a private temporary copy that is deleted when the job finishes.
- Besides the result file you name, only a small state database and short-lived scratch files are
written. They stay in `.scratch/` or `data/`, never in the system temporary directory, and are
readable by the owner only (`0600`/`0700` on POSIX, a protected ACL on Windows).
- Logs and error messages carry names, sizes, ids and counts, not file contents.
- Input is parsed and never evaluated: there is no `eval`, `pickle` or dynamic import on the data
path, and JSON, CSV and regular-expression handling use bounded standard-library tools.
- Each record is written exactly once. This was tested by killing real processes at three
different points while mining a 50 MiB file; the output matched the planted records exactly, in
order, every time. A power cut is the one case that can repeat work, never corrupt it (see
[Limitations](#limitations)).
## Limitations
- Binary formats (Parquet, SQLite, PDF) and compressed formats other than gzip, bzip2, xz and
single-file zip (zstandard, 7-zip) are rejected with a clear error; convert them first.
- The built-in extraction finds `local@domain.tld` addresses. It does not match quoted local parts,
the rarer local-part characters (`!#$&'*/=?^{|}~`) or hosts without a dot.
- A record is a physical line. A quoted CSV value that spans lines is searched line by line, and
a record count of such a file is given in lines. There is no column targeting: to work on one
column, write a pattern for that column's values.
- Mining runs on one thread, about 15 to 25 MiB/s when every row has a match. A library caller can
set `OrchestratorConfig(max_workers=...)` to mine chunks on several threads, but the work is
CPU-bound Python under the interpreter lock, so the gain measured was small (at most about 1.2
times). Unique mining, the command line and the MCP tools always mine one chunk at a time.
- Mining supports ASCII-compatible encodings (ASCII, UTF-8, cp1252, cp1254 for Turkish,
ISO-8859-1) directly, and UTF-16/32 through the conversion copy, which needs temporary disk space
for the converted text. A file that is not UTF-8 has its encoding guessed (the profile says how
sure it is); addresses made of plain ASCII letters come out right whatever the guess.
- Text in which no table, log or JSON layout fits (a list of addresses, a web page, an e-mail
message, a log with a layout of its own) is read as plain lines (`format` is `text`): it has no
columns, so `quality` cannot check it, but it can be profiled and mined.
- A table's first row is taken for a header when it holds short names that start with a letter and
contain no `@`. A list whose first row holds an address keeps it as data; a headerless table whose
first row is plain words (`Ada,Istanbul`) loses that row, so give such a file a header row.
- The CLI, `run_mining` and the MCP tools hold a cross-process lock per job, so a second process
on the same job is refused ("already running"). Code that drives `create_orchestrator` directly
must provide its own single-writer guarantee.
- If the source is truncated or replaced while it is being mined, the affected chunks fail with an
explicit reason instead of silently dropping records. A file replaced by another of the same
size cannot be detected.
- A failed chunk that succeeds on a retry is appended after the others, so output order can then
differ from file order (no record is lost or duplicated). With `--unique`, "first occurrence" then
means first in processing order.
- A match longer than 4 KiB that lies inside a line longer than 1 MiB can be cut in two.
- The MCP server handles one request at a time; use `wait=false` for long runs. It cannot interrupt
a running call, but you can stop the process safely at any moment and resume later.
- A power cut can lose the most recent checkpoints, though it never corrupts data, so a few chunks
may be mined again. That is harmless because extraction is idempotent.
## Troubleshooting
[docs/troubleshooting.md](docs/troubleshooting.md) lists the messages you may meet, what they mean
and what to do, and explains speed, disk use and clean-up. Open an issue with the output of
`datamining-skill --version` and a small synthetic file that reproduces the problem.
## Development
```bash
git clone https://github.com/coreworkstr-art/datamining-skill && cd datamining-skill
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
python -m ruff check . # lint
python -m mypy # strict type checking
python -m pytest # full suite: crash recovery, memory bounds, path security, MCP protocol
```
GitHub Actions runs the lint, the type check and the tests on Linux, macOS and Windows with
Python 3.11 to 3.14, then builds the package and smoke-tests the installed wheel in a clean
environment. Validation scripts you can run yourself:
```bash
python scripts/generate_dummy_csv.py --size-gb 2 --output data/dummy_2gb.csv # memory check
python scripts/simulate_crash_resume.py # 100 chunks, killed at 50, resumed
python scripts/simulate_mining_crash.py # 50 MiB mined, killed at chunk 3, three crash points
python scripts/mcp_smoke_test.py # MCP handshake and tools against the installed command
```
Project layout:
```text
src/datamining_skill/
domain/ value objects, exceptions, ports (no dependencies)
application/ profiler, chunking engine, worker, strategies, aggregator, orchestrator
infrastructure/ streaming, encoding, source conversion, SQLite state, scratch/output stores,
background jobs, MCP server, logging
bootstrap.py composition root (run_mining, profile_source, create_mcp_server, ...)
cli.py command line: profile, quality, mine, preview, clean, mcp
.claude-plugin/ Claude Code plugin and marketplace manifests
skills/datamining/ the skill that teaches an assistant to use the tools
tests/ pytest suite scripts/ validation and release scripts
docs/ architecture, tool reference, security and troubleshooting notes
examples/ a small sample file to try the commands on
```
## Contributing
You are welcome to contribute under any name or pseudonym. [CONTRIBUTING.md](CONTRIBUTING.md) says how
and the [Code of Conduct](CODE_OF_CONDUCT.md) says how we treat each other. Report security problems
privately, as [SECURITY.md](SECURITY.md) describes. What changed in each version is in
[CHANGELOG.md](CHANGELOG.md).
## Using it responsibly
E-mail addresses are personal data under the GDPR in the EU, the KVKK in Turkey and comparable
laws elsewhere. The program extracts whatever you point it at and cannot tell whether you may.
Make sure you have a lawful basis for processing the source data and for what you do with the
result, and keep result files as well protected as the source. Extracting addresses to send
unsolicited mail is a misuse of this tool.
DataMining Skill is an independent project. It is not affiliated with, endorsed or sponsored by
Anthropic, Cursor or the Model Context Protocol project. Claude, Claude Code, Claude Desktop and
Cursor are names of their owners, used here only to say which clients it works with.
## License
MIT. See [LICENSE](LICENSE). Copyright (c) 2026 DataMining Skill Contributors. CoreWorks Koray Uğrik
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues