DataMining Skill

DataMining Skill pulls data out of very large text files (CSV, TSV, JSON, JSON Lines, logs and plain text) on an ordinary computer. It is a command-line tool, a Python library and an MCP server in one.

The file is read in pieces, so memory use follows the settings and not the size of the file. Progress is saved as it goes: if a run is interrupted, the same command carries on from the last checkpoint.

It was written for one job: getting a clean list of e-mail addresses out of a big export or log. You can also give it a regular expression of your own. Use it from a terminal, call it from Python, or point an assistant that speaks the Model Context Protocol (Claude Code, Claude Desktop, Cursor and others) at the MCP server and the skill that come with it.

A few things worth knowing before you start:

• Everything stays on your machine. There is no network code, no telemetry and no third-party package, and your data is never sent anywhere.

• Memory use is set by the configuration, not by the file. The tests check it, and the Performance figures include a 10 GiB file.

• If a run is killed or the power goes, running the same command again gives the same records an uninterrupted run would have. Limitations has the details, such as the order of records after a retry.

• gzip, bzip2, xz and single-file zip archives are read directly, and so are Excel's "Unicode Text" (UTF-16) files. One command turns a messy export into a list of distinct, lower-case addresses.

• For assistants there are six MCP tools, a Claude Code plugin and a skill that says when to use them. File access is limited to the directories you list.

Contents

Install | Quick start | Use it from an AI assistant | Command line | Recipes | Python API | How it works | Performance | Guarantees | Limitations | Troubleshooting | Development | Contributing

Related MCP server: mix_server

Install

Python 3.11 or newer; nothing else is needed at runtime.

# with uv: the command gets its own isolated environment
uv tool install git+https://github.com/coreworkstr-art/datamining-skill

# or with pip
pip install git+https://github.com/coreworkstr-art/datamining-skill

# or from a clone, which also gives you the tests and the example data
git clone https://github.com/coreworkstr-art/datamining-skill && cd datamining-skill
pip install .

Check it works: datamining-skill --version. If the command is not found, see Troubleshooting or run it as python -m datamining_skill.

The examples below use a small sample file, examples/contacts.csv. A clone has it; otherwise download it or use any CSV, JSON, log or text file of your own.

Quick start

Inspect a file without reading it into memory, extract a clean address list, check the result:

# What is in this file? (milliseconds, constant memory)
datamining-skill profile examples/contacts.csv

# Is the file in good shape? Malformed rows, empty values, type changes (counts and line numbers only)
datamining-skill quality examples/contacts.csv

# Every distinct e-mail address, lower-cased, one per line (resumable, crash-safe)
datamining-skill mine examples/contacts.csv emails.csv --unique --lowercase

# Look at the first records of the result
datamining-skill preview emails.csv --rows 5

profile prints a JSON description: size, encoding, format, columns or keys, and the record count with its unit (a quoted CSV value can span lines, so a CSV is counted in lines). mine writes emails.csv and prints a JSON summary:

{
  "output_name": "emails.csv", "resumed": false, "chunks_total": 1, "chunks_processed": 1,
  "chunks_failed": 0, "records_written": 15, "duplicates_skipped": 3, "already_complete": false,
  "oversized_lines": 0, "source_transform": null, "first_error": null
}

quality reads a CSV, TSV or JSON Lines file once and reports what is wrong with it: records that are malformed (a row with too few or too many columns, a broken quote, a line that is not JSON) with the first line numbers, how many values of each field are empty or missing, the types found in each field, and fields whose values change type (a column of numbers with a few words in it). Each field also gets an approximate count of its distinct values and, if it is numeric with at least ten values, its mean, spread and estimated quartiles. It keeps counts, line numbers and small summaries of fixed size, never values, so its memory does not grow with the file. It checks up to 1,000,000 records by default (--max-records, and --max-seconds if you want a time limit) and says whether it read the whole file. On a laptop it takes about 30 seconds for a million CSV rows with six columns, and about 20 seconds for half a million JSON Lines records. The file's format is detected first: text with no table in it (a list of addresses, a web page, or a table too broken to recognise) has no columns to check, so quality says so; it can still be mined.

If mine is interrupted, run the same command again: it resumes where it stopped. Run it again after it finished and it says already_complete instead of doing the work twice. Changing any setting (the pattern, --unique, ...) starts a new job.

Use it from an AI assistant

datamining-skill mcp runs a Model Context Protocol server on standard input/output, so an assistant can inspect and mine local files on your behalf.

Every argument is described in docs/mcp-tools.md.

The assistant can only reach the directories you list. Every file argument is resolved (symlinks and .. included) and must lie inside a directory given with --allow-dir; anything else is refused. The first --allow-dir is the workspace for relative paths.

Claude Code

The repository is a Claude Code plugin: it starts the server for you through uv (install it first; without it, register the server yourself, below), restricted to the project directory, and adds the datamining skill, which tells Claude when to use the tools and how to choose their arguments.

claude plugin marketplace add coreworkstr-art/datamining-skill
claude plugin install datamining-skill@datamining-skill

Then ask, for example: "Profile events.csv, then give me a clean list of the distinct e-mail addresses in it, lower-cased." Run /mcp to see that the server is connected.

Prefer to register the server yourself? After installing the command (see Install):

claude mcp add --scope project datamining-skill -- \
  datamining-skill mcp --allow-dir /absolute/path/to/your/data

or add the equivalent entry to .mcp.json at the project root:

{
  "mcpServers": {
    "datamining-skill": {
      "command": "datamining-skill",
      "args": ["mcp", "--allow-dir", "/absolute/path/to/your/data"]
    }
  }
}

Claude Desktop

Edit claude_desktop_config.json (macOS: ~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\) and restart the app:

{
  "mcpServers": {
    "datamining-skill": {
      "command": "datamining-skill",
      "args": ["mcp", "--allow-dir", "/absolute/path/to/your/data"]
    }
  }
}

Cursor

Create .cursor/mcp.json in your project (or ~/.cursor/mcp.json for all projects):

{
  "mcpServers": {
    "datamining-skill": {
      "command": "datamining-skill",
      "args": ["mcp", "--allow-dir", "${workspaceFolder}"]
    }
  }
}

Configuration notes

• If the client can't find the command, give it the full path. This happens when you installed into a virtual environment. Point at the console script (/path/to/.venv/bin/datamining-skill, or ...\.venv\Scripts\datamining-skill.exe on Windows) or at the interpreter: "command": "/path/to/.venv/bin/python", "args": ["-m", "datamining_skill", "mcp", "--allow-dir", "..."].

• In a JSON file, Windows paths need doubled backslashes ("C:\\Users\\you\\data"). Forward slashes work too.

• If Windows says an Application Control policy blocked the program (os error 4551), start the server through Python instead of the command: "command": "python", "args": ["-m", "datamining_skill", "mcp", "--allow-dir", "..."]. The plugin already does.

• You can repeat --allow-dir, or set DATAMINING_SKILL_ALLOWED_DIRS (paths separated by ; on Windows, : elsewhere). With neither, the server uses its working directory, and refuses to if that is the filesystem root.

• Results, state and scratch files go to <first allowed dir>/.scratch/ and to the result path you name. Add .scratch/ to your .gitignore.

• Custom patterns (the pattern and fields arguments) are off by default. A regular expression written by a model can be made to backtrack until your machine stalls, so start the server with --allow-custom-patterns only if you accept that. The built-in e-mail extraction is not affected: it uses bounded patterns, and the test suite runs it against hostile lines a megabyte long under a time limit.

• What the assistant sees is masked. In preview_result rows and in the column names from profile_dataset, e-mail addresses, IPv4 and IPv6 addresses, card numbers, T.C. kimlik numbers, IBANs and secrets (tokens, API keys) become tags such as [EMAIL]. It is a best-effort safety net, not anonymisation (see docs/mcp-tools.md), and --unmasked-preview turns it off.

• To keep a record of what the assistant did, pass --audit-log /path/outside/your/data/audit.jsonl. It gets one line per tool call with the time, the tool, paths and counts, and never any contents.

• To test an installation without a client, run python scripts/mcp_smoke_test.py. It does a full handshake, profile and mine against the installed command and exits non-zero on any problem.

The server is written against the standard library only and speaks both generations of MCP: the stateful initialize handshake (revisions 2024-11-05 to 2025-11-25) and the stateless 2026-07-28 revision (server/discover, per-request metadata). Nothing but protocol messages is ever written to stdout. Because tool arguments come from a language model they are treated as untrusted; see docs/privacy-and-security.md for the threat model.

Command line

datamining-skill profile PATH [--config FILE]
datamining-skill quality PATH [--max-records N] [--max-seconds N] [--workspace DIR] [--mask]
datamining-skill mine SOURCE OUTPUT [--workspace DIR] [--unique] [--lowercase]
                                    [--pattern REGEX [--fields a,b]] [--overwrite]
                                    [--csv-formula-guard] [--json-escapes {auto,on,off}]
                                    [--max-expanded-gib GIB] [--no-progress]
datamining-skill preview PATH [--rows N] [--mask] [--sample [--seed N]]
datamining-skill clean [--workspace DIR] [--all] [--dry-run]
datamining-skill mcp [--allow-dir DIR ...] [--allow-custom-patterns]
                     [--unmasked-preview] [--audit-log FILE]

Every command also accepts --log-level {DEBUG,INFO,WARNING,ERROR} (structured JSON logs on stderr, WARNING by default), --no-logs and --debug. Results go to stdout. A failure prints a single error: line, never a Python traceback; add --debug to see the traceback as well.

A fresh run refuses to overwrite an existing output file that it did not create. State and scratch files live in ./.scratch/; datamining-skill clean removes those of finished jobs and never touches a result file or a running job.

Recipes

# A clean list of the distinct addresses in a big export
datamining-skill mine export.csv addresses.csv --unique --lowercase

# Straight from a compressed log, as JSON Lines
datamining-skill mine app.log.gz addresses.jsonl --unique --lowercase

# An Excel "Unicode Text" export (UTF-16) works like any other file
datamining-skill mine customers.txt addresses.csv --unique --lowercase

# A JSON export, including addresses spelled with \u0040 escapes
datamining-skill mine users.json addresses.csv --unique --lowercase

# Your own pattern: IPv4 addresses (note the non-capturing group), one column
datamining-skill mine access.log hosts.csv --pattern "(?:\d{1,3}\.){3}\d{1,3}" --fields ip

# A pattern with capture groups: each group becomes a column
datamining-skill mine app.log errors.csv --pattern "(\d{4}-\d\d-\d\d) ERROR (\w+)" --fields date,code

# Opening the result in a spreadsheet? Neutralise formula injection
datamining-skill mine export.csv addresses.csv --csv-formula-guard

# Start over, ignoring earlier progress
datamining-skill mine export.csv addresses.csv --overwrite

Typical gotchas are collected in docs/troubleshooting.md.

Python API

from datamining_skill import profile_source, run_mining

# Profile: size, encoding, format, structure, estimated record count (a plain dictionary)
profile = profile_source("events.jsonl")
print(profile["format"], profile["records"]["count"], profile["structure"]["fields"])

# Mine with managed state: resumes automatically if interrupted
summary = run_mining("events.csv", "emails.csv", unique=True, lowercase=True,
                     on_progress=lambda p: print(p))
print(summary.to_dict())

Each stage is also available on its own:

from datamining_skill import StateManager, create_chunking_engine, create_orchestrator, create_profiler

profile = create_profiler().profile("events.csv")

# 1. Plan: memory-aware, newline-aligned byte ranges (a lazy generator; nothing is read)
for chunk in create_chunking_engine().plan("events.csv", profile):
    print(chunk.chunk_id, chunk.start_byte, chunk.end_byte)

# 2. Mine with your own state database
with StateManager("data/run-001.sqlite3") as state:
    orchestrator = create_orchestrator(output_path="data/results.csv", state=state)
    summary = orchestrator.run("events.csv")      # safe to re-run after a crash

Write your own mining logic

An ExtractionStrategy turns one line into zero or more records. It performs no I/O; reading, writing and checkpointing are handled for you.

from collections.abc import Iterator, Sequence
from datamining_skill import StateManager, create_orchestrator

class StatusCodes:
    fields = ("status",)

    def extract(self, line: str) -> Iterator[Sequence[str]]:
        if " 500 " in line:
            yield ("server-error",)

with StateManager("data/run-002.sqlite3") as state:
    create_orchestrator(
        output_path="data/errors.jsonl", state=state, strategy=StatusCodes()
    ).run("access.log")

A strategy may also take start and stop (def extract(self, line, start=0, stop=None)) and report only the matches that begin in [start, stop). It is then searched exactly even in lines longer than 1 MiB, which arrive in overlapping windows; one that takes only the line sees each window's own span, so a match that straddles two windows can be missed.

How it works

flowchart LR
    F[("Huge file<br/>or .gz / .zip / UTF-16")] -->|"converted to a private<br/>UTF-8 copy if needed"| P["Data Profiler<br/>size, encoding, format,<br/>structure, record estimate"]
    P --> C["Chunking Engine<br/>newline-aligned byte ranges<br/>sized from free RAM"]
    C --> S[("State Manager<br/>SQLite, WAL")]
    S -->|"claim next PENDING chunk"| W["Miner Worker<br/>read blocks of lines,<br/>apply strategy, drop repeats"]
    W --> T["chunk_N.tmp"]
    T --> A["Result Aggregator<br/>truncate to last commit,<br/>append, fsync"]
    A --> O[("results.csv / .jsonl")]
    A -->|"COMPLETED + output length<br/>in one transaction"| S

The code follows Clean Architecture; dependencies point inward only:

flowchart TB
    CLI["cli.py and bootstrap.py<br/>composition root and delivery"] --> I
    I["infrastructure (adapters)<br/>file streaming, SQLite state, source conversion,<br/>scratch and output stores, MCP server, JSON logging"] --> A
    A["application (use cases)<br/>profiler, chunking engine, worker and strategies,<br/>aggregator, orchestrator"] --> D
    D["domain<br/>models, ports (Protocols), exceptions"]

Design notes, diagrams and the crash-recovery analysis are in docs/architecture.md.

Performance

Measured on a Windows laptop with Python 3.14; speeds vary a lot between machines and moments, so treat them as orders of magnitude.

The built-in extraction looks for each @ with a plain string search and runs the pattern only around it, so sparse text is read at close to disk speed; text with an address on every line is bound by the interpreter. Mining runs on one thread by default (Limitations); docs/troubleshooting.md says what helps.

Guarantees

• The runtime imports no networking modules and has no dependencies.

• Source files are only read. They are never modified, indexed or kept. A compressed or UTF-16 source is converted to a private temporary copy that is deleted when the job finishes.

• Besides the result file you name, only a small state database and short-lived scratch files are written. They stay in .scratch/ or data/, never in the system temporary directory, and are readable by the owner only (0600/0700 on POSIX, a protected ACL on Windows).

• Logs and error messages carry names, sizes, ids and counts, not file contents.

• Input is parsed and never evaluated: there is no eval, pickle or dynamic import on the data path, and JSON, CSV and regular-expression handling use bounded standard-library tools.

• Each record is written exactly once. This was tested by killing real processes at three different points while mining a 50 MiB file; the output matched the planted records exactly, in order, every time. A power cut is the one case that can repeat work, never corrupt it (see Limitations).

Limitations

• Binary formats (Parquet, SQLite, PDF) and compressed formats other than gzip, bzip2, xz and single-file zip (zstandard, 7-zip) are rejected with a clear error; convert them first.

• The built-in extraction finds local@domain.tld addresses. It does not match quoted local parts, the rarer local-part characters (!#$&'*/=?^{|}~) or hosts without a dot.

• A record is a physical line. A quoted CSV value that spans lines is searched line by line, and a record count of such a file is given in lines. There is no column targeting: to work on one column, write a pattern for that column's values.

• Mining runs on one thread, about 15 to 25 MiB/s when every row has a match. A library caller can set OrchestratorConfig(max_workers=...) to mine chunks on several threads, but the work is CPU-bound Python under the interpreter lock, so the gain measured was small (at most about 1.2 times). Unique mining, the command line and the MCP tools always mine one chunk at a time.

• Mining supports ASCII-compatible encodings (ASCII, UTF-8, cp1252, cp1254 for Turkish, ISO-8859-1) directly, and UTF-16/32 through the conversion copy, which needs temporary disk space for the converted text. A file that is not UTF-8 has its encoding guessed (the profile says how sure it is); addresses made of plain ASCII letters come out right whatever the guess.

• Text in which no table, log or JSON layout fits (a list of addresses, a web page, an e-mail message, a log with a layout of its own) is read as plain lines (format is text): it has no columns, so quality cannot check it, but it can be profiled and mined.

• A table's first row is taken for a header when it holds short names that start with a letter and contain no @. A list whose first row holds an address keeps it as data; a headerless table whose first row is plain words (Ada,Istanbul) loses that row, so give such a file a header row.

• The CLI, run_mining and the MCP tools hold a cross-process lock per job, so a second process on the same job is refused ("already running"). Code that drives create_orchestrator directly must provide its own single-writer guarantee.

• If the source is truncated or replaced while it is being mined, the affected chunks fail with an explicit reason instead of silently dropping records. A file replaced by another of the same size cannot be detected.

• A failed chunk that succeeds on a retry is appended after the others, so output order can then differ from file order (no record is lost or duplicated). With --unique, "first occurrence" then means first in processing order.

• A match longer than 4 KiB that lies inside a line longer than 1 MiB can be cut in two.

• The MCP server handles one request at a time; use wait=false for long runs. It cannot interrupt a running call, but you can stop the process safely at any moment and resume later.

• A power cut can lose the most recent checkpoints, though it never corrupts data, so a few chunks may be mined again. That is harmless because extraction is idempotent.

Troubleshooting

docs/troubleshooting.md lists the messages you may meet, what they mean and what to do, and explains speed, disk use and clean-up. Open an issue with the output of datamining-skill --version and a small synthetic file that reproduces the problem.

Development

git clone https://github.com/coreworkstr-art/datamining-skill && cd datamining-skill
python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

python -m ruff check .  # lint
python -m mypy          # strict type checking
python -m pytest        # full suite: crash recovery, memory bounds, path security, MCP protocol

GitHub Actions runs the lint, the type check and the tests on Linux, macOS and Windows with Python 3.11 to 3.14, then builds the package and smoke-tests the installed wheel in a clean environment. Validation scripts you can run yourself:

python scripts/generate_dummy_csv.py --size-gb 2 --output data/dummy_2gb.csv   # memory check
python scripts/simulate_crash_resume.py        # 100 chunks, killed at 50, resumed
python scripts/simulate_mining_crash.py        # 50 MiB mined, killed at chunk 3, three crash points
python scripts/mcp_smoke_test.py               # MCP handshake and tools against the installed command

Project layout:

src/datamining_skill/
  domain/           value objects, exceptions, ports (no dependencies)
  application/      profiler, chunking engine, worker, strategies, aggregator, orchestrator
  infrastructure/   streaming, encoding, source conversion, SQLite state, scratch/output stores,
                    background jobs, MCP server, logging
  bootstrap.py      composition root (run_mining, profile_source, create_mcp_server, ...)
  cli.py            command line: profile, quality, mine, preview, clean, mcp
.claude-plugin/     Claude Code plugin and marketplace manifests
skills/datamining/  the skill that teaches an assistant to use the tools
tests/              pytest suite          scripts/    validation and release scripts
docs/               architecture, tool reference, security and troubleshooting notes
examples/           a small sample file to try the commands on

Contributing

You are welcome to contribute under any name or pseudonym. CONTRIBUTING.md says how and the Code of Conduct says how we treat each other. Report security problems privately, as SECURITY.md describes. What changed in each version is in CHANGELOG.md.

Using it responsibly

E-mail addresses are personal data under the GDPR in the EU, the KVKK in Turkey and comparable laws elsewhere. The program extracts whatever you point it at and cannot tell whether you may. Make sure you have a lawful basis for processing the source data and for what you do with the result, and keep result files as well protected as the source. Extracting addresses to send unsolicited mail is a misuse of this tool.

DataMining Skill is an independent project. It is not affiliated with, endorsed or sponsored by Anthropic, Cursor or the Model Context Protocol project. Claude, Claude Code, Claude Desktop and Cursor are names of their owners, used here only to say which clients it works with.

License

MIT. See LICENSE. Copyright (c) 2026 DataMining Skill Contributors. CoreWorks Koray Uğrik