datasentry
Summary: DataSentry is a data-quality MCP server that discovers, explains, repairs, and verifies data issues across files, databases, and cloud storage, with optional scheduled jobs and PII redaction management.
Scan data — run quality scans on CSV/Parquet/JSONL/XLSX/DuckDB files, PostgreSQL/MySQL tables via DSN, or s3:// / gs:// / az:// objects, with sampling methods (random, stratified, reservoir, time-based, rare oversampling, none), seeds, detector whitelists, tags, and optional dataset IDs.
Inspect findings — list issues filtered by severity or scan run, and retrieve six-dimension quality scores (completeness, validity, uniqueness, consistency, integrity, timeliness) where dimensions under 60 signal risk.
Analyze history — compare drift between two scans or the two most recent scans (schema, row count, score, issue distribution), view per-dataset quality trends, fetch scan-time column profiles, and build same-dataset historical run comparisons.
Export reports — retrieve the canonical JSON report of any scan run.
Discover detectors — list all 39 registered detectors with IDs, dimensions, capabilities, and default thresholds.
Repair safely — generate rule-based repair proposals (read-only), preview or apply repairs that write a repaired copy plus before-snapshot (never overwriting the source), verify by re-scanning the repaired copy (fixed/persistent/new issues), and roll back applied repairs from snapshots.
Validate contracts — check a data contract YAML file and return the structured contract or validation errors.
Schedule jobs — create, list, trigger, update, and remove cron-scheduled scans with quality gates (0–100), webhook URLs, and retry settings.
Manage PII — list encryption sessions, restore plaintext from redacted placeholders, delete sessions, rotate the vault encryption key, and purge sessions older than N days.
Safety controls — mutating operations require explicit
confirm: true, and tools carry read-only/destructive annotations.
Enables adding DataSentry data-quality gates to dbt pipelines, with integration examples provided.
Provides a reusable GitHub Actions workflow to run DataSentry scans as CI quality gates, failing on configured severity levels and uploading HTML reports.
Allows scanning MySQL databases for data-quality issues, including evidence-backed issue detection, quality scoring, and safe repair workflows.
Provides optional AI-assisted repair proposals using local Ollama LLMs, with human approval required for state-changing repairs.
Provides optional AI-assisted repair proposals using OpenAI models, with human approval required for state-changing repairs.
Allows scanning PostgreSQL databases for data-quality issues, including evidence-backed issue detection, quality scoring, and safe repair workflows.
Allows scanning SQLite databases for data-quality issues, including evidence-backed issue detection, quality scoring, and safe repair workflows.
中文导读:DataSentry 会先自动发现数据质量问题,再给出样本、比例、置信度等证据。修复采用
propose → preview → apply to a copy → verify → rollback的保守流程。检测与评分不依赖 LLM,AI 只作为可选辅助,数据可以完全留在本机。
Try it in 30 seconds
Install the current PyPI release and run the zero-config product tour:
pip install --upgrade datasentry-ai
datasentry demodatasentry-demo is also available as a direct console alias.
The demo generates synthetic dirty data, runs the built-in detectors, exports JSON + HTML reports, applies one safe repair to a copy, re-scans the repaired copy, and prints a rollback command. It needs no dataset, cloud service, API key, or LLM.
synthetic dirty CSV
↓
deterministic detectors
↓
evidence-backed issues + quality score
↓
preview → repaired copy
↓
re-scan → verify new/persistent issuesPrefer scanning your own data immediately?
curl -L https://raw.githubusercontent.com/Jackxiaozhiren/datasentry/main/demo-data/orders.csv -o orders.csv
datasentry scan orders.csv
datasentry issues list --severity highOr launch the interactive interfaces:
datasentry # terminal UI
datasentry-server # Web UI + REST API at http://localhost:8000/ui/Related MCP server: aegis-dq
Start with your problem
If you need to... | Start here |
Find suspicious data before you know every rule |
|
See the full discovery → repair → verify loop |
|
Block severe data issues in CI | |
Add quality gates to dbt / Airflow | |
Inspect and repair issues without overwriting the source | |
Give AI agents deterministic data-quality tools |
The default path is intentionally low-commitment: scan first, inspect evidence, then decide whether a finding deserves a permanent rule or a repair.
Why DataSentry exists
Most data-quality tools are excellent once you already know the expectations, checks, or contracts you want to enforce. Real incidents often start one step earlier: you do not yet know what is wrong.
DataSentry is built around the complete remediation loop:
Find → Explain → Fix safely → VerifyFind — discover common quality problems without writing every rule first.
Explain — attach samples, affected counts/ratios, detector evidence, and confidence.
Fix safely — preview changes and apply repairs to a copy instead of mutating the source.
Verify — re-scan the repaired copy and surface persistent or newly introduced issues.
DataSentry fits best when you are looking at data you do not fully trust yet, want evidence before codifying checks, or need a conservative remediation path. It complements expectation- and contract-driven tools rather than requiring you to replace them.
What it catches automatically
DataSentry ships with a registry of deterministic detectors covering common failure modes such as:
missing and placeholder values;
invalid emails, URLs, dates, and encodings;
duplicate identifiers and uniqueness violations;
inconsistent categories and cross-field contradictions;
foreign-key and referential-integrity problems;
numeric and statistical outliers;
schema, row-count, score, and issue-distribution drift.
Re-count them from the registry the installed build actually ships:
datasentry detectors | python3 -c "import json,sys; print(len(json.load(sys.stdin)['detectors']))"
# 39That number is the registry, not what any single file triggers: a detector whose required column types are absent is skipped, so the count that actually runs depends on the schema you scan.
Every scan produces an evidence-backed issue list and a six-dimension quality score across completeness, validity, uniqueness, consistency, integrity, and timeliness.
Safe repair, not blind mutation
# inspect the highest-severity findings
datasentry issues list --severity high
# propose a repair without changing data
datasentry repair propose <issue_id> --file orders.csv
# preview the exact effect
datasentry repair preview <issue_id> --file orders.csv
# apply to a repaired copy; the original is not overwritten
datasentry repair apply <issue_id> --file orders.csv
# re-scan the repaired copy and detect regressions
datasentry repair verify <run_id>
# inspect or undo the repair
datasentry repair diff <run_id>
datasentry repair rollback <run_id>Repairs are fingerprinted, auditable, and reversible. AI-generated repair proposals remain human-approved state changes.
Quality gates for CI
datasentry scan orders.csv --fail-on highFor GitHub repositories, reuse DataSentry's maintained workflow_call gate instead of duplicating installation and exit-code handling:
jobs:
datasentry:
uses: Jackxiaozhiren/datasentry/.github/workflows/datasentry-quality-gate.yml@main
with:
path: data/orders.csv
fail_on: highSee docs/GITHUB_ACTIONS.md for inputs, artifacts, security boundaries, and version-pinning guidance.
Reports can be exported as JSON, Markdown, HTML, JUnit, and SARIF. The versioned, machine-readable JSON report Schema is included for downstream validation. The GitHub Actions example fails the workflow on severe findings while still uploading an HTML report for review.
Give AI agents deterministic data-quality tools
DataSentry includes an MCP stdio server:
datasentry mcp --project /path/to/projectMCP-capable clients can scan files, inspect evidence-backed issues, read quality scores and trends, compare drift, validate contracts, manage scheduled jobs, and call DataSentry tools without bypassing the same underlying safety rules used by the CLI and REST API.
Copy-paste setup recipes for VS Code and Claude Desktop are in docs/MCP.md.
Hosts that declare MCP Apps support also get a read-only report card from report_export; see
docs/MCP_APPS.md for the negotiation contract and its verification limits.
Boundary: AI may propose; humans approve state-changing repairs.
Where it fits next to popular data-quality projects
This is a positioning guide, not a winner/loser feature scorecard. These projects solve overlapping but different jobs; check their upstream documentation for current capabilities.
Project | Core mental model | A strong fit when you want... |
DataSentry | discover → explain → repair → verify | automatic issue discovery plus a controlled, reversible remediation loop |
Expectations / expressive data tests | explicit validation rules, validation results, and generated data-quality documentation | |
data contracts and quality checks | YAML contracts and verification across a broad data stack | |
“unit tests for data” on Spark | large-scale data verification in Spark-centric environments | |
one-line profiling / EDA | fast exploratory profiling and shareable analysis reports |
DataSentry is intentionally not trying to replace a metadata catalog, lineage platform, or every validator. Its focus is narrower: find bad data, show why it was flagged, and close the repair loop without gambling on the source.
Local-first by design
deterministic detection and scoring run locally;
DuckDB powers core local execution;
OpenAI/Ollama assistance is optional;
PII redaction, encrypted mappings, and LLM audit records are available when AI is enabled;
the original source file is not overwritten by repair workflows.
Data sources
CSV, Parquet, JSONL, XLSX
DuckDB and SQLite
PostgreSQL and MySQL
s3://,gs://, andaz://objectssingle files, batches, and globs
History and drift
Persist scans and compare data over time:
datasentry drift latest orders
datasentry scoreTracked signals include schema changes, row-count movement, quality-score changes, and issue-distribution drift.
Architecture
flowchart LR
Sources[Files / DBs / cloud objects] --> DuckDB[Local execution]
DuckDB --> Detect[Detectors]
Detect --> Evidence[Evidence fusion]
Evidence --> Score[6-dimension score]
Score --> Reports[Reports / history / gates]
Reports --> CLI[CLI / TUI]
Reports --> Web[Web / REST]
Reports --> MCP[MCP]
Evidence --> Proposal[Repair proposal]
Proposal --> Preview[Preview]
Preview --> Apply[Apply to copy]
Apply --> Verify[Verify by re-scan]
Verify --> Rollback[Rollback artifact]
LLM[Optional OpenAI / Ollama] -. proposes .-> ProposalReproducible benchmark
uv sync
uv run python benchmarks/bench_scan.py 1000000 42The benchmark generates synthetic dirty data and measures profiling, detection/fusion/scoring, numeric-outlier detection, JSONL reading, sampling, score drift, and memory high-water marks. See docs/BENCHMARKS.md.
Development
uv sync
make check # lint + mypy --strict + tests/coverage
make demo # exercise the public datasentry demo path
make bench # benchmark
make build # distributionsContributing
Useful contributions include new detectors, connectors, integration examples, reproducible benchmark cases, documentation/translations, minimal bug reproductions, and CLI/TUI/Web usability improvements.
Want a small first contribution? Start with good first issue #14 or good first issue #15.
See CONTRIBUTING.md, CODE_OF_CONDUCT.md, SECURITY.md, and ROADMAP.md. Small contributions are tracked with good first issue and help wanted labels.
Documentation
docs/FAQ.md— evaluation, safety boundaries, and how DataSentry fits alongside other data-quality toolsexamples/— scenario-first runnable examplesdocs/MCP.md— VS Code and Claude Desktop MCP setupdocs/GITHUB_ACTIONS.md— reusable GitHub quality gateexamples/integrations/github-actions/— copy-paste CI gatedocs/BENCHMARKS.md— benchmark policydocs/DEVELOPMENT.md— engineering notes
License
Apache-2.0 — see LICENSE.
Available Tools
25 toolscomparison_buildARead-only
Same-dataset historical run comparison (V6, ADR-064): per-run overall/dimension scores, issue counts and delta vs previous run. Use the current run id as current_run_id. Returns ok:true with comparison null when fewer than two runs exist.
Inputs: dataset_id (required), current_run_id (required).
contract_validateBRead-only
Validate a data contract YAML file and return the structured contract or validation errors.
Inputs: path (required).
detectors_listARead-only
List all registered quality detectors with id, dimension, capabilities and default thresholds.
Inputs: no named parameters.
drift_compareBRead-only
Compare two historical scans of a dataset: schema, row-count, score and issue-distribution drift. Thresholds mirror the CLI (drift compare --row-ratio-threshold/--score-threshold, defaults 0.20/5.0; D1-06).
Inputs: current_run_id (required), reference_run_id (required), score_threshold, row_ratio_threshold.
drift_latestARead-only
Drift between the two most recent scans of a dataset. Fails if fewer than two completed scans exist. Thresholds mirror the CLI (defaults 0.20/5.0; D1-06).
Inputs: dataset_id (required), score_threshold, row_ratio_threshold.
issues_listARead-only
List quality issues across scans, optionally filtered by minimum severity (info/low/medium/high/critical) or a scan id.
Inputs: scan_run_id, severity_at_least.
job_createB
Register a scheduled scan job (Step 51): cron expression, data file path, optional quality gate threshold (gate_quality_min, 0-100) and optional webhook URL. Returns the created job view.
Inputs: cron (required), name (required), path (required), confirm (required), +5 more.
job_removeBDestructive
Delete a scheduled job (Step 88, ADR-088). Returns removed=true.
Inputs: job_id (required), confirm (required).
jobs_listARead-only
List scheduled scan jobs (Step 51/52): cron, enabled, status, next run time and last result. Optionally filter by status (idle/running/dead).
Inputs: status.
job_triggerA
Immediately run a scheduled job once (Step 51). Returns the run id and outcome; mutual exclusion: 409-style error if already running.
Inputs: job_id (required), confirm (required).
job_updateADestructive
Update a scheduled job (Step 88, ADR-088): enabled/cron/retry_attempts/webhook_url/gate_quality_min, partial update; cron change recomputes next run time. Returns the updated view.
Inputs: job_id (required), confirm (required), cron, enabled, +3 more.
pii_delete_sessionBDestructive
Delete a PII encryption session (Step 100, ADR-100). Removing the encrypted mapping is not reversible and does not require the encryption key. Returns deleted=true or ok:false when the session does not exist.
Inputs: confirm (required), session_id (required).
pii_purge_sessionsADestructive
Delete PII encryption sessions older than N days (Step 103, ADR-103). Sessions are matched by creation time only; the encryption key is not required. Returns purged (count of deleted sessions) or ok:false when olderThanDays is invalid (< 1).
Inputs: confirm (required), olderThanDays (required).
pii_restoreA
Restore plaintext in a text by substituting REDACTED placeholders from a PII session's mapping (Step 100, ADR-100). Explicit authorization semantics: calling this tool is the authorization to view plaintext. Returns ok:false with an error when the session does not exist, the mapping cannot be decrypted (missing or wrong key), or no encryption key is configured.
Inputs: text (required), confirm (required), session_id (required).
pii_rotate_keyADestructive
Rotate the PII vault encryption key (Step 102, ADR-102): re-encrypts all existing mappings with a new key and writes it to the local key file, making the previous key invalid for existing sessions. Optional newKey supplies the key material; a random key is generated when omitted (mirrors CLI rotate-key --new-key). Returns ok:false with an error when no encryption key is configured.
Inputs: confirm (required), newKey.
pii_sessionsARead-only
List PII encryption sessions (Step 100, ADR-100) with key version and creation time (no ciphertext, no plaintext). Returns ok:false with an error when no encryption key is configured (set DATASENTRY_ENCRYPTION_KEY or run 'datasentry llm rotate-key').
Inputs: no named parameters.
profiles_getCRead-only
Scan-time column profile sidecar of a scan run (V6, ADR-061) as raw JSON. Returns ok:false with an error when no profile exists.
Inputs: scan_run_id (required).
quality_scoreARead-only
Six-dimension quality score of a scan run (use issues_list or scan_file to obtain scan_run_id). Dimensions below 60 indicate risk.
Inputs: scan_run_id (required).
repair_apply_batchADestructive
Apply repair proposals for one or more issues of a scan run. WRITES DATA: each repair writes a repaired copy plus a before snapshot under .datasentry/repairs/; the source file is never overwritten. D5-08: prefer repair_propose_batch first. preview_only is mandatory: true reviews operations without writing, false applies them. Issues without a proposal are skipped (no_proposal), not failed. Returns applied runs and per-issue errors.
Inputs: confirm (required), scan_run_id (required), source_path (required), preview_only (required), +1 more.
repair_propose_batchB
Generate repair proposals for one or more issues of a scan run (rule engine, read-only, never writes data). issue_ids: array of issue ids; omit to cover all issues of the run. Returns proposals, unsupported issues and per-issue errors.
Inputs: scan_run_id (required), source_path (required), issue_ids.
repair_rollback_batchADestructive
Roll back applied repairs by run id. WRITES DATA: restores the pre-repair snapshot for each run. Returns rolled-back runs and per-run errors.
Inputs: confirm (required), repair_run_ids (required).
repair_verifyB
Verify a repair run: re-scan the repaired copy and report remaining issues vs the original scan (fixed/persistent/new issue types). Read-only. Returns the verify scan run id and comparison counts.
Inputs: repair_run_id (required).
report_exportARead-only
Export the canonical JSON report of a scan run (same payload as CLI report export --as json / REST GET /scans/{id}/report; D1-06 capability parity). Read-only.
Inputs: scan_run_id (required).
scan_fileA
Scan a data file (CSV/Parquet/JSONL/XLSX/DuckDB), a PostgreSQL table (postgresql://DSN, Step 55), a MySQL table (mysql://DSN, Step 56) or a cloud storage file (s3:// gs:// az:// CSV/Parquet/JSONL, Step 57) and persist the quality report. Returns scan id, status, row count, quality score and issue counts. Optional sampling (sampling_size or sampling_ratio; sampling_method random|stratified|reservoir|time_based|rare_oversampling|none, default reservoir; sampling_seed, default 42), detector whitelist and tags mirror the CLI (Step 76, ADR-076).
Inputs: path (required), seed, tags, detectors, +6 more.
trends_listARead-only
Cross-scan quality trends per dataset (V7 data surface, ADR-065): score/issues series with latest score, delta and direction. Optionally filter to one dataset.
Inputs: dataset_id.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 25 tools
Most tools target distinct resources and actions (PII sessions, scans, repairs, jobs, contracts), but a few pairs like drift_compare/drift_latest and trends_list/comparison_build have overlapping purposes that descriptions mostly clarify.
All names use snake_case, but the pattern mixes verb_noun (scan_file, repair_propose_batch) with noun_verb (issues_list, drift_compare, job_create), and pii_sessions is an outlier, making the convention inconsistent.
25 tools is heavy for the stated purpose, though they are divided across coherent domains (PII, scanning, repair, scheduling, contracts). Borderline; some reduction could improve focus.
Core domains have strong lifecycle coverage: PII sessions (list/restore/delete/rotate/purge), scans (create/list/score/drift/report/trends/profiles/compare), repairs (propose/apply/rollback/verify), and jobs (list/create/trigger/update/remove). The main gap is contracts, which only has contract_validate with no create/update/delete surface.
Maintenance
Related MCP Connectors
Monitor MCP servers, API contracts and AI outputs for schema drift. Alerts on breaking changes.
Hosted MCP server for AI-driven data ops. Create apps, manage schemas, and CRUD structured data.
Hosted MCP server for RudderStack, the agentic CDP that runs on your warehouse. Ask Claude, ChatGPT, Cursor or Codex why events failed, which warehouse loads broke, or where your tracking plan is violated. Build and test transformations and audiences. OAuth login, your permissions, PII masked.
MCP server for your apps' tools and custom tools, plus hosted AI agents and approval-gated workflows
Related MCP Servers
- AlicenseAqualityBmaintenanceMCP server with 32 tools for ETL ingestion, AI-generated data quality rules, AI transformations, vector search, and natural-language SQL. Works across Postgres, MongoDB, Kafka, S3/MinIO, HashiCorp Vault, and five vector stores (Qdrant, Weaviate, Milvus, Chroma, pgvector).9014Apache 2.0
- AlicenseBqualityDmaintenanceAgentic data quality MCP server — runs structured validation rules against warehouses (DuckDB, BigQuery, Athena, Databricks, Postgres), diagnoses failures with LLM root cause analysis, and proposes SQL remediations. Full audit trail of every AI decision.633 PyPI6Apache 2.0
- AlicenseNot gradedqualityCmaintenanceAI-driven MCP server that audits, profiles, detects schema drift, and auto-generates documentation for dbt projects, enabling natural language interaction with your dbt project's health.105MIT
- AlicenseAqualityBmaintenanceLocal-first production-readiness MCP server for AI-built apps. It runs read-only checks, produces an evidence-based readiness score, and guides fixes before launch.11224 npm7Apache 2.0