Skip to main content
Glama

中文导读:DataSentry 会先自动发现数据质量问题,再给出样本、比例、置信度等证据。修复采用 propose → preview → apply to a copy → verify → rollback 的保守流程。检测与评分不依赖 LLM,AI 只作为可选辅助,数据可以完全留在本机。

Try it in 30 seconds

Install the current PyPI release and run the zero-config product tour:

pip install --upgrade datasentry-ai
datasentry demo

datasentry-demo is also available as a direct console alias.

The demo generates synthetic dirty data, runs the built-in detectors, exports JSON + HTML reports, applies one safe repair to a copy, re-scans the repaired copy, and prints a rollback command. It needs no dataset, cloud service, API key, or LLM.

synthetic dirty CSV
        ↓
deterministic detectors
        ↓
evidence-backed issues + quality score
        ↓
preview → repaired copy
        ↓
re-scan → verify new/persistent issues

Prefer scanning your own data immediately?

curl -L https://raw.githubusercontent.com/Jackxiaozhiren/datasentry/main/demo-data/orders.csv -o orders.csv
datasentry scan orders.csv
datasentry issues list --severity high

Or launch the interactive interfaces:

datasentry          # terminal UI
datasentry-server   # Web UI + REST API at http://localhost:8000/ui/

Related MCP server: aegis-dq

Start with your problem

If you need to...

Start here

Find suspicious data before you know every rule

datasentry scan data.csv

See the full discovery → repair → verify loop

datasentry demo

Block severe data issues in CI

docs/GITHUB_ACTIONS.md

Add quality gates to dbt / Airflow

examples/integrations/

Inspect and repair issues without overwriting the source

Safe repair

Give AI agents deterministic data-quality tools

docs/MCP.md

The default path is intentionally low-commitment: scan first, inspect evidence, then decide whether a finding deserves a permanent rule or a repair.

Why DataSentry exists

Most data-quality tools are excellent once you already know the expectations, checks, or contracts you want to enforce. Real incidents often start one step earlier: you do not yet know what is wrong.

DataSentry is built around the complete remediation loop:

Find → Explain → Fix safely → Verify
  • Find — discover common quality problems without writing every rule first.

  • Explain — attach samples, affected counts/ratios, detector evidence, and confidence.

  • Fix safely — preview changes and apply repairs to a copy instead of mutating the source.

  • Verify — re-scan the repaired copy and surface persistent or newly introduced issues.

DataSentry fits best when you are looking at data you do not fully trust yet, want evidence before codifying checks, or need a conservative remediation path. It complements expectation- and contract-driven tools rather than requiring you to replace them.

What it catches automatically

DataSentry ships with a registry of deterministic detectors covering common failure modes such as:

  • missing and placeholder values;

  • invalid emails, URLs, dates, and encodings;

  • duplicate identifiers and uniqueness violations;

  • inconsistent categories and cross-field contradictions;

  • foreign-key and referential-integrity problems;

  • numeric and statistical outliers;

  • schema, row-count, score, and issue-distribution drift.

Re-count them from the registry the installed build actually ships:

datasentry detectors | python3 -c "import json,sys; print(len(json.load(sys.stdin)['detectors']))"
# 39

That number is the registry, not what any single file triggers: a detector whose required column types are absent is skipped, so the count that actually runs depends on the schema you scan.

Every scan produces an evidence-backed issue list and a six-dimension quality score across completeness, validity, uniqueness, consistency, integrity, and timeliness.

Safe repair, not blind mutation

# inspect the highest-severity findings
datasentry issues list --severity high

# propose a repair without changing data
datasentry repair propose <issue_id> --file orders.csv

# preview the exact effect
datasentry repair preview <issue_id> --file orders.csv

# apply to a repaired copy; the original is not overwritten
datasentry repair apply <issue_id> --file orders.csv

# re-scan the repaired copy and detect regressions
datasentry repair verify <run_id>

# inspect or undo the repair
datasentry repair diff <run_id>
datasentry repair rollback <run_id>

Repairs are fingerprinted, auditable, and reversible. AI-generated repair proposals remain human-approved state changes.

Quality gates for CI

datasentry scan orders.csv --fail-on high

For GitHub repositories, reuse DataSentry's maintained workflow_call gate instead of duplicating installation and exit-code handling:

jobs:
  datasentry:
    uses: Jackxiaozhiren/datasentry/.github/workflows/datasentry-quality-gate.yml@main
    with:
      path: data/orders.csv
      fail_on: high

See docs/GITHUB_ACTIONS.md for inputs, artifacts, security boundaries, and version-pinning guidance.

Reports can be exported as JSON, Markdown, HTML, JUnit, and SARIF. The versioned, machine-readable JSON report Schema is included for downstream validation. The GitHub Actions example fails the workflow on severe findings while still uploading an HTML report for review.

Give AI agents deterministic data-quality tools

DataSentry includes an MCP stdio server:

datasentry mcp --project /path/to/project

MCP-capable clients can scan files, inspect evidence-backed issues, read quality scores and trends, compare drift, validate contracts, manage scheduled jobs, and call DataSentry tools without bypassing the same underlying safety rules used by the CLI and REST API.

Copy-paste setup recipes for VS Code and Claude Desktop are in docs/MCP.md. Hosts that declare MCP Apps support also get a read-only report card from report_export; see docs/MCP_APPS.md for the negotiation contract and its verification limits.

Boundary: AI may propose; humans approve state-changing repairs.

This is a positioning guide, not a winner/loser feature scorecard. These projects solve overlapping but different jobs; check their upstream documentation for current capabilities.

Project

Core mental model

A strong fit when you want...

DataSentry

discover → explain → repair → verify

automatic issue discovery plus a controlled, reversible remediation loop

Great Expectations

Expectations / expressive data tests

explicit validation rules, validation results, and generated data-quality documentation

Soda Core

data contracts and quality checks

YAML contracts and verification across a broad data stack

Deequ

“unit tests for data” on Spark

large-scale data verification in Spark-centric environments

ydata-profiling

one-line profiling / EDA

fast exploratory profiling and shareable analysis reports

DataSentry is intentionally not trying to replace a metadata catalog, lineage platform, or every validator. Its focus is narrower: find bad data, show why it was flagged, and close the repair loop without gambling on the source.

Local-first by design

  • deterministic detection and scoring run locally;

  • DuckDB powers core local execution;

  • OpenAI/Ollama assistance is optional;

  • PII redaction, encrypted mappings, and LLM audit records are available when AI is enabled;

  • the original source file is not overwritten by repair workflows.

Data sources

  • CSV, Parquet, JSONL, XLSX

  • DuckDB and SQLite

  • PostgreSQL and MySQL

  • s3://, gs://, and az:// objects

  • single files, batches, and globs

History and drift

Persist scans and compare data over time:

datasentry drift latest orders
datasentry score

Tracked signals include schema changes, row-count movement, quality-score changes, and issue-distribution drift.

Architecture

flowchart LR
    Sources[Files / DBs / cloud objects] --> DuckDB[Local execution]
    DuckDB --> Detect[Detectors]
    Detect --> Evidence[Evidence fusion]
    Evidence --> Score[6-dimension score]
    Score --> Reports[Reports / history / gates]
    Reports --> CLI[CLI / TUI]
    Reports --> Web[Web / REST]
    Reports --> MCP[MCP]
    Evidence --> Proposal[Repair proposal]
    Proposal --> Preview[Preview]
    Preview --> Apply[Apply to copy]
    Apply --> Verify[Verify by re-scan]
    Verify --> Rollback[Rollback artifact]
    LLM[Optional OpenAI / Ollama] -. proposes .-> Proposal

Reproducible benchmark

uv sync
uv run python benchmarks/bench_scan.py 1000000 42

The benchmark generates synthetic dirty data and measures profiling, detection/fusion/scoring, numeric-outlier detection, JSONL reading, sampling, score drift, and memory high-water marks. See docs/BENCHMARKS.md.

Development

uv sync
make check          # lint + mypy --strict + tests/coverage
make demo           # exercise the public datasentry demo path
make bench          # benchmark
make build          # distributions

Contributing

Useful contributions include new detectors, connectors, integration examples, reproducible benchmark cases, documentation/translations, minimal bug reproductions, and CLI/TUI/Web usability improvements.

Want a small first contribution? Start with good first issue #14 or good first issue #15.

See CONTRIBUTING.md, CODE_OF_CONDUCT.md, SECURITY.md, and ROADMAP.md. Small contributions are tracked with good first issue and help wanted labels.

Documentation

License

Apache-2.0 — see LICENSE.


Available Tools

25 tools
comparison_buildA
Read-only

Same-dataset historical run comparison (V6, ADR-064): per-run overall/dimension scores, issue counts and delta vs previous run. Use the current run id as current_run_id. Returns ok:true with comparison null when fewer than two runs exist.

Inputs: dataset_id (required), current_run_id (required).

contract_validateB
Read-only

Validate a data contract YAML file and return the structured contract or validation errors.

Inputs: path (required).

detectors_listA
Read-only

List all registered quality detectors with id, dimension, capabilities and default thresholds.

Inputs: no named parameters.

drift_compareB
Read-only

Compare two historical scans of a dataset: schema, row-count, score and issue-distribution drift. Thresholds mirror the CLI (drift compare --row-ratio-threshold/--score-threshold, defaults 0.20/5.0; D1-06).

Inputs: current_run_id (required), reference_run_id (required), score_threshold, row_ratio_threshold.

drift_latestA
Read-only

Drift between the two most recent scans of a dataset. Fails if fewer than two completed scans exist. Thresholds mirror the CLI (defaults 0.20/5.0; D1-06).

Inputs: dataset_id (required), score_threshold, row_ratio_threshold.

issues_listA
Read-only

List quality issues across scans, optionally filtered by minimum severity (info/low/medium/high/critical) or a scan id.

Inputs: scan_run_id, severity_at_least.

job_createB

Register a scheduled scan job (Step 51): cron expression, data file path, optional quality gate threshold (gate_quality_min, 0-100) and optional webhook URL. Returns the created job view.

Inputs: cron (required), name (required), path (required), confirm (required), +5 more.

job_removeB
Destructive

Delete a scheduled job (Step 88, ADR-088). Returns removed=true.

Inputs: job_id (required), confirm (required).

jobs_listA
Read-only

List scheduled scan jobs (Step 51/52): cron, enabled, status, next run time and last result. Optionally filter by status (idle/running/dead).

Inputs: status.

job_triggerA

Immediately run a scheduled job once (Step 51). Returns the run id and outcome; mutual exclusion: 409-style error if already running.

Inputs: job_id (required), confirm (required).

job_updateA
Destructive

Update a scheduled job (Step 88, ADR-088): enabled/cron/retry_attempts/webhook_url/gate_quality_min, partial update; cron change recomputes next run time. Returns the updated view.

Inputs: job_id (required), confirm (required), cron, enabled, +3 more.

pii_delete_sessionB
Destructive

Delete a PII encryption session (Step 100, ADR-100). Removing the encrypted mapping is not reversible and does not require the encryption key. Returns deleted=true or ok:false when the session does not exist.

Inputs: confirm (required), session_id (required).

pii_purge_sessionsA
Destructive

Delete PII encryption sessions older than N days (Step 103, ADR-103). Sessions are matched by creation time only; the encryption key is not required. Returns purged (count of deleted sessions) or ok:false when olderThanDays is invalid (< 1).

Inputs: confirm (required), olderThanDays (required).

pii_restoreA

Restore plaintext in a text by substituting REDACTED placeholders from a PII session's mapping (Step 100, ADR-100). Explicit authorization semantics: calling this tool is the authorization to view plaintext. Returns ok:false with an error when the session does not exist, the mapping cannot be decrypted (missing or wrong key), or no encryption key is configured.

Inputs: text (required), confirm (required), session_id (required).

pii_rotate_keyA
Destructive

Rotate the PII vault encryption key (Step 102, ADR-102): re-encrypts all existing mappings with a new key and writes it to the local key file, making the previous key invalid for existing sessions. Optional newKey supplies the key material; a random key is generated when omitted (mirrors CLI rotate-key --new-key). Returns ok:false with an error when no encryption key is configured.

Inputs: confirm (required), newKey.

pii_sessionsA
Read-only

List PII encryption sessions (Step 100, ADR-100) with key version and creation time (no ciphertext, no plaintext). Returns ok:false with an error when no encryption key is configured (set DATASENTRY_ENCRYPTION_KEY or run 'datasentry llm rotate-key').

Inputs: no named parameters.

profiles_getC
Read-only

Scan-time column profile sidecar of a scan run (V6, ADR-061) as raw JSON. Returns ok:false with an error when no profile exists.

Inputs: scan_run_id (required).

quality_scoreA
Read-only

Six-dimension quality score of a scan run (use issues_list or scan_file to obtain scan_run_id). Dimensions below 60 indicate risk.

Inputs: scan_run_id (required).

repair_apply_batchA
Destructive

Apply repair proposals for one or more issues of a scan run. WRITES DATA: each repair writes a repaired copy plus a before snapshot under .datasentry/repairs/; the source file is never overwritten. D5-08: prefer repair_propose_batch first. preview_only is mandatory: true reviews operations without writing, false applies them. Issues without a proposal are skipped (no_proposal), not failed. Returns applied runs and per-issue errors.

Inputs: confirm (required), scan_run_id (required), source_path (required), preview_only (required), +1 more.

repair_propose_batchB

Generate repair proposals for one or more issues of a scan run (rule engine, read-only, never writes data). issue_ids: array of issue ids; omit to cover all issues of the run. Returns proposals, unsupported issues and per-issue errors.

Inputs: scan_run_id (required), source_path (required), issue_ids.

repair_rollback_batchA
Destructive

Roll back applied repairs by run id. WRITES DATA: restores the pre-repair snapshot for each run. Returns rolled-back runs and per-run errors.

Inputs: confirm (required), repair_run_ids (required).

repair_verifyB

Verify a repair run: re-scan the repaired copy and report remaining issues vs the original scan (fixed/persistent/new issue types). Read-only. Returns the verify scan run id and comparison counts.

Inputs: repair_run_id (required).

report_exportA
Read-only

Export the canonical JSON report of a scan run (same payload as CLI report export --as json / REST GET /scans/{id}/report; D1-06 capability parity). Read-only.

Inputs: scan_run_id (required).

scan_fileA

Scan a data file (CSV/Parquet/JSONL/XLSX/DuckDB), a PostgreSQL table (postgresql://DSN, Step 55), a MySQL table (mysql://DSN, Step 56) or a cloud storage file (s3:// gs:// az:// CSV/Parquet/JSONL, Step 57) and persist the quality report. Returns scan id, status, row count, quality score and issue counts. Optional sampling (sampling_size or sampling_ratio; sampling_method random|stratified|reservoir|time_based|rare_oversampling|none, default reservoir; sampling_seed, default 42), detector whitelist and tags mirror the CLI (Step 76, ADR-076).

Inputs: path (required), seed, tags, detectors, +6 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.3/5.0

    Scored across 25 tools

    Disambiguation4/5

    Most tools target distinct resources and actions (PII sessions, scans, repairs, jobs, contracts), but a few pairs like drift_compare/drift_latest and trends_list/comparison_build have overlapping purposes that descriptions mostly clarify.

    Naming Consistency3/5

    All names use snake_case, but the pattern mixes verb_noun (scan_file, repair_propose_batch) with noun_verb (issues_list, drift_compare, job_create), and pii_sessions is an outlier, making the convention inconsistent.

    Tool Count3/5

    25 tools is heavy for the stated purpose, though they are divided across coherent domains (PII, scanning, repair, scheduling, contracts). Borderline; some reduction could improve focus.

    Completeness4/5

    Core domains have strong lifecycle coverage: PII sessions (list/restore/delete/rotate/purge), scans (create/list/score/drift/report/trends/profiles/compare), repairs (propose/apply/rollback/verify), and jobs (list/create/trigger/update/remove). The main gap is contracts, which only has contract_validate with no create/update/delete surface.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      B
      maintenance
      MCP server with 32 tools for ETL ingestion, AI-generated data quality rules, AI transformations, vector search, and natural-language SQL. Works across Postgres, MongoDB, Kafka, S3/MinIO, HashiCorp Vault, and five vector stores (Qdrant, Weaviate, Milvus, Chroma, pgvector).
      90
      14
      Apache 2.0
    • A
      license
      B
      quality
      D
      maintenance
      Agentic data quality MCP server — runs structured validation rules against warehouses (DuckDB, BigQuery, Athena, Databricks, Postgres), diagnoses failures with LLM root cause analysis, and proposes SQL remediations. Full audit trail of every AI decision.
      6
      33 PyPI
      6
      Apache 2.0
    • A
      license
      Not graded
      quality
      C
      maintenance
      AI-driven MCP server that audits, profiles, detects schema drift, and auto-generates documentation for dbt projects, enabling natural language interaction with your dbt project's health.
      105
      MIT