ingest_sql_log
Ingest SQL log files (CSV, JSONL, .gz) to parse table and column references, redact sensitive data, and roll up usage per fingerprint, enabling unused column and data health analysis.
Instructions
Ingest a SQL log file (pg_stat_statements CSV or generic JSONL, .gz transparently) into the per-dataset runtime tables. Each query is parsed for table + column refs, redacted at the chokepoint (string + numeric literals + cell-PII registry), and rolled up into runtime_query_calls keyed by (fingerprint, table, column). Tables in the log that don't match any indexed dataset count as unmapped. Foundational primitive for find_unused_columns, check_column_drop_safe, and data_health_radar (v1.6.0 sibling-parity Phase 1).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| redact | No | Scrub PII / literals before persisting. Default true. | |
| source | No | pg_stat_statements | jsonl | auto (default — sniff by extension). | auto |
| max_rows | No | Hard cap on ingested rows. Default 100000. | |
| file_path | Yes | Path to a CSV / JSONL / .gz log file. |