ingest_sql_log
Ingest SQL log files (CSV/JSONL/.gz) into runtime tables, parse query references, redact PII/literals, and aggregate usage by column to support unused-column and drop-safety analysis.
Instructions
Ingest a SQL log file (pg_stat_statements CSV or generic JSONL, .gz transparently) into the per-dataset runtime tables. Each query is parsed for table + column refs, redacted at the chokepoint (string + numeric literals + cell-PII registry), and rolled up into runtime_query_calls keyed by (fingerprint, table, column). Tables in the log that don't match any indexed dataset count as unmapped. Foundational primitive for find_unused_columns, check_column_drop_safe, and data_health_radar (v1.6.0 sibling-parity Phase 1).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| redact | No | Scrub PII / literals before persisting. Default true. | |
| source | No | pg_stat_statements | jsonl | auto (default — sniff by extension). | auto |
| max_rows | No | Hard cap on ingested rows. Default 100000. | |
| file_path | Yes | Path to a CSV / JSONL / .gz log file. |