ingest_sql_log
Ingest SQL logs (CSV/JSONL/gz) to parse table and column references, redact PII and literals, and roll up query fingerprints for usage analysis.
Instructions
Ingest a SQL log file (pg_stat_statements CSV or generic JSONL, .gz transparently) into the per-dataset runtime tables. Each query is parsed for table + column refs, redacted at the chokepoint (string + numeric literals + cell-PII registry), and rolled up into runtime_query_calls keyed by (fingerprint, table, column). Tables in the log that don't match any indexed dataset count as unmapped. Foundational primitive for find_unused_columns, check_column_drop_safe, and data_health_radar (v1.6.0 sibling-parity Phase 1).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| redact | No | Scrub PII / literals before persisting. Default true. | |
| source | No | pg_stat_statements | jsonl | auto (default — sniff by extension). | auto |
| max_rows | No | Hard cap on ingested rows. Default 100000. | |
| file_path | Yes | Path to a CSV / JSONL / .gz log file. |