Refresh the local catalog cache from the live Socrata Discovery API. Pulls the full dataset listing (~280 datasets: id, name, description, fields, tags, update cadence) into data/catalog.parquet. Run first if search_datasets reports a missing cache, or when the portal may have new datasets. Needs no credentials; the app token in .env only lifts the anonymous rate limit. Returns the count of datasets cached.

Search the cached dataset catalog by keyword. Matches name, description, category, and tags. Returns each result's id, name, category, link, and -- crucially -- columns_field_name, the machine field names a query actually accepts (NOT columns_name, the human display names; querying with a display name silently returns zero rows). Use this to find a dataset, then describe_dataset to get its full schema before querying.

Full schema and coverage profile for one dataset. Returns the machine field names (query with these, not the display names), their types, display names, tags, update cadence, and -- when the dataset is cached -- the row count and the dataset's own earliest/latest date (the era-gap signal: a zero result for an old date may mean the dataset predates it, not that nothing happened). Set probe=true to also get coverage_flags: fields that are only partly populated -- the recent-window traps to check before trusting any total or rate.

Pull every row of a dataset via the SODA API and cache it as Parquet (data/raw/.parquet), so it can be cross-referenced in SQL. Supports a server-side $where filter and $select projection. Re-running overwrites the cache; check describe_dataset's cached/fetched_at first to avoid re-pulling time-sensitive data unnecessarily. Large datasets (millions of rows) take time; prefer a $where filter when a subset suffices.

Probe how ONE field is actually populated across a cached dataset -- the defence against the recent-window trap. Reports the populated fraction and, for populated rows, the date range they span, and states a warning when the pattern looks like a recent operational window rather than the full history (e.g. a 'balance' field only filled for the last year). Always run this before summing or rate-computing over a field. The field must be a machine name from columns_field_name; a display name raises an actionable error listing the real fields.

Automated field-population audit: scan every field of a cached dataset and return the least-populated first -- the automated version of 'check field population before trusting a total'. This is how you discover, up front, that a 'complete-looking' column is really a recent window. Use it instead of hand-checking fields one at a time.

List the dataset ids whose rows are cached locally in data/raw/. Only cached datasets can be queried in SQL or audited for coverage; use fetch_dataset to pull one that is not yet cached.

Run a DuckDB SELECT over the cached datasets, exposed as one SQL view per dataset named ds_<id_with_underscores> (e.g. ixg8-tyau -> ds_ixg8_tyau). This is the cross-referencing step: join/aggregate across ds_* views. Every Socrata column is a string -- use TRY_CAST(col AS DOUBLE) for numeric work, and expect join keys (e.g. department names) to need normalization before they line up. NOTE: runs arbitrary SQL over local Parquet only; a trusted single-user tool, not sandboxed. Prefer describe_dataset + coverage_audit first so you know the fields exist and are fully populated.