nport

Normalized fund holdings and service-provider graph built from SEC Form N-PORT and Form N-CEN, joined to the GLEIF LEI register, served to agents over MCP.

Design: docs/design.md. Empirical notes on the source data: docs/data-notes.md.

Status: ingest layer only. Raw and Parquet layers land end to end; the dbt transform, MCP server, and Terraform are not built yet.

Quickstart

uv sync
export SEC_USER_AGENT="Your Name your@email.com"   # the SEC throttles anonymous traffic

uv run nport-ingest periods nport    # what quarters are published upstream
uv run nport-ingest all              # N-PORT + N-CEN + GLEIF, latest available
uv run nport-ingest status           # what is landed, from the manifests

Landing all three sources takes a few minutes and writes ~1 GB under ./data. To land a single source or an older quarter:

uv run nport-ingest land nport --period 2026q1
uv run nport-ingest land ncen --raw-only     # skip Parquet conversion
uv run nport-ingest land gleif --force       # re-download and re-convert

Related MCP server: civicgraph-mcp

Storage

NPORT_DATA_ROOT controls where everything goes and is the only difference between local iteration and running in ECS:

export NPORT_DATA_ROOT=./data                 # default
export NPORT_DATA_ROOT=s3://my-bucket/nport   # needs the `s3` extra: uv sync --extra s3

Layout, per docs/design.md §2:

{root}/raw/{source}/{period}/{archive}.zip             original bytes, immutable
{root}/parquet/{table}/period={period}/data.parquet     zstd, all columns VARCHAR
{root}/parquet/_manifest/{source}/period={period}/
    manifest.json    source URL, raw SHA-256, per-table row counts
    schema.json      column names, types, descriptions and PKs, from the
                     CSVW sidecar the SEC ships inside each archive

Re-running a landed period is a no-op unless --force is passed, which makes the task safe to retry from a schedule. If the SEC republishes a quarter under the same filename, the run warns that the byte count moved rather than silently serving stale data.

Design decisions in the ingest layer

Every Parquet column is VARCHAR. Typing belongs in dbt staging/. The sources are full of values that break type inference — dates as 28-FEB-2026, numbers as -.0091723714, and N/A appearing in numeric columns — so inference here would either fail or silently coerce, and it would put business logic in the one layer that has to stay reproducible from raw bytes.

Download URLs are discovered, not constructed. Most SEC archives follow {period}_{source}.zip, but several N-CEN quarters shipped as 2025q1_ncen_0.zip after a republish. The index page is scraped for hrefs, with the conventional pattern as a fallback.

The archives' own schema metadata is extracted. Both SEC ZIPs contain a CSVW *_metadata.json declaring per-table primary keys, column types and column descriptions. That is the authoritative input for the grain assertions in transform/, so it is written out as schema.json rather than left in a ZIP.

Layout

src/nport/ingest/        fetch -> validate -> land as Parquet
    sources/             one module per upstream source
    convert.py           ZIP of delimited text -> Parquet, via DuckDB
    land.py              orchestration and manifests
    storage.py           URI addressing over fsspec (local or S3)
transform/               dbt project (dbt-duckdb)          [not built]
src/nport/server/        FastMCP server                    [not built]
evals/                   scored question set               [not built]
infra/                   Terraform                         [not built]

Development

uv run ruff check src/ && uv run ruff format src/
uv run pytest