# nport

Normalized fund holdings and service-provider graph built from SEC Form N-PORT
and Form N-CEN, joined to the GLEIF LEI register, served to agents over MCP.

Design: [docs/design.md](docs/design.md).
Empirical notes on the source data: [docs/data-notes.md](docs/data-notes.md).

Status: **ingest layer only.** Raw and Parquet layers land end to end; the dbt
transform, MCP server, and Terraform are not built yet.

## Quickstart

```bash
uv sync
export SEC_USER_AGENT="Your Name your@email.com"   # the SEC throttles anonymous traffic

uv run nport-ingest periods nport    # what quarters are published upstream
uv run nport-ingest all              # N-PORT + N-CEN + GLEIF, latest available
uv run nport-ingest status           # what is landed, from the manifests
```

Landing all three sources takes a few minutes and writes ~1 GB under `./data`.
To land a single source or an older quarter:

```bash
uv run nport-ingest land nport --period 2026q1
uv run nport-ingest land ncen --raw-only     # skip Parquet conversion
uv run nport-ingest land gleif --force       # re-download and re-convert
```

## Storage

`NPORT_DATA_ROOT` controls where everything goes and is the only difference
between local iteration and running in ECS:

```bash
export NPORT_DATA_ROOT=./data                 # default
export NPORT_DATA_ROOT=s3://my-bucket/nport   # needs the `s3` extra: uv sync --extra s3
```

Layout, per [docs/design.md](docs/design.md) §2:

```
{root}/raw/{source}/{period}/{archive}.zip             original bytes, immutable
{root}/parquet/{table}/period={period}/data.parquet     zstd, all columns VARCHAR
{root}/parquet/_manifest/{source}/period={period}/
    manifest.json    source URL, raw SHA-256, per-table row counts
    schema.json      column names, types, descriptions and PKs, from the
                     CSVW sidecar the SEC ships inside each archive
```

Re-running a landed period is a no-op unless `--force` is passed, which makes
the task safe to retry from a schedule. If the SEC republishes a quarter under
the same filename, the run warns that the byte count moved rather than silently
serving stale data.

## Design decisions in the ingest layer

**Every Parquet column is VARCHAR.** Typing belongs in dbt `staging/`. The
sources are full of values that break type inference — dates as `28-FEB-2026`,
numbers as `-.0091723714`, and `N/A` appearing in numeric columns — so
inference here would either fail or silently coerce, and it would put business
logic in the one layer that has to stay reproducible from raw bytes.

**Download URLs are discovered, not constructed.** Most SEC archives follow
`{period}_{source}.zip`, but several N-CEN quarters shipped as
`2025q1_ncen_0.zip` after a republish. The index page is scraped for hrefs, with
the conventional pattern as a fallback.

**The archives' own schema metadata is extracted.** Both SEC ZIPs contain a CSVW
`*_metadata.json` declaring per-table primary keys, column types and column
descriptions. That is the authoritative input for the grain assertions in
`transform/`, so it is written out as `schema.json` rather than left in a ZIP.

## Layout

```
src/nport/ingest/        fetch -> validate -> land as Parquet
    sources/             one module per upstream source
    convert.py           ZIP of delimited text -> Parquet, via DuckDB
    land.py              orchestration and manifests
    storage.py           URI addressing over fsspec (local or S3)
transform/               dbt project (dbt-duckdb)          [not built]
src/nport/server/        FastMCP server                    [not built]
evals/                   scored question set               [not built]
infra/                   Terraform                         [not built]
```

## Development

```bash
uv run ruff check src/ && uv run ruff format src/
uv run pytest
```