# Paperless Document Assets

Licensed under the [MIT License](LICENSE).

Prepare versioned, reusable AI assets beside Paperless-ngx. Paperless remains the source of original files and archive metadata; MinerU produces derived Markdown, images, structure, and locator data.

## Capabilities

- Read documents and metadata from a configured Paperless API.
- Run MinerU as a separate worker and publish immutable, versioned bundles with a manifest.
- Serve prepared Markdown, images, structure, and original-file references through an authenticated Reader and read-only MCP tools.
- Optionally send explicitly selected prepared Markdown to WeKnora through its existing ingestion interface.
- Preserve source identity and input digests so later source revisions produce distinct bundles.

Preparation does not add documents to a WeKnora knowledge base. The optional bridge is explicit and does not subscribe to future Paperless documents. This project does not update Paperless originals or claim that generated OCR/Markdown is error-free.

## Requirements

- Python 3.12–3.14 for the service and command-line tools.
- A Paperless API account and token with only the access needed to read source documents.
- A writable data directory for bundle and catalog storage.
- MinerU 4.0.10 in the worker environment for PDF and office-document preparation. The Reader/API gateway runs without the worker extra; Markdown and text assets can be prepared without MinerU.
- PostgreSQL for the persistent catalog configuration shown in the Compose examples.

## Configuration

Copy [the gateway environment template](config/gateway.env.example) to a private file outside the repository. Set credentials there; do not commit real values. Set the Compose variables `ASSET_ENV_FILE`, `ASSET_DATA_DIR`, `PAPERLESS_URL`, `PAPERLESS_INSTANCE`, and `ASSET_PUBLIC_URL` in the deployment environment. Replace the example host names with endpoints reachable from the relevant containers.

The gateway Compose example binds to loopback by default. If it must be reachable from another machine, place it behind TLS and an authentication boundary, and restrict network access to trusted clients. Keep the Reader token separate from any publishing token. Do not expose service credentials in a browser or public URL.

## Minimal local Reader example

The base package is enough to run the gateway and process a Markdown fixture. This example uses a local SQLite catalog and loopback access; it does not connect to Paperless or WeKnora.

```sh
uv sync
mkdir -p .local/catalog .local/fixtures
python3 -c 'import pathlib,secrets; p=pathlib.Path(".local/api-token"); p.write_text(secrets.token_urlsafe(32)); p.chmod(0o600)'
printf '# Example document\n\nPrepared Markdown is readable through the gateway.\n' > .local/fixtures/example.md
export ASSET_API_TOKEN="$(cat .local/api-token)"
uv run document-assets serve --root .local/catalog --fixtures .local/fixtures
```

In another terminal, set the same token and prepare/read the fixture:

```sh
export ASSET_API_TOKEN="$(cat .local/api-token)"
uv run document-assets prepare-fixture .local/fixtures/example.md
BUNDLE_ID="$(uv run document-assets worker-once | python3 -c 'import ast,sys; print(ast.literal_eval(sys.stdin.read())["bundle_id"])')"
uv run document-assets read "$BUNDLE_ID"
```

The `worker` extra is needed for parsing PDFs and office files. Install it only when that processing capability is needed:

```sh
uv sync --extra worker
```

For a configuration-only check, run:

```sh
docker compose -f compose.gateway.yml config
```

Review the rendered mounts, URLs, and ports before starting services. `compose.nas.yml` is a deployment staging template; copy the listed source files into a deployment directory so its root-relative mounts resolve:

```sh
DEPLOY_DIR=/srv/paperless-document-assets-deploy
mkdir -p "$DEPLOY_DIR"
cp compose.nas.yml Dockerfile.gateway Dockerfile.paperless-ocr pyproject.toml uv.lock README.md "$DEPLOY_DIR/"
cp -R src fixtures "$DEPLOY_DIR/"
cp config/asset-reader.nginx.conf "$DEPLOY_DIR/asset-reader.nginx.conf"
cp scripts/source_directory_bridge.py "$DEPLOY_DIR/source_directory_bridge.py"
cp config/nas-stack.env.example "$DEPLOY_DIR/nas-stack.env"
cp config/nas-asset.env.example "$DEPLOY_DIR/nas-asset.env"
chmod 600 "$DEPLOY_DIR/nas-stack.env" "$DEPLOY_DIR/nas-asset.env"
```

Fill both environment files privately, then build the gateway and OCR base images and render the staged Compose file:

```sh
cd "$DEPLOY_DIR"
docker build -f Dockerfile.gateway -t paperless-document-assets-gateway:nas-stage .
docker build -f Dockerfile.paperless-ocr -t paperless-ngx:3.2.1-chi-sim-local .
docker compose --env-file nas-stack.env -f compose.nas.yml --profile assets config
```

The optional source-directory bridge requires an explicit, narrow `BRIDGE_INCLUDE_GLOB` pattern and keeps its state outside both source and Paperless consume directories.

## Development

The default development profile does not install the large MinerU worker runtime. Core tests stay green and worker integration tests are reported as skipped:

```sh
uv sync --extra test --extra server --extra agent --extra ops
uv run python -m pytest -q
```

To exercise PDF/Office/MinerU parsing as well, install the optional worker runtime and run the same suite. Worker-marked tests are then executed instead of skipped:

```sh
uv sync --extra test --extra server --extra agent --extra ops --extra worker
uv run python -m pytest -q
```

Use `uv run python -m pytest -q -m 'not worker'` for an explicit core-only run. Tests use synthetic fixtures and mocks; even the worker profile does not by itself prove connectivity or OCR quality in a live Paperless instance.

## Data and security boundary

Paperless owns originals. This project owns derived asset bundles and their manifest. WeKnora owns its ingested copy, chunks, and indexes. A Reader credential grants read access only; any write or publication capability must use a distinct private credential. Store database backups, Paperless exports, original documents, runtime logs, local environment files, and private certificates outside this repository.

Supported document formats and extraction quality depend on the pinned MinerU release and the configured worker. Always review generated material against the Paperless original before relying on it.