QLD Legislation Export & Ingest

An end-to-end pipeline for semantic research over current/in-force Queensland legislation, in three stages:

1. Extract - enumerate in-force documents from the public legislation.qld.gov.au browse data source into a manifest, then download the authorised XML for each.

2. Ingest - walk the authorised QuILLS XML structure (Chapter / Part / Division / Subdivision / section, schedules, dictionary definitions) into a ChromaDB vector store using bge-m3 embeddings.

3. Serve - a standalone MCP server exposing semantic search over that store.

No API account or authentication is required for the extract stage.

Repo layout

config/settings.example.json   tuning knobs / output paths / db + model
src/qld_legislation/            extract package
  config.py                     loads config/settings.json (or the example)
  http.py                       shared GET with exponential-backoff retry
  enumerate.py                  builds the manifest from the browse data source
  fetch.py                      downloads XML for every manifest entry
  cli.py                        `enumerate` / `fetch` subcommands
  ingest/                       XML -> ChromaDB stage
    parse.py                    QuILLS XML tree -> structural units
    chunk.py                    units -> embedding-ready chunks + metadata
    store.py                    upsert into ChromaDB (torch or onnx-int8)
    cli.py                      ingest CLI (has a dep-free --dry-run)
    triage.py                   per-Act "which Act governs this?" documents
    triage_cli.py               builds the small qld_acts triage index
  cth/                          Commonwealth (FRL) parallel pipeline
    enumerate.py                in-force principal Acts via the FRL OData API
    fetch.py                    download compilation Epubs (resumable)
    parse.py                    Epub XHTML (OPC drafting styles) -> units
    cli.py                      `enumerate` / `fetch` / `ingest` subcommands
  update.py                     differential updater (diff vs saved state)
  rerank.py                     cross-encoder reranking of search hits
  export_onnx.py                one-time bge-m3 ONNX int8 export + benchmark
  mcp_server.py                 standalone MCP server (semantic search tools)
Dockerfile                      self-contained image for the remote MCP server
docker-compose.yml              run the server + mount the store as a volume
scripts/run_full_extract.sh     enumerate then fetch, in one call
tests/                          unit tests for the pure (non-network) logic
data/                           gitignored - manifest.json, xml/, chroma/

Related MCP server: mcp-context

Setup

Install the package (editable) with the extras you need. This puts the qld-legis* commands on your PATH and makes the package importable from any directory - no PYTHONPATH juggling. Use a virtual env (on macOS the system Python is locked down):

python3 -m venv .venv && source .venv/bin/activate

pip install -e .                # extract stage only (just requests)
pip install -e '.[ingest]'      # + ingest stage (chromadb + bge-m3)
pip install -e '.[mcp]'         # + standalone MCP server (adds the mcp SDK)

cp config/settings.example.json config/settings.json   # optional, see Configuration

Prefer plain requirements files? requirements.txt, requirements-ingest.txt and requirements-mcp.txt install the same dependency sets, but then you must run modules with PYTHONPATH=src python -m qld_legislation.....

Commands below use the console scripts (qld-legis, qld-legis-ingest, qld-legis-mcp); each is equivalent to python -m qld_legislation.<module>.

Usage

Extract

# Stage 1: enumerate every in-force Act into a manifest
qld-legis enumerate --out data/manifest.json

# ... also include in-force Subordinate Legislation (regulations etc.)
qld-legis enumerate --out data/manifest.json --include-sl

# Stage 2: download XML for everything in the manifest
qld-legis fetch --manifest data/manifest.json --out-dir data/xml

Or both stages via the convenience script: scripts/run_full_extract.sh [--include-sl].

The fetch stage is resumable: it skips any file that already exists in --out-dir unless --overwrite is passed. Anything that fails after retries is recorded in <out-dir>/_failed.json (id, title, error) so a re-run can be targeted at just the failures if needed.

Ingest

# Inspect what would be ingested - no ML deps needed, nothing written.
qld-legis-ingest data/xml --dry-run

# Build (or rebuild) the whole vector store from the downloaded corpus.
# On an Apple-silicon Mac add --device mps to use the GPU for the embed.
qld-legis-ingest data/xml --reset --device mps

# Ingest / update a single Act (re-ingest upserts, it does not duplicate).
qld-legis-ingest data/xml/act-2001-064_*.xml

# Optional: build the Act-triage index (one vector per Act) for broad
# "which Act governs this?" queries. Tiny/fast; doesn't touch the main store.
qld-legis-ingest-acts data/xml

Subordinate legislation ingests through the same pipeline - the parser handles the <subordleg> DTD, and a structured schedule (a code of practice with its own clauses) is exploded into per-clause units citing as sch 8 s 12.

Keep it current

qld-legis-update performs a differential update: it enumerates the in-force list, compares each document's version.desc.id against the saved state (data/state.json), and fetches + re-ingests only new/changed documents (old chunks are deleted first, so renumbered provisions don't linger). Repealed documents are removed from the store. Byte-identical XML skips the re-embed even if the version id churned. State is saved after every document, so an interrupted run resumes.

qld-legis-update --seed          # first run on an existing store: baseline only
qld-legis-update --dry-run       # show what would change
qld-legis-update                 # update Acts
qld-legis-update --include-sl    # ... and Subordinate Legislation
                                 # (FIRST run with this flag ingests all
                                 #  in-force SL - a long job on CPU)

Stop the MCP server/container before updating - ChromaDB should not be written while another process queries it. On Windows, scripts/update.ps1 wraps stop -> update -> start for Task Scheduler.

Faster embedding: ONNX int8

bge-m3 query/ingest embedding can run through a dynamic-int8-quantized ONNX export instead of torch fp32 - typically 2-4x faster on CPUs with VNNI int8 instructions (Intel 12th-gen+, Apple Silicon via arm64, most servers). Vectors stay compatible with an existing torch-built store (measured cosine 0.99+ between backends), and both backends persist the same embedding-function name in ChromaDB, so no re-ingest or migration is needed.

pip install -e ".[onnx]"
qld-legis-export-onnx                # one-time export + quantize (~5-10 min)
qld-legis-export-onnx --benchmark    # A/B torch vs onnx-int8 on THIS machine

Then set in config/settings.json:

  "embedding_backend": "onnx-int8",
  "onnx_model_dir": "data/models/bge-m3-onnx"

Every ingest command and the MCP server pick the backend up from settings (env: QLD_EMBEDDING_BACKEND / QLD_ONNX_MODEL_DIR). The Docker image bakes the quantized model and defaults to onnx-int8. Run --benchmark before a big ingest: on CPUs without VNNI, int8 can lose on batch throughput - use whichever backend wins there.

Commonwealth legislation

The Federal Register of Legislation (legislation.gov.au) is covered by a parallel pipeline (src/qld_legislation/cth/). Enumeration uses the FRL's public OData API; the text comes from each compilation's Epub (served at the stable URL /{titleId}/latest/latest/text/original/epub), whose XHTML preserves the OPC drafting styles (ActHead1..5, subsection, Definition, Penalty, ENote*) - so the structure parses natively, stdlib-only.

qld-legis-cth enumerate            # ~3,900 in-force principal Acts -> manifest
qld-legis-cth fetch                # download Epubs (resumable, skip-existing)
qld-legis-cth ingest --dry-run     # parse + chunk stats, nothing written
qld-legis-cth ingest               # embed into the cth_legislation collection
qld-legis-cth ingest --ids C2004A03712 C2004A00818   # or start with a subset

Ingest is resumable (per-document state saves) and re-runs skip unchanged Epubs by content hash. Endnotes are dropped, definitions become per-term units, and schedule clauses cite as sch 1 s 6 (e.g. the Australian Privacy Principles).

Cross-corpus search stays fast by design: Cth chunks live in a separate collection, the query is embedded ONCE and the same vector fans out across the QLD + Cth HNSW indexes (~ms each), and results merge by score - so jurisdiction='all' costs virtually the same as a single-corpus search, and scoped searches touch only their own index. All tools accept jurisdiction='qld'|'cth'|'all'; get_provision routes by doc-id shape (act-2001-064 vs C2004A03712); Cth Acts join the triage index for find_acts with a jurisdiction tag.

Separate store dirs + zero-downtime updates

ChromaDB is single-writer per directory, so ingesting into the directory the MCP server is serving requires the server down. To keep one corpus live while another is (re)built, give each its own store directory:

• db_path - the QLD store (chroma).

• cth_db_path - the Cth store (chroma_cth). Empty = share db_path.

The server opens a client per directory and fans queries across both, so a Cth ingest into chroma_cth never locks the QLD store - QLD stays up for the whole (long) Cth run. Set cth_db_path in settings and mount both dirs in docker-compose.yml (see the commented lines there).

This also gives a blue/green hot-swap for updating a served corpus with seconds of downtime instead of hours:

1. Copy the live store dir: chroma_cth -> chroma_cth.new.

2. Ingest updates into chroma_cth.new (server keeps serving chroma_cth).

3. Point cth_db_path at chroma_cth.new and restart the container (docker compose up -d - a ~30-60s blip), then delete the old dir.

The ingest target can be a directory, a single file, or a glob, and it does not need to be inside the repo - point it at wherever your XML lives (qld-legis-ingest /path/to/xml_corpus --reset). Output goes to db_path from settings (default data/chroma), relative to your current directory, so run these from the repo root unless you pass an absolute --db.

Each section becomes one chunk carrying its full structural path (Chapter/Part/Division/Subdivision), heading, citation and currency date (as_at) as metadata; dictionary definitions each become their own chunk; schedules become one chunk each. Amendment historynotes are dropped and cross-references (intref/legref) are flattened to plain text. Chunk ids are deterministic ({doc_id}:{provision_id}), so ingesting an amended Act updates its provisions in place. When a provision_id legitimately repeats within one document - e.g. a Cth Act whose Schedules each restart section numbering, so two real s 48 exist - the duplicate is suffixed deterministically (...~2) in stable parse order, keeping ids unique without breaking idempotent re-ingest. QLD provision ids come from unique XML ids, so this never fires there. All QLD chunks land in a single qld_legislation collection with the Act identity in metadata, so search spans the whole corpus with optional filter-by-Act.

Serve (MCP)

qld-legis-mcp        # stdio transport

Tools exposed (every search tool takes jurisdiction='qld'|'cth'|'all', default 'all', fanning out across the QLD + Cth stores):

• search_legislation(query, limit=5, act="", kind="", must_contain="", rerank=False, jurisdiction="all") - semantic search; filter by Act and kind, require an exact substring (must_contain, hybrid keyword+semantic), or rerank the top candidates with a cross-encoder for sharper precision (slower).

• keyword_search(text, limit=10, act="", kind="", jurisdiction="all") - exact-substring search, no embedding, near-instant; for specific terms/phrases and broad sweeps.

• find_acts(query, limit=5, jurisdiction="all") - triage over the qld_acts index to identify the governing Act(s) for a broad topic, then drill in with search_legislation's act= filter. Cth Acts join the same index with a jurisdiction tag. (Requires the triage index; see qld-legis-ingest-acts.)

• get_provision(doc_id, ref, context=0, include_definitions=False) - fetch a provision verbatim; optionally with neighbouring sections (context=1 adds s N-1/s N+1) and the dictionary definitions of terms it uses. Routes to the right store by doc-id shape (act-2001-064 vs C2004A03712). No embedding.

• list_acts() - the Acts in the store, with currency date and provision count.

Every hit carries cite_as - the citation including the consolidation currency date (e.g. "... (current as at 2026-04-27)") - and the server's instructions tell the client to cite with it.

Local (stdio) - Claude Desktop launches the server as a subprocess. Point your MCP client at the console script:

{
  "mcpServers": {
    "qld-legislation": {
      "command": "/path/to/.venv/bin/qld-legis-mcp",
      "env": { "QLD_SETTINGS": "/path/to/repo/config/settings.json" }
    }
  }
}

Remote (shareable) - serve over Streamable HTTP so others connect to a URL.

Claude's "Add custom connector" UI accepts only a URL and OAuth - there is no field for a bearer token. So for friends-with-a-key sharing, put the secret in the URL path: serve the endpoint at an unguessable path and the URL itself is the key (wrong path -> 404).

# pick a secret path once, e.g. /mcp-7f3k9x2q.../mcp
QLD_SETTINGS=config/settings.json \
  qld-legis-mcp --http --host 127.0.0.1 --port 8000 --path "/mcp-<random>/mcp"

Then expose 127.0.0.1:8000 through a tunnel that gives a public HTTPS URL (Tailscale Funnel, Cloudflare Tunnel) and share <https-tunnel>/mcp-<random>/mcp as the connector URL. Bind to 127.0.0.1 - let the tunnel terminate TLS; don't expose the raw port.

QLD_MCP_TOKEN additionally requires an Authorization: Bearer <token> header (constant-time checked, 401 otherwise). That works for curl/programmatic clients but not Claude's connector, so it's an optional extra on top of the secret path. --sse selects the legacy SSE transport.

The MCP SDK's DNS-rebinding protection rejects any non-localhost Host with 421, which breaks access through a tunnel. The server therefore disables that check by default when serving HTTP (the tunnel + secret path are the gate). To re-enable it and pin the accepted host(s), pass --allowed-hosts <host> or set QLD_MCP_ALLOWED_HOSTS (comma-separated, e.g. your tunnel hostname).

The server loads bge-m3 and the multi-GB store on the host, so a remote deployment needs a few GB of RAM and disk co-located with it. Query-time embedding is one string at a time, so CPU is fine.

Serve with Docker

The tidiest way to run the remote server. The image bakes in bge-m3 (starts offline, no runtime download) and CPU-only torch; the ChromaDB store is mounted as a volume, never baked in.

1. Edit docker-compose.yml: set the store volume's host path to your chroma directory, and replace CHANGE_ME_SECRET in QLD_MCP_PATH with a random string (python -c "import secrets; print(secrets.token_urlsafe(24))").

2. Build and run:

   docker compose up -d --build
   docker compose logs -f          # watch it start

3. Point your tunnel (Tailscale Funnel / Cloudflare Tunnel) at host port 8000, and share <https-tunnel>/mcp-<your-secret>/mcp as the connector URL.

The container is configured entirely by environment (QLD_DB_PATH, QLD_COLLECTION, QLD_MCP_PATH, QLD_EMBEDDING_DEVICE) - no settings file to manage inside it. The store must have been built by a compatible ChromaDB version; if the container can't read it, match the version that built it.

Configuration

Runtime knobs (base URL, page size, throttle/timeout/retry settings, default output paths, and the ingest db_path / collection / embedding_model) live in a JSON file rather than being hardcoded. Resolution order:

1. --config path/to/file.json on the CLI

2. $QLD_SETTINGS environment variable

3. config/settings.json (gitignored - your local copy)

4. config/settings.example.json (committed default, used as-is if you never create a settings.json)

Copy config/settings.example.json to config/settings.json and edit it to override paths or tuning without touching code.

The public endpoints

1. Enumerate - browse data source

GET https://www.legislation.qld.gov.au/projectdata
    ?ds=OQPC-BrowseDataSource
    &start=<N>&count=50
    &sortField=sort.title&sortDirection=asc
    &expression=<expression>
    &subset=browse

Returns JSON: {"data": [...]}, where each row is a document version record. Fields are wrapped as {"__type__": ..., "__value__": ...} and unwrapped by enumerate.py. Relevant fields: id (e.g. act-2023-027), title, year, no, print.type. Page through with start/count until a page returns fewer rows than count.

Expression syntax - a CCL-like filter string:

PrintType="act.reprint" AND PitValid=@pointInTime("YYYYMMDDhhmmss") AND Repealed="N"

• PrintType - "act.reprint" for consolidated in-force Acts; "reprint" for consolidated in-force Subordinate Legislation (regulations etc.).

• PitValid=@pointInTime("...") - pins results to a single current version per document as at that point in time. Without this clause the source returns multiple historical version rows per Act. We use "now" (i.e. the current in-force version).

• Repealed="N" - excludes repealed instruments.

As of this writing this returns 574 in-force Acts.

The browse source emits several rows per document (different renditions/versions of the same Act); enumerate.dedupe() collapses these to one manifest entry per lowercase id.

2. Fetch - XML rendition

GET https://www.legislation.qld.gov.au/view/xml/inforce/current/{id}

Returns the authorised XML for the current in-force version of that document (e.g. id=act-1984-051). No auth needed.

The official API (documented, not used)

api.legislation.qld.gov.au exposes a /v1/documents + rendition API that takes username/password auth. As of this writing its document and rendition endpoints return 500 / error-code-4900 database errors, so it isn't usable. It's noted here as the "proper" path in case OQPC fixes it - the public website endpoints above are the working path this repo relies on and should keep being used unless the official API is confirmed working again.

The authorised XML structure (QuILLS)

The fetched documents use OQPC's "QuILLS Act" DTD. ingest/parse.py walks it natively (no geometry heuristics). Key tags:

Licensing

The legislation text retrieved by this tool is © State of Queensland, published under [Creative Commons Attribution 4.0 (CC BY 4.0)] (https://creativecommons.org/licenses/by/4.0/). Any redistribution of the downloaded XML/text must retain that attribution. This repository's own code is licensed separately - see LICENSE.

Testing

pip install -e . pytest
pytest tests/

The tests cover the pure logic only (expression building, dedup, filename slugging, and the XML parse/chunk pipeline against a self-contained fixture), so they need no network and none of the heavy ML deps.

Roadmap

Done since the first cut: full-scale QLD ingest (574 Acts), Subordinate Legislation end-to-end, the Act-triage index, cross-encoder reranking, the ONNX int8 backend, the differential updater, and the Commonwealth pipeline with separate-store-dir cross-corpus search. Remaining / next:

• ONNX-quantise for ingest as well as query (currently torch is used for ingest to avoid the int8 attention-matrix OOM on the longest Acts).

• Optionally index the running-header/endnote currency metadata per provision.

• Commonwealth legislative instruments (F* disallowable instruments), the Cth analogue of Subordinate Legislation, once their Epub structure is confirmed to match the principal-Act styles.