ABS Data — Australian Bureau of Statistics data for your AI assistant

Ask the question you actually have. Add one web address to Claude, ChatGPT or Copilot, say what you care about, and your assistant finds the right ABS data, fetches it live, and explains it — with its caveats. Free, anonymous and read-only.

Start here: https://abs-data-front-door.aicolab.workers.dev — setup guides for each AI tool, things to ask, and a feedback form.

An independent, unofficial connector — not an ABS product, and not endorsed by the Australian Bureau of Statistics. Data: ABS, CC BY 4.0.

How it was built

Before offering anything, this project measured what the ABS Data API actually serves — every dataflow, by retrieval — so the connector only ever offers tables and options that have data, and a selection that doesn't exist comes back with the ones that do.

• 621,536,714 series exist across 1,227 dataflows. ABS's metadata implies 14,761,557,818 — a density of 4.21%, overstated 23.8×.

• The per-dimension metadata is exact: every advertised option exists, and nothing real is unadvertised. The entire overstatement is combinatorial.

• 81.0% of series sit in flows whose sparsity is principled and verified in the keys — almost entirely hierarchical geography encoded as separate dimensions (STATE is a function of REGION in 664 flows). The remaining 19% is not reducible to a single-dimension dependency.

• detail=serieskeysonly — the endpoint that should answer "what exists" — is broken (HTTP 200, malformed JSON). Eight of fifteen documented behaviours do not conform. See reports/.

DESIGN.md is the full record: verified API findings, every decision and why, and the corrections made along the way.

Related MCP server: abs

Layout

packages/schema      Drizzle schema (declared vs observed, never joined), shared CSV/period parsing
packages/crawler     structural crawl, observed probe, conformance suite, sizing
packages/catalogue   delta report, sparsity analysis, contract generator, OpenAPI emitter, artifact
packages/contract    the corrected contract: stable Zod verbs + generated tables/options (per crawl)
packages/worker      the front door: MCP door (SDK v2) + HTTP door (oRPC + Scalar) + RPC door (Cap'n Web) on one Worker;
                     src/ops/ holds new operations in @aicolab/app-kit's shape (DESIGN.md decision 34)
packages/site        the pages (landing, /admin) and the MCP App views — Solid 2 + @aicolab/ui-solid, built by Vite
packages/ui-solid    git submodule: the AI CoLab component system (public); packages/solid is its Solid helpers
packages/sdk         @abs/sdk — typed TypeScript client over the RPC door (connect(), invalidSelectionFromError())
tooling/tsconfig     copy of the portal's shared tsconfigs, for the submodules (deleted at the move to app-kit)
reports/             delta.md/json, sparsity.md/json, openapi.json, abs-report-draft.md
data/                local SQLite catalogue (~400GB), raw archive, D1 import SQL — not committed

Running it

git submodule update --init           # packages/ui-solid and packages/solid
pnpm install
pnpm db:migrate                       # local SQLite at data/abs-catalogue.sqlite

pnpm crawl:structural                 # 7 bulk requests, ~1 minute
pnpm crawl:probe -- '--key-values=*'  # every dataflow by retrieval; ~24h, 20GB heap for the giants
pnpm report:delta && pnpm report:sparsity
pnpm generate:contract                # -> packages/contract/src/generated, data/d1/*.sql
pnpm --filter @abs/catalogue emit:openapi

cd packages/worker
npx wrangler d1 execute abs-catalogue --local --file=../../data/d1/00-schema.sql   # then each data file
npx wrangler d1 execute abs-catalogue --local --file=../schema/migrations/0003_public_micromacro.sql  # feedback table
printf 'FEEDBACK_ADMIN_TOKEN=local-dev-token\nFEEDBACK_SALT=local-dev-salt\n' > .dev.vars  # gitignored
pnpm dev                                                                           # builds packages/site, then wrangler dev
ABS_MCP_URL=http://localhost:8787/mcp FEEDBACK_ADMIN_TOKEN=local-dev-token npx tsx test/protocol.ts  # ~80 checks, all doors
pnpm --filter @abs/sdk example                                                     # the SDK against production
ABS_URL=http://localhost:8787 npx tsx test/limits.ts                               # rate limits (spends a minute's budget)

# The in-chat views in a dev-only MCP Apps host, against the local Worker:
pnpm --filter @abs/site dev            # then http://localhost:5180/harness.html

Deploying (pnpm run deploy in packages/worker builds the site first — not pnpm deploy, which is pnpm's own command) needs, once: the feedback migration on the remote D1 (wrangler d1 execute abs-catalogue --remote --file=../schema/migrations/0003_public_micromacro.sql) and two secrets (wrangler secret put FEEDBACK_ADMIN_TOKEN, wrangler secret put FEEDBACK_SALT).

Rate limits, per client address: 600 requests/min overall; 120/min for data pulls (each is a live ABS call); 20/min for Code Mode execute; 5/min for feedback (and 20 per day, in D1). Over the limit, HTTP doors answer 429 with Retry-After; an MCP tool call gets a tool error.

Using the SDK from TypeScript:

import { connect, invalidSelectionFromError } from "@abs/sdk";

const abs = connect(); // https://abs-data-front-door.aicolab.workers.dev/rpc
const [hits, cpi] = await Promise.all([          // one HTTP batch
  abs.searchTables({ query: "cpi rent", limit: 3 }),
  abs.describeTable("CPI"),
]);
try {
  const data = await abs.getData({ table: "CPI", select: { REGION: "Sydney", INDEX: "10001" }, lastN: 4 });
} catch (err) {
  const invalid = invalidSelectionFromError(err); // typed: reason, dimension, validOptions from observation
}

The probe is resumable: rerunning skips flows already recorded as complete. For the largest census flows run with NODE_OPTIONS=--max-old-space-size=20480.

Gotcha: wrangler keys the local D1 state by database_id. Changing the id in wrangler.toml (e.g. after d1 create) gives wrangler dev a fresh, empty local database — re-run the local import.

Principles

• Observed-only. Declared metadata steers the crawler and feeds the defect report; it never enters what a caller sees. Availability is empirical: observed options from the catalogue, exact combination checks live against ABS's availableconstraint (which agrees with the probe on every dimension of every flow).

• One contract, derived doors. Every crawl regenerates one contract; the MCP verbs, the HTTP routes, the OpenAPI document and the docs all read or render it. Nothing hardcodes the surface, so nothing drifts.

• Only calls that return data. A selection is verified before it is forwarded; an invalid one comes back with the observed valid options — as an MRTR input_required on MCP, a typed 422 on HTTP.