sydney-rental-data-pipeline

The data pipeline behind SydneyRealEstateFindings: it searches realestate.com.au through a real Chrome, measures how far every listing is from where Errol needs to be, scores it, publishes its photos, and writes the result into the findings repo's data/. The site next door renders that and nothing else (its docs/adr/0005).

It is also an MCP server, thinly: search_listings and get_listing for asking questions from Claude Code. That used to be the whole program; now it is one subcommand.

SydneyRentalDataPipeline/        this repo
  src/            REA scrape + parse, geocode, route (Valhalla/ORS/Google), TfNSW,
                  images — and cli.ts, mcp.ts, setup.ts
  src/stages/     the run, one module per step, each exporting main(argv)
  src/lib/        scoring, ledger, search planning, walkability, photos/R2
  test/           node --test suites (the former check:* scripts)
  .env            every key: TFNSW_API_KEY, GOOGLE_MAPS_API_KEY, R2_*, …
  writes to  →    ../SydneyRealEstateFindings/data/   (and its public/images mirror)

SydneyRealEstateFindings/        the site: src/ + data/ + .env.local, five self-checks
  packages/schema/               the zod schema both repos depend on

The two checkouts must sit side by side. The schema lives with the data it describes, as the sydney-rental-schema package inside the findings repo, and this one depends on it by path (file:../SydneyRealEstateFindings/packages/schema), so npm links it rather than copying it. FINDINGS_DIR relocates the data, not the code or the package. The moves are recorded in MIGRATION.md (Phase 1) and PHASE2.md (Phase 2).

Requirements

• Node.js 20.12+ (process.loadEnvFile)

• Google Chrome installed (the real browser — see How it works)

• The findings repo checked out as ../SydneyRealEstateFindings, with its own npm install (that is what builds packages/schema, and the linked package resolves zod from there — both repos pin 4.4.3)

Related MCP server: zillow-mcp

Setup

npm install && npm run build
cp .env.example .env        # then fill in the keys — see Configuration
node dist/cli.js setup      # one-time warm-up: a Chrome window opens for a few seconds

Register the MCP adapter with Claude Code (user scope, so it works from either repo):

claude mcp add realestate -s user -- node "E:/Personal Projects/SydneyRealEstate/SydneyRentalDataPipeline/dist/cli.js" mcp

Running the pipeline

One entry point, node dist/cli.js <command>; every npm run script in package.json is an alias for one of them, kept so the run protocol's commands still read the same.

Paths handed to a command — a capture, --out, --cache, --places-out — resolve against the current directory, so prefer absolute ones. The committed runs' captures live at E:/Personal Projects/SydneyRealEstate/captures/.

run — the whole thing at once

node dist/cli.js run --capture="E:/Personal Projects/SydneyRealEstate/captures/<capture>.json"
# … the absence gate prints its table and stops …
node dist/cli.js run --resume

run does capture → absence gate → build → enrich walk · travel · transit → replay → validate in one process, keeping its place in scratch/run-state.json. The replay is the step it exists to stop you forgetting: the enrichers write the ledger, because a routed minute is a fact about a place rather than about a moment, and a replay is what folds their answers into the run just built.

It stops at both of AGENT.md's human gates:

• Absence — it fetches every active-but-absent listing's page, writes proposed verdicts into the capture's gone map, prints the table and stops. "Page gone" and "leased" are the same evidence read two ways, so the reading is yours. Anything it could not reach gets no verdict and is left out of the map. --skip-absence skips the whole step; say so in the commentary.

• Commentary — checked at the end. run will not call a run ready to commit without it.

--dry-run walks all of it and writes nothing. --no-enrich stops after the build. --search is the explicit opt-in to taking a fresh capture, which is a real pass against REA — ask Errol first, as everywhere else here.

The protocol around it — what to commit, and what to say in the commentary — is the findings repo's AGENT.md; the run happens here and commits there, because data/ is versioned there. Before a data commit, this repo's gate is:

npm run typecheck && npm run build && npm test && npm run validate:data

validate:data -- --check-remote also verifies every photo path against R2 when the keys are in. The findings repo has its own, smaller gate for the site.

The MCP adapter

What Claude Code talks to. Interactive use only: the pipeline calls the same functions directly — get_listing is src/lib/listing-detail.ts, which run's absence gate calls too.

geocode_places and route_places used to be tools too. Their only caller was the pipeline, which no longer needs a wire to reach them.

Returned fields

Address (with suburb/state/postcode), price display, bedrooms, bathrooms, car spaces, studies, land and building size, property type, agency (with average rating and review count), agents (name, job title, profile URL), inspection times and auction dates as ISO 8601 with offset, description, photo and floorplan URLs.

On a walk, a route that is really a ferry crossing carries mislabelled — the implied speed that gave it away, the threshold it beat, and confirmation from the timetable that a ferry serves that address. The minutes are left exactly as the router returned them; the flag is the interpretation, kept separate so you can disagree with it. Needs TFNSW_API_KEY, and is skipped without one. On transit answered by TfNSW, each leg's journey reports what the trip is actually made of. See CLAUDE.md for both.

Results include listings REA blends in from neighbouring suburbs; those are flagged isSurrounding: true. They can be across water from the suburb you searched, so filter on each listing's own suburb for anything geographic.

Photos and token cost

get_listing_photos sends real image data into the conversation, which is not free. REA's image URLs carry a {size} placeholder, so resolution is chosen at the CDN rather than downscaled locally. Cost is roughly (width × height) / 750 tokens per image:

Keep limit small. It's a separate tool from get_listing precisely so routine lookups don't drag a dozen images into context.

Note that many REA listings are virtually staged — the furniture is digitally inserted and the listing usually says so in the description. Rooms are real, furnishings are not; use the floorplan for dimensions.

How it works

realestate.com.au is behind Kasada bot protection. Everything in src/ is shaped by that:

• Raw HTTP does not work. Every endpoint — including lexa.realestate.com.au/graphql and services.realestate.com.au — returns 429 with a x-kpsdk-ct header. Those APIs are scoped to signed mobile-app tokens.

• Headless does not work from a cold start. Plain Playwright, Playwright with anti-automation flags, and even patched Chrome all get 429 on a fresh profile.

• What works: patchright driving your real installed Chrome with a persistent profile, warmed up headed once. Kasada issues its challenge, the browser solves it, and the token persists in the profile directory. Headless works from then on.

That's why setup opens a visible window once, and why bundled Chromium isn't used — it's fingerprinted and blocked.

Data is read from the page's server-side hydration blob rather than the DOM:

<script>window.ArgonautExchange={ "resi-property_listing-experience-web": {
  "urqlClientCache": "{\"<hash>\":{\"data\":\"{…}\"}}" }}</script>
     └─> buySearch.results.exact.items[].listing   ← the real Lexa GraphQL objects

Two things that will bite you if you modify the parser:

• The app deletes window.ArgonautExchange after hydration. Reading the live global returns undefined on a page that loaded perfectly fine. Parse the <script> tag's source text.

• Results are split into results.exact and results.surrounding. Reading only exact silently drops up to two-thirds of a page.

Configuration

Everything lives in .env at this package's root — copy .env.example. It is read from the package root whichever directory a command is run from, and whoever starts the process: a script, the CLI, or Claude Code spawning the MCP adapter. What is already in the environment wins over the file, so keep the MCP entry's env block empty. Keys belong in .env; it is gitignored.

The REALESTATE_MCP_* names predate the rename and are hand-maintained in two .env files; they stay.

Photo hosting

Listing photos are not in git. build publishes them to a Cloudflare R2 bucket and mirrors them into the findings repo's public/images/listings/ (gitignored there) so validate can check them and so a re-upload never means re-downloading from REA.

The findings repo's PLAN.md §2 originally committed photos to the repo, with §11 holding an escape hatch open — every image path in JSON is site-relative and every render goes through the site's src/lib/images.ts, so moving the files costs one env var and zero JSON changes. That hatch was taken before the first real run rather than after, because photos entering git history is the one part that would have been expensive to undo. R2 rather than the alternatives because egress is free, and the site's whole job is serving the same photos to phones over and over.

One-time setup

1. Bucket — Cloudflare dashboard → R2 → Create bucket, any name — it goes in R2_BUCKET. Location Automatic (or APAC).

2. Public access — one of:

   • Public Development URL (bucket → Settings): free, instant, gives https://pub-<hash>.r2.dev. Cloudflare rate-limits it and labels it development-only — fine for a handful of viewers.

   • Custom domain: needs a domain on Cloudflare. Proper CDN caching, no rate limit. Better if you have one.

3. API token — R2 → Manage API tokens → Create API token, permission Object Read & Write, scoped to that bucket. Copy the Access Key ID and Secret Access Key; the secret is shown once.

4. Account ID — on the R2 overview page.

5. .env at this package's root (gitignored, never reaches Vercel) — copy .env.example and fill:

   R2_ACCOUNT_ID=…
   R2_ACCESS_KEY_ID=…
   R2_SECRET_ACCESS_KEY=…
   R2_BUCKET=<your-bucket>
   R2_PUBLIC_BASE_URL=https://pub-<hash>.r2.dev

6. Verify — npm run check:r2. It uploads a real WebP, reads it back over the public URL the way a phone would, and cleans up. The public read is the part worth having: a token can write perfectly while public access is still off, and you would not find out until photos 404 on the live site.

7. Site env — set NEXT_PUBLIC_IMAGE_BASE_URL in the findings repo (.env.local, and its Vercel project) to the same public base URL.

No CORS configuration is needed: photos are rendered with plain <img> tags, not fetched.

build refuses to run without R2 configured, rather than writing image paths that would resolve to nothing (--local-images skips uploading, for local testing only). A photo is only recorded once it has uploaded and been mirrored — upload happens first, so a failed upload leaves nothing behind for a later run to mistake for a real file.

Development

Two configs, one command. npm run build runs both: tsconfig.json compiles src/ to dist/, which is what the CLI and the MCP adapter run, and tsconfig.test.json compiles src/ and test/ together to dist-test/, which node --test runs. npm run typecheck is the second one with --noEmit. Nothing compiles TypeScript at run time any more — there is no tsx here.

npm test is node dist/cli.js check: 174 assertions over the scoring model, the ledger merge, search planning, walkability geometry and the transit classifier. They print their working out as well as passing, because that is what they are for — read check:scoring's arithmetic before trusting a run's numbers.

Everything under src/stages/ and src/lib/ came from the findings repo's scripts/ and still computes what it computed there. The proof is replay reproducing both committed runs byte for byte, which is worth re-running after any change under src/lib/:

node dist/cli.js replay "E:/Personal Projects/SydneyRealEstate/captures/2026-08-24-walk15.json"    --run-id=2026-08-24a
node dist/cli.js replay "E:/Personal Projects/SydneyRealEstate/captures/2026-08-24-transit25.json" --run-id=2026-08-25a
git -C ../SydneyRealEstateFindings diff --stat data/     # must print nothing

(git status there will still flag the two files — the script writes LF and the working copy is CRLF. git checkout -- data/runs/ puts them back.)

The MCP adapter runs compiled JS as a child of Claude Code — so edits to src/ do nothing until that process is replaced:

./scripts/reload-mcp.ps1

Rebuilds, kills any Chrome holding the browser profile, kills the server process(es), and reports whether the profile is still warm. -CheckOnly inspects without changing anything; -NoBuild skips compilation. It aborts before killing anything if the build fails.

It only targets node processes referencing this repo's dist/cli.js and Chrome processes referencing the realestate-mcp profile — your normal browser is untouched.

Three different actions for three different changes:

The MCP tool list is negotiated once during the connection handshake and cached for the session. Killing the server picks up new code immediately, but a newly added tool stays invisible until Claude Code restarts.

setup is independent of both — it warms a Kasada token into the profile directory, which survives restarts and reboots.

One profile, one process. Chrome takes an exclusive lock on its user-data-dir. capture uses the same profile as the MCP adapter on purpose (it is the warm one), so a capture cannot run while Claude Code has the server up — disconnect it, or run the reload script first. Ad-hoc scripts that call fetchPage should point at their own profile:

$env:REALESTATE_MCP_PROFILE = "<scratch dir>"

Verify new URL filters by result count, not status code. REA silently ignores malformed filter segments and still returns HTTP 200 with a normal-looking page. See CLAUDE.md for the segment grammar.

Troubleshooting

"Blocked by realestate.com.au bot protection" — the profile has gone cold. Re-run node dist/cli.js setup. Token lifetime is not documented; expect to re-warm occasionally. Check current state without changing anything with ./scripts/reload-mcp.ps1 -CheckOnly.

"Could not open the browser profile — another process is using it" — Chrome allows one process per profile directory. Something else holds it: a setup run still open, a stray Chrome, a capture in progress, or the MCP adapter Claude Code spawned. ./scripts/reload-mcp.ps1 clears it.

setup appears to do nothing, and tools stay blocked — this was a real bug, now fixed. The server used to hold the profile lock for its whole lifetime, so setup in another terminal could never acquire it, and the running Chrome kept serving the cookies it loaded at startup. The browser now closes after REALESTATE_MCP_IDLE, and fetchPage retries once with a fresh context when it hits a block.

"Could not launch Google Chrome" — install Chrome, or set REALESTATE_MCP_CHANNEL=msedge.

Warm-up fails repeatedly — rapid requests raise your Kasada score and it takes a while to decay. Wait a few minutes, or try from a different network.

A filter seems to be ignored — it probably is. REA drops malformed filter segments silently and still returns 200. Compare totalResults against an unfiltered search.

Cannot find module 'sydney-rental-schema' — the findings repo is not checked out beside this one, or has not had npm install run in it. This package depends on file:../SydneyRealEstateFindings/packages/schema, and that install is what builds the dist/ the link points at. FINDINGS_DIR relocates the data, not the package.

the test suites are not built — npm run build. It builds both configs; dist-test/ is where node --test looks.

"R2 is not configured — missing … in the pipeline's .env" — fill the five R2_* lines in .env (see Photo hosting). build and reset refuse without them; validate only needs them with --check-remote.