Gyeongsangbuk-do Data Compass

A hackathon vertical slice that queries population, local tax, policy, and council data for Gyeongsangbuk-do and its cities/counties through purpose-built MCP tools, presenting conclusions, evidence, sources, and limitations together.

[!IMPORTANT] Operational data and basic queries use only official_observed. When using results, you must also verify the original source and reference date. demo_sample operational rows and automatic fallback have been removed.

Implementation Scope

• Responsive Korean web UI with 4 representative questions

• MCP server based on the official @modelcontextprotocol/sdk (local stdio + Streamable HTTP for hosting)

• HTTP API and demo orchestration that call the same tools

• 10 tools: population/local tax trends, cross-comparison, policy eligibility explanations, council search, press release search, per-member bill search, claim verification, source lookup, and evidence-constrained administrative document drafting

• CP949/UTF-8 CSV and XLSX ETL that identifies files by header/content rather than relying on NFD filenames

• KRW unit normalization, quality validation, isolation, and official-first querying for 9 local tax source grains and the Ministry of the Interior and Safety HWPX yearbook

• SHA-256 catalog generation for all provided CSV, PDF, and XLSX files

• Crawl of 13,707 Gyeongsangbuk-do press releases (2020~2026) with a search tool for descriptive context only

• Crawl of 64 Gyeongsangbuk-do Council members, 1,454 proposed bills, and 24,178 sponsor relationships, with a per-member bill search tool that separates lead sponsors from co-sponsors

• PDF.js-based page extraction, chunking, and local embedding loading for 408 meeting minutes, audit reports, and provincial administration work plans

• Page-level Korean plausibility/mojibake indicators, repeated header removal, speaker/table boundary chunking, and golden regression evaluation

• Lifecycle that searches only approved PDF processing versions, purge based on retention period and confirmation tokens, and capacity reporting

• PostgreSQL normalized schema, migrations, and Docker Compose

• PostgreSQL hybrid repository combining Korean FTS/trigram with stored embeddings

• Feature-flag-based PostgreSQL MCP repository, readiness, and redacted tool audit logging

• Vitest-based domain, search, ETL, tool, and HTTP tests

Quick Start

Requirements: Node.js 22 or later, pnpm 10 or later

pnpm install
pnpm test
pnpm build
pnpm start

The UI is the Next.js app in prototype/, and 8787 forwards everything except its own paths (/mcp, /health, /ready, /api/*) to it. Run both together.

pnpm --filter @gb/prototype dev                # 3100
PROTOTYPE_ORIGIN=http://127.0.0.1:3100 pnpm start   # 8787

Open http://localhost:8787 in a browser. If PROTOTYPE_ORIGIN is empty, UI paths return 404 and only MCP and API remain.

DATA_REPOSITORY=auto is the default, and startup fails if DATABASE_URL is missing or PostgreSQL is not ready. Explicit memory mode starts with an empty repository. The server reads .env automatically. The local HTTP auth default is HTTP_AUTH_MODE=off and must not be exposed externally. The server bind default is also HTTP_HOST=127.0.0.1; only Docker explicitly uses 0.0.0.0.

Development mode:

pnpm dev

Docker

HTTP_AUTH_TOKEN="$(openssl rand -hex 32)" docker compose up --build

• Web/API: http://localhost:8787

• PostgreSQL: localhost:5432

• Initial schema: db/schema.sql

Compose enforces HTTP_AUTH_MODE=token and will not start without a token. Production deployments should sit behind a reverse proxy that terminates TLS, with the token injected via a secret manager or deployment secret. Do not commit tokens to the repository.

To apply migrations to an existing PostgreSQL:

pnpm db:schema

DATABASE_URL is read from environment variables or the repository's .env; the command output does not expose the connection string.

MCP stdio Configuration

First run pnpm install && pnpm build, then register with your MCP client as follows.

{
  "mcpServers": {
    "gyeongbuk-public-ai": {
      "command": "node",
      "args": [
        "/absolute/path/to/gyeongbuk-public-ai/services/mcp-server/dist/index.js",
        "--stdio"
      ]
    }
  }
}

During development, you can run the stdio server directly with the following command.

pnpm --filter @gb/mcp-server dev:stdio

The server uses stdout exclusively for JSON-RPC. HTTP authentication and rate limit settings do not apply to the local stdio transport.

MCP Remote Hosting (Streamable HTTP)

Running in HTTP mode serves the MCP standard Streamable HTTP transport at the /mcp path. It exposes exactly the same tool list as stdio.

HTTP_HOST=0.0.0.0 \
PORT=8787 \
HTTP_AUTH_MODE=token \
HTTP_AUTH_TOKEN=<32바이트 이상 토큰> \
DATABASE_URL=postgresql://... \
DATA_REPOSITORY=postgres \
node services/mcp-server/dist/index.js --http

Remote MCP client registration:

{
  "mcpServers": {
    "gyeongbuk-public-ai": {
      "type": "http",
      "url": "https://<host>/mcp",
      "headers": { "Authorization": "Bearer <HTTP_AUTH_TOKEN>" }
    }
  }
}

• For public hosting, HTTP_AUTH_MODE=token must be set. On auth failure, no session is created and the connection is closed with 401.

• If both MCP_ALLOWED_HOSTS and MCP_ALLOWED_ORIGINS are empty, DNS rebinding defense is disabled. Set them unless you are behind a trusted proxy.

• Protocol traffic on /mcp consumes the cheap budget, and expensive tools among tools/call additionally consume the expensive budget. See docs/CHANGES_HOSTED_MCP_SERVER.md for details.

HTTP API

curl http://localhost:8787/health

export HTTP_AUTH_TOKEN="$(openssl rand -hex 32)"
HTTP_AUTH_MODE=token pnpm start

curl -X POST http://localhost:8787/api/agent \
  -H "authorization: Bearer $HTTP_AUTH_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"question":"포항시는 2018년부터 2024년까지 인구는 줄었는데 지방세 세수는 늘었나?"}'

curl -X POST http://localhost:8787/api/tools/search_policies \
  -H "authorization: Bearer $HTTP_AUTH_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"region":"상주시","age":34,"query":"스마트팜","stage":"예비창업"}'

curl -X POST http://localhost:8787/api/tools/search_council_records \
  -H "authorization: Bearer $HTTP_AUTH_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"query":"지방소멸","dateFrom":"2020-01-01","dateTo":"2024-12-31","limit":5}'

# 보도자료는 서술적 맥락 전용이라 dataStatuses를 받지 않는다
curl -X POST http://localhost:8787/api/tools/search_press_releases \
  -H "authorization: ******" \
  -H 'content-type: application/json' \
  -d '{"query":"청년 일자리","dateFrom":"2024-01-01","limit":5}'

# 의원별 의안. 대표발의(lead)와 공동발의(co)는 절대 합산하지 않는다
curl -X POST http://localhost:8787/api/tools/search_member_bills \
  -H "authorization: ******" \
  -H 'content-type: application/json' \
  -d '{"member":"김대진","query":"산림","role":"lead","limit":5}'

All factual responses include top-level data_status, data_statuses, sources, and limitations. The input's optional dataStatuses can specify official_observed/demo_sample, but the default allows official data only. If more than one actual status is returned, data_status is "mixed" and individual statuses are preserved in data_statuses and records. However, derived statistics are not considered safe based on label alone. summary is computed only when there are two or more points where dataStatus, the metric's accounting basis, amount meaning, source grain, and raw unit all match, and for population the measurement basis and reference date precision also match. Otherwise, summary: null and summary_unavailable_reason are returned while preserving the original points and homogeneous series_segments. Population/local tax match verdicts are generated only when both sides are comparable official series.

HTTP Access Control

• HTTP_AUTH_MODE=token requires Authorization: Bearer or X-API-Key on /api/**. The token must be at least 32 bytes, and weak placeholders are rejected at startup.

• /health, the minimal-information /ready, and the static UI are public. In token mode, the public UI does not put the API token in the browser, so API call functionality does not work without reverse proxy auth integration.

• Token comparison uses a SHA-256 fixed-length digest with constant-time comparison; tokens/raw bodies are never stored in logs or audit tables.

• The per-client in-memory token bucket defaults to 120 cheap calls per minute, 12 expensive calls (PDF hybrid/demo/draft, etc.), and 60 readiness calls. Exceeding the limit returns 429 with Retry-After.

• The client is the socket IP by default. X-Forwarded-For is ignored. Only when HTTP_TRUST_PROXY=loopback or explicit is set with HTTP_TRUST_PROXY_IPS to precisely trust the immediate proxy is the validated first IP used.

• JSON POSTs apply application/json, a default 64KiB limit, and depth 12/node 5,000 limits. Fixed route/method/tool allowlists and CSP/frame/referrer/permissions security headers are used, and cross-origin wildcard CORS is not provided.

• All responses return a safe X-Request-ID. Only strictly formatted inbound IDs are accepted and linked to the PostgreSQL audit correlation_id.

• /api/admin/storage is available only in token mode. The public /ready does not expose DB size, version, or PostgreSQL settings.

• Node request/operation timeouts and PostgreSQL connection/statement/query timeouts are tunable via .env.example variables. Default DB statement/query timeouts are 8s/10s.

Runtime Repository

All SQL in PostgreSQL mode uses purpose-built parameterized/bounded queries. The pool is closed on shutdown, and tool execution records only a salted SHA-256 of the request body plus field types/lengths to mcp_tool_audit on a best-effort basis. Audit write failures remain operational warnings but do not turn successful read results into failures. If cross-deployment hash correlation is needed, pin AUDIT_HASH_SALT from a secret manager.

Source Data ETL

Source files are treated as read-only and are never copied or modified.

# 파일을 쓰지 않고 전체 인벤토리·파싱 결과 확인
pnpm etl:dry-run

# data/generated/에 결정적 JSON 산출물 기록(같은 입력이면 unchanged)
pnpm etl

# 공개 포항시의회 의안 갱신(약관 검토 필수; robots 확인 불가 시 별도 승인)
pnpm pohang:crawl -- --acknowledge-terms --acknowledge-robots-unavailable

# 경상북도 보도자료 수집(서술적 맥락용; 통계 근거가 아님)
pnpm gb:press:crawl --accept-terms --since=2020-01-01

# 경상북도의회 의원 명부와 발의의안 수집(약 25분)
pnpm gb:council:crawl --accept-terms

# ETL 재생성 + 스키마 적용 + 구조화 데이터와 PDF 전체 적재
pnpm db:normalize

# 이미 생성한 산출물을 적재(제한된 DB 사용자는 스키마 적용 생략 가능)
pnpm db:load -- --skip-schema

# PDF만 페이지 추출/chunk/로컬 임베딩 후 적재
pnpm pdf:normalize

# 개발·테스트용 파일 제한
pnpm pdf:normalize -- --limit 5

# golden page와 corpus 품질 분포 평가; report는 data/generated/에 기록
pnpm pdf:evaluate
pnpm pdf:evaluate -- --limit 5

# PDF 단계를 생략하거나 db:normalize 안에서 제한
pnpm db:normalize -- --skip-pdf
pnpm db:normalize -- --pdf-limit 5

New Data Intake

Newly received source files should not be placed directly in datasets/; put them in data/intake/inbox/. They go through six stages: intake → validation → duplicate determination → storage → processing → ledger recording.

# 실제 이동 없이 처리 계획만 확인
pnpm intake:dry-run

# 접수 실행(저장 후 정규화·적재까지)
pnpm intake

# 원장 요약(접수/중복/격리 건수)
pnpm intake:status

• Validation inspects the actual byte signature, not the extension. HTML error pages saved as .csv or .pdf files that are not PDFs are quarantined with a reason sidecar.

• Duplicate determination checks against SHA-256 across the entire source repository, so re-receiving the same file does not cause double loading.

• Storage location is determined by the identified content type, not the filename.

• All determinations are recorded in the raw_intake_file ledger for post-hoc traceability.

The ETL performs the following:

1. Catalogs all files with path (NFC-normalized), size, SHA-256, reference date/collection time, and format

2. Interprets CSV as UTF-8 first, falling back to CP949 on failure

3. Preserves the 년월 date in Pohang's yearly population data as-is and identifies various council bill CSVs

4. Finds 2026년 7월 in the population XLSX title and normalizes Gyeongbuk/city-county values as of 2026-07-31

5. Recursively scans datasets/new/ and identifies the 9 local tax grains and separate quarantine targets by CSV headers or HWPX internal chapter/table markers, not filenames

6. Converts explicit won/thousand-won units to KRW integers, and determines Gimcheon's unmarked payment amounts as thousand-won only when there is an exact match across 6 tax items with the explicitly marked thousand-won collection data

7. Records raw/cleaned page hashes, extraction method, and quality indicators for PDF sources, promoting only readable official documents to official_observed

8. Removes only high-confidence repeated headers/footers across documents and chunks speaker transitions and table rows atomically

9. In addition to the Unicode Korean ratio, checks administrative terms, valid bigrams, particles, rare syllables, and whitespace/punctuation to distinguish extracted_ok, mojibake_suspected, unusable_for_retrieval, and empty

10. Preserves only hash, classification, and reason for excluded pages; they are not included in chunk/embedding/search

11. Exposes parsing errors and local tax quality issues via errors.json, local-tax.quality.report.json, and exit codes

data/generated/ is regenerable and therefore excluded from Git.

PostgreSQL Loading

The loader guarantees:

• Canonical region keys for Gyeongsangbuk-do and the current 22 cities/counties (excluding Gunwi County)

• Source version preservation based on path+SHA-256. The current run loads 977 sources into the catalog, including the HWPX yearbook, additional separate PDFs, press releases, and council bills

• Parameterized upserts in 500-row batches and single-fact transactions

• Deterministic council record IDs based on source SHA/path/row number for general CSVs and on original record IDs for Pohang public bills, plus source_locator for population and council data

• Press releases identified by board post ID (gb-press-<B_STEP>) so re-crawls do not add rows

• Gyeongsangbuk-do Council bills identified by the site's 32-hex bill_uid. Bill numbers are reused across terms, so they are not used as keys; sponsors are loaded separately by role (lead/co)

• Upserts without duplicates on re-run, and existing rows absent from a new run are not deleted

• Fact transaction rollback on failure and ingestion_run.status = failed recording where possible

• Unresolved regions/invalid rows are recorded in the reject count and error report without fabricating arbitrary IDs

• PDF pages/chunks get deterministic IDs from source version, extractor/chunker version, page, and sequence; previous rows are not deleted, but chunks absent from the current extraction are excluded via is_current = false

• Search uses only approved_current chunks from the single approved version in the lifecycle

• Embeddings are loaded into PostgreSQL per generation batch so the full vector corpus is never held in memory

• Local tax totals are defensively selected in the order of same-scope municipal collection receipts, city/county tax subtotals from the Ministry of the Interior and Safety yearbook, total local tax from the per-capita burden source, and payment amounts, preserving amount_semantics and source_grain

• Combined provincial tax + city/county tax totals are preserved with headlineEligible=false and excluded from the tool's city/county tax headline

• The operational load plan creates only official_observed and catalog_only; it does not generate demo_sample

Expected current row counts for the provided sources and samples (document_chunk/embedding based on the is_current join):

The 2021, 2023, 2025, and 2026 provincial administration work plans in datasets/extra/ are searchable as official page passages. The 2020 and 2022 plans have CID mojibake, and the 2024 plan has no usable text layer, so only catalog/provenance and exclusion reasons are preserved; they are not included in search.

The overlap between official population and official headline local tax is 16 region-years/5 regions (Bonghwa 20172024, Gumi 20232025, Cheongdo 20202022, Gyeongsan 2025, Pohang 2024). Gumi 20232025 and Cheongdo 20202022 form official time series using the population stated in the tax-year reports. These figures grow as new data is loaded, so tests do not pin values; they only verify that the overlap actually exists in both tables. The previous representative period 20222024 was dropped due to insufficient endpoints, and the representative question was changed to the Cheongdo case of 2020~2022 where official comparison is possible.

Gyeongsangbuk-do press releases are descriptive context, not statistical facts, so they are stored in a separate press_release table outside the fact model.

Gyeongsangbuk-do Council member and bill statistics are not facts but legislative activity records, so they are stored separately in council_member / council_bill / council_bill_sponsor.

[!IMPORTANT] The existing council_record is the Pohang City Council meeting minutes, and the three council_* tables are for the Gyeongsangbuk-do Council. They are different institutions, so they must not be joined or summed. Summing lead sponsorship (lead) and co-sponsorship (co) also overestimates legislative activity.

SELECT data_status, count(*) FROM data_source GROUP BY data_status;
SELECT data_status, count(*) FROM population_observation GROUP BY data_status;
SELECT data_status, count(*) FROM local_tax_revenue GROUP BY data_status;
SELECT severity, issue_code, count(*) FROM data_quality_issue
GROUP BY severity, issue_code ORDER BY severity, issue_code;
SELECT data_status, count(*) FROM council_record GROUP BY data_status;
SELECT content_classification, count(*) FROM press_release
GROUP BY content_classification;
SELECT entity_type, count(*) FROM source_locator GROUP BY entity_type;
SELECT text_class, count(*) FROM document_page WHERE is_current GROUP BY text_class;
SELECT content_classification, count(*) FROM document_page
WHERE is_current GROUP BY content_classification;
SELECT is_current, count(*) FROM document_chunk GROUP BY is_current;

PDF source lifecycle and storage operations

New extractors/chunkers are registered as candidate without automatically changing production search. Approval/rollback switches visibility in a single transaction after confirming that retained chunks and embeddings are complete. Past version rollback can temporarily hide new documents that did not exist at the time, so returned document/page/chunk coverage must be verified.

# version, current/stale 행, relation/index 크기, 안전한 DB 설정과 경고
pnpm lifecycle:status

# candidate 승인 또는 retained retired version rollback
pnpm lifecycle:approve -- \
  --version 'extractor::chunker' \
  --actor operator-name \
  --reason 'Golden/corpus evaluation passed'

# active approved version은 직접 retire할 수 없음
pnpm lifecycle:retire -- \
  --version 'extractor::old-chunker' \
  --actor operator-name \
  --reason 'Superseded and retained for rollback'

# 기본은 30일 보존 dry-run. 출력의 exact token 없이는 삭제 불가
pnpm lifecycle:purge -- --actor operator-name --retention-days 30
pnpm lifecycle:purge -- \
  --execute --confirm PLAN_TOKEN --batch-size 500 \
  --max-rows 10000 --max-runtime-seconds 60

# 만료 audit도 먼저 dry-run하고, 삭제 전 일별 aggregate를 보존
pnpm lifecycle:audit-purge -- --actor operator-name

# 기본은 allowlist 테이블 ANALYZE-only, 최대 10분
pnpm lifecycle:maintain -- --actor operator-name

Purge uses advisory locks, a minimum retention period, approved/current protection, FK-safe batches, and an exact plan hash. If the candidate changes, existing tokens are not executed. VACUUM is not run automatically and requires both --vacuum --confirm-vacuum. The previously observed 53100 is shared memory exhaustion, so do not blindly retry; check /dev/shm, shm_size, disk, maintenance_work_mem, and parallel maintenance settings.

After approving a new policy extraction version, the previous 23,795 and legacy 17,487, totaling 41,282 stale chunks/embeddings are preserved. The actual stale corpus is not deleted until operator approval. Purge does not delete base document/data_source provenance; it only cleans up per-version processing status. Physical space reclamation after logical deletion is the responsibility of a separate maintenance-window VACUUM.

PDF embedding and search

The default provider is the 384-dimensional local/hashed-korean-token/1-d384, which does not use the network. It is a retrieval-only lexical vector created by feature-hashing Korean words and 2/3-character tokens, and does not claim semantic embedding quality. Embeddings with the same content hash are not regenerated on re-execution.

# 기본: 로컬, 무료, 결정적
pnpm pdf:normalize -- --provider hashed

# 명시적으로 외부 전송을 허용한 경우에만 OpenAI 사용
OPENAI_API_KEY=... pnpm pdf:normalize -- \
  --provider openai --allow-external

Since dimensions can differ by provider, embeddings are stored not as JSONB but as document_chunk_embedding.embedding real[] with a dimension check. The current environment does not require pgvector ANN; it uses the pg_trgm and pgcrypto extensions installed by the schema and PostgreSQL FTS to produce up to 200 lexical candidates, then performs cosine reranking. The dedicated PostgresHybridSearchRepository is connected to the council search of the PostgreSQL MCP repository and returns structured bills and PDF chunks together.

Full local execution results are 408 PDFs, 12,522 pages, and 24,565 current chunks/embeddings. Pages are extracted_ok 12,178, mojibake_suspected 172, unusable_for_retrieval 166, empty 6, and 0 chunks were generated from excluded pages. Re-scanning only the existing 401-document corpus, all 11,712 pages are extracted_ok. Repeated header lines (543 lines) were removed.

pnpm pdf:evaluate evaluates 6 local golden pages with short markers stored and the full corpus distribution, producing data/generated/pdf.quality.report.json. Golden failures, failed pages exceeding 1%, excluded-page chunk contamination, and policy documents that are readable but fail to verify year/document-type evidence are treated as severe regressions.

Quality commands

pnpm lint
pnpm typecheck
pnpm test
pnpm build
pnpm pdf:evaluate

Repository structure

prototype/                Next.js 화면(8787이 여기로 넘긴다)
services/mcp-server/      MCP 서버(stdio · Streamable HTTP) + HTTP API
packages/domain/          Zod 입력, 공통 타입, 분석·초안 로직
packages/retrieval/       한글 lexical/hybrid 검색
packages/evaluation/      평가 결과 판정
pipelines/                원천 파일 인벤토리와 정규화
db/                       PostgreSQL 스키마와 마이그레이션
services/mcp-server/test-fixtures/demo/  demo 거부 동작만 검증하는 테스트 fixture
evals/                    대표 평가 질문
docs/                     설계·도구·데이터 사전·변경 기록

Current limitations

• Re-collecting the bodies of the 2020, 2022, and 2024 work plans is outside this scope. Without a future separately approved source replacement or OCR decision, search coverage is not claimed.

• Press releases are narrative announcements published by the province, not observed statistics. The figures in the body are plans as of the announcement date, so they cannot be used as evidence for verify_claim_with_data; mentioned_regions only means that city/county names appear in the body, not jurisdiction.

• The golden set covers only 6 pages of 5 local PDFs and contains no actual repeated-footer samples, so it is verified only with synthetic unit tests.

• The default embedding is a lexical feature hash, not a semantic model, and does not use pgvector ANN.

• The numbers and policy/meeting content in the sample are not official facts.

• Growth rates are not calculated across demo and official values, or across different monetary meanings/accounting standards/grain/units. Demo values cannot be used for claim verification or as public-document evidence.

• The official local tax total covers all current 22 cities/counties as of 2024, and combined with the existing CSV period totals 46 region-years. Multi-year homogeneous series still exist only in some regions such as Gyeongsan, Gumi, Bonghwa, and Cheongdo, so growth rates are not calculated for regions with only a single point.

• The 720 rows of Yeongcheon tax exemptions and reductions were not loaded because the meaning of the amount/count columns is unverified.

• The 40,694 rows of standard assessed values for general buildings by address in Gimcheon were excluded from general analysis, search, and embedding.

• The mere fact that population and local taxes moved in opposite directions does not establish causation.

• The token is a single shared secret; there is no tenant/role permission model.

• Rate limits are process-local, so multiple replicas require a shared Redis/gateway.

• TLS certificates and secret rotation are the responsibility of the reverse proxy/deployment platform.

• The 41,282 stale items are in safe retention; actual purge and physical space reclamation require operator approval/inspection.

For details, see Architecture, Data Dictionary, and MCP Tool Specification.

License

The code is licensed under the MIT License. The licenses of the source public data follow the conditions of each providing institution, and this repository does not include the source datasets/.