DataPulse MY

Live dashboard: https://www.data-pulse.my

Open in Google Colab:

🤖 AI-agent-ready — Wire DataPulse MY into Claude Desktop, Cursor, Cline, or any MCP-compatible client with one config block. Your agent gets 389 official Malaysian datasets — including 30 GTFS transit feeds (KTMB, Prasarana, BAS.MY) — with declared licences and an honest ten-status trust taxonomy instead of a blanket green checkmark.

→ Connect your AI agent in 30 seconds

DataPulse MY is an open-source trust layer for Malaysian public data. It makes official datasets easier to assess and reuse by publishing a small manifest, human-readable health reports, and machine-readable health envelopes.

It does not replace the official source. It documents what is available, whether it is fresh, how its schema behaves, and which collection quirks users need to handle.

Who it is for

• AI builders and AI-agent / LLM-app developers wiring a model to the read-only MCP server so it can check a Malaysian figure's freshness, licence, and provenance before it cites the number.

• AI agents as consumers discovering and querying the catalogue through llms.txt, agent.json, and the read-only MCP endpoint without a browser, receiving the same freshness, licence, schema-drift, and provenance evidence a human reviewer would.

• University students, academics, and researchers using the Colab notebook to see which Malaysian official series is genuinely current versus reference or stale before grounding coursework, a thesis, or a replication.

• Data scientists and analysts distinguishing a series that stopped updating (stale/discontinued) from a lookup/reference table that is fine as-is, before building a model or dashboard.

• Compliance, validation, and regulatory-monitoring teams keeping an evidence/attestation trail (probe timestamp, content date, source, attestation) that an official figure was checked before it appears in a product or external communication.

• Journalists and fact-checkers verifying a fuel price, inflation/CPI, rate, ridership, or policy figure is current before publishing.

• Civic technologists building reproducible pipelines on transparent health metadata.

• Public servants seeing the discoverability and reliability status of the open data their agency publishes.

Related MCP server: Data.gov.my MCP Server

Use this for

• AI builder / agent developer: give your agent a pre-citation trust check — before it answers from a Malaysian dataset, have it call verify_dataset and get_freshness_summary and caveat anything stale instead of asserting currentness.

• AI agent consumer: discover and query the catalogue via llms.txt plus the read-only MCP endpoint, and receive freshness status, licence, schema-drift, and provenance as structured context.

• Student / researcher: ground a coursework or thesis figure in a dataset whose freshness and licence you can actually verify, via the Colab notebook.

• Data scientist / analyst: before modelling a series, tell one that stopped updating (stale/discontinued) apart from a lookup/reference table that is fine as-is.

• Compliance / validation team: keep an evidence trail that an official figure was checked at a known time before it appears in your product or external communication.

• Journalist / fact-checker: before citing a fuel price, inflation/CPI, rate, or ridership figure, check its freshness so you are not publishing a stale number.

• Civic technologist: consume the machine-readable health envelopes in a reproducible pipeline with transparent health metadata.

• Public servant: see how discoverable and reliably described your agency's published open data is.

Dataset health

Health is reported as fresh, aging, stale, discontinued, degraded, browser-dependent, unreachable, unknown, unknown-freshness, or reference. Unknown freshness means the URL and content shape work, but neither a Last-Modified header nor a parseable content date proves when the data was updated. Reference means versioned lookup data is reachable and its record count is measured, while date-based freshness does not apply. The reference family splits into plain reference, policy-reference (policy state that stays valid until superseded — BNM OPR is current while unchanged, not stale), and reference-current (a lookup that must still pass freshness, such as a bank-rate table that can itself go stale). The public _trust_summary shows the distribution and explicitly counts missing freshness and row-count signals.

Discontinued — The source has stopped publishing new data. The data is frozen at the last known content date. This is not a freshness failure — it's a publisher decision.

Current distribution (_trust_summary): 87 fresh · 132 aging · 152 stale · 1 discontinued · 1 degraded · 5 browser-dependent · 11 reference

Subscribe: RSS feed — get notified when dataset health changes.

⚠️ Status: active development. Dashboards and health snapshots update as probes complete. Per-dataset reports under data/{id}.md may briefly lag behind the live health snapshot in health/latest.json (a regeneration gap that is being closed). Coverage and quality improve with each tagged release; expect rough edges. Track progress via the GitHub Releases page — v0.4.0 is the current milestone.

Browser-dependent datasets

Five sources (currently 1.4% of the catalogue) require a real browser to probe because their source pages render client-side JavaScript: eperolehan-diklankan, doe_apims, doe_rqims, doe_mqims, and kkm_idengue.

DataPulse uses Camofox, a self-hosted patched headless-Chromium sidecar, to probe these. The probe path is check.sh → Camofox sidecar → DOM snapshot → content-date extraction.

To enable browser probing:

1. Run the Camofox Docker sidecar on a reachable address (default http://localhost:9377). The probe script and the GitHub Actions workflow pick this up from the CAMOFOX_BASE_URL environment variable; nothing in this repo encodes a public IP.

2. Set CAMOFOX_BASE_URL to that address.

3. Restart the timer with systemctl restart datapulse-health.timer.

Without Camofox, those five datasets will sit at browser-dependent — the honest status: DataPulse cannot probe them without a browser, so it says so rather than failing silently. See scripts/smoke_browser_probes.sh for isolated smoke tests.

Legal

DataPulse probes publicly-published open-data sources. We do not bypass authentication, CAPTCHAs, or terms-of-service restrictions. Every source we probe is publicly available without login; the data is aggregate/non-personal; and the probe respects each dataset's declared refresh frequency.

All scraping is rate-limited (5-minute cadence, dataset-tier cadence applied) and identifies itself via User-Agent. Sources we cannot probe without authentication, CAPTCHA bypass, or ToS violation are marked unreachable or browser-dependent — never silently scraped through a workaround.

If you are a data source maintainer and would like DataPulse to adjust its probe cadence, exclude a dataset, or remove it from the manifest, please open a GitHub issue or contact the maintainers.

AI-agent-ready — what it means for you

Give your organisation's AI tools current, licensed, and verified Malaysian public data without first building a custom integration. DataPulse MY makes the full portfolio discoverable from one self-describing index, ready for agents, RAG systems, and internal knowledge tools to consume.

What being AI-ready gives you

• Zero integration work: an AI agent or LLM/RAG system fetches one llms.txt and can use the entire portfolio immediately — no scraping, API-key setup, or data-format reverse-engineering.

• Honest freshness signals: a 5-minute timer probes datasets when their cadence tier is due, separating HTTP reachability, browser dependency, schema validity, and source freshness so missing evidence is visible instead of being labelled healthy. Consumers can also use anomaly_detected as an explainable, orthogonal freshness-delta signal without changing the ten-status taxonomy.

• Machine-readable and licence-clear: every dataset has a JSON envelope with its schema, licence, and refresh cadence, giving legal and engineering teams the information they need to approve and integrate it.

• Trustworthy for AI: verified official sources and explicit licences let agents cite and use the data without permission ambiguity.

• RAG and knowledge-base ready: drop the envelopes into a retrieval pipeline to ground chatbots and AI tools in current Malaysian public data.

Every manifest dataset declares either CC BY 4.0 or OGL licensing and is assessed with the honest ten-status trust taxonomy. Each entry retains its human-readable steward and supplies a stable custodian ID resolved through custodians.json for publisher-level provenance.

Why this matters now. AI agents are increasingly the buyers of structured public data — they retrieve and act on resources at machine speed, and the proportion of traffic that hits the human web from agents is already displacing the old "crawl → index → human visits → ad impression" bargain that funded open publishers. The workloads that survive that shift are the ones that publish clean, attested, licence-clear resources an agent can consume without re-deriving trust from scratch. DataPulse MY is built for that: one self-describing index, one MCP endpoint, ten honest health statuses, and signed probe attestations — so an agent can answer a question about Malaysian public data without first having to verify the data exists, is current, and is licensed for the use it is about to make of it.

MCP server (read-only)

DataPulse MY also exposes an AI-ready, read-only MCP server so agents can query the catalogue natively:

• Endpoint: https://mcp.data-pulse.my/mcp (Streamable HTTP, no auth) Verified by mcpgrade: 100/100 (Grade A). The canonical tool count lives in mcp.json / agent.json.

• 18 tools: search_datasets, get_dataset, find_stale, find_anomalies, find_deteriorating, find_recovering, find_unreliable, find_schema_drift, check_reconciliation, get_provenance, get_evidence, verify_dataset, get_freshness_summary, verify_evidence, trust_verdict, verify_attestation, find_by_licence, usage_summary

The public endpoint serves all 18 read-only tools over the 389-dataset catalogue.

• 8 resources plus 1 resource template, including datapulse://attestations and the signed daily probe-attestation index.

get_evidence exposes pipeline receipts; verify_evidence performs cached transport-only live checks and does not update health.

Connect from Claude Desktop:

{
  "mcpServers": {
    "datapulse-my": {
      "transport": "streamable-http",
      "url": "https://mcp.data-pulse.my/mcp"
    }
  }
}

See llms.txt for the full discovery index, and docs/mcp-deploy.md for the deployment architecture.

Authenticated buyer API

Paying integrations use the separate, versioned /api/v1/ buyer API. It uses X-API-Key authentication, durable per-key request limits, and audit logs; the public MCP endpoint above remains intentionally unauthenticated. See the buyer API reference for endpoint and operator details.

How to consume the data

• LLM index

• Agent manifest

• MCP advertisement

• Sitemap

• MCP endpoint

For humans wiring their own agents: see the MCP server section below for the Claude Desktop / Cursor / Cline config block, or the full integration guide at docs/mcp-deploy.md.

Wire it into Claude Desktop via claude_desktop_config.json (30 seconds, no API key):

{
  "mcpServers": {
    "datapulse-my": {
      "transport": "streamable-http",
      "url": "https://mcp.data-pulse.my/mcp"
    }
  }
}

Restart Claude Desktop, confirm the hammer icon shows "datapulse-my" with the read-only tools listed above. Cursor / Cline use the same JSON in their MCP config panel.

Included datasets

• Malaysian Fuel Prices — Samples: CSV, JSON

• ePerolehan Tender Notices (DIIKLANKAN) — Sample JSON

• PriceCatcher (Grocery Prices) — Samples: main CSV, item lookup, premise lookup

Daily reference data

Daily-published reference datasets from official Malaysian sources. Each agency publishes on its own schedule, declared in refresh_frequency; the dashboard combines the date-only source value with the publication time and does not infer a time from midnight or UTC conversion.

Bank Negara Malaysia (BNM)

• BNM Daily Exchange Rates (0900) — Sample JSON

• BNM Daily Exchange Rates (1130) — Sample JSON

• BNM Daily Exchange Rates (1200) — Sample JSON

• BNM Daily Exchange Rates (1700) — Sample JSON

• Overnight Policy Rate (OPR)

• Base Rates / BLR / Effective LR

• Kuala Lumpur USD/MYR Reference Rate

• Interest Rates: Banking Institutions

• Interest Volume: Banking Institutions

• Interbank Swap

• Kijang Emas (Gold Reference Price)

• Malaysia Overnight Rate (MYOR)

MET Malaysia

• MET Malaysia Weather Forecast — Samples: CSV, JSON

Department of Environment (DOE)

• DOE APIMS Air Quality (Hourly) — Sample JSON

• DOE RQIMS River Water Quality (Continuous) — Sample JSON

• DOE MQIMS Marine Water Quality (Monthly) — Sample JSON

KKM (Ministry of Health)

• KKM iDengue Weekly Dengue Cases — Sample JSON

OpenDOSM (DOSM open data portal)

• OpenDOSM Crime by District (Annual) — Sample JSON

• OpenDOSM Monthly CPI by State — Sample JSON

• OpenDOSM Annual Real GDP by State — Sample JSON

• OpenDOSM Quarterly Real GDP — Sample JSON

• OpenDOSM Annual Real GDP by Supply Sector — Sample JSON

• OpenDOSM Monthly Trade Headline — Sample JSON

• OpenDOSM Monthly CPI Inflation by Division — Sample JSON

• OpenDOSM Monthly Trade by End Use (BEC) — Sample JSON

• OpenDOSM Quarterly Labour Force Statistics — Sample JSON

• OpenDOSM Quarterly Labour Force Statistics by State — Sample JSON

• OpenDOSM Annual Employment by Sector and Sex — Sample JSON

• OpenDOSM Annual Population by State — Sample JSON

• OpenDOSM Annual Nominal GDP by Supply Sector — Sample JSON

• OpenDOSM Quarterly Nominal GDP — Sample JSON

• OpenDOSM Quarterly Real GDP (Seasonally Adjusted) — Sample JSON

• OpenDOSM Annual Nominal GDP and GNI — Sample JSON

• OpenDOSM Monthly Core CPI Inflation by Division — Sample JSON

• OpenDOSM Monthly CPI Inflation by State and Division — Sample JSON

• OpenDOSM Monthly Producer Price Index — Sample JSON

• OpenDOSM Annual Labour Force Statistics — Sample JSON

• OpenDOSM Monthly Labour Force Statistics — Sample JSON

• OpenDOSM Monthly Trade by SITC Section — Sample JSON

• OpenDOSM IPI for Export-Oriented Divisions — Sample JSON

• OpenDOSM IPI for Domestic-Oriented Divisions — Sample JSON

• OpenDOSM Annual Births by State — Sample JSON

• OpenDOSM Annual Deaths by State — Sample JSON

• OpenDOSM Annual Maternal Deaths by State — Sample JSON

• OpenDOSM Annual Marriages by State and Sex — Sample JSON

• OpenDOSM Household Income, Malaysia — Sample JSON

• OpenDOSM Household Income by State — Sample JSON

• OpenDOSM Household Income by District — Sample JSON

• OpenDOSM Poverty, Malaysia — Sample JSON

• OpenDOSM Poverty by State — Sample JSON

• OpenDOSM Poverty by District — Sample JSON

• OpenDOSM Income Inequality, Malaysia — Sample JSON

• OpenDOSM Income Inequality by State — Sample JSON

• OpenDOSM Income Inequality by District — Sample JSON

• OpenDOSM Household Expenditure by DUN — Sample JSON

• OpenDOSM Household Expenditure by Parliamentary Constituency — Sample JSON

• OpenDOSM Annual Population, Malaysia — Sample JSON

• OpenDOSM Annual Population by Parliamentary Constituency — Sample JSON

• OpenDOSM Annual Deaths by District and Sex — Sample JSON

• OpenDOSM Annual Marriages by State, Age, and Sex — Sample JSON

• OpenDOSM Annual Fertility — Sample JSON

• OpenDOSM Annual Maternal Deaths, Malaysia — Sample JSON

data.gov.my

• data.gov.my Monthly Interest Rates — Sample JSON

• data.gov.my Quarterly Federal Government Revenue — Sample JSON

• data.gov.my Quarterly Federal Operating Expenditure — Sample JSON

• data.gov.my Monthly Money Aggregates — Sample JSON

• data.gov.my Monthly Payment Systems — Sample JSON

• data.gov.my Monthly Payment Instruments — Sample JSON

• data.gov.my Monthly Payment Channels — Sample JSON

• data.gov.my Annual Interest Rates — Sample JSON

• data.gov.my Annual EPF Dividend Rates — Sample JSON

• data.gov.my Monthly Vehicle Registrations by Type and Fuel — Sample JSON

• data.gov.my Daily FPX Transactions — Sample JSON

• data.gov.my Healthcare Staff by State and Staff Type — Sample JSON

• data.gov.my Daily Blood Donations by Blood Group and State — Sample JSON

• data.gov.my Infant Immunisation Coverage — Sample JSON

• data.gov.my Sexually Transmitted Diseases by State — Sample JSON

• data.gov.my Daily PeKaB40 Health Screenings by State — Sample JSON

• data.gov.my Malaysian National Health Accounts Expenditure — Sample JSON

• data.gov.my Monthly Electricity Supply — Sample JSON

• data.gov.my Water Production by State — Sample JSON

• data.gov.my Access to Treated Water by State and Strata — Sample JSON

• data.gov.my Monthly KTMB Ridership — Sample JSON

• data.gov.my Cellular Subscribers by Plan Type — Sample JSON

• data.gov.my Prisoners by State and Sex — Sample JSON

• data.gov.my Drug Addicts by State and Age Group — Sample JSON

• data.gov.my Female Representation in Local Authorities — Sample JSON

• data.gov.my Female Representation in Parliament — Sample JSON

• data.gov.my Monthly Marine Fish Landings by State — Sample JSON

• data.gov.my Crop Area and Production by State — Sample JSON

• data.gov.my Public Education Institutions by District — Sample JSON

GTFS transit feeds

The transport namespace adds 16 GTFS Static schedule ZIPs and 14 GTFS Realtime vehicle-position protobuf feeds for KTMB, Prasarana, and BAS.MY services. Static samples are under samples/gtfs-static/ and realtime snapshots are under samples/gtfs-realtime/.

DataPulse MY currently tracks the portfolio declared in datapulse.json.

Current coverage

Refresh cadence

How to use it

Start with datapulse.json to discover datasets and their official sources. Follow each health_report link for a plain-language assessment, or consume the matching file under data/json/ in an automated workflow.

For example, a data pipeline can inspect status, content_freshness_date, and freshness_signal_source before processing a source, while a researcher can review the known quirks before designing a collection method.

Verify before trust

Every dataset in this catalogue ships with a publicly-signed Sigstore DSSE evidence receipt that an agent can verify offline, without trusting the DataPulse server. An agent (human or MCP) can obtain, for any dataset, the full health row + evidence + signed-receipt-verification in three MCP tool calls or fewer: search_datasets → verify_dataset → get_freshness_summary. The standard offline path is:

cosign verify-blob \
  --bundle https://www.data-pulse.my/data/fuelprice.receipt.sigstore.json \
  --certificate-identity 'https://github.com/r3dz4r/datapulse-my/.github/workflows/deploy-cloudflare-pages.yml@refs/heads/main' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  --certificate-github-workflow-repository r3dz4r/datapulse-my \
  --certificate-github-workflow-ref refs/heads/main \
  --certificate-github-workflow-name "Datapulse MY canonical Pages" \
  fuelprice

For the portfolio-level health bundle, verify /signatures/health.latest.sigstore.json with the exact companion manifest at /signatures/datapulse.json. That signed-manifest snapshot is distinct from /datapulse.json, the current discovery manifest: the latter can change when generated metadata is refreshed. A valid signature proves the integrity of DataPulse's attested observation, not that upstream data is semantically true. Every refresh publishes signed bundles to the public Rekor log.

See notes/2026-08-30-phase-5-verify-before-trust-sovereignty-one-pager.md for the full positioning one-pager (sovereignty framing, no-competitor gap, distribution status).

Monitoring

• The VPS datapulse-health.timer wakes every 5 minutes and runs only the datasets whose cadence tier is due.

• GitHub Actions performs a full weekly probe as a fallback and republishes the generated health, badge, feed, README, catalog snapshot, and delta artifacts.

• RSS feed — available.

• Status badges — available.

• More datasets — planned.

Adopt a dataset

Know a Malaysian public dataset that deserves dependable health metadata? Adopt it: verify its source and licence, document its schema and quirks, and submit a health report. See CONTRIBUTING.md for the expected three-file contribution model.

New contributors can start with the repository's Good first issues or propose a dataset through the GitHub issue forms. Maintainers use good first issue (yellow), adopt-a-dataset (blue), freshness-check (blue), bug (red), documentation (blue), question (purple), and wontfix (gray) to route contributions.

Licence

DataPulse MY is released under the MIT License. Source datasets remain subject to the licences and attribution requirements stated in their individual health reports.

Privacy

See PRIVACY.md for what DataPulse collects (transient operational logs for rate limiting and usage aggregation) and what it does not collect (no credentials, no accounts, no personal data).