DataPulse MY

Live dashboard: https://www.data-pulse.my

Open in Google Colab:

🤖 AI-agent-ready — Wire DataPulse MY into Claude Desktop, Cursor, Cline, or any MCP-compatible client with one config block. Your agent gets 375 official Malaysian datasets — including 30 GTFS transit feeds (KTMB, Prasarana, BAS.MY) — with declared licences and an honest nine-status trust taxonomy instead of a blanket green checkmark.

→ Connect your AI agent in 30 seconds

DataPulse MY is an open-source trust layer for Malaysian public data. It makes official datasets easier to assess and reuse by publishing a small manifest, human-readable health reports, and machine-readable health envelopes.

It does not replace the official source. It documents what is available, whether it is fresh, how its schema behaves, and which collection quirks users need to handle.

Who it is for

• Journalists and researchers checking whether a public dataset is usable.

• Civic technologists building reproducible data pipelines.

• Public servants improving the discoverability and reliability of open data.

• Developers who need stable, machine-readable dataset health metadata.

Related MCP server: Data.gov.my MCP Server

Use this for

• Journalist fact-check: before citing a fuel price figure, check fuelprice freshness to make sure it's current.

• Pipeline health gate: fail the build when a required dataset probe has remained unavailable for more than 24 hours.

• RAG knowledge base: consume the JSON envelopes as structured context for a chatbot answering "what's the latest BNM rate?"

Dataset health

Health is reported as fresh, aging, stale, discontinued, degraded, browser-dependent, unreachable, unknown, unknown-freshness, or reference. Unknown freshness means the URL and content shape work, but neither a Last-Modified header nor a parseable content date proves when the data was updated. Reference means versioned lookup data is reachable and its record count is measured, while date-based freshness does not apply. The public _trust_summary shows the distribution and explicitly counts missing freshness and row-count signals.

Discontinued — The source has stopped publishing new data. The data is frozen at the last known content date. This is not a freshness failure — it's a publisher decision.

Current distribution (_trust_summary): 85 fresh · 59 aging · 73 stale · 139 discontinued · 5 browser-dependent · 1 unreachable · 13 reference

Subscribe: RSS feed — get notified when dataset health changes.

⚠️ Status: active development. Dashboards and health snapshots update as probes complete. Per-dataset reports under data/{id}.md may briefly lag behind the live health snapshot in health/latest.json (a regeneration gap that is being closed). Coverage and quality improve with each tagged release; expect rough edges. Track progress via the GitHub Releases page — v0.4.0 is the current milestone.

Browser-dependent datasets

Five sources (currently 1.4% of the catalogue) require a real browser to probe because their source pages render client-side JavaScript: eperolehan-diklankan, doe_apims, doe_rqims, doe_mqims, and kkm_idengue.

DataPulse uses Camofox, a self-hosted patched headless-Chromium sidecar, to probe these. The probe path is check.sh → Camofox sidecar → DOM snapshot → content-date extraction.

To enable browser probing:

1. Run the Camofox Docker sidecar on a reachable address (default http://localhost:9377). The probe script and the GitHub Actions workflow pick this up from the CAMOFOX_BASE_URL environment variable; nothing in this repo encodes a public IP.

2. Set CAMOFOX_BASE_URL to that address.

3. Restart the timer with systemctl restart datapulse-health.timer.

Without Camofox, those five datasets will sit at browser-dependent — the honest status: DataPulse cannot probe them without a browser, so it says so rather than failing silently. See scripts/smoke_browser_probes.sh for isolated smoke tests.

Legal

DataPulse probes publicly-published open-data sources. We do not bypass authentication, CAPTCHAs, or terms-of-service restrictions. Every source we probe is publicly available without login; the data is aggregate/non-personal; and the probe respects each dataset's declared refresh frequency.

All scraping is rate-limited (15-minute cadence, dataset-tier cadence applied) and identifies itself via User-Agent. Sources we cannot probe without authentication, CAPTCHA bypass, or ToS violation are marked unreachable or browser-dependent — never silently scraped through a workaround.

If you are a data source maintainer and would like DataPulse to adjust its probe cadence, exclude a dataset, or remove it from the manifest, please open a GitHub issue or contact the maintainers.

AI-agent-ready — what it means for you

Give your organisation's AI tools current, licensed, and verified Malaysian public data without first building a custom integration. DataPulse MY makes the full portfolio discoverable from one self-describing index, ready for agents, RAG systems, and internal knowledge tools to consume.

What being AI-ready gives you

• Zero integration work: an AI agent or LLM/RAG system fetches one llms.txt and can use the entire portfolio immediately — no scraping, API-key setup, or data-format reverse-engineering.

• Honest freshness signals: a 15-minute timer probes datasets when their cadence tier is due, separating HTTP reachability, browser dependency, schema validity, and source freshness so missing evidence is visible instead of being labelled healthy.

• Machine-readable and licence-clear: every dataset has a JSON envelope with its schema, licence, and refresh cadence, giving legal and engineering teams the information they need to approve and integrate it.

• Trustworthy for AI: verified official sources and explicit licences let agents cite and use the data without permission ambiguity.

• RAG and knowledge-base ready: drop the envelopes into a retrieval pipeline to ground chatbots and AI tools in current Malaysian public data.

Every manifest dataset declares either CC BY 4.0 or OGL licensing and is assessed with the honest nine-status trust taxonomy.

MCP server (read-only)

DataPulse MY also exposes an AI-ready, read-only MCP server so agents can query the catalogue natively:

• Endpoint: https://mcp.data-pulse.my/mcp (Streamable HTTP, no auth)

• 5 tools: search_datasets, get_dataset, find_stale, get_provenance, find_by_licence

• 3 resources: datapulse://index, datapulse://licences, datapulse://{dataset_id}

The public endpoint is live and serves all 5 read-only tools over the 375-dataset catalogue.

Connect from Claude Desktop:

{
  "mcpServers": {
    "datapulse-my": {
      "transport": "streamable-http",
      "url": "https://mcp.data-pulse.my/mcp"
    }
  }
}

See llms.txt for the full discovery index, and docs/mcp-deploy.md for the deployment architecture.

How to consume the data

Verify access:

curl -s https://r3dz4r.github.io/datapulse-my/llms.txt

• llms.txt — curated dataset index

• datapulse.json — manifest with a declared $schema

• health/latest.json — latest freshness snapshot

• feed.xml — dataset health change feed

• datapulse.schema.json — manifest schema

To consume the portfolio:

1. Fetch llms.txt for the curated index.

2. Fetch datapulse.json for the machine-readable manifest, including licence, refresh cadence, and geographic coverage.

3. Fetch health/latest.json to check freshness before use.

4. Cite each dataset according to its licence and attribution requirements.

robots.txt allows all agents; scripts/verify_agent_ready.sh is the agent-consumer self-test.

For humans wiring their own agents: see the MCP server section below for the Claude Desktop / Cursor / Cline config block, or the full integration guide at docs/mcp-deploy.md.

Wire it into Claude Desktop via claude_desktop_config.json (30 seconds, no API key):

{
  "mcpServers": {
    "datapulse-my": {
      "transport": "streamable-http",
      "url": "https://mcp.data-pulse.my/mcp"
    }
  }
}

Restart Claude Desktop, confirm the hammer icon shows "datapulse-my" with 5 tools: search_datasets, get_dataset, find_stale, get_provenance, find_by_licence. Cursor / Cline use the same JSON in their MCP config panel.

Included datasets

• Malaysian Fuel Prices — Samples: CSV, JSON

• ePerolehan Tender Notices (DIIKLANKAN) — Sample JSON

• PriceCatcher (Grocery Prices) — Samples: main CSV, item lookup, premise lookup

Daily reference data

Daily-published reference datasets from official Malaysian sources. Each agency publishes on its own schedule, declared in refresh_frequency; the dashboard combines the date-only source value with the publication time and does not infer a time from midnight or UTC conversion.

Bank Negara Malaysia (BNM)

• BNM Daily Exchange Rates (0900) — Sample JSON

• BNM Daily Exchange Rates (1130) — Sample JSON

• BNM Daily Exchange Rates (1200) — Sample JSON

• BNM Daily Exchange Rates (1700) — Sample JSON

MET Malaysia

• MET Malaysia Weather Forecast — Samples: CSV, JSON

Department of Environment (DOE)

• DOE APIMS Air Quality (Hourly) — Sample JSON

• DOE RQIMS River Water Quality (Continuous) — Sample JSON

• DOE MQIMS Marine Water Quality (Monthly) — Sample JSON

KKM (Ministry of Health)

• KKM iDengue Weekly Dengue Cases — Sample JSON

OpenDOSM (DOSM open data portal)

• OpenDOSM Crime by District (Annual) — Sample JSON

• OpenDOSM Monthly CPI by State — Sample JSON

• OpenDOSM Annual Real GDP by State — Sample JSON

• OpenDOSM Quarterly Real GDP — Sample JSON

• OpenDOSM Annual Real GDP by Supply Sector — Sample JSON

• OpenDOSM Monthly Trade Headline — Sample JSON

• OpenDOSM Monthly CPI Inflation by Division — Sample JSON

• OpenDOSM Monthly Trade by End Use (BEC) — Sample JSON

• OpenDOSM Quarterly Labour Force Statistics — Sample JSON

• OpenDOSM Quarterly Labour Force Statistics by State — Sample JSON

• OpenDOSM Annual Employment by Sector and Sex — Sample JSON

• OpenDOSM Annual Population by State — Sample JSON

• OpenDOSM Annual Nominal GDP by Supply Sector — Sample JSON

• OpenDOSM Quarterly Nominal GDP — Sample JSON

• OpenDOSM Quarterly Real GDP (Seasonally Adjusted) — Sample JSON

• OpenDOSM Annual Nominal GDP and GNI — Sample JSON

• OpenDOSM Monthly Core CPI Inflation by Division — Sample JSON

• OpenDOSM Monthly CPI Inflation by State and Division — Sample JSON

• OpenDOSM Monthly Producer Price Index — Sample JSON

• OpenDOSM Annual Labour Force Statistics — Sample JSON

• OpenDOSM Monthly Labour Force Statistics — Sample JSON

• OpenDOSM Monthly Trade by SITC Section — Sample JSON

• OpenDOSM IPI for Export-Oriented Divisions — Sample JSON

• OpenDOSM IPI for Domestic-Oriented Divisions — Sample JSON

• OpenDOSM Annual Births by State — Sample JSON

• OpenDOSM Annual Deaths by State — Sample JSON

• OpenDOSM Annual Maternal Deaths by State — Sample JSON

• OpenDOSM Annual Marriages by State and Sex — Sample JSON

• OpenDOSM Household Income, Malaysia — Sample JSON

• OpenDOSM Household Income by State — Sample JSON

• OpenDOSM Household Income by District — Sample JSON

• OpenDOSM Poverty, Malaysia — Sample JSON

• OpenDOSM Poverty by State — Sample JSON

• OpenDOSM Poverty by District — Sample JSON

• OpenDOSM Income Inequality, Malaysia — Sample JSON

• OpenDOSM Income Inequality by State — Sample JSON

• OpenDOSM Income Inequality by District — Sample JSON

• OpenDOSM Household Expenditure by DUN — Sample JSON

• OpenDOSM Household Expenditure by Parliamentary Constituency — Sample JSON

• OpenDOSM Annual Population, Malaysia — Sample JSON

• OpenDOSM Annual Population by Parliamentary Constituency — Sample JSON

• OpenDOSM Annual Deaths by District and Sex — Sample JSON

• OpenDOSM Annual Marriages by State, Age, and Sex — Sample JSON

• OpenDOSM Annual Fertility — Sample JSON

• OpenDOSM Annual Maternal Deaths, Malaysia — Sample JSON

data.gov.my

• data.gov.my Monthly Interest Rates — Sample JSON

• data.gov.my Quarterly Federal Government Revenue — Sample JSON

• data.gov.my Quarterly Federal Operating Expenditure — Sample JSON

• data.gov.my State Government Expenditure — Sample JSON

• data.gov.my Monthly Money Aggregates — Sample JSON

• data.gov.my Monthly Currency in Circulation — Sample JSON

• data.gov.my Monthly Payment Systems — Sample JSON

• data.gov.my Monthly Payment Instruments — Sample JSON

• data.gov.my Monthly Payment Channels — Sample JSON

• data.gov.my Annual Interest Rates — Sample JSON

• data.gov.my Annual EPF Dividend Rates — Sample JSON

• data.gov.my Monthly Vehicle Registrations by Type and Fuel — Sample JSON

• data.gov.my Daily FPX Transactions — Sample JSON

• data.gov.my Hospital Beds by State and Hospital Type — Sample JSON

• data.gov.my Healthcare Staff by State and Staff Type — Sample JSON

• data.gov.my Daily Blood Donations by Blood Group and State — Sample JSON

• data.gov.my Infant Immunisation Coverage — Sample JSON

• data.gov.my Sexually Transmitted Diseases by State — Sample JSON

• data.gov.my Daily PeKaB40 Health Screenings by State — Sample JSON

• data.gov.my Malaysian National Health Accounts Expenditure — Sample JSON

• data.gov.my Monthly Electricity Consumption — Sample JSON

• data.gov.my Monthly Electricity Supply — Sample JSON

• data.gov.my Water Consumption by State and Sector — Sample JSON

• data.gov.my Water Production by State — Sample JSON

• data.gov.my Access to Treated Water by State and Strata — Sample JSON

• data.gov.my Daily Public Transport Ridership — Sample JSON

• data.gov.my Monthly KTMB Ridership — Sample JSON

• data.gov.my Cellular Subscribers by Plan Type — Sample JSON

• data.gov.my Prisoners by State and Sex — Sample JSON

• data.gov.my Drug Addicts by State and Age Group — Sample JSON

• data.gov.my Female Representation in Local Authorities — Sample JSON

• data.gov.my Female Representation in Parliament — Sample JSON

• data.gov.my Monthly Marine Fish Landings by State — Sample JSON

• data.gov.my Crop Area and Production by State — Sample JSON

• data.gov.my Public Education Institutions by District — Sample JSON

GTFS transit feeds

The transport namespace adds 16 GTFS Static schedule ZIPs and 14 GTFS Realtime vehicle-position protobuf feeds for KTMB, Prasarana, and BAS.MY services. Static samples are under samples/gtfs-static/ and realtime snapshots are under samples/gtfs-realtime/.

DataPulse MY currently tracks the portfolio declared in datapulse.json.

Current coverage

Refresh cadence

How to use it

Start with datapulse.json to discover datasets and their official sources. Follow each health_report link for a plain-language assessment, or consume the matching file under data/json/ in an automated workflow.

For example, a data pipeline can inspect status, content_freshness_date, and freshness_signal_source before processing a source, while a researcher can review the known quirks before designing a collection method.

Monitoring

• The VPS datapulse-health.timer wakes every 15 minutes and runs only the datasets whose cadence tier is due.

• GitHub Actions performs a full weekly probe as a fallback and republishes the generated health, badge, feed, README, and changelog artifacts.

• RSS feed — available.

• Status badges — available.

• More datasets — planned.

Adopt a dataset

Know a Malaysian public dataset that deserves dependable health metadata? Adopt it: verify its source and licence, document its schema and quirks, and submit a health report. See CONTRIBUTING.md for the expected three-file contribution model.

New contributors can start with the repository's Good first issues or propose a dataset through the GitHub issue forms. Maintainers use good first issue (yellow), adopt-a-dataset (blue), freshness-check (blue), bug (red), documentation (blue), question (purple), and wontfix (gray) to route contributions.

Licence

DataPulse MY is released under the MIT License. Source datasets remain subject to the licences and attribution requirements stated in their individual health reports.