IBGE Microdata MCP Server

Local-first MCP server for discovering, downloading, unpacking, converting, querying, and analyzing official public IBGE microdata.

This project does not host IBGE datasets. It uses IBGE download servers as the source of truth, downloads only files explicitly requested by the user, mirrors them into a local cache, and runs analysis on local files.

IBGE's raw public download host is ftp.ibge.gov.br. The server accepts both https:// and http:// URLs for that host. It tries to keep official HTTPS URLs where possible, but can fall back to official HTTP for public files when HTTPS is slow or unavailable. Tool output reports the resolved URL, transport used, and SHA-256 hash after download.

Why Local-First

IBGE microdata files are public, but many are large enough that an MCP server should not return them directly in chat responses. The practical workflow is:

discover official files -> inspect size -> download selected archive -> inspect/extract entries -> convert selected variables to Parquet -> query with DuckDB

In plain terms:

• DuckDB is a local analytical SQL engine. It can query large local files without running a database server.

• Parquet is a compressed columnar file format. Convert fixed-width TXT microdata once, then query only the columns you need.

Related MCP server: recife-open-data-mcp

Core Generic Tools

The generic path is discovery, caching, official metadata inspection/search, Parquet conversion, profiling, and DuckDB querying. These tools are the main public surface of the server.

Optional Survey-Specific Helpers

These helpers are layered on top of the same local-first workflow. They are useful shortcuts for known IBGE formats, but they are not required for the generic workflow.

Prerequisites

This MCP is a local tool. Users are expected to have:

• Node.js 18.20 or newer.

• pnpm.

• R with Rscript available on PATH.

R is included as a project prerequisite because PNAD Contínua and Data Zoom workflows are best supported by the existing Brazilian R ecosystem. The MCP server itself still runs as a Node/TypeScript process and returns MCP-friendly JSON, Markdown, and local file paths.

Install

1. Install system runtimes.

On macOS with Homebrew:

brew install node r
npm install -g pnpm@11.7.0

On Windows with winget:

winget install OpenJS.NodeJS
winget install RProject.R
npm install -g pnpm@11.7.0

On Ubuntu/Debian:

sudo apt update
sudo apt install -y nodejs npm r-base
sudo npm install -g pnpm@11.7.0

Linux package managers sometimes ship older Node.js versions. If node --version is below 18.20, install a newer Node.js release from NodeSource, nvm, or the official Node.js installer.

You can also install the same tools from their official installers:

• Node.js: https://nodejs.org/

• pnpm: https://pnpm.io/installation

• R: https://cran.r-project.org/

Check that all three are available:

node --version
pnpm --version
Rscript --version

2. Clone the GitHub repository:

git clone https://github.com/emmanueltsallis/ibge-microdata-mcp.git
cd ibge-microdata-mcp

3. Install Node dependencies and build the local MCP server:

pnpm install
pnpm run build

4. Install the baseline R packages used by the R-backed IBGE workflows:

Rscript -e 'install.packages(c("PNADcIBGE", "survey", "jsonlite", "arrow"), repos = "https://cloud.r-project.org")'
Rscript -e 'install.packages("datazoom.social", repos = c("https://datazoompuc.r-universe.dev", "https://cloud.r-project.org"))'

Run

node dist/index.js

Generic MCP Client Setup

Any MCP client that supports stdio can run this server. After cloning, installing dependencies, and running pnpm run build, configure your client to start the built Node entrypoint.

Use an absolute path to dist/index.js in client configuration. Most clients use the same basic shape:

{
  "mcpServers": {
    "ibge-microdata": {
      "command": "node",
      "args": ["/absolute/path/to/ibge-microdata-mcp/dist/index.js"]
    }
  }
}

If your client uses TOML or another config format, the same idea applies: command is the Node executable, and args contains the absolute path to this server's built dist/index.js file.

Most MCP clients start configured servers when the app or session starts, so restart or reload the client after changing MCP configuration. If the tools do not appear immediately, restart the client first before debugging the server.

MCP uses stdout for protocol messages. This server writes startup and fatal logs to stderr, which keeps stdio protocol traffic clean for clients.

You can smoke-test the server independently of any specific client:

node --input-type=module - <<'NODE'
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';

const transport = new StdioClientTransport({
  command: 'node',
  args: ['/absolute/path/to/ibge-microdata-mcp/dist/index.js'],
});
const client = new Client({ name: 'ibge-microdata-smoke', version: '0.0.0' });
await client.connect(transport);
const tools = await client.listTools();
console.log(tools.tools.map((tool) => tool.name).sort().join('\n'));
await client.close();
NODE

The smoke test should list tools such as ibge_microdata_list_surveys, ibge_microdata_metadata_inventory, and ibge_microdata_fixed_width_zip_to_parquet.

For a shorter generic walkthrough, see examples/generic-workflow.md. For a starter harmonization recipe, see examples/harmonization-recipe.json. For external harmonization sources that can inform recipes, see docs/harmonization-sources.md.

R-Backed PNADc Workflow

Use the R status tool first:

ibge_microdata_r_status({})

Download PNAD Contínua through PNADcIBGE and write a Parquet file that the MCP can query with DuckDB:

ibge_microdata_pnadc_r_download({
  "year": 2024,
  "quarter": 4,
  "vars": ["UF", "V1028"],
  "outputPath": "/Users/you/.cache/ibge-microdata-mcp/converted/pnadc_2024q4.parquet"
})

Use datazoom.social when you want Data Zoom's PNAD Contínua processing or panel identifiers:

ibge_microdata_datazoom_pnadc_load({
  "outputDir": "/Users/you/.cache/ibge-microdata-mcp/datazoom/pnadc",
  "years": [2024],
  "quarters": [1, 2, 3, 4],
  "panel": "basic",
  "outputFormat": "parquet"
})

Generic Workflow

1. Find public files from a known survey family or an official directory:

ibge_microdata_list_surveys({})

ibge_microdata_connectivity_check({})

ibge_microdata_list_files({
  "survey": "pof"
})

ibge_microdata_discover({
  "rootUrl": "https://ftp.ibge.gov.br/",
  "maxDepth": 3,
  "maxDirectories": 50
})

ibge_microdata_discover_metadata({
  "rootUrl": "https://ftp.ibge.gov.br/",
  "maxDepth": 4,
  "maxDirectories": 100
})

2. Inspect file metadata before downloading:

ibge_microdata_file_info({
  "url": "https://ftp.ibge.gov.br/path/to/public/archive.zip"
})

3. Download to a local cache:

ibge_microdata_download_file({
  "url": "https://ftp.ibge.gov.br/path/to/public/archive.zip",
  "cacheRoot": "/Users/you/.cache/ibge-microdata-mcp"
})

The downloader mirrors the official ftp.ibge.gov.br path under cacheRoot. On repeated calls, it checks IBGE content-length metadata first and returns a cache hit when the existing local file has the expected byte size.

If HTTPS to ftp.ibge.gov.br times out, the downloader may retry the same public file over http://ftp.ibge.gov.br. This does not send credentials or private data; it only downloads public IBGE files. The response reports transport, usedFallback, and sha256 so the transfer remains auditable in headless MCP use.

4. List the cache later if you need to rediscover local paths:

ibge_microdata_list_cache({
  "cacheRoot": "/Users/you/.cache/ibge-microdata-mcp",
  "limit": 50,
  "offset": 0
})

5. Preview cache cleanup when storage grows:

ibge_microdata_cleanup_cache({
  "cacheRoot": "/Users/you/.cache/ibge-microdata-mcp",
  "dryRun": true,
  "olderThanDays": 30,
  "minBytes": 100000000
})

The cleanup tool defaults to dryRun: true, requires at least one filter, and only considers files under cacheRoot/ftp.ibge.gov.br. Set dryRun: false only after reviewing the preview.

6. Inspect archive contents:

ibge_microdata_zip_entries({
  "zipPath": "/Users/you/.cache/ibge-microdata-mcp/ftp.ibge.gov.br/path/to/public/archive.zip"
})

7. Inventory official metadata and search variables:

ibge_microdata_metadata_inventory({
  "zipPaths": ["/Users/you/.cache/ibge-microdata-mcp/ftp.ibge.gov.br/path/to/public/documentation.zip"],
  "search": "weight",
  "variableLimit": 50
})

ibge_microdata_search_variables({
  "zipPaths": ["/Users/you/.cache/ibge-microdata-mcp/ftp.ibge.gov.br/path/to/public/documentation.zip"],
  "query": "rendimento",
  "limit": 50
})

The metadata tools parse official SAS/TXT input layouts, POF-style Excel dictionaries, generic Excel dictionary tables, and plain-text dictionary tables with recognizable position/width/variable columns. When available, they return value labels/categories such as state codes or response categories. If a file cannot be parsed, the inventory reports which parser attempts were tried.

8. Export Base dos Dados-style local documentation CSVs when you want portable metadata files:

ibge_microdata_export_architecture_csv({
  "zipPaths": ["/Users/you/.cache/ibge-microdata-mcp/ftp.ibge.gov.br/path/to/public/documentation.zip"],
  "outputPath": "/Users/you/.cache/ibge-microdata-mcp/extra/architecture/variables.csv"
})

ibge_microdata_export_dictionary_csv({
  "zipPaths": ["/Users/you/.cache/ibge-microdata-mcp/ftp.ibge.gov.br/path/to/public/documentation.zip"],
  "outputPath": "/Users/you/.cache/ibge-microdata-mcp/extra/dicionario.csv"
})

The architecture export writes one row per parsed variable. The dictionary export writes one row per parsed category/code label. These are local documentation artifacts inspired by Base dos Dados workflows; they do not upload anything to BigQuery and do not require Base dos Dados credentials.

9. Inspect a fixed-width layout directly when you already have the layout file:

ibge_microdata_inspect_layout({
  "layoutPath": "/path/to/official-input-layout.txt",
  "search": "weight",
  "limit": 50
})

10. Convert selected variables to Parquet:

ibge_microdata_fixed_width_zip_to_parquet({
  "layoutPath": "/path/to/official-input-layout.txt",
  "zipPath": "/Users/you/.cache/ibge-microdata-mcp/ftp.ibge.gov.br/path/to/public/archive.zip",
  "entryName": "MICRODATA.txt",
  "outputPath": "/Users/you/.cache/ibge-microdata-mcp/converted/sample.parquet",
  "selectedVariables": ["record_id", "region", "sample_weight", "target_value"]
})

11. Profile the Parquet file before writing custom SQL:

ibge_microdata_profile_parquet_views({
  "views": [
    {
      "name": "microdata",
      "parquetPaths": ["/Users/you/.cache/ibge-microdata-mcp/converted/sample.parquet"]
    }
  ],
  "columns": ["region", "sample_weight", "target_value"],
  "topK": 10,
  "sampleRows": 3
})

If columns is omitted, the tool profiles the first 25 columns by default. This keeps wide microdata files manageable while still giving enough information to choose variables and write queries.

12. Query the Parquet file with DuckDB:

ibge_microdata_query_parquet({
  "parquetPaths": ["/Users/you/.cache/ibge-microdata-mcp/converted/sample.parquet"],
  "sql": "select region, sum(sample_weight * target_value) / sum(sample_weight) as weighted_mean from microdata group by region order by region",
  "maxRows": 100
})

The query tools accept only SELECT or WITH queries, reject semicolons and write-oriented keywords, and cap returned rows.

Harmonization Recipes

Recipes are optional JSON files that make harmonization assumptions explicit and reusable. The MCP does not ship one universal harmonization standard; instead, a recipe declares the required input views/columns, an output SELECT transformation, optional source references, and validation checks.

ibge_microdata_validate_recipe({
  "recipePath": "/path/to/harmonization-recipe.json",
  "views": [
    {
      "name": "microdata",
      "parquetPaths": ["/Users/you/.cache/ibge-microdata-mcp/converted/sample.parquet"]
    }
  ],
  "sampleRows": 5
})

If the validation output says requirements and validations passed, write the harmonized Parquet file:

ibge_microdata_apply_recipe({
  "recipePath": "/path/to/harmonization-recipe.json",
  "views": [
    {
      "name": "microdata",
      "parquetPaths": ["/Users/you/.cache/ibge-microdata-mcp/converted/sample.parquet"]
    }
  ],
  "outputPath": "/Users/you/.cache/ibge-microdata-mcp/converted/harmonized.parquet",
  "sampleRows": 5
})

Recipe SQL accepts only SELECT or WITH statements. The validation tool reports missing input columns, output schema, sample output rows, and validation results without writing a file. The apply tool writes the harmonized output only when requirements and validations pass.

Weighted Distributions

Use ibge_microdata_weighted_distribution when a Parquet file contains one row per analytical unit, a numeric value column, and a numeric survey/sample weight column:

ibge_microdata_weighted_distribution({
  "views": [
    {
      "name": "microdata",
      "parquetPaths": ["/Users/you/.cache/ibge-microdata-mcp/converted/sample.parquet"]
    }
  ],
  "unitSql": "select region, target_value as value, sample_weight as weight from microdata",
  "valueColumn": "value",
  "weightColumn": "weight",
  "groupColumn": "region",
  "topPercents": [0.01, 0.05, 0.1]
})

The tool ranks units by the value column, applies weights, reports total weight, total value, weighted mean, optional group shares, and top-bracket shares. If a top bracket cuts through tied values at the cutoff, the tied bucket is allocated proportionally.

Relational Records

Some surveys publish multiple record files. Convert each record to Parquet, inspect the resulting schemas, then join named views:

ibge_microdata_describe_parquet_views({
  "views": [
    {
      "name": "record_a",
      "parquetPaths": ["/Users/you/.cache/ibge-microdata-mcp/converted/record_a.parquet"]
    },
    {
      "name": "record_b",
      "parquetPaths": ["/Users/you/.cache/ibge-microdata-mcp/converted/record_b.parquet"]
    }
  ],
  "includeRowCounts": true,
  "sampleRows": 3
})

ibge_microdata_query_parquet_views({
  "views": [
    {
      "name": "record_a",
      "parquetPaths": ["/Users/you/.cache/ibge-microdata-mcp/converted/record_a.parquet"]
    },
    {
      "name": "record_b",
      "parquetPaths": ["/Users/you/.cache/ibge-microdata-mcp/converted/record_b.parquet"]
    }
  ],
  "sql": "select a.region, count(*) as rows from record_a a join record_b b using (record_id) group by a.region order by a.region",
  "maxRows": 100
})

POF Dictionaries

POF editions use Excel dictionary workbooks. Use the manifest tool to map dictionary sheets to TXT entries before converting records:

ibge_microdata_pof_manifest({
  "dictionaryPath": "/path/to/dictionary.xls",
  "dataZipPath": "/Users/you/.cache/ibge-microdata-mcp/ftp.ibge.gov.br/path/to/Dados.zip",
  "search": "weight",
  "variableLimit": 20
})

ibge_microdata_pof_zip_record_to_parquet({
  "dictionaryPath": "/path/to/dictionary.xls",
  "zipPath": "/Users/you/.cache/ibge-microdata-mcp/ftp.ibge.gov.br/path/to/Dados.zip",
  "recordName": "Domicílio",
  "outputPath": "/Users/you/.cache/ibge-microdata-mcp/converted/pof_record.parquet",
  "selectedVariables": ["UF", "ESTRATO_POF", "TIPO_SITUACAO_REG"]
})

The POF converter applies implied decimal scaling from the dictionary and writes DuckDB-queryable Parquet files.

Tests

Offline unit tests:

pnpm test

Live smoke tests against official IBGE endpoints:

RUN_IBGE_SMOKE=1 pnpm test -- tests/smoke.test.ts

Smoke tests list official directories, read HEAD metadata, and download the smaller POF documentation ZIP to verify dictionary parsing. They do not download large microdata data ZIPs.

Local R setup smoke test:

RUN_R_SMOKE=1 pnpm test -- tests/r-smoke.test.ts

The R smoke test checks Rscript and baseline R package availability. It does not download PNAD microdata.

Current Limits

• This is a local-first MCP server, not a hosted warehouse of all IBGE microdata.

• Discovery is deliberately bounded; broad root crawls should use explicit maxDepth and maxDirectories values to avoid excessive requests.

• Generic fixed-width conversion, official metadata inventory/search, Parquet profiling/querying, weighted distribution summaries, and POF dictionary conversion are implemented.

• Metadata parsing supports common IBGE SAS/TXT input layouts, POF-style Excel dictionaries, generic Excel dictionary tables, and generic TXT dictionary tables. PDF-only/prose-only codebooks and highly irregular historical layouts may still need parser adapters.

• Additional survey-specific harmonized recipes can be added as optional layers without changing the generic workflow.

License

MIT. See LICENSE.