mcp-anac-crawler

MCP stdio server for crawling the Italian ANAC Pubblicità Legale tender platform (pubblicitalegale.anticorruzione.it), built to the requirements in docs/req-web-crawel.

The finding that shaped the design

The requirements assume an HTML crawler ("analizzare pagina web"). The target is not an HTML site: /bandi is an Angular single-page application. The served HTML is a 37 KB shell containing no tender data at all — scraping it returns nothing.

All data is delivered by a JSON API that the SPA's own BackendService calls. That API is undocumented, so the contract was reverse-engineered from the production bundle (main.<hash>.js) and verified against live responses. See endpoints.py for the full contract.

Two consequences that drive most of this codebase:

1. Two incompatible pagination models coexist. /avvisi is offset-based (Spring Page, with totals); /avvisi-full-text is cursor-based — page N+1 requires the token returned with page N. Conflating them silently returns page 0 forever.

2. Documents are not on ANAC's domain. documenti_di_gara_link points at arbitrary third-party contracting-authority portals, and never directly at a file — verified across a full day of notices, zero links end in .pdf. Reaching a PDF therefore needs an HTML hop, and every such fetch is an untrusted-URL fetch (SSRF surface).

Quick start

python -m venv .venv && .venv/Scripts/pip install -e ".[dev]"
.venv/Scripts/python -m anac_crawler          # speaks MCP over stdio

Register with an MCP client:

{
  "mcpServers": {
    "anac-crawler": {
      "command": "C:/path/to/.venv/Scripts/python.exe",
      "args": ["-m", "anac_crawler"],
      "env": { "ANAC_LOG_LEVEL": "INFO", "ANAC_RATELIMIT_REQUESTS_PER_SECOND": "4" }
    }
  }
}

Tools

Typical flow: anac_reference_data → anac_search_notices → anac_inspect_page on a documenti_di_gara_link → anac_download_document on a discovered PDF.

Requirements coverage

Configuration

Everything is env-driven and validated at startup (ANAC_ prefix) — see .env.example and config.py. The knobs that matter most in production:

ANAC_RATELIMIT_REQUESTS_PER_SECOND=4     # politeness toward a public service
ANAC_RATELIMIT_MAX_CONCURRENCY=4
ANAC_DOWNLOAD_HOST_ALLOWLIST=            # pin the egress surface (strongly recommended)
ANAC_DOWNLOAD_MAX_BYTES=67108864
ANAC_CA_BUNDLE=                          # for corporate TLS interception
ANAC_LOG_FORMAT=json                     # structured logs on stderr

TLS note

Document downloads default to the OS trust store (via truststore), not certifi. This is functional, not cosmetic: many Italian PA portals serve an incomplete certificate chain, which OpenSSL cannot resolve but the Windows/macOS verifiers can (via AIA). Verified against a live portal — certifi fails, the OS store completes TLS 1.3. Details in http/tls.py.

Verification

.venv/Scripts/python -m pytest -q          # 241 tests, no network
.venv/Scripts/python -m mypy src           # strict, clean
.venv/Scripts/python -m ruff check src tests scripts
.venv/Scripts/python scripts/smoke_live.py   # live API contract check
.venv/Scripts/python scripts/smoke_stdio.py  # real JSON-RPC handshake over stdio

smoke_stdio.py is the one that matters most: it spawns the server as a subprocess and drives it exactly as a client would, which is the only way to catch a stray byte on stdout.

Further reading

• docs/ARCHITECTURE.md — layering, request lifecycle, design decisions

• docs/ENTERPRISE.md — what is production-ready, what is not, and the recommended roadmap with known limitations stated explicitly