# mcp-anac-crawler

MCP **stdio** server for crawling the Italian ANAC *Pubblicità Legale* tender platform
(`pubblicitalegale.anticorruzione.it`), built to the requirements in
[docs/req-web-crawel](docs/req-web-crawel).

## The finding that shaped the design

The requirements assume an HTML crawler ("analizzare pagina web"). The target is not an
HTML site: **`/bandi` is an Angular single-page application.** The served HTML is a 37 KB
shell containing no tender data at all — scraping it returns nothing.

All data is delivered by a JSON API that the SPA's own `BackendService` calls. That API is
undocumented, so the contract was reverse-engineered from the production bundle
(`main.<hash>.js`) and verified against live responses. See
[endpoints.py](src/anac_crawler/anac/endpoints.py) for the full contract.

Two consequences that drive most of this codebase:

1. **Two incompatible pagination models coexist.** `/avvisi` is offset-based (Spring
   `Page`, with totals); `/avvisi-full-text` is **cursor-based** — page *N+1* requires the
   token returned with page *N*. Conflating them silently returns page 0 forever.
2. **Documents are not on ANAC's domain.** `documenti_di_gara_link` points at arbitrary
   third-party contracting-authority portals, and never directly at a file — verified
   across a full day of notices, zero links end in `.pdf`. Reaching a PDF therefore needs
   an HTML hop, and every such fetch is an **untrusted-URL fetch** (SSRF surface).

## Quick start

```bash
python -m venv .venv && .venv/Scripts/pip install -e ".[dev]"
.venv/Scripts/python -m anac_crawler          # speaks MCP over stdio
```

Register with an MCP client:

```json
{
  "mcpServers": {
    "anac-crawler": {
      "command": "C:/path/to/.venv/Scripts/python.exe",
      "args": ["-m", "anac_crawler"],
      "env": { "ANAC_LOG_LEVEL": "INFO", "ANAC_RATELIMIT_REQUESTS_PER_SECOND": "4" }
    }
  }
}
```

## Tools

| Tool | Purpose | Notes |
|---|---|---|
| `anac_reference_data` | Taxonomies, value bands, publication dates, categories, news | Call first to build valid filters |
| `anac_search_notices` | Search with **offset** pagination | Reports totals; use for page jumps |
| `anac_search_notices_full_text` | Full-text search with **cursor** pagination | Pass `next_token` from previous page |
| `anac_collect_notices` | Multi-page traversal in one call | Hard record/page ceilings |
| `anac_get_notice` | Single notice by `idAvviso` | Flattened record |
| `anac_get_notice_history` | Revision chronology | Upstream 404 → empty list |
| `anac_inspect_page` | Fetch an HTML landing page, discover PDF links | Untrusted third-party content |
| `anac_download_document` | Download a document | **Not read-only**; writes to disk, egress-guarded |
| `anac_health` | Breaker state, cache stats, latency percentiles, schema-drift signal | |

Typical flow: `anac_reference_data` → `anac_search_notices` → `anac_inspect_page` on a
`documenti_di_gara_link` → `anac_download_document` on a discovered PDF.

## Requirements coverage

| Requirement | Where |
|---|---|
| Python | 3.11+, `src/anac_crawler/` |
| stdio | [mcp_server/server.py](src/anac_crawler/mcp_server/server.py) + stdout guard |
| HTTP GET/POST | [http/client.py](src/anac_crawler/http/client.py) (`request`, `post_json`) |
| Cookies | Persistent JSON jar + Azure `ARRAffinity` re-scoping |
| Sessions | One pooled, keep-alive `AsyncClient` shared across all tool calls |
| Web protocol | HTTP/2, conditional requests, `Retry-After`, redirects, robots.txt |
| Page analysis | [documents/html.py](src/anac_crawler/documents/html.py) (selectolax) |
| Caching | [http/cache.py](src/anac_crawler/http/cache.py) — 2-tier, ETag revalidation, single-flight, stale-if-error |
| Navigation | Unified pagination + bounded traversal in [repository.py](src/anac_crawler/anac/repository.py) |
| Targeted file access + PDF download | [documents/fetcher.py](src/anac_crawler/documents/fetcher.py) |

## Configuration

Everything is env-driven and validated at startup (`ANAC_` prefix) — see
[.env.example](.env.example) and [config.py](src/anac_crawler/config.py).
The knobs that matter most in production:

```bash
ANAC_RATELIMIT_REQUESTS_PER_SECOND=4     # politeness toward a public service
ANAC_RATELIMIT_MAX_CONCURRENCY=4
ANAC_DOWNLOAD_HOST_ALLOWLIST=            # pin the egress surface (strongly recommended)
ANAC_DOWNLOAD_MAX_BYTES=67108864
ANAC_CA_BUNDLE=                          # for corporate TLS interception
ANAC_LOG_FORMAT=json                     # structured logs on stderr
```

### TLS note

Document downloads default to the **OS trust store** (via `truststore`), not certifi.
This is functional, not cosmetic: many Italian PA portals serve an incomplete certificate
chain, which OpenSSL cannot resolve but the Windows/macOS verifiers can (via AIA).
Verified against a live portal — certifi fails, the OS store completes TLS 1.3. Details in
[http/tls.py](src/anac_crawler/http/tls.py).

## Verification

```bash
.venv/Scripts/python -m pytest -q          # 241 tests, no network
.venv/Scripts/python -m mypy src           # strict, clean
.venv/Scripts/python -m ruff check src tests scripts
.venv/Scripts/python scripts/smoke_live.py   # live API contract check
.venv/Scripts/python scripts/smoke_stdio.py  # real JSON-RPC handshake over stdio
```

`smoke_stdio.py` is the one that matters most: it spawns the server as a subprocess and
drives it exactly as a client would, which is the only way to catch a stray byte on stdout.

## Further reading

- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — layering, request lifecycle, design decisions
- [docs/ENTERPRISE.md](docs/ENTERPRISE.md) — what is production-ready, what is not, and the
  recommended roadmap with known limitations stated explicitly