# goaimoat-web-scrape-mcp

Turn any URL into clean, LLM-ready Markdown — or crawl a site and get Markdown per page.

Built for AI agents that need to actually read a page: full-text reading, research,
and RAG ingestion.

- **Hosted (no install):** `https://scrape.mcp.goaimoat.com/mcp`
- **Registry name:** `com.goaimoat/web-scrape`
- **Free:** 3 calls per email. Then a $29/yr license.

## Tools

| Tool | What it does |
|---|---|
| `scrape_url(url)` | One URL → clean Markdown |
| `crawl_website(url, limit)` | Crawl a site → Markdown per page (default 10) |

Both accept optional `email` (records free quota) and `license_key`.

## Use the hosted endpoint

Add to Claude Desktop / Cursor / Cline (`streamable-http`):

```json
{
  "mcpServers": {
    "web-scrape": {
      "type": "streamable-http",
      "url": "https://scrape.mcp.goaimoat.com/mcp"
    }
  }
}
```

Then just call `scrape_url` with your `email` — no key, no signup for the first 3 calls.

## Run it yourself

Requires a `MONID_API_KEY` (the upstream data provider). Get one at [monid.ai](https://monid.ai).

### Docker

```bash
docker build -t goaimoat-web-scrape-mcp .
docker run --rm -i -e MONID_API_KEY=your_key goaimoat-web-scrape-mcp
```

The container runs on **stdio** by default, which is what directory scanners (Glama and
similar) use for introspection.

### Local

```bash
pip install -r requirements.txt
MONID_API_KEY=your_key python server.py                                  # stdio
MCP_TRANSPORT=streamable-http MCP_PORT=8026 MONID_API_KEY=... python server.py
```

Without `MONID_API_KEY` the server still starts and answers introspection, but tool calls
return a clear "no credential configured" error rather than fabricating a result.

## Why pay for this instead of rolling your own

The page is unreachable to a model with no browsing tool, and the upstream call has a real
per-call cost. This server routes through a cheaper provider than the well-known
alternatives, so the per-call price stays a small fraction of a monthly scraping plan.

## Compliance

Public web only. This is a scraping utility — it does **not** proxy, unblock, or otherwise
circumvent restricted network access for any region. Respect each site's robots.txt and
terms of service.

## Part of GoAI Moat

A suite of MCP servers giving agents data they cannot reach themselves: verified work
emails, company records, Google search, and web scraping.

- Catalog: https://goaimoat.com/mcp-catalog.html
- All servers (monorepo): https://github.com/jayniebingyu-cyber/goaimoat-mcp

## License

MIT