# Web Scraper MCP Server

A **Model Context Protocol (MCP) server** that provides web scraping capabilities using **FastMCP**.
It allows AI agents to fetch, clean, and extract readable content from web pages, optionally including links and images.


## Features

* 🌐 Fetches and parses web pages
* 🧹 Cleans and normalizes extracted text
* 🔗 Optionally extracts links
* 🖼️ Optionally extracts image URLs
* ⚡ Async HTTP requests using `httpx`
* 🧠 Exposed as an MCP tool (`scrape_url`)
* 🐳 Dockerized for easy deployment


## Project Structure

```
. 
├── .env.example
├── .gitignore
├── Dockerfile         # Container image definition
├── Makefile           # Common development and deployment commands
├── README.md          # This file
├── docker-compose.yml # Compose file for easy local development
├── main.py            # MCP server and scraping logic
└── requirements.txt   # Dependencies file
```


## `main.py`

This file contains the MCP server implementation.

### Key Components

* **FastMCP Server**

  * Runs an MCP-compatible HTTP server
  * Exposes tools to AI agents

* **`scrape_url` Tool**

  * Fetches and extracts content from a given URL

### Tool Signature

```python
scrape_url(
    url: str,
    include_links: bool = True,
    include_images: bool = False,
    clean_text: bool = True
) -> str
```

### Behavior

* Validates URLs before fetching
* Removes `<script>` and `<style>` tags
* Extracts readable text
* Optionally appends:

  * All links (`<a href>`)
  * All images (`<img src>`)
* Returns a formatted plain-text result

### Configuration (Environment Variables)

| Variable    | Default   | Description   |
| ----------- | --------- | ------------- |
| `HOST_ADDR` | `0.0.0.0` | Bind address  |
| `HOST_PORT` | `9292`    | Server port   |
| `LOG_LEVEL` | `INFO`    | Logging level |


## Dockerfile

The `Dockerfile` builds a lightweight containerized version of the MCP server.

### Highlights

* Based on `python:3.12-slim`
* Optimized for production use
* Sets helpful defaults:

  * No bytecode
  * Unbuffered output
  * Reduced pip noise

### Example Build & Run

```bash
docker build -t mcp-web-scraper .
docker run -p 9292:9292 mcp-web-scraper
```


## Makefile

The `Makefile` provides shortcuts for common tasks.

### Common Targets

| Target        | Description                                       |
| ------------- | ------------------------------------------------- |
| `clean`       | Remove local virtual environment                  |
| `dev-compose` | Start development environment with Docker Compose |
| `dev-down`    | Stop development Docker Compose stack             |
| `infra`       | Apply Terraform infrastructure                    |

### Example Usage

```bash
make dev-compose
make dev-down
make clean
```


## Running Locally (Without Docker)

```bash
pip install -r requirements.txt
python main.py
```

The server will start at:

```
http://localhost:9292
```


## MCP Integration

This server is designed to be consumed by **MCP-compatible clients or agents**, enabling them to:

* Read web pages
* Extract structured textual content
* Use scraped data as model context