# WebcrawlerAPI MCP Server

[WebcrawlerAPI](https://webcrawlerapi.com) is a powerful web scraping and website crawling service that allows you to extract data from any website with ease. 

This MCP (Model Context Protocol) server integrates WebcrawlerAPI's scraping capabilities directly into MCP-compatible applications like Claude Code, enabling integration within your AI workflows. Get started with your free API key at the [WebcrawlerAPI Dashboard](https://dash.webcrawlerapi.com).

## Setup

1. Get your API key from [WebcrawlerAPI Dashboard](https://dash.webcrawlerapi.com/access)
2. Set the environment variable:
   ```bash
   export WEBCRAWLER_API_KEY="your-api-key-here"
   ```

## Using with Claude Code

Add this server to your Claude Code configuration using npx:

```bash
npx webcrawler-mcp
```

Or add to your MCP settings configuration:

```json
{
  "mcpServers": {
    "webcrawler": {
      "command": "npx",
      "args": ["-y", "webcrawler-mcp"],
      "env": {
        "WEBCRAWLER_API_KEY": "your-api-key-here"
      }
    }
  }
}
```

## Available Tools

### webcrawler-scrape

Scrapes content from any webpage and returns it as markdown.

**Parameters:**
- `url` (required): The URL of the webpage to scrape
- `prompt` (optional): Optional prompt to extract specific information from the page

**Example usage:**
```
Use the webcrawler-scrape tool to get the content from https://example.com
```

**With a prompt for targeted extraction:**
```
Use the webcrawler-scrape tool to scrape https://news.ycombinator.com with the prompt "Extract all article titles and their corresponding URLs"
```

The tool returns the scraped content as markdown, along with the page title and HTTP status code. When using a prompt, the API will focus on extracting the specific information you requested.

## Running as Standalone App

You can also run the server as a standalone application:

### Via npm
```bash
npm install
npm run build
npm start
```

### Via npx
```bash
npx webcrawler-mcp
```

### HTTP Mode
To run in HTTP mode instead of stdio:
```bash
USE_HTTP=true npx webcrawler-mcp
# or
npm run start:http
```

The HTTP server will start on port 8080 by default, with endpoints:
- Health check: `http://localhost:8080/health`
- MCP endpoint: `http://localhost:8080/mcp`

## Environment Variables

- `WEBCRAWLER_API_KEY`: Your WebcrawlerAPI.com API key (required)
- `USE_HTTP`: Set to "true" to use HTTP transport instead of stdio
- `PORT`: HTTP server port (default: 8080)

## Development

```bash
npm install
npm run dev  # Watch mode
npm run build
```