# SpiderFoot MCP Agent

A Node.js implementation of the Model Context Protocol (MCP) server that exposes SpiderFoot's functionality as tools. This project provides both an MCP server and a web client for interacting with the SpiderFoot web interface.

## Features

- **MCP Server**: Exposes SpiderFoot functionality through the Model Context Protocol
- **Web Client**: Programmatic interface to interact with SpiderFoot's web interface
- **TypeScript Support**: Full TypeScript support for better development experience
- **Docker Support**: Easy deployment using Docker
- **Modular Design**: Easy to extend with new functionality

## Requirements

- Node.js 18+ (recommended 20+)
- A local SpiderFoot instance (Docker or direct installation)
  - Default web interface URL: `http://127.0.0.1:5001`
- Docker (optional, for containerized deployment)

## Setup

### Prerequisites

1. Ensure you have a running instance of SpiderFoot
2. Clone this repository:
   ```bash
   git clone https://github.com/yourusername/Spiderfoot-MCP-Agent.git
   cd Spiderfoot-MCP-Agent
   ```

### Installation

1. Install dependencies:
   ```bash
   npm install
   ```

2. Configure environment:
   ```bash
   cp .env.example .env
   ```
   
   Edit the `.env` file with your SpiderFoot details:
   ```env
   # Base URL of your SpiderFoot instance
   SPIDERFOOT_BASE_URL=http://127.0.0.1:5001
   
   # Authentication (if enabled in SpiderFoot)
   # SPIDERFOOT_USER=username
   # SPIDERFOOT_PASS=password
   
   # Allow starting scans through the API
   ALLOW_START_SCAN=true
   ```

## Usage

### Running the MCP Server

#### Development Mode (stdio transport)
```bash
npm run dev
```

#### Development Mode (HTTP transport)
```bash
npm run dev:http
```

#### Production Build
```bash
# Build the project
npm run build

# Start the server
npm start
```

### Using the Web Client

The package includes a web client that can be used to interact with the SpiderFoot web interface programmatically.

```javascript
import { SpiderFootWebClient } from './spiderfoot-web-client.js';

// Create a new client instance
const client = new SpiderFootWebClient('http://127.0.0.1:5001');

// List all scans
const scans = await client.listScans();
console.log('Existing scans:', scans);

// Start a new scan
try {
  const result = await client.startScan('example.com', ['type_DNS_TEXT'], 'domain', 'test-scan');
  console.log('Scan started:', result);
} catch (error) {
  console.error('Failed to start scan:', error);
}
```

## Development

### Building the Project
```bash
npm run typecheck
npm run build
```

Start from compiled output:

```bash
npm start            # stdio transport
npm run start:http   # HTTP transport (dist/index-http.js)
```

## Tools

The server registers the following tools:

- `spiderfoot_ping` – GET `/ping`
- `spiderfoot_modules` – GET `/modules`
- `spiderfoot_event_types` – GET `/eventtypes`
- `spiderfoot_scans` – GET `/scanlist`
- `spiderfoot_scan_info` – GET `/scanopts?id=<sid>`
- `spiderfoot_start_scan` – POST `/startscan` (guarded by `ALLOW_START_SCAN`)
- `spiderfoot_scan_data` – POST `/scaneventresults`
- `spiderfoot_scan_data_unique` – POST `/scaneventresultsunique`
- `spiderfoot_scan_logs` – POST `/scanlog`
- `spiderfoot_export_json` – POST `/scanexportjsonmulti`

Dangerous endpoints like `/query` are intentionally omitted.

## HTTP vs stdio transports

- `src/index.ts` uses the stdio transport (`StdioServerTransport`). This is commonly used when an IDE/agent launches your process and communicates via stdio.
- `src/index-http.ts` uses the Streamable HTTP transport, listening on `/:port/mcp` (default port `3000`). Use this for remote/HTTP-based MCP clients.

Environment variable for HTTP port:

- `MCP_HTTP_PORT` (default: `3000`)

## Docker usage

This repo includes a `Dockerfile` and `docker-compose.yml` to run the MCP server in Docker.

Build the image:

```bash
docker build -t spiderfoot-mcp:local .
```

Run with Docker directly:

```bash
docker run --rm -p 3000:3000 \
  -e SPIDERFOOT_BASE_URL=http://host.docker.internal:5001 \
  -e ALLOW_START_SCAN=true \
  -e MCP_HTTP_PORT=3000 \
  --name spiderfoot-mcp spiderfoot-mcp:local
```

Or with Compose:

```bash
docker-compose up --build
```

Compose file (`docker-compose.yml`) configures:

- Service: `spiderfoot-mcp`
- Port mapping: `3000:3000`
- Default env points to your host’s SpiderFoot at `http://host.docker.internal:5001`

Notes:

- On Linux, replace `host.docker.internal` with your host IP or use the container network to reach your SpiderFoot service.
- Ensure SpiderFoot is reachable on port `5001` from inside the MCP container.

## Environment variables

- `SPIDERFOOT_BASE_URL` — Base URL of your SpiderFoot web UI/API.
- `ALLOW_START_SCAN` — `true|false`. Enables/disables `spiderfoot_start_scan` tool. Default `true`.
- `SPIDERFOOT_USER`, `SPIDERFOOT_PASS` — Optional HTTP Digest credentials if you enable auth in SpiderFoot.
- `MCP_HTTP_PORT` — Port for HTTP transport (if using `index-http.ts`). Default `3000`.

## Project layout

- `src/index.ts` — MCP server (stdio transport) and tool registration.
- `src/index-http.ts` — MCP server (HTTP transport) with session management.
- `src/spiderfootClient.ts` — Axios-based client for SpiderFoot endpoints.
- `Dockerfile` — Multi-stage image: builds TS → runs HTTP server.
- `docker-compose.yml` — Runs container with env defaults.

## Using with IDEs and MCP-compatible clients

This section provides JSON-based configuration examples for connecting this MCP server from popular IDEs and tools. Two transport modes are supported:

1. Stdio transport: the IDE launches your local process
2. HTTP transport: the IDE connects to a running server at `http://localhost:5002/mcp` (Docker with compose) or `http://localhost:3000/mcp` when running `npm run dev:http` locally

You can use both; add two separate entries if your IDE supports it.

### Docker-based JSON (stdio inside container)

If you prefer your IDE to launch the MCP server inside Docker (without needing a long-running compose service), use this stdio-in-container configuration. It runs the stdio entrypoint (`dist/index.js`) and communicates over stdin/stdout.

```json
{
  "mcpServers": {
    "spiderfoot-mcp-docker-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--add-host=host.docker.internal:host-gateway",
        "-e",
        "SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
        "spiderfoot-mcp:local",
        "node",
        "dist/index.js"
      ],
      "env": {}
    }
  }
}
```

Copy-paste Claude Desktop block (Docker stdio + HTTP):

```json
{
  "mcpServers": {
    "spiderfoot-mcp-docker-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--add-host=host.docker.internal:host-gateway",
        "-e",
        "SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
        "spiderfoot-mcp:local",
        "node",
        "dist/index.js"
      ]
    },
    "spiderfoot-mcp-http": {
      "type": "http",
      "url": "http://localhost:5002/mcp"
    }
  }
}
```

Notes:

- Make sure you have built the image (`docker build -t spiderfoot-mcp:local .` or `docker-compose build`).
- This approach does not expose a port; it uses stdio via Docker (`-i`).
- The host SpiderFoot URL is passed via `-e SPIDERFOOT_BASE_URL=http://host.docker.internal:5001`.

### Common configuration examples

__Stdio (local process)__

```json
{
  "mcpServers": {
    "spiderfoot-mcp-stdio": {
      "type": "stdio",
      "command": "node",
      "args": [
        "./node_modules/tsx/dist/cli.mjs",
        "src/index.ts"
      ],
      "cwd": "C:/dev-env.local/project-repos/Spiderfoot-MCP-Agent",
      "env": {
        "SPIDERFOOT_BASE_URL": "http://127.0.0.1:5001",
        "ALLOW_START_SCAN": "true"
      }
    }
  }
}
```

__HTTP (connect to running server)__

```json
{
  "mcpServers": {
    "spiderfoot-mcp-http": {
      "type": "http",
      "url": "http://localhost:5002/mcp"
    }
  }
}
```

Notes:
- If you prefer `npm start` instead of `tsx`, update `command`/`args` accordingly, e.g. `command: "npm", args: ["run", "dev"]`.
- On Windows, keep forward slashes in `cwd` or escape backslashes (e.g., `C:\\dev-env.local\\project-repos\\Spiderfoot-MCP-Agent`).
- Ensure SpiderFoot is reachable at `SPIDERFOOT_BASE_URL` from the MCP server.

### Windsurf

Steps:
1. Open `Settings` → `MCP` (or Tools/Integrations section that manages MCP servers).
2. Add a new server entry.
3. Paste one of the JSON examples above into your MCP server configuration, merging with any existing `mcpServers` entries. Recommended options:
   - Docker stdio: `spiderfoot-mcp-docker-stdio` (uses `command: docker`)
   - HTTP: `serverUrl` to `http://localhost:5002/mcp`
4. Save settings.
5. Start the server if using HTTP mode (Docker Compose or `npm run dev:http`). For stdio, Windsurf will launch it automatically when needed.

Windsurf – Option 2: HTTP via serverUrl

```json
{
  "mcpServers": {
    "spiderfoot-mcp-http": {
      "serverUrl": "http://localhost:5002/mcp"
    }
  }
}
```

Windsurf – Option 1: Docker stdio

```json
{
  "mcpServers": {
    "spiderfoot-mcp-docker-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--add-host=host.docker.internal:host-gateway",
        "-e",
        "SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
        "spiderfoot-mcp:local",
        "node",
        "dist/index.js"
      ]
    }
  }
}
```

Notes:

- Make sure you have built the image (`docker build -t spiderfoot-mcp:local .` or `docker-compose build`).
- This approach does not expose a port; it uses stdio via Docker (`-i`).
- The host SpiderFoot URL is passed via `-e SPIDERFOOT_BASE_URL=http://host.docker.internal:5001`.

### Cursor

Steps:
1. Open Cursor settings for MCP integrations.
2. Add a new MCP server.
3. Use the Docker stdio JSON to launch in a container, or the HTTP example to connect to `http://localhost:5002/mcp`.
4. Save and test by listing tools from the MCP panel.

Cursor – Option 1: Docker stdio

```json
{
  "mcpServers": {
    "spiderfoot-mcp-docker-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--add-host=host.docker.internal:host-gateway",
        "-e",
        "SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
        "spiderfoot-mcp:local",
        "node",
        "dist/index.js"
      ]
    }
  }
}
```

Cursor – Option 2: HTTP

```json
{
  "mcpServers": {
    "spiderfoot-mcp-http": {
      "type": "http",
      "url": "http://localhost:5002/mcp"
    }
  }
}
```

### Claude Desktop

Claude Desktop reads a JSON configuration file that can include the `mcpServers` map shown above.

Typical configuration file locations:
- Windows: `%APPDATA%/Claude/claude_desktop_config.json`
- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
- Linux: `~/.config/Claude/claude_desktop_config.json`

Add or merge one of the following under a top-level `mcpServers` object if your extension reads from it, or under the extension-specific key (e.g., `"cline.mcpServers"`).

Claude Desktop – Option 1: Docker stdio

```json
{
  "mcpServers": {
    "spiderfoot-mcp-docker-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--add-host=host.docker.internal:host-gateway",
        "-e",
        "SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
        "spiderfoot-mcp:local",
        "node",
        "dist/index.js"
      ]
    }
  }
}
```

Claude Desktop – Option 2: HTTP

```json
{
  "mcpServers": {
    "spiderfoot-mcp-http": {
      "type": "http",
      "url": "http://localhost:5002/mcp"
    }
  }
}
```

### VS Code (Continue)

Configuration is typically stored in VS Code `settings.json`.

Common locations:

- Windows: `%APPDATA%/Code/User/settings.json`
- macOS: `~/Library/Application Support/Code/User/settings.json`
- Linux: `~/.config/Code/User/settings.json`

Add or merge the following under a top-level `mcpServers` object if your extension reads from it, or under the extension-specific key (e.g., `"continue.mcpServers"`).

VS Code (Continue) – Option 1: Docker stdio

```json
{
  "mcpServers": {
    "spiderfoot-mcp-docker-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--add-host=host.docker.internal:host-gateway",
        "-e",
        "SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
        "spiderfoot-mcp:local",
        "node",
        "dist/index.js"
      ]
    }
  }
}
```

VS Code (Continue) – Option 2: HTTP

```json
{
  "mcpServers": {
    "spiderfoot-mcp-http": {
      "type": "http",
      "url": "http://localhost:5002/mcp"
    }
  }
}
```

Notes:

- Some VS Code MCP extensions expect a namespaced key (e.g., `continue.mcpServers`). If so, copy the object assigned to `mcpServers` above into that namespaced setting.
- Ensure the working directory (`cwd`) points at `Spiderfoot-MCP-Agent/`.

### VS Code (Cline)

VS Code (Cline) – Option 1: Docker stdio

```json
{
  "mcpServers": {
    "spiderfoot-mcp-docker-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--add-host=host.docker.internal:host-gateway",
        "-e",
        "SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
        "spiderfoot-mcp:local",
        "node",
        "dist/index.js"
      ]
    }
  }
}
```

VS Code (Cline) – Option 2: HTTP

```json
{
  "mcpServers": {
    "spiderfoot-mcp-http": {
      "type": "http",
      "url": "http://localhost:5002/mcp"
    }
  }
}
```

### JetBrains (Continue plugin)

Open your JetBrains IDE settings → Continue → MCP (or Tools/Integrations) and add a server using the same JSON entries shown above.

If your IDE stores a JSON configuration file, place the same `mcpServers` map in that file and restart the IDE. Use stdio or HTTP entries per your preference.

JetBrains (Continue) – Option 1: Docker stdio

```json
{
  "mcpServers": {
    "spiderfoot-mcp-docker-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--add-host=host.docker.internal:host-gateway",
        "-e",
        "SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
        "spiderfoot-mcp:local",
        "node",
        "dist/index.js"
      ]
    }
  }
}
```

JetBrains (Continue) – Option 2: HTTP

```json
{
  "mcpServers": {
    "spiderfoot-mcp-http": {
      "type": "http",
      "url": "http://localhost:5002/mcp"
    }
  }
}
```

### Zed

Open Zed settings JSON (e.g., `~/.config/zed/settings.json`) and add an MCP servers map. For many setups, a root-level `mcpServers` object works; otherwise, consult Zed’s MCP documentation for the exact key.

Zed – Option 1: Docker stdio

```json
{
  "mcpServers": {
    "spiderfoot-mcp-docker-stdio": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--add-host=host.docker.internal:host-gateway",
        "-e",
        "SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
        "spiderfoot-mcp:local",
        "node",
        "dist/index.js"
      ]
    }
  }
}
```

Zed – Option 2: HTTP

```json
{
  "mcpServers": {
    "spiderfoot-mcp-http": {
      "type": "http",
      "url": "http://localhost:5002/mcp"
    }
  }
}
```

### MCP Inspector (testing)

- Stdio: run `npm run dev` and point Inspector to that command.
- HTTP: run Docker Compose (or `npm run dev:http`) and connect Inspector to `http://localhost:5002/mcp`.

## Notes

- Source files are in `src/`:
  - `src/index.ts` – MCP server definition and tool registration (stdio).
  - `src/index-http.ts` – Streamable HTTP transport variant.
  - `src/spiderfootClient.ts` – HTTP wrapper around SpiderFoot endpoints using `axios`.
- The project uses ESM (`"type": "module"`), TypeScript 5, and zod for input validation.
- Default behavior allows starting scans; disable by setting `ALLOW_START_SCAN=false`.