# web-explorer

External web tools for any MCP client. Search the web, crawl pages, drive an
anti-detect Firefox browser past bot walls, and pull structured data from
Wikipedia, Wikidata, GitHub, and YouTube.

Two things worth stating up front:

- **Free APIs only.** No paid services, no metered quotas. It uses DuckDuckGo,
  Wikipedia, Wikidata, the public GitHub API, and YouTube's own caption
  endpoints. The only credential any tool reads is a GitHub token, which is free
  and which only `github` uses (for code search and issue actions; plain file
  and repo reads work without one). See Credentials below.
- **Runs off your own IP.** Every request goes out from the machine running the
  server, straight to the source. No proxy, no relay, no third-party scraping
  service in the middle reselling your traffic.

## Tools

| Tool | What it does |
|------|--------------|
| `web` | DuckDuckGo search and direct page crawl with readable-text extraction. News search too. |
| `fx` | Anti-detect Firefox (Camoufox) for bot-protected sites (Cloudflare, Datadome, verification walls). Persistent profile keeps cookies and solved challenges across restarts. Optional install, see below. |
| `wiki` | Wikipedia search and full-article fetch. |
| `wikidata_query` | Wikidata SPARQL queries for structured entity data (people, places, orgs, relationships). |
| `github` | Fetch files, analyze repo structure, search code, and read or manage issues via the public GitHub API. Code search and issue actions need a free `GITHUB_TOKEN`; file and repo reads do not. |
| `youtube` | Transcripts and metadata for any video. youtube-transcript-api first, yt-dlp fallback. Any URL shape or bare video id. |

Every tool takes an `action` argument, except `wikidata_query` which takes a
SPARQL `query`. Call `list_tools` from your client for the exact schema of every
action and parameter. What each action does:

### `web`

- `search` - web results as title, url, and snippet. `news=true` switches to the
  DuckDuckGo news endpoint (adds date and source per result); `timelimit` filters
  to `d`, `w`, or `m`.
- `crawl` - fetch one URL and extract readable article text (trafilatura, with a
  BeautifulSoup fallback), including PDFs. `summary=true` (default) returns a
  short version, `summary=false` the fuller text.

Results are cached for 20 minutes. If a page is behind bot protection, `crawl`
points you at `fx`.

### `fx` (optional install)

A real Camoufox (hardened Firefox) session for sites that block plain fetches.

- `read` - navigate, wait out JS challenges, extract clean text, auto-close. The
  common one-shot case.
- Step-by-step automation: `navigate`, `click`, `fill`, `type`, `press`,
  `scroll`, `wait`, `extract` (raw HTML), `text` (body text), `evaluate` (run
  JavaScript), `screenshot` (element or `full_page`), `cookies` / `add_cookies`,
  and `close`.
- `status` - report the current session.

Options include `visible` (show the window), `stealth` (humanized cursor), and
`load_images`. The profile persists to disk; see the note under Safety.

### `wiki`

- `search` - article titles matching a query.
- `page` - full article text for a title.

Cached for one hour.

### `wikidata_query`

Run a read-only SPARQL query against Wikidata for structured entity data. Limits:
5000 character query, 15 second timeout, 1000 results. Update operations (INSERT,
DELETE, DROP, and so on) are rejected.

### `github`

- `fetch` - raw file content. `analyze` - repo metadata, languages, file count.
- `search` - code search (requires a token).
- Issues: `get_issue`, `list_issues`, `create_issue`, `comment_issue`,
  `update_issue`. The write actions require a token. See Credentials.

Cached for 10 minutes.

### `youtube`

- `transcript` - captions only. `metadata` - title, channel, duration,
  description, chapters. `info` - both at once (the common case).
- `list_languages` - available caption tracks. `search` - find a phrase within a
  transcript, returning timestamped matches with `t=` deep links.

Transcript `format` can be `text` (default), `timestamped`, `srt`, `vtt`, or
`json`. Long transcripts auto-paginate with `segment_offset` and `max_segments`.

### Saving context: `harvest=True`

`web`, `fx`, `github`, and `youtube` accept `harvest=True`. Instead of returning
a large page or transcript into your context window, the content is written to
`harvested/<harvest_dest>/` as clean markdown and only the file path plus a short
summary come back. Point `HARVEST_ROOT` at a different directory if you want.

## Install

Python 3.10 or newer.

Install as a package to get a `web-explorer` command on your PATH:

```
pipx install git+https://github.com/cutlerbenjamin1-cmd/web-explorer
```

From a local checkout: `pip install .` Or just install the dependencies and run `server.py` directly:

```
pip install -r requirements.txt
```

That covers the server and the five free-API tools (`web`, `wiki`,
`wikidata_query`, `github`, `youtube`).

### Optional: the `fx` browser

The anti-detect browser is a heavier dependency, so it is opt-in. The server and
the other tools work fine without it; `fx` just returns a clear "not installed"
message until you add it.

```
pip install -r requirements-fx.txt   # or: pip install "web-explorer[fx]"
python -m camoufox fetch
```

`python -m camoufox fetch` is a one-time download of a patched Firefox build
(about 150 MB). camoufox also pulls in playwright and browserforge.

## Use it with an MCP client

Speaks MCP over stdio. If you installed the package, point your client at the `web-explorer` command:

```json
{
  "mcpServers": {
    "web-explorer": {
      "command": "web-explorer"
    }
  }
}
```

Otherwise point `python` at `server.py` with an absolute path:

```json
{
  "mcpServers": {
    "web-explorer": {
      "command": "python",
      "args": ["/absolute/path/to/web-explorer/server.py"]
    }
  }
}
```

## Running over HTTP (optional)

To serve this over MCP streamable HTTP instead of stdio, use
[mcp-http-bridge](https://github.com/cutlerbenjamin1-cmd/mcp-http-bridge): run it
from this directory and point your client at the bridge. Keep the bridge's tool
filter tight if you expose it beyond localhost, since `fx` can drive a real
browser.

## Safety

- **Untrusted-content banner.** Everything these tools return is data from
  outside your machine. Each response is tagged as external and untrusted so an
  agent does not treat page content as instructions.
- **SSRF protection.** `web(crawl)` refuses private and loopback address ranges,
  so a crawled URL cannot be pointed back at your internal network.
- **Polite rate limiting.** Per-tool cooldowns space out requests so you do not
  hammer a host from your own IP and trip a 429.
- **The `fx` profile is real browser state.** `fx` keeps a persistent Firefox
  profile at `data/fx_profile/`. Anything you do in it sticks: if you sign into a
  site through `fx`, that session and its cookies are written to disk there in the
  clear, exactly like a normal browser profile. That is what lets logins and
  solved Cloudflare challenges survive a restart. The folder is gitignored so it
  will not be committed, but treat it like a password store: do not share or copy
  it, and delete `data/fx_profile/` to wipe every saved session.

## Credentials

Only the `github` tool reads a credential, and it is an ordinary free GitHub
personal access token.

- Without a token: `github` file fetch and repo analyze work, subject to
  GitHub's anonymous rate limit (about 60 requests per hour).
- With a token: that limit rises to 5000 per hour, and `github` code search plus
  the issue actions (create, comment, update) become available. They return a
  clear "token required" error otherwise.

Set it in the environment before launching the server:

```
set GITHUB_TOKEN=ghp_your_token_here        # Windows
export GITHUB_TOKEN=ghp_your_token_here      # macOS / Linux
```

No other tool needs an account or key. DuckDuckGo, Wikipedia, Wikidata, and
YouTube are all used unauthenticated.

## Environment variables

Every variable below is optional.

| Variable | Effect |
|----------|--------|
| `GITHUB_TOKEN` | GitHub auth. See Credentials above. |
| `HARVEST_ROOT` | Directory for `harvest=True` output. Default: `./harvested`. |
| `YOUTUBE_COOKIES_BROWSER` | Browser name (for example `firefox`) so yt-dlp can borrow its cookies when a video is IP-blocked. |
| `YOUTUBE_COOKIES_PROFILE` | Browser profile path for the cookies above. |
| `YOUTUBE_BIND_IP` | Local source IP to bind outbound YouTube requests to. |
| `MCP_DEBUG` | Set to `true` for stderr debug logging. |
| `OUTPUT_MAX_CHARS` | Hard cap on a single tool response. Default: 15000. |
| `GITHUB_MAX_RESPONSE_BYTES` | Max GitHub file size fetched. Default: about 2MB. |
| `WEB_CRAWL_MAX_BYTES` | Max HTML or text size for `web(crawl)`. Default: about 2MB. |
| `WEB_CRAWL_MAX_PDF_BYTES` | Max PDF size for `web(crawl)`. Default: about 50MB. |

## License

MIT. See LICENSE.