Python Web Search & Scraping MCP
by jatsus22
README.md
# GitHub Backend — Python Web Search & Scraping MCP
Independent Python MCP server for **Google Custom Search** and Python-only public-page extraction. This is the standalone deployment companion for the existing ChatGPT **Web Search API** plugin.
## Routes
| Route | Purpose | Authentication |
|---|---|---|
| `GET /health` | Readiness endpoint | Public |
| `POST /extract` | Python page extraction for existing plugin | Bearer token |
| `/mcp` | Streamable HTTP MCP with five tools | Bearer token |
MCP tools: `connection_status`, `google_search`, `scrape_page`, `scrape_pages` (up to 20), and `search_and_scrape`.
This scraper does **not** render JavaScript. It only accesses public HTTPS resources and reports publisher restrictions, including HTTP 403. Publishing the source does not bypass such restrictions.
### Official documentation fallback
When `https://6sense.com/platform/revvyai` returns HTTP 403, extraction makes one independent request to the publicly accessible, publisher-owned `https://support.6sense.com/docs/revvyai-in-sales-intelligence`. The fallback is limited to this curated RevvyAI URL mapping; arbitrary sites returning 403 remain restricted. The implementation does not attempt to defeat a publisher's access controls.
Successful fallback responses retain the requested URL in `url`, identify the actual alternative in `fallback_source_url`, and carry `original_http_status: 403`, `source_type: official_documentation_fallback`, and `fallback_used: true`. The `http_status` describes the **alternative document**, not the blocked original. If the alternative fails, the original `restricted` result is retained with `fallback_attempted_url` and `fallback_status` metadata. Consumers should attribute retrieved text to the alternative document, never the original page.
## Publish public source on GitHub
1. Use the public repository **[jatsus22/python-web-search-mcp](https://github.com/jatsus22/python-web-search-mcp)**. This repository is the source for the Python MCP backend.
2. Upload the **contents of this folder**, preserving `python_mcp/`, `tests/`, `.github/workflows/`, and the root `Dockerfile`. The ZIP is a transfer artifact; do not upload the ZIP itself as the only repository file.
3. Or, from this folder using Git:
```bash
git init
git add .
git commit -m "Initial GitHub Python MCP backend"
git branch -M main
git remote add origin https://github.com/jatsus22/python-web-search-mcp.git
git push -u origin main
```
**Never commit API keys or `SCRAPER_SERVICE_TOKEN`.** Source defaults intentionally contain no secret. Set secrets only in the deployment configuration.
## Deploy the Python MCP backend from GitHub (Render)
GitHub stores the source. A Python-capable host runs the server and provides the HTTPS endpoints. Render can build the repository's Dockerfile using the included `render.yaml` Blueprint; choose a hosting plan suitable for your uptime and cost requirements.
1. Push this repository to GitHub without including any `.env` file or secret.
2. In the [Render dashboard](https://dashboard.render.com/), choose **New → Blueprint** and connect this GitHub repository. Render reads `render.yaml` and builds the root `Dockerfile`.
3. When prompted for deployment environment variables, enter:
| Variable | Value |
|---|---|
| `SCRAPER_SERVICE_TOKEN` | A strong random secret, generated locally and stored only in Render and your Web Search API Site's environment |
| `GOOGLE_API_KEY` | Your Google Custom Search API key; optional for page scraping, required for Google search |
`MCP_HOST=0.0.0.0` and `GOOGLE_CX=871edc6cb49394af9` are declared in `render.yaml`. The Dockerfile defaults to port `8080`. Never add the token or Google key to the public repository.
4. After deployment, open `https://YOUR_SERVICE.onrender.com/health`. A JSON record with `"status":"ok"` confirms that the Python process is running. Test extraction with a bearer token from a private terminal:
```bash
curl -fsS https://YOUR_SERVICE.onrender.com/health
curl -fsS -X POST https://YOUR_SERVICE.onrender.com/extract \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $SCRAPER_SERVICE_TOKEN" \
-d '{"url":"https://www.python.org/"}'
```
5. The direct MCP endpoint is `https://YOUR_SERVICE.onrender.com/mcp` (Streamable HTTP). Clients must send the same bearer token. Integrating it into ChatGPT may require compatible OAuth or another supported connection/authentication mechanism.
6. To connect the existing **Web Search API Site** to this Python service, set its production environment variables:
```text
PYTHON_SCRAPER_URL=https://YOUR_SERVICE.onrender.com/extract
SCRAPER_SERVICE_TOKEN=<the same bearer secret>
```
Verify the Site's `connection_status` reports `python_scraper.configured: true`, then call `scrape_page_python` and inspect the genuine extraction result. GitHub deployment alone cannot make that status true.
## Run locally
```bash
python -m pip install -r requirements.txt
export GOOGLE_API_KEY='your-key' # optional, required for Google search
export SCRAPER_SERVICE_TOKEN='test-only-secret'
MCP_HOST=127.0.0.1 PORT=8897 python -m python_mcp.server
```
In another terminal:
```bash
python -m unittest discover -s tests -v
MCP_URL=http://127.0.0.1:8897/mcp python -m python_mcp.smoke_test
```
`smoke_test.py` automatically sends the bearer header when `SCRAPER_SERVICE_TOKEN` is set. Never disclose a production token in test logs.
## Notes
- A public GitHub repository is **source hosting**, not runtime hosting. GitHub Pages cannot run Python server code; Render or another Python-compatible host runs the MCP service.
- Running the container on a public interface without `SCRAPER_SERVICE_TOKEN` is intentionally rejected.
- `/mcp` and `/extract` fail closed with HTTP 503 if the bearer token is missing, even when a generic ASGI launcher is used.
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues