Skip to main content
Glama
Python Web Search & Scraping MCP

Python Web Search & Scraping MCP

by jatsus22

GitHub Backend — Python Web Search & Scraping MCP

Independent Python MCP server for Google Custom Search and Python-only public-page extraction. This is the standalone deployment companion for the existing ChatGPT Web Search API plugin.

Routes

Route

Purpose

Authentication

GET /health

Readiness endpoint

Public

POST /extract

Python page extraction for existing plugin

Bearer token

/mcp

Streamable HTTP MCP with five tools

Bearer token

MCP tools: connection_status, google_search, scrape_page, scrape_pages (up to 20), and search_and_scrape.

This scraper does not render JavaScript. It only accesses public HTTPS resources and reports publisher restrictions, including HTTP 403. Publishing the source does not bypass such restrictions.

Official documentation fallback

When https://6sense.com/platform/revvyai returns HTTP 403, extraction makes one independent request to the publicly accessible, publisher-owned https://support.6sense.com/docs/revvyai-in-sales-intelligence. The fallback is limited to this curated RevvyAI URL mapping; arbitrary sites returning 403 remain restricted. The implementation does not attempt to defeat a publisher's access controls.

Successful fallback responses retain the requested URL in url, identify the actual alternative in fallback_source_url, and carry original_http_status: 403, source_type: official_documentation_fallback, and fallback_used: true. The http_status describes the alternative document, not the blocked original. If the alternative fails, the original restricted result is retained with fallback_attempted_url and fallback_status metadata. Consumers should attribute retrieved text to the alternative document, never the original page.

Related MCP server: Google Search MCP Server

Publish public source on GitHub

  1. Use the public repository jatsus22/python-web-search-mcp. This repository is the source for the Python MCP backend.

  2. Upload the contents of this folder, preserving python_mcp/, tests/, .github/workflows/, and the root Dockerfile. The ZIP is a transfer artifact; do not upload the ZIP itself as the only repository file.

  3. Or, from this folder using Git:

    git init
    git add .
    git commit -m "Initial GitHub Python MCP backend"
    git branch -M main
    git remote add origin https://github.com/jatsus22/python-web-search-mcp.git
    git push -u origin main

Never commit API keys or SCRAPER_SERVICE_TOKEN. Source defaults intentionally contain no secret. Set secrets only in the deployment configuration.

Deploy the Python MCP backend from GitHub (Render)

GitHub stores the source. A Python-capable host runs the server and provides the HTTPS endpoints. Render can build the repository's Dockerfile using the included render.yaml Blueprint; choose a hosting plan suitable for your uptime and cost requirements.

  1. Push this repository to GitHub without including any .env file or secret.

  2. In the Render dashboard, choose New → Blueprint and connect this GitHub repository. Render reads render.yaml and builds the root Dockerfile.

  3. When prompted for deployment environment variables, enter:

    Variable

    Value

    SCRAPER_SERVICE_TOKEN

    A strong random secret, generated locally and stored only in Render and your Web Search API Site's environment

    GOOGLE_API_KEY

    Your Google Custom Search API key; optional for page scraping, required for Google search

    MCP_HOST=0.0.0.0 and GOOGLE_CX=871edc6cb49394af9 are declared in render.yaml. The Dockerfile defaults to port 8080. Never add the token or Google key to the public repository.

  4. After deployment, open https://YOUR_SERVICE.onrender.com/health. A JSON record with "status":"ok" confirms that the Python process is running. Test extraction with a bearer token from a private terminal:

    curl -fsS https://YOUR_SERVICE.onrender.com/health
    curl -fsS -X POST https://YOUR_SERVICE.onrender.com/extract \
      -H 'Content-Type: application/json' \
      -H "Authorization: Bearer $SCRAPER_SERVICE_TOKEN" \
      -d '{"url":"https://www.python.org/"}'
  5. The direct MCP endpoint is https://YOUR_SERVICE.onrender.com/mcp (Streamable HTTP). Clients must send the same bearer token. Integrating it into ChatGPT may require compatible OAuth or another supported connection/authentication mechanism.

  6. To connect the existing Web Search API Site to this Python service, set its production environment variables:

    PYTHON_SCRAPER_URL=https://YOUR_SERVICE.onrender.com/extract
    SCRAPER_SERVICE_TOKEN=<the same bearer secret>

    Verify the Site's connection_status reports python_scraper.configured: true, then call scrape_page_python and inspect the genuine extraction result. GitHub deployment alone cannot make that status true.

Run locally

python -m pip install -r requirements.txt
export GOOGLE_API_KEY='your-key'   # optional, required for Google search
export SCRAPER_SERVICE_TOKEN='test-only-secret'
MCP_HOST=127.0.0.1 PORT=8897 python -m python_mcp.server

In another terminal:

python -m unittest discover -s tests -v
MCP_URL=http://127.0.0.1:8897/mcp python -m python_mcp.smoke_test

smoke_test.py automatically sends the bearer header when SCRAPER_SERVICE_TOKEN is set. Never disclose a production token in test logs.

Notes

  • A public GitHub repository is source hosting, not runtime hosting. GitHub Pages cannot run Python server code; Render or another Python-compatible host runs the MCP service.

  • Running the container on a public interface without SCRAPER_SERVICE_TOKEN is intentionally rejected.

  • /mcp and /extract fail closed with HTTP 503 if the bearer token is missing, even when a generic ASGI launcher is used.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      B
      quality
      D
      maintenance
      Enables web searching and content scraping through Google Custom Search API. Provides tools to search the internet, extract webpage content, and automatically scrape search results for comprehensive information gathering.
      3
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables Google search functionality through the Google Custom Search API with enterprise-grade security authentication. Supports streamable HTTP transport protocol for real-time search operations with configurable result limits.
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Enables JavaScript-rendered Google searches and readable text extraction from public URLs via a real Chrome instance, along with health and CAPTCHA status monitoring.
      3
      69
      MIT
    • F
      license
      Not graded
      quality
      C
      maintenance
      Enables aggregated web searches across multiple providers with automatic failover, exposing search and source-listing tools via a streamable HTTP MCP endpoint secured by bearer token authentication.
      -