Scrapling MCP Server
Summary: This MCP server exposes session-based web scraping and browser automation capabilities, not the README’s 18 knowledge-ingestion tools.
Open, list, and close reusable stealth browser sessions or HTTP sessions with proxies, cookies, locale/timezone, CDP, and Chrome options.
Send HTTP requests (GET/POST/PUT/DELETE) with auth, headers, cookies, retries, redirects, TLS verification, HTTP/3, and browser impersonation.
Fetch pages via HTTP or JS-rendered browser, including Cloudflare challenge solving, resource blocking, waits, and CSS selectors.
Extract pages/elements as Markdown, HTML, text, or ARIA snapshots, with optional regex search over snapshots.
Automate browser pages: mouse actions, scrolling, form filling, checkbox/radio/select fields, key presses, dialogs, and waits.
Execute JavaScript on the page and return JSON-compatible results.
Inspect recorded network traffic: list/filter/paginate requests and read request/response headers and bodies.
Capture PNG/JPEG screenshots of the full page or individual elements.
Save/load browser state (cookies, localStorage, IndexedDB) to/from Playwright-compatible JSON files.
Provides anonymous web search through DuckDuckGo, returning result titles, URLs, and snippets for agent research.
Enables GitHub repository ingestion, raw file retrieval, issue fetching, and code search, including private repositories via token.
Supports GraphQL schema introspection, returning structured types, queries, and mutations from GraphQL APIs.
Fetches and parses OpenAPI 3.x / Swagger 2.x specs in JSON or YAML into summarized endpoint maps for API discovery.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Scrapling MCP Server Extract the article at https://example.com/blog into markdown"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Scrapling Knowledge Engine — MCP Server
A FastMCP server that gives AI agents and IDEs a single endpoint for knowledge ingestion: web scraping, docs-site crawling, GitHub/repo ingestion, API introspection, PDF extraction, and context bundling — all returned as clean, LLM-ready data.
Overview
Modern AI applications constantly need fresh, structured, external knowledge. Rather than reimplementing scraping, repo ingestion, and API discovery inside every RAG system or agent, this project exposes those capabilities through the Model Context Protocol (MCP).
The server started as a Scrapling-powered web extractor and has grown into a full knowledge engine: 18 tools spanning search, scraping, crawling, GitHub, API introspection, document extraction, and context assembly. It runs over streamable-http on port 8080 and plugs into any MCP-compatible client.
┌──────────────────────┐
│ AI IDE / AI Agent │
│ Kiro / MCP Client │
└──────────┬────────────┘
│ MCP (streamable-http :8080)
▼
┌──────────────────────────────────────┐
│ Scrapling Knowledge Engine │
│ search · scrape · crawl · github · │
│ api · pdf · context · manifest │
└──────────┬────────────────────────────┘
▼
Web · Docs · GitHub · REST/GraphQL · PDFs
▼
Clean Markdown / Structured JSON
▼
RAG · Embeddings · Vector DB · Agent MemoryRelated MCP server: read-website-fast
Tool Catalog (18 tools)
The server is organized into five phases. Every tool returns a consistent dict or list[dict] shape with an error key on failure, and never crashes the server.
Phase 0 — Core Scraping
Tool | Description |
| Anonymous DuckDuckGo search → title + URL + snippet |
| Fast HTTP page fetch → clean LLM markdown |
| Headless browser fetch bypassing Cloudflare/Akamai |
| Batch fetch up to 5 URLs in one call |
Phase 1 — Repo & API Ingestion
Tool | Description |
| Full file tree + raw content for matched extensions (public & private via token) |
| Fetch a single raw file from any repo (auto |
| Issues with title, body, labels, comments (PRs skipped) |
| GitHub code search with repo/language filters + content preview |
| Parse OpenAPI 3.x / Swagger 2.x (JSON or YAML) → summarised endpoint map |
| Introspection query → structured types, queries, mutations |
Phase 2 — Docs & Site Crawling
Tool | Description |
| BFS crawl following internal links → clean markdown per page |
| Parse |
| Download a PDF → extract clean text via |
Phase 3 — Context Management
Tool | Description |
| Classify a URL (docs/repo/openapi/graphql/pdf/blog/generic) → suggested tool |
| Compress content to a char budget, preserving headings + code blocks |
| Multi-source ingest → chunk → rank by topic → single budgeted context block |
Phase 4 — Live API & Tool Calling
Tool | Description |
| Generic authenticated REST caller (any verb, bearer auth, JSON/text decode) |
| Self-describing manifest of all tools (name, schema, example) via introspection |
Health Check
Endpoint | Description |
| Health check for Lightsail / load balancers |
Project Structure
.
├── app/
│ ├── server.py # FastMCP server — registers all 18 tools
│ ├── __init__.py
│ └── tools/
│ ├── __init__.py
│ ├── search.py # search_web
│ ├── scrape.py # fetch_page_knowledge, fetch_stealth_knowledge, batch_knowledge_extract
│ ├── github.py # ingest_github_repo, fetch_github_file, fetch_github_issues, search_github_code
│ ├── api.py # fetch_openapi_spec, fetch_graphql_schema, call_rest_api
│ ├── crawl.py # scrape_docs_site, scrape_sitemap
│ ├── documents.py # fetch_pdf_content
│ ├── context.py # build_context_bundle, summarize_for_context, detect_content_type
│ └── manifest.py # list_available_tools
├── tests/ # pytest suite (mocked, offline)
├── .github/workflows/ # CI: test + build/push to GHCR
├── Dockerfile
├── requirements.txt
├── pyproject.toml # pytest + coverage config
└── README.mdTechnology Stack
Component | Technology |
Protocol | Model Context Protocol (MCP) via FastMCP |
Web Extraction | Scrapling (HTTP + stealth browser) |
HTML → Markdown | markdownify |
Search | duckduckgo_search |
API parsing | PyYAML (OpenAPI), requests (REST/GraphQL) |
Crawling | BeautifulSoup + lxml (sitemap parsing) |
Documents | pypdf |
Language | Python 3.11+ |
Containerization | Docker |
Cloud | AWS Lightsail |
CI/CD | GitHub Actions → GHCR |
Running Locally
1. Clone the repository
git clone https://github.com/Santhosh-p653/scrapling.git
cd scrapling2. Create a virtual environment and install dependencies
python -m venv .venv
# Linux / macOS
source .venv/bin/activate
# Windows
.venv\Scripts\activate
pip install -r requirements.txt
# Download browser binaries for Scrapling stealth mode
scrapling install3. Run the server
python -m app.serverThe server starts on http://0.0.0.0:8080 with the MCP endpoint at /mcp and a health check at /healthz.
Run with Docker
Build the image:
docker build -t scrapling-mcp .Run the container:
docker run -d \
--name scrapling-mcp \
-p 8080:8080 \
scrapling-mcpCheck health:
curl http://localhost:8080/healthzConnecting to Kiro (or any MCP client)
Add the server to your MCP config (~/.kiro/settings/mcp.json for user-level, or .kiro/settings/mcp.json for workspace-level):
{
"mcpServers": {
"scrapling-mcp": {
"url": "http://localhost:8080/mcp",
"disabled": false
}
}
}Then start the server (python -m app.server) and the client will discover all 18 tools. Call list_available_tools from the client to get the full self-describing manifest.
Testing
Tests are fully mocked and run offline (no live network needed).
pip install pytest pytest-cov pytest-mock
pytest tests/ --cov=app --cov-report=term-missingCoverage config and an 80% gate live in pyproject.toml.
CI/CD — GitHub Actions → GHCR
.github/workflows/docker-publish.yml runs two jobs:
test — installs deps, runs pytest with coverage (80% gate).
build-and-push — on push to
mainor av*.*.*tag, builds the image and pushes to GitHub Container Registry (ghcr.io/<owner>/<repo>) withlatest,sha-<short>, and semver tags.
Pull requests run tests only; images are published on merges and tags.
Deploy to AWS Lightsail
GitHub → push → GitHub Actions → test → build → GHCR → Lightsail container service
▼
Scrapling Knowledge Engine
▼
MCP-compatible AI clientsCreate an AWS Lightsail container service.
Point it at the image published to GHCR by CI (or build and push manually).
Expose port
8080and set the health check path to/healthz.Keep any secrets (e.g.
GITHUB_TOKENfor private repo ingestion) in environment variables, never in the repo.
Example Agent Workflows
Research a topic across many sources in one call:
build_context_bundle(
sources=[
"https://github.com/pallets/flask", # repo → ingest_github_repo
"https://flask.palletsprojects.com/", # docs → scrape_docs_site
"flask blueprints best practices", # bare string → search_web
],
topic="flask blueprints and app factories",
max_tokens=4000,
)Discover and call an API:
spec = fetch_openapi_spec("https://api.example.com/openapi.json")
# agent reads the endpoint map, then:
call_rest_api(
url="https://api.example.com/v1/users",
method="POST",
body={"name": "Ada"},
auth_token="...",
)Let the agent auto-route an unknown URL:
route = detect_content_type("https://example.com/paper.pdf")
# → { "type": "pdf", "suggested_tool": "fetch_pdf_content", ... }Design Principles
Consistent shapes — every tool returns
dict/list[dict]with anerrorkey on failure.Never crashes — all tools handle exceptions gracefully.
No storage layer — the server returns raw/cleaned content; embedding and indexing are the agent's job.
No auth on the server itself — handle authentication at the deployment/network level.
Modular — each tool category lives in its own module under
app/tools/.
Security Considerations
Use HTTPS in production and restrict network access.
Validate/allowlist requested URLs where appropriate.
Keep tokens and secrets in environment variables or a secrets manager.
Apply rate limits and monitor resource consumption.
Respect target sites' terms of service and applicable laws.
Future Improvements
Authentication / access control for MCP clients
URL/domain allowlisting
Content caching
Vector DB integration for persistent knowledge
Crawl scheduling and queue-based ingestion
Observability dashboard
Author
Santhosh P — building around AI Agents, RAG, MCP, cloud infrastructure, and web data pipelines.
GitHub: @Santhosh-p653
Key Idea: Deploy the knowledge engine once. Plug it into any AI workflow that needs fresh, structured external knowledge — web, docs, repos, APIs, or PDFs — all through one MCP endpoint.
Available Tools
16 toolsbrowser_actionsBrowser actionsADestructive
Run mouse, field, key, dialog, and wait actions in order; return plain text without a snapshot. Stop on the first error, identifying its action; partial effects remain. No retries. Element mouse targets auto-wait and scroll into view; coordinates and wheel do not wait for navigation or scrolling. Cancelled/timed-out clicks attempt button release. Load waits observe the current document, not future navigation.
Inputs: actions (required), session_id (required), slowly.
browser_evaluateRun JavaScriptADestructive
Run JavaScript on the current page, await promises, and return JSON text without a snapshot. Return JSON-compatible values; null/undefined become null. Script effects are not undone on failure. Cancellation stops waiting, but may not stop the script.
Inputs: expression (required), session_id (required), arg, isolated_context.
browser_extractExtract page contentARead-only
Read the current page or one element as snapshot, HTML, Markdown, or text without navigating. Return plain text; snapshot search keeps refs, ancestor paths and three lines of nearby context.
Inputs: session_id (required), boxes, depth, search, +3 more.
browser_fetchFetch in browser sessionBRead-only
Fetch a URL in an open stealthy browser session. Options apply only to this request.
Inputs: url (required), session_id (required), wait, timeout, +13 more. Returns: Request's response information structure.
browser_fetch_onceFetch page in browserBRead-only
Fetch a page with a stealth browser and JavaScript rendering.
Inputs: url (required), wait, proxy, locale, +23 more. Returns: Request's response information structure.
browser_load_stateLoad browser stateADestructive
Replace browser cookies, local storage and IndexedDB from a Playwright-compatible JSON file.
Leaves sessionStorage and open tabs unchanged. Fetch or reload pages to use the restored login.
Inputs: path (required), session_id (required).
browser_network_requestRead network requestARead-only
Read a complete saved request part without resending it, with structured data and availability notes.
Inputs: request_id (required), session_id (required), part. Returns: One complete saved request part, with an optional note.
browser_network_requestsList network requestsARead-only
List completed requests across page changes, with IDs and pagination. Recording starts when the browser session opens.
Inputs: session_id (required), limit, after_id, url_pattern, +1 more. Returns: Recorded request summaries and pagination details.
browser_openOpen browserA
Open a reusable stealthy browser session. Pass per-request options to browser_fetch.
Inputs: proxy, locale, cdp_url, cookies, +10 more. Returns: Response returned when a new session is created.
browser_save_stateSave browser stateBDestructive
Save browser cookies, local storage and IndexedDB as Playwright-compatible JSON. Excludes sessionStorage.
Inputs: path (required), session_id (required).
browser_screenshotTake browser session screenshotARead-only
Capture the current page or one element without navigating; return the image and current URL. Element capture waits and scrolls into view. Use browser_actions for other waits before capture.
Inputs: session_id (required), ref, quality, timeout, +3 more.
close_sessionClose sessionB
Close a session and free its resources.
Inputs: session_id (required). Returns: Response returned when a session is closed.
list_sessionsList sessionsBRead-only
List registered sessions, their status, and effective settings.
Inputs: no named parameters. Returns: result
make_requestSend HTTP requestBRead-only
Make an HTTP request without JavaScript. Suitable for low-to-mid protection.
Inputs: url (required), auth, data, json, +18 more. Returns: Request's response information structure.
open_request_sessionOpen HTTP sessionA
Open a reusable HTTP session for session_make_request, keeping cookies, connections, and request settings.
Inputs: proxy, session_id, impersonate. Returns: Response returned when a new session is created.
session_make_requestSend HTTP request in HTTP sessionBRead-only
Make an HTTP request using the session's cookies, connections, impersonation, and proxy. Options apply to this request.
Inputs: url (required), session_id (required), auth, data, +16 more. Returns: Request's response information structure.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 16 tools
The set contains several close pairs whose boundaries require careful reading of descriptions: make_request vs session_make_request, browser_fetch vs browser_fetch_once, and browser_network_requests vs browser_network_request. The distinctions exist (stateless vs session-bound, one-off vs persistent, list vs detail), but an agent can easily misselect without parsing the long descriptions.
There are two coherent families: browser_* prefixed tools and session/verb-first tools (open_request_session, close_session, list_sessions, session_make_request, make_request). Within each family it is readable, but the mixed conventions across the set (browser_open vs open_request_session, bare make_request vs session_make_request) create inconsistency.
16 tools is borderline heavy for the surface. Many are justified (fetch variants, network inspection), but the fetch/request pairs (browser_fetch vs browser_fetch_once, make_request vs session_make_request) inflate the count and add selection burden without adding fundamentally new capability.
The surface covers the full scraping lifecycle: session lifecycle (open/close/list), state save/load, multiple fetch/request modes, extraction, actions, evaluation, network inspection, and screenshots. Minor gaps exist (no explicit browser teardown distinct from close_session, no cookie-specific tool), but the core workflow is well covered.
Maintenance
Related MCP Connectors
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Web scraping, browser automation, screenshots and AI data extraction for AI agents via MCP.
Turn public web pages into clean Markdown for LLMs, RAG and AI agents
Web pages as Markdown, site maps and crawls, AI data extraction and screenshots.
1
Related MCP Servers
- FlicenseNot gradedqualityAmaintenanceEnables AI agents and MCP clients to fetch webpages and crawl sites, extract main content, and convert it into clean Markdown with preserved links for fast, token-efficient scraping.-
- AlicenseNot gradedqualityAmaintenanceEnables AI agents and MCP clients to fetch web pages or crawl sites and convert them into clean, token-efficient Markdown while preserving links.MIT
- AlicenseNot gradedqualityAmaintenanceEnables MCP clients and AI agents to fetch webpages and crawl sites, strip noise, and convert content into clean, link-preserving Markdown with caching and configurable multi-page depth while minimizing token usage.MIT
- AlicenseNot gradedqualityAmaintenanceEnables AI agents and MCP clients to fetch webpages, strip noise, and convert them into clean, token-efficient Markdown. Supports optional crawling, caching, robots.txt compliance, and link preservation.MIT