Skip to main content
Glama
Scrapling MCP Server

Scrapling MCP Server

Scrapling Knowledge Engine — MCP Server

A FastMCP server that gives AI agents and IDEs a single endpoint for knowledge ingestion: web scraping, docs-site crawling, GitHub/repo ingestion, API introspection, PDF extraction, and context bundling — all returned as clean, LLM-ready data.

AWS MCP Python Docker Scrapling

Overview

Modern AI applications constantly need fresh, structured, external knowledge. Rather than reimplementing scraping, repo ingestion, and API discovery inside every RAG system or agent, this project exposes those capabilities through the Model Context Protocol (MCP).

The server started as a Scrapling-powered web extractor and has grown into a full knowledge engine: 18 tools spanning search, scraping, crawling, GitHub, API introspection, document extraction, and context assembly. It runs over streamable-http on port 8080 and plugs into any MCP-compatible client.

┌──────────────────────┐
│   AI IDE / AI Agent   │
│  Kiro / MCP Client    │
└──────────┬────────────┘
           │ MCP (streamable-http :8080)
           ▼
┌──────────────────────────────────────┐
│       Scrapling Knowledge Engine       │
│  search · scrape · crawl · github ·    │
│  api · pdf · context · manifest        │
└──────────┬────────────────────────────┘
           ▼
   Web · Docs · GitHub · REST/GraphQL · PDFs
           ▼
     Clean Markdown / Structured JSON
           ▼
   RAG · Embeddings · Vector DB · Agent Memory

Related MCP server: read-website-fast

Tool Catalog (18 tools)

The server is organized into five phases. Every tool returns a consistent dict or list[dict] shape with an error key on failure, and never crashes the server.

Phase 0 — Core Scraping

Tool

Description

search_web

Anonymous DuckDuckGo search → title + URL + snippet

fetch_page_knowledge

Fast HTTP page fetch → clean LLM markdown

fetch_stealth_knowledge

Headless browser fetch bypassing Cloudflare/Akamai

batch_knowledge_extract

Batch fetch up to 5 URLs in one call

Phase 1 — Repo & API Ingestion

Tool

Description

ingest_github_repo

Full file tree + raw content for matched extensions (public & private via token)

fetch_github_file

Fetch a single raw file from any repo (auto main→master fallback)

fetch_github_issues

Issues with title, body, labels, comments (PRs skipped)

search_github_code

GitHub code search with repo/language filters + content preview

fetch_openapi_spec

Parse OpenAPI 3.x / Swagger 2.x (JSON or YAML) → summarised endpoint map

fetch_graphql_schema

Introspection query → structured types, queries, mutations

Phase 2 — Docs & Site Crawling

Tool

Description

scrape_docs_site

BFS crawl following internal links → clean markdown per page

scrape_sitemap

Parse sitemap.xml (incl. index files) → batch-fetch pages as markdown

fetch_pdf_content

Download a PDF → extract clean text via pypdf (handles encryption)

Phase 3 — Context Management

Tool

Description

detect_content_type

Classify a URL (docs/repo/openapi/graphql/pdf/blog/generic) → suggested tool

summarize_for_context

Compress content to a char budget, preserving headings + code blocks

build_context_bundle

Multi-source ingest → chunk → rank by topic → single budgeted context block

Phase 4 — Live API & Tool Calling

Tool

Description

call_rest_api

Generic authenticated REST caller (any verb, bearer auth, JSON/text decode)

list_available_tools

Self-describing manifest of all tools (name, schema, example) via introspection

Health Check

Endpoint

Description

GET /healthz

Health check for Lightsail / load balancers

Project Structure

.
├── app/
│   ├── server.py              # FastMCP server — registers all 18 tools
│   ├── __init__.py
│   └── tools/
│       ├── __init__.py
│       ├── search.py          # search_web
│       ├── scrape.py          # fetch_page_knowledge, fetch_stealth_knowledge, batch_knowledge_extract
│       ├── github.py          # ingest_github_repo, fetch_github_file, fetch_github_issues, search_github_code
│       ├── api.py             # fetch_openapi_spec, fetch_graphql_schema, call_rest_api
│       ├── crawl.py           # scrape_docs_site, scrape_sitemap
│       ├── documents.py       # fetch_pdf_content
│       ├── context.py         # build_context_bundle, summarize_for_context, detect_content_type
│       └── manifest.py        # list_available_tools
├── tests/                     # pytest suite (mocked, offline)
├── .github/workflows/         # CI: test + build/push to GHCR
├── Dockerfile
├── requirements.txt
├── pyproject.toml             # pytest + coverage config
└── README.md

Technology Stack

Component

Technology

Protocol

Model Context Protocol (MCP) via FastMCP

Web Extraction

Scrapling (HTTP + stealth browser)

HTML → Markdown

markdownify

Search

duckduckgo_search

API parsing

PyYAML (OpenAPI), requests (REST/GraphQL)

Crawling

BeautifulSoup + lxml (sitemap parsing)

Documents

pypdf

Language

Python 3.11+

Containerization

Docker

Cloud

AWS Lightsail

CI/CD

GitHub Actions → GHCR

Running Locally

1. Clone the repository

git clone https://github.com/Santhosh-p653/scrapling.git
cd scrapling

2. Create a virtual environment and install dependencies

python -m venv .venv

# Linux / macOS
source .venv/bin/activate
# Windows
.venv\Scripts\activate

pip install -r requirements.txt

# Download browser binaries for Scrapling stealth mode
scrapling install

3. Run the server

python -m app.server

The server starts on http://0.0.0.0:8080 with the MCP endpoint at /mcp and a health check at /healthz.

Run with Docker

Build the image:

docker build -t scrapling-mcp .

Run the container:

docker run -d \
  --name scrapling-mcp \
  -p 8080:8080 \
  scrapling-mcp

Check health:

curl http://localhost:8080/healthz

Connecting to Kiro (or any MCP client)

Add the server to your MCP config (~/.kiro/settings/mcp.json for user-level, or .kiro/settings/mcp.json for workspace-level):

{
  "mcpServers": {
    "scrapling-mcp": {
      "url": "http://localhost:8080/mcp",
      "disabled": false
    }
  }
}

Then start the server (python -m app.server) and the client will discover all 18 tools. Call list_available_tools from the client to get the full self-describing manifest.

Testing

Tests are fully mocked and run offline (no live network needed).

pip install pytest pytest-cov pytest-mock
pytest tests/ --cov=app --cov-report=term-missing

Coverage config and an 80% gate live in pyproject.toml.

CI/CD — GitHub Actions → GHCR

.github/workflows/docker-publish.yml runs two jobs:

  1. test — installs deps, runs pytest with coverage (80% gate).

  2. build-and-push — on push to main or a v*.*.* tag, builds the image and pushes to GitHub Container Registry (ghcr.io/<owner>/<repo>) with latest, sha-<short>, and semver tags.

Pull requests run tests only; images are published on merges and tags.

Deploy to AWS Lightsail

GitHub → push → GitHub Actions → test → build → GHCR → Lightsail container service
                                                              ▼
                                                 Scrapling Knowledge Engine
                                                              ▼
                                                   MCP-compatible AI clients
  1. Create an AWS Lightsail container service.

  2. Point it at the image published to GHCR by CI (or build and push manually).

  3. Expose port 8080 and set the health check path to /healthz.

  4. Keep any secrets (e.g. GITHUB_TOKEN for private repo ingestion) in environment variables, never in the repo.

Example Agent Workflows

Research a topic across many sources in one call:

build_context_bundle(
    sources=[
        "https://github.com/pallets/flask",      # repo → ingest_github_repo
        "https://flask.palletsprojects.com/",     # docs → scrape_docs_site
        "flask blueprints best practices",        # bare string → search_web
    ],
    topic="flask blueprints and app factories",
    max_tokens=4000,
)

Discover and call an API:

spec = fetch_openapi_spec("https://api.example.com/openapi.json")
# agent reads the endpoint map, then:
call_rest_api(
    url="https://api.example.com/v1/users",
    method="POST",
    body={"name": "Ada"},
    auth_token="...",
)

Let the agent auto-route an unknown URL:

route = detect_content_type("https://example.com/paper.pdf")
# → { "type": "pdf", "suggested_tool": "fetch_pdf_content", ... }

Design Principles

  • Consistent shapes — every tool returns dict/list[dict] with an error key on failure.

  • Never crashes — all tools handle exceptions gracefully.

  • No storage layer — the server returns raw/cleaned content; embedding and indexing are the agent's job.

  • No auth on the server itself — handle authentication at the deployment/network level.

  • Modular — each tool category lives in its own module under app/tools/.

Security Considerations

  • Use HTTPS in production and restrict network access.

  • Validate/allowlist requested URLs where appropriate.

  • Keep tokens and secrets in environment variables or a secrets manager.

  • Apply rate limits and monitor resource consumption.

  • Respect target sites' terms of service and applicable laws.

Future Improvements

  • Authentication / access control for MCP clients

  • URL/domain allowlisting

  • Content caching

  • Vector DB integration for persistent knowledge

  • Crawl scheduling and queue-based ingestion

  • Observability dashboard

Author

Santhosh P — building around AI Agents, RAG, MCP, cloud infrastructure, and web data pipelines.

GitHub: @Santhosh-p653


Key Idea: Deploy the knowledge engine once. Plug it into any AI workflow that needs fresh, structured external knowledge — web, docs, repos, APIs, or PDFs — all through one MCP endpoint.

Available Tools

16 tools
browser_actionsBrowser actionsA
Destructive

Run mouse, field, key, dialog, and wait actions in order; return plain text without a snapshot. Stop on the first error, identifying its action; partial effects remain. No retries. Element mouse targets auto-wait and scroll into view; coordinates and wheel do not wait for navigation or scrolling. Cancelled/timed-out clicks attempt button release. Load waits observe the current document, not future navigation.

Inputs: actions (required), session_id (required), slowly.

browser_evaluateRun JavaScriptA
Destructive

Run JavaScript on the current page, await promises, and return JSON text without a snapshot. Return JSON-compatible values; null/undefined become null. Script effects are not undone on failure. Cancellation stops waiting, but may not stop the script.

Inputs: expression (required), session_id (required), arg, isolated_context.

browser_extractExtract page contentA
Read-only

Read the current page or one element as snapshot, HTML, Markdown, or text without navigating. Return plain text; snapshot search keeps refs, ancestor paths and three lines of nearby context.

Inputs: session_id (required), boxes, depth, search, +3 more.

browser_fetchFetch in browser sessionB
Read-only

Fetch a URL in an open stealthy browser session. Options apply only to this request.

Inputs: url (required), session_id (required), wait, timeout, +13 more. Returns: Request's response information structure.

browser_fetch_onceFetch page in browserB
Read-only

Fetch a page with a stealth browser and JavaScript rendering.

Inputs: url (required), wait, proxy, locale, +23 more. Returns: Request's response information structure.

browser_load_stateLoad browser stateA
Destructive

Replace browser cookies, local storage and IndexedDB from a Playwright-compatible JSON file.

Leaves sessionStorage and open tabs unchanged. Fetch or reload pages to use the restored login.

Inputs: path (required), session_id (required).

browser_network_requestRead network requestA
Read-only

Read a complete saved request part without resending it, with structured data and availability notes.

Inputs: request_id (required), session_id (required), part. Returns: One complete saved request part, with an optional note.

browser_network_requestsList network requestsA
Read-only

List completed requests across page changes, with IDs and pagination. Recording starts when the browser session opens.

Inputs: session_id (required), limit, after_id, url_pattern, +1 more. Returns: Recorded request summaries and pagination details.

browser_openOpen browserA

Open a reusable stealthy browser session. Pass per-request options to browser_fetch.

Inputs: proxy, locale, cdp_url, cookies, +10 more. Returns: Response returned when a new session is created.

browser_save_stateSave browser stateB
Destructive

Save browser cookies, local storage and IndexedDB as Playwright-compatible JSON. Excludes sessionStorage.

Inputs: path (required), session_id (required).

browser_screenshotTake browser session screenshotA
Read-only

Capture the current page or one element without navigating; return the image and current URL. Element capture waits and scrolls into view. Use browser_actions for other waits before capture.

Inputs: session_id (required), ref, quality, timeout, +3 more.

close_sessionClose sessionB

Close a session and free its resources.

Inputs: session_id (required). Returns: Response returned when a session is closed.

list_sessionsList sessionsB
Read-only

List registered sessions, their status, and effective settings.

Inputs: no named parameters. Returns: result

make_requestSend HTTP requestB
Read-only

Make an HTTP request without JavaScript. Suitable for low-to-mid protection.

Inputs: url (required), auth, data, json, +18 more. Returns: Request's response information structure.

open_request_sessionOpen HTTP sessionA

Open a reusable HTTP session for session_make_request, keeping cookies, connections, and request settings.

Inputs: proxy, session_id, impersonate. Returns: Response returned when a new session is created.

session_make_requestSend HTTP request in HTTP sessionB
Read-only

Make an HTTP request using the session's cookies, connections, impersonation, and proxy. Options apply to this request.

Inputs: url (required), session_id (required), auth, data, +16 more. Returns: Request's response information structure.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.4/5.0

    Scored across 16 tools

    Disambiguation3/5

    The set contains several close pairs whose boundaries require careful reading of descriptions: make_request vs session_make_request, browser_fetch vs browser_fetch_once, and browser_network_requests vs browser_network_request. The distinctions exist (stateless vs session-bound, one-off vs persistent, list vs detail), but an agent can easily misselect without parsing the long descriptions.

    Naming Consistency3/5

    There are two coherent families: browser_* prefixed tools and session/verb-first tools (open_request_session, close_session, list_sessions, session_make_request, make_request). Within each family it is readable, but the mixed conventions across the set (browser_open vs open_request_session, bare make_request vs session_make_request) create inconsistency.

    Tool Count3/5

    16 tools is borderline heavy for the surface. Many are justified (fetch variants, network inspection), but the fetch/request pairs (browser_fetch vs browser_fetch_once, make_request vs session_make_request) inflate the count and add selection burden without adding fundamentally new capability.

    Completeness4/5

    The surface covers the full scraping lifecycle: session lifecycle (open/close/list), state save/load, multiple fetch/request modes, extraction, actions, evaluation, network inspection, and screenshots. Minor gaps exist (no explicit browser teardown distinct from close_session, no cookie-specific tool), but the core workflow is well covered.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents and MCP clients to fetch webpages and crawl sites, extract main content, and convert it into clean Markdown with preserved links for fast, token-efficient scraping.
      -
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables MCP clients and AI agents to fetch webpages and crawl sites, strip noise, and convert content into clean, link-preserving Markdown with caching and configurable multi-page depth while minimizing token usage.
      MIT