krwl3r

██╗  ██╗██████╗ ██╗    ██╗██╗     ██████╗ ██████╗
██║ ██╔╝██╔══██╗██║    ██║██║     ╚════██╗██╔══██╗
█████╔╝ ██████╔╝██║ █╗ ██║██║      █████╔╝██████╔╝
██╔═██╗ ██╔══██╗██║███╗██║██║      ╚═══██╗██╔══██╗
██║  ██╗██║  ██║╚███╔███╔╝███████╗██████╔╝██║  ██║
╚═╝  ╚═╝╚═╝  ╚═╝ ╚══╝╚══╝ ╚══════╝╚═════╝ ╚═╝  ╚═╝

// it crawls so your agents don't have to

KRWL3R is written in 1337speak, referencing Linkin Park's "KRWLNG" from the Reanimation album (2002) — where "Crawling" was reimagined without vowels. This project does the same: reimagines web crawling for the AI agent era.

What is KRWL3R

KRWL3R is a web intelligence engine purpose-built for AI agents. It combines two battle-tested open source projects into a unified, agent-friendly interface:

• Scrapling — adaptive scraping with auto-healing selectors that survive website redesigns

• PinchTab — headless browser control with intelligent text extraction (~800 tokens per page)

Instead of dumping raw HTML at your LLM, KRWL3R extracts clean, structured, token-efficient content — and exposes it through MCP, HTTP API, CLI, and ACP interfaces so any agent can use it.

Related MCP server: Browser Automation MCP

Features

Quick Start

Install

pip install krwl3r

Scrape a page

from krwl3r import Scraper

scraper = Scraper()
result = scraper.extract("https://example.com")

print(result.title)       # Page title
print(result.content)     # Clean text, ~800 tokens
print(result.metadata)    # Structured metadata

Control a browser

from krwl3r import Browser

async with Browser() as browser:
    page = await browser.new_page("https://example.com")
    await page.click("button#load-more")
    content = await page.extract()
    print(content.text)

Use with Claude Desktop (MCP)

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "krwl3r": {
      "command": "krwl3r",
      "args": ["mcp"]
    }
  }
}

Then ask Claude: "Scrape the pricing page at example.com and summarize the plans."

Compatibility

KRWL3R works with any AI tool that supports MCP, HTTP, or CLI interfaces.

Architecture

                         ┌─────────────────────────────┐
                         │        AI  AGENTS            │
                         │  Claude, Gemini, Codex, ...  │
                         └──────────┬──────────────────┘
                                    │
              ┌─────────────────────┼─────────────────────┐
              │                     │                      │
         ┌────▼────┐          ┌────▼────┐           ┌────▼────┐
         │   MCP   │          │  HTTP   │           │   ACP   │
         │ Server  │          │  API    │           │ Server  │
         └────┬────┘          └────┬────┘           └────┬────┘
              │                     │                      │
              └─────────────────────┼──────────────────────┘
                                    │
                         ┌──────────▼──────────┐
                         │    KRWL3R  CORE     │
                         │                     │
                         │  ┌───────────────┐  │
                         │  │  Orchestrator │  │
                         │  └───────┬───────┘  │
                         │          │          │
                         │   ┌──────┴──────┐   │
                         │   │             │   │
                         │ ┌─▼──┐     ┌───▼─┐ │
                         │ │Scrp│     │Pnch │ │
                         │ │lng │     │Tab  │ │
                         │ └─┬──┘     └───┬─┘ │
                         │   │             │   │
                         └───┼─────────────┼───┘
                             │             │
                      ┌──────▼──┐    ┌────▼─────┐
                      │  HTTP   │    │ Headless  │
                      │Requests │    │ Chrome    │
                      └─────────┘    └──────────┘

Layer 1 — Protocol Adapters: MCP, HTTP REST, ACP, and CLI interfaces that translate agent requests into unified internal calls.

Layer 2 — Core Orchestrator: Routes requests, manages concurrency, handles retries, and selects the optimal extraction strategy.

Layer 3 — Extraction Engines: Scrapling for fast HTTP-based extraction with auto-healing selectors. PinchTab for full browser control when JavaScript rendering or interaction is required.

Layer 4 — Transport: Raw HTTP requests for static content, headless Chrome instances for dynamic pages.

Powered By

KRWL3R stands on the shoulders of two exceptional open source projects:

Scrapling

D4Vinci/Scrapling — BSD-3-Clause — ~20k stars

An undetectable, powerful web scraping library with automatic anti-bot evasion and adaptive selectors that survive website changes. Scrapling's auto-healing selector engine is what makes KRWL3R resilient — when a site redesigns, selectors adapt instead of breaking.

PinchTab

pinchtab/pinchtab — MIT — ~3k stars

A Go-based browser control and text extraction engine that produces clean, ~800-token page representations. PinchTab's intelligent content extraction is what makes KRWL3R token-efficient — agents get structured content instead of raw HTML soup.

License

MIT — use it, fork it, ship it.

Contributing

Contributions are welcome. See docs/contributing.md for guidelines.

Quick version:

1. Fork the repo

2. Create a feature branch (git checkout -b feat/my-feature)

3. Commit with conventional commits (feat:, fix:, docs:, chore:)

4. Open a pull request

Please be respectful of the upstream projects (Scrapling and PinchTab) — KRWL3R integrates them, it does not fork or replace them.

// 2026 — built for the agent era