Scrapy MCP Server

English | 简体中文

✨ Why Negentropy Perceives?

In the vast ecosystem of AI agent projects, the "dirty work" of information perception often degenerates into fragile, unmaintainable chaos over time. Grounded in our core engineering philosophy of Orthogonal Decomposition and Entropy Reduction (Negentropy), we completely quarantine the mess of low-level network communications and format deconstruction. We only inject pure, undisputed certainty into your sandbox:

• 🕵️ Web Page to Markdown: Facing heavily-rendered SPAs and fortified anti-scraping defenses? The engine comes armed with a built-in 5-tier penetration mechanism (ranging from hyper-concurrency to headless stealth browser rotation). "What You See Is What You Get" — tearing through waterfall setups is a walk in the park.

• 📑 PDF to Markdown: Stop compromising over misaligned tables and mangled characters. Powered by our proprietary "Engine Arena" mechanism, engaging Smart mode summons an LLM as the ultimate referee. It coordinates 7 specialized engines (including Docling, PyMuPDF, etc.) performing concurrent deconstruction to precisely extract LaTeX formulas, gnarly table matrices, and deep layout structures.

• 🦾 Heavy-Duty Infrastructure: Abandon toy-grade SDK wrappers. Our core is hardwired with resilient exponential backoffs, multi-layered rate-limiting circuit breakers, and aggressive memory caching mechanisms. Riding on full-duplex asyncio, it maxes out the absolute throughput limit of a single node.

• 🔌 Native MCP Integration: We firmly embrace the pristine Model Context Protocol specification. Leveraging standard HTTP / STDIO / SSE transports, it abandons redundant glue code for seamless, zero-friction injection into Claude Desktop or Cursor environments.

Related MCP server: local-fetch

Quick Start

1. Millisecond Loading

# We recommend using uv (Python 3.13+ required)
uv add negentropy-perceives

2. Ignite the Engine

uv run negentropy-perceives  # Defaults to listening on localhost:2992, HTTP mode

💡 Advanced Arsenal: Upon first launch, Negentropy Perceives will auto-generate its configuration at ~/.negentropy/perceives.config.yaml. Hidden inside are the switches for high-tier warfare.

3. Witness True Perception

import asyncio
from negentropy.perceives.sdk import NegentropyPerceivesClient

async def perceive_world():
    async with NegentropyPerceivesClient() as client:
        result = await client.parse_webpage_to_markdown(
            url="https://en.wikipedia.org/wiki/Entropy",
        )
        print("====== Pure Nectar Extracted ======")
        print(result.markdown_content[:250], "......\n")
        print(f"📊 Pure words retrieved from the noise: {result.word_count}")

asyncio.run(perceive_world())

4. Connect the MCP Client

Add the following to your claude_desktop_config.json in Claude Desktop:

{
  "mcpServers": {
    "negentropy-perceives": {
      "type": "http",
      "url": "http://localhost:2992/mcp"
    }
  }
}

Supports three transport modes: STDIO (local dev), HTTP (production-recommended), and SSE (compatibility mode). See the User Guide for the comprehensive configuration.

Core Capabilities

Toolkit Overview

Web Scraping Strategies

PDF Engines

In auto mode, the system cascades through a graceful degradation chain: Docling → MinerU → Marker → PyMuPDF → PyPDF. Activating smart mode enlists an LLM to orchestrate a competitive parallel run across engines, ultimately fusing the optimum output.

Architectural Landscape

graph TD
    A["SDK Layer<br/>NegentropyPerceivesClient"] -.->|"HTTP Transport"| T["MCP Tool Layer<br/>6 Tools · @app.tool()"]
    T --> P["Pipeline Layer<br/>Stage Orchestration · Competition/Fallback"]
    T --> B["Processing Engine Layer<br/>Scraping · PDF · Markdown"]
    P --> B
    B --> C["Infrastructure Layer<br/>RateLimiter · Cache · Metrics · ErrorHandler · Retry"]
    C --> D["Configuration Layer<br/>pydantic-settings · Env Vars"]

    style A fill:#4c1d95,stroke:#a78bfa,color:#ffffff
    style T fill:#1e3a8a,stroke:#3b82f6,color:#ffffff
    style P fill:#b45309,stroke:#f59e0b,color:#ffffff
    style B fill:#166534,stroke:#22c55e,color:#ffffff
    style C fill:#134e4a,stroke:#14b8a6,color:#ffffff
    style D fill:#581c87,stroke:#9333ea,color:#ffffff

A 5-tier orthogonal architecture: SDK → MCP Tools → Pipeline Orchestration → Processing Engines → Infrastructure, with the Configuration Layer interweaving through everything. Featuring a 10-Stage PDF Pipeline and a 12-Stage WebPage Pipeline that strictly enforce both fallback and competitive execution models.

Documentation Navigator

Community & Contributions

Beyond the World Wide Web and massive unstructured texts lies an abyss of noise. Only through relentless code evolution can we forge ahead steadily. If you hold the inspiration to pull chaos back into order, please do not hesitate to share:

1. Before striking your keyboard, flip through the Developer Guide along the way.

2. Hurl your paradigm-shifting ideas at our Issues or directly submit a Pull Request armed with game-changing power.