pulpie-mcp

An MCP server that gives AI agents their own tools for turning web pages into clean Markdown. It runs the Pulpie content extraction model locally, so tables, code blocks, links, and images come through intact instead of getting summarized away.

The agent can read a page inline, save it as a .md file, crawl a whole docs section into a folder, and check what it already saved before fetching again.

Why I made it

I already had a little local UI for Pulpie that I used to pull docs and articles into Markdown, and it works really well. But I was still the one doing the pulling. Most agent fetch tools run pages through a small model that summarizes or trims them, which is not what you want when the agent needs the actual API reference.

So this gives the agent the same thing I was using. It can grab full documentation whenever it needs it and save it somewhere useful, and I don't have to go do it by hand.

Related MCP server: WebforAI Text Extractor

What it does

A few details worth knowing:

• Where files go is up to the agent. For project work it passes a folder inside the project, like /path/to/project/DOCS/reference/uv. For general research it leaves directory empty and the file goes into the global library at ~/.pulpie/library/<site>/.

• Every saved file has frontmatter with title, source, and fetched, so you (and the agent) can always tell where a file came from and how old it is.

• Crawling checks the sitemap first. It looks at robots.txt, then <docs path>/sitemap.xml, then /sitemap.xml. If none of those exist, it follows links from the pages instead. It only crawls pages on the same host and under the starting path.

• Markdown and plain-text URLs pass straight through. If a URL already serves text/markdown or text/plain (like llms.txt), you get the file as is.

How it works

There are two pieces:

1. The MCP server is what your agent launches. It's small and doesn't load the model.

2. The backend is a local HTTP server that holds the Pulpie model in memory and does the actual fetching and extraction.

You never have to start the backend yourself. The first time a tool needs it, the MCP server starts it in the background and waits for the model to load. Every agent session shares that one backend, so you only ever have one copy of the model on your GPU. After 30 minutes with no requests it shuts itself down and frees the memory, and the next request starts it again.

The model is feyninc/pulpie-orange-small, a 210M parameter encoder. It uses around 420 MB of VRAM. On my RTX 4090 laptop, a cold start (including loading an already downloaded model) took about 10 seconds. After that, a single docs page saved in under a second, and an 8-page crawl of the uv docs took about 3 seconds. Most of that is network time.

Requirements

• Python 3.10 or newer

• uv (recommended) or pip

• An NVIDIA GPU is nice but not required. Pulpie uses CUDA if it can, then Apple MPS, then falls back to CPU.

Heads up: this pulls in PyTorch, so the install is a few GB.

I've only tested this on Linux with an NVIDIA GPU so far. It should work on macOS, and Windows is untested.

Installation

Install it as a tool so the pulpie-mcp command is on your PATH:

uv tool install git+https://github.com/pinkpixel-dev/pulpie-mcp

Or from a local clone:

git clone https://github.com/pinkpixel-dev/pulpie-mcp
cd pulpie-mcp
uv tool install .

The model downloads from Hugging Face the first time the backend starts.

Adding it to your agent

Claude Code

claude mcp add --scope user pulpie -- pulpie-mcp

Codex

In ~/.codex/config.toml:

[mcp_servers.pulpie]
command = "pulpie-mcp"

Other MCP clients

Most clients take a JSON config like this:

{
  "mcpServers": {
    "pulpie": {
      "command": "pulpie-mcp"
    }
  }
}

If you add environment variables (see below), put them in the client's env block for this server.

Using it

You don't need special prompts. Just ask for what you want:

• "Pull the uv docs into this project's DOCS/reference folder."

• "Read the asyncio queue docs and tell me how join() works."

• "Do we already have the FastAPI docs saved somewhere?"

The server tells the agent to save project docs inside the project and general research in the global library, so it usually picks the right spot on its own.

Configuration

Everything is optional.

Private addresses

By default the server won't fetch localhost, 127.0.0.1, LAN addresses like 192.168.x.x, or cloud metadata addresses. Agents follow links they find on the open web, so I'd rather block those than have a random page send the agent poking around your network. This is checked on every redirect too.

If you want to pull docs from a local dev server, set PULPIE_ALLOW_PRIVATE=1.

Commands

pulpie-mcp            # run the MCP server over stdio (your agent does this)
pulpie-mcp serve      # run the backend in the foreground, handy for debugging
pulpie-mcp status     # show whether the backend is running, and on which device
pulpie-mcp stop       # stop the backend

If something goes wrong with the backend, check ~/.pulpie/backend.log.

Limitations

• Pages that render entirely in JavaScript come back mostly empty, because the backend fetches the HTML the server sends and doesn't run a browser. Most docs sites (Docusaurus, MkDocs, VitePress, Sphinx, Next.js) render on the server, so this hasn't been an issue for docs so far.

• Code blocks are indented, not fenced. Pulpie converts HTML to Markdown with html2text, which writes code as 4-space indented blocks without a language tag. It's valid Markdown, just not as pretty.

• No PDFs. Non-HTML pages other than Markdown and plain text are rejected.

• Saved pages are untrusted web content. Anything an agent reads from them is data, not instructions.

Model license

pulpie-mcp doesn't bundle or redistribute the model weights. They're downloaded from Hugging Face on your machine.

Development

uv sync
uv run pytest

The integration test runs the real backend and model against a live page, so it's skipped unless you opt in:

PULPIE_INTEGRATION=1 uv run pytest tests/test_integration.py

License

Apache 2.0. See LICENSE.

Pulpie is made by Feyn.

Made with 💖 by Pink Pixel