An MCP server that gives AI agents their own tools for turning web pages into clean Markdown. It runs the Pulpie content extraction model locally, so tables, code blocks, links, and images come through intact instead of getting summarized away.
The agent can read a page inline, save it as a .md file, crawl a whole docs section into a folder, and search everything it saved later. Over time the library turns into a local reference collection you can reuse across projects.
I already had a little local UI for Pulpie that I used to pull docs and articles into Markdown, and it works really well. But I was still the one doing the pulling. Most agent fetch tools run pages through a small model that summarizes or trims them, which is not what you want when the agent needs the actual API reference.
So this gives the agent the same thing I was using. It can grab full documentation whenever it needs it and save it somewhere useful, and I don't have to go do it by hand.
Related MCP server: WebforAI Text Extractor
Tool | What it's for |
| Read a page right now. The Markdown comes back inline, cut off at |
| Save one page as a |
| Save a documentation section as a folder of |
| List saved docs with their source URL and fetch time, optionally filtered by a search term. |
| Search saved docs by what they say and get back the matching sections, each with its heading path, source URL, and file path. |
A few details worth knowing:
Where files go is up to the agent. For project work it passes a folder inside the project, like /path/to/project/DOCS/reference/uv. For general research it leaves directory empty and the file goes into the global library at ~/.pulpie/library/<site>/.
Every saved file has frontmatter with title, source, and fetched, so you (and the agent) can always tell where a file came from and how old it is.
Crawling checks the sitemap first. It looks at robots.txt, then <docs path>/sitemap.xml, then /sitemap.xml. If none of those exist, it follows links from the pages instead. It only crawls pages on the same host and under the starting path.
Search works on sections, not whole pages. Each page is split at its #, ##, and ### headings, so a search for useEffect cleanup returns the cleanup section instead of the whole hooks page. It searches the global library by default, or any folder you pass as directory.
Markdown and plain-text URLs pass straight through. If a URL already serves text/markdown or text/plain (like llms.txt), you get the file as is.
There are two pieces:
The MCP server is what your agent launches. It's small and doesn't load the model.
The backend is a local HTTP server that holds the Pulpie model in memory and does the actual fetching and extraction.
You never have to start the backend yourself. The first time a tool needs it, the MCP server starts it in the background and waits for the model to load. Every agent session shares that one backend, so you only ever have one copy of the model on your GPU. After 30 minutes with no requests it shuts itself down and frees the memory, and the next request starts it again.
The model is feyninc/pulpie-orange-small, a 210M parameter encoder. It uses around 420 MB of VRAM. On my RTX 4090 laptop, a cold start (including loading an already downloaded model) took about 10 seconds. After that, a single docs page saved in under a second, and an 8-page crawl of the uv docs took about 3 seconds. Most of that is network time.
Python 3.10 or newer
uv (recommended) or pip
An NVIDIA GPU is nice but not required. Pulpie uses CUDA if it can, then Apple MPS, then falls back to CPU.
Heads up: this pulls in PyTorch, so the install is a few GB.
I've only tested this on Linux with an NVIDIA GPU so far. It should work on macOS, and Windows is untested.
Install it as a tool so the pulpie-mcp command is on your PATH:
uv tool install git+https://github.com/pinkpixel-dev/pulpie-mcpOr from a local clone:
git clone https://github.com/pinkpixel-dev/pulpie-mcp
cd pulpie-mcp
uv tool install .The model downloads from Hugging Face the first time the backend starts.
claude mcp add --scope user pulpie -- pulpie-mcpIn ~/.codex/config.toml:
[mcp_servers.pulpie]
command = "pulpie-mcp"Most clients take a JSON config like this:
{
"mcpServers": {
"pulpie": {
"command": "pulpie-mcp"
}
}
}If you add environment variables (see below), put them in the client's env block for this server.
You don't need special prompts. Just ask for what you want:
"Pull the uv docs into this project's DOCS/reference folder."
"Read the asyncio queue docs and tell me how join() works."
"Do we already have the FastAPI docs saved somewhere?"
"Search the library for how FastAPI handles background tasks."
The server tells the agent to save project docs inside the project and general research in the global library, so it usually picks the right spot on its own.
Everything is optional.
Variable | Default | What it does |
|
| Where the backend runs. Only local URLs get auto-started. |
|
| Holds the global library, the search index ( |
|
| Where saved docs go when no directory is given. |
| off | Set to |
|
| Seconds without requests before the backend exits. |
|
| How long to wait for the backend to start. The first run includes the model download. |
By default the server won't fetch localhost, 127.0.0.1, LAN addresses like 192.168.x.x, or cloud metadata addresses. Agents follow links they find on the open web, so I'd rather block those than have a random page send the agent poking around your network. This is checked on every redirect too.
If you want to pull docs from a local dev server, set PULPIE_ALLOW_PRIVATE=1.
pulpie-mcp # run the MCP server over stdio (your agent does this)
pulpie-mcp serve # run the backend in the foreground, handy for debugging
pulpie-mcp status # show whether the backend is running, and on which device
pulpie-mcp stop # stop the backendIf something goes wrong with the backend, check ~/.pulpie/backend.log.
Pages that render entirely in JavaScript come back mostly empty, because the backend fetches the HTML the server sends and doesn't run a browser. Most docs sites (Docusaurus, MkDocs, VitePress, Sphinx, Next.js) render on the server, so this hasn't been an issue for docs so far.
Code blocks are indented, not fenced. Pulpie converts HTML to Markdown with html2text, which writes code as 4-space indented blocks without a language tag. It's valid Markdown, just not as pretty.
Search is keyword search, not semantic search. It uses SQLite full-text search with stemming, so caching finds cache and readOnlyHint finds read_only_hint. A question phrased in totally different words than the docs use can still miss. If nothing matches every word, the results say they only match some of the words.
No PDFs. Non-HTML pages other than Markdown and plain text are rejected.
Saved pages are untrusted web content. Anything an agent reads from them is data, not instructions.
This project's code is Apache 2.0, but the Pulpie model it downloads,feyninc/pulpie-orange-small, is licensed CC BY-NC 4.0. That means non-commercial use only. The pulpie Python library itself is Apache 2.0. If you want to use this in a commercial product, check the model license with Feyn first.
pulpie-mcp doesn't bundle or redistribute the model weights. They're downloaded from Hugging Face on your machine.
uv sync
uv run pytestThe integration test runs the real backend and model against a live page, so it's skipped unless you opt in:
PULPIE_INTEGRATION=1 uv run pytest tests/test_integration.pyApache 2.0. See LICENSE.
Pulpie is made by Feyn.
Made with 💖 by Pink Pixel
Crawl a documentation site and save every page as Markdown, plus an index.md.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Where to start, usually the docs home page. | |
| directory | No | Absolute folder to save into. Use a folder inside the user's project for project reference docs. Leave empty to use the global library. | |
| max_pages | No | Stop after this many pages. | |
| path_prefix | No | Only crawl URLs whose path starts with this, like /docs/. Defaults to the start URL's path, so pass a broader prefix when starting from a page deep inside the docs. |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It discloses the output artifact (Markdown files plus index.md) but says nothing about rate limiting/politeness, robots handling, overwrite or failure behavior, or what happens on partial crawls — significant gaps for a tool that writes many files to disk.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence stating action and output with zero filler. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return-value explanation is correctly omitted, and the schema fully documents inputs. However, in a five-sibling docs/fetch/save library, the description omits the routing information (crawl-site vs fetch-single-page) and crawl-scope behavior that an agent needs to pick this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and every parameter (url, directory, max_pages, path_prefix) is well documented with defaults and rationale in the schema itself. The description adds nothing about parameters, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (crawl), a specific resource (a documentation site) and the concrete outcome (every page saved as Markdown plus index.md). It clearly distinguishes the action from the sibling names fetch_markdown/save_markdown/list_library, though it never explicitly contrasts itself with them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no when-to-use guidance, no alternatives, and no exclusions. It never tells the agent to prefer this over fetch_markdown for a single page, or when a crawl is inappropriate (e.g., huge sites, auth-protected docs). Usage is only loosely implied by the word 'crawl'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch a web page and return its main content as Markdown, without saving it.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The http(s) URL to read. | |
| max_chars | No | Cut the Markdown off after this many characters. |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses that the tool fetches and converts to Markdown without saving, but doesn't mention potential limitations such as handling of paywalls, authentication, rate limits, or the fact that only main content is extracted (implying some content may be omitted). The presence of an output schema means return format details are covered, but operational behavior remains underspecified.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, well-structured sentence that front-loads the core action and includes a key differentiator. Every part earns its place with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (2 parameters, output schema exists), the description is nearly complete. It covers the main purpose and the no-save behavior. A minor gap is the lack of explicit guidance on when to use this versus similar tools, but for a straightforward fetch operation this is acceptable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema fully documents the parameters (url and max_chars). The description adds no further parameter details. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb ('Fetch') and resource ('a web page'), and specifies the transformation ('return its main content as Markdown'). The contrast with sibling 'save_markdown' is evident from the phrase 'without saving it', making it unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implicitly differentiates from 'save_markdown' by stating 'without saving it', which tells the agent this tool is for when the content is needed temporarily or for processing. However, it doesn't provide explicit when/when-not guidance or mention alternatives like 'crawl_docs'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List Markdown docs saved by pulpie, with their source URL and fetch time.
| Name | Required | Description | Default |
|---|---|---|---|
| query | No | Only show docs whose title, URL, or path contains this. | |
| directory | No | Absolute folder to list. Leave empty to list the global library. |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, and it does disclose the return payload (source URL and fetch time), which clarifies this is a read-only listing. However it says nothing about ordering, result limits, or that a query narrows results rather than replacing the list. Adequate but thin for a zero-annotation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence that identifies the actor (pulpie), the resource, and the returned fields with zero filler. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple (2 optional params, no nesting) and both an output schema and full parameter descriptions exist, so the description needn't explain returns. It is nearly complete, missing only a statement about result scope/ordering or how it relates to search_library.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so both 'query' (title/URL/path substring match) and 'directory' (absolute folder vs global library) are already fully documented in the schema. The description adds no parameter meaning beyond that, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('List') and a specific resource ('Markdown docs saved by pulpie') and names the returned fields. It does not explicitly differentiate itself from the closely-named sibling search_library, leaving the list-vs-search boundary to inference.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No when-to-use guidance is given. The nearby sibling search_library handles filtered retrieval and the tool itself accepts a 'query' filter, but the description never says when to call this instead of search_library or whether this returns the entire library. The agent must infer usage from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch a web page and save its main content as a Markdown file. Returns the path and outline.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The http(s) URL to save. | |
| filename | No | Optional file name. Defaults to one built from the URL path. | |
| directory | No | Absolute folder to save into. Use a folder inside the user's project for project reference docs. Leave empty to use the global library. |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It correctly signals a two-part side effect (network fetch + file write) and mentions the return shape, but omits overwrite semantics, whether directories are created, and failure behavior for unreachable URLs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, front-loaded with the primary action and followed by the return value. No filler whatsoever.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so the brief return note is a bonus rather than a requirement, and the params are fully covered by the schema. The remaining gap is the absence of routing guidance relative to the four sibling tools for a write operation with no annotations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all three parameters (including the global-vs-project distinction for directory) are already documented. The description adds no parameter-level detail beyond the schema, which is the expected baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Names a specific verb (save) and resource (web page main content as Markdown file), plus a brief note on what it returns. It is distinguishable from fetch_markdown by the word 'save', but the description never explicitly contrasts the two, so an agent must infer the persistence difference.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no statement of when to use this versus fetch_markdown, crawl_docs, or the library tools, and no prerequisites or exclusions. Usage must be inferred entirely from the tool name.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Search saved docs by content. Returns the best matching sections with their file paths.
| Name | Required | Description | Default |
|---|---|---|---|
| site | No | Only search docs from this host, like react.dev. | |
| limit | No | Most sections to return. | |
| query | Yes | Words to look for, like 'useEffect cleanup'. | |
| directory | No | Absolute folder to search, like a project's DOCS/reference. Leave empty to search the global library. |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses the return shape ('best matching sections with their file paths'), which hints at ranked/sectioned results, but says nothing about read-only semantics, ranking behavior, or result truncation beyond what the schema's limit already encodes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, zero waste, with the core action front-loaded and the return behavior second. Nothing is redundant or padded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be explained in detail, and the annotations-less safety profile is low-risk for a search tool. The description is adequate, though it could note that results come from the previously saved/crawled library rather than the live web.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so site, limit, query, and directory are all documented in the schema with examples. The description adds no parameter-level detail, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource - searching saved docs by content - and adds scope via 'saved docs' versus the live-web sibling crawl_docs. It does not explicitly distinguish itself from list_library, but the 'by content' framing makes the retrieval style clear.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by 'search saved docs by content', and the schema note about leaving directory empty implies a global-library fallback. However, no alternative tool is named and no when-not-to-use condition is given, so an agent must infer that list_library covers enumeration and fetch_markdown covers retrieval.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0crawl_docsfetch_markdownlist_librarysave_markdownsearch_libraryScored across 5 tools
fetch_markdown and save_markdown are the only pair with overlapping purposes, but the save/no-save distinction is clearly stated. list_library, search_library, and crawl_docs each target a distinct operation with no real ambiguity.
All five tools follow a clean verb_noun snake_case pattern (fetch_markdown, save_markdown, crawl_docs, list_library, search_library). The convention is applied uniformly and reads predictably.
Five tools is a tight, well-scoped set for a fetch/save/search docs server. Each tool earns its place with no redundancy.
The lifecycle covers fetch, persist, bulk-crawl, browse, and search. Minor gaps exist: no tool to delete or remove a saved doc, and no direct read of a specific saved file, but these are workaroundable.
MCP server (stdio): fetch web pages as clean readable markdown via the AgentForge API
Turn any public website into an MCP server for agents to search, read and navigate.
Web scraping MCP server: any URL to Markdown or JSON, with JavaScript rendering and batch jobs.
Docs: https://docs.keenable.ai/mcp-server Keenable is a free, remote MCP server that gives agents access to the web index. Search the web with ranked results and date/site filters, then fetch any indexed page as clean markdown. Works out of the box with no account or API key.