Skip to main content
Glama

crawler

npm CI Pages license

This repository holds polite-crawler, a polite web crawler for research and for AI agents, and its site.

polite-crawler fetches a source one time, keeps it, and reads it many times. It obeys robots.txt, sends one request at a time to each site, and answers each later question from a local full-text index.

npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler
npx polite-crawler find "robots exclusion"
claude mcp add polite-crawler -- npx -y polite-crawler-mcp

What the crawler does for you and for the sites:

  • It stops a host after a challenge, a 401, a 402 or a 403, for 1 to 24 hours. It does not try a different identity.

  • By default, it fetches only public addresses. --allow-private permits loopback and private addresses.

  • The MCP server gives page text in an <untrusted-web-content> envelope, so an assistant can tell data from instructions. A page that refuses AI input stays held back until a person confirms.

  • The library is polite-crawler, and the MCP server is the package polite-crawler-mcp (its README).

The package README tells the details.

Development

The repository is a pnpm workspace. It needs Node.js 22.19 or later and pnpm 10 or later.

pnpm install
pnpm browser   # the headless shell for the renderer test
pnpm check     # typecheck, lint, test, lint:ste, build and the smoke test of the package

The prose of the repository (the README files, the doc comments and the site) follows ASD-STE100 Simplified Technical English. The linter ste-lint examines it in CI.

Related MCP server: pyaireader

License

MIT

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      D
      maintenance
      Ad-free web scraping and search exposed as 3 MCP tools. fetch_page, fetch_pages_batch, search_web. Works with Claude Desktop, Cursor, Cline.
      3
      69 PyPI
      2
      MIT
    • A
      license
      C
      quality
      C
      maintenance
      MCP server for safely reading public URLs for AI agents, providing tools to fetch, extract, cache, and inspect web content as evidence.
      15
      MIT
    • F
      license
      B
      quality
      B
      maintenance
      Exposes web research tools to MCP agents so they can search the web, open and read pages, grep fetched content, manage a local SQLite corpus, and generate citations. Includes a containerized fetcher for headless browsing and PDF extraction.
      7
      -
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to fetch web pages, extract clean Markdown, save and search versions offline, crawl sites within bounds, and retrieve stable citations through MCP tools.
      12 npm
      Apache 2.0