Which integrations are available for this server?

Provides a scrape tool for CrewAI agents, enabling web scraping and crawling within CrewAI workflows. Supports self-hosting via Docker Compose with API (port 8080), MCP SSE (port 8081), Redis, and optional crawl workers. Provides web search functionality via DuckDuckGo as a fallback search engine when SearXNG is not configured. Provides AgentFetchTools for LangChain agents, enabling web scraping, crawling, search, and extraction as LangChain tools. Provides local LLM-based structured data extraction via Ollama, configured with OLLAMA_URL and OLLAMA_MODEL, without API costs. Provides tools for OpenAI function calling, enabling web scraping, crawling, search, and extraction via get_tools. Provides optional Redis-backed caching and job queue for horizontal scaling of crawl operations. Provides web search functionality via SearXNG with optional result scraping, configured via SEARXNG_URL.

How do I use agentfetch?

1. Click on "Install Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@agentfetch fetch https://en.wikipedia.org/wiki/OpenAI as markdown" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

agentfetch

by SID1ART

Overview Schema Related Servers Score Discussions

Python

Local

agentfetch

Open-source web retrieval & research agent built for AI agents.

License: MIT Python 3.10+ Tests

agentfetch is a free, local alternative to Firecrawl, Exa, Parallel.ai, and Tavily. It fetches any webpage, crawls any site, searches the web, and researches any topic — returning clean markdown and structured reports that AI agents can consume directly.

Works with LangChain, LlamaIndex, CrewAI, AutoGen, Claude MCP, OpenAI function calling, Gemini, Groq, and plain REST. No vendor lock-in, no API keys required.

Install

Standard

pip install git+https://github.com/SID1ART/agentfetch.git

Cloud notebooks (Colab, Jupyter, Kaggle)

pip install https://github.com/SID1ART/agentfetch/archive/main.zip

With extra integrations

pip install "agentfetch[langchain,llamaindex,crewai] @ git+https://github.com/SID1ART/agentfetch.git"
pip install "agentfetch[search] @ git+https://github.com/SID1ART/agentfetch.git"   # adds Google search engine

No PyPI account, no API tokens, no sign-up needed. GitHub is the source.

What makes it different

Research Agent — Tavily-style deep research: auto-decomposes questions into sub-queries, searches multiple engines, gathers full content, and synthesizes a structured report with citations via Ollama or Claude. Supports iterative follow-up (depth="deep"), structured output schemas, and four citation formats (numbered, MLA, APA, Chicago).
Smart Mode Router — detects JavaScript-heavy SPAs (Next.js, Nuxt, React) and falls back to Playwright headless browser automatically. Static pages use direct HTTP.
5-layer extraction pipeline — trafilatura → newspaper3k → readability-lxml → BeautifulSoup → plain text. Best-effort extraction from any HTML.
Never raises exceptions — always returns structured FetchResult with confidence scores, error fields, and injection detection. Agents can trust the output.
Post-extraction quality check — detects SPA shell text, low prose ratio, missing sentence structure, and downgrades confidence so agents can retry with browser engine.
In-memory LRU cache — deduplicates repeated URL fetches within a session. Configurable size/TTL via env vars. No Redis required.
Information saturation crawling — no arbitrary depth limits. CrawlStopper detects vocabulary saturation and content redundancy, stopping when enough data is gathered.
Persistent crawl store — SQLite-backed job persistence when Redis is not configured. Results survive process restarts.
Prompt injection firewall — 13 patterns detected and redacted to [REDACTED BY AGENTFETCH].
Cloudflare bypass — optional curl_cffi integration with 12 TLS fingerprint profiles (Chrome 99–124, Safari 15/17) and auto-rotation.
Browser stealth — optional playwright-stealth integration for advanced anti-detection (WebGL vendor, canvas fingerprint, navigator.webdriver removal). Enabled by default.
Robots.txt compliance — optional async parser with caching, crawl-delay, and sitemap discovery.
Proxy rotation — round-robin or random proxy pools with automatic failure tracking.
Local LLM extraction — optional Ollama integration for structured data extraction without API costs.
Redis-backed job queue — horizontal scaling for crawl operations with background workers.

Related MCP server: superFetch MCP Server

Tools

Tool	Description
`agent_scrape`	Fetch any URL; auto-detects browser need. Supports ScrapeConfig (wait_for selectors, tag filtering, citation markers, proxies, JA3 profile).
`agent_crawl`	Recursive crawl with information saturation stopping, robots.txt compliance, deduplication.
`agent_search`	Web search via SearXNG, DuckDuckGo, Google, or Bing with optional result scraping.
`agent_extract`	Structured data extraction by JSON schema via Ollama, Anthropic Claude, or CSS fallback.
`agent_map`	Discover all URLs on a website via sitemap.xml and BFS crawling.
`agent_status`	Poll crawl job progress (in-memory or Redis).
`agent_research`	Research a topic: decomposes into sub-queries, searches multi-engine, gathers content, synthesizes a structured report with citations via LLM. Supports deep iterative follow-up and structured output schemas.

Library API

Function	Description
`smart_fetch(url, config=)`	Fetch a single URL; auto-detects browser need. Returns `FetchResult`.
`batch_fetch(urls, concurrency=)`	Fetch multiple URLs concurrently. Returns `list[FetchResult]`.
`search_fetch(query, sources=, max_results=)`	Search and optionally scrape results. Returns `SearchResult`.
`parallel_search(query, sources=, max_results=)`	Search engine results without scraping. Returns `tuple[list[EngineResult], list[str], dict[str, str]]`.
`smart_research(prompt, config=)`	Research a topic: decomposes query, gathers sources, synthesizes report with citations via LLM. Returns `ResearchResult`.

Quickstart

LangChain

from agentfetch.integrations.langchain.tools import AgentFetchTools
tools = AgentFetchTools
# Use with any LangChain agent

MCP (Claude Desktop, Cursor, etc.)

pip install git+https://github.com/SID1ART/agentfetch.git
agentfetch-mcp
# configure in Claude Desktop or any MCP host

REST API

pip install git+https://github.com/SID1ART/agentfetch.git
agentfetch serve

# Scrape
curl -X POST http://localhost:8080/agent_scrape \
  -d '{"url": "https://example.com"}'

# Research (async — returns job ID, poll for result)
curl -X POST http://localhost:8080/agent_research \
  -d '{"prompt": "Latest AI developments", "max_sources": 10}'
# → {"request_id":"abc123","status":"pending",...}

# Poll research result
curl http://localhost:8080/agent_research/abc123
# → {"request_id":"abc123","status":"complete","answer":"# Report...","sources":[...]}

# Research with streaming (SSE)
curl -X POST http://localhost:8080/agent_research/stream \
  -d '{"prompt": "Compare OpenAI and Anthropic pricing", "citation_format": "apa"}'
# → event: progress, event: result (full report)

Python library

import asyncio
from agentfetch import smart_fetch, search_fetch
from agentfetch.core.schema import ScrapeConfig

# Fetch a single URL
result = asyncio.run(smart_fetch(
    "https://en.wikipedia.org/wiki/Obsession_(2025_film)",
    config=ScrapeConfig(
        wait_for=".main-content",
        exclude_tags=["nav", "footer"],
        citation_links=True,
    )
))
print(result.content)  # clean markdown
print(result.citations)  # [1], [2] URLs

# Search with multiple engines
sr = asyncio.run(search_fetch(
    "latest AI news",
    sources=["duckduckgo", "google", "bing"],
    max_results=5,
))
print(sr.results)      # list[FetchResult]
print(sr.errors)       # per-engine errors, e.g. {"google": "rate limited (429)"}
print(sr.sources_used) # engines that returned results

# Research a topic (uses Ollama or Claude for query decomposition and synthesis)
from agentfetch import smart_research, ResearchConfig

report = asyncio.run(smart_research(
    "Compare pricing of OpenAI, Anthropic, and Google AI APIs",
    config=ResearchConfig(
        max_sources=15,
        citation_format="apa",
        depth="deep",
    )
))
print(report.answer)        # Comprehensive report with [Author, Year] citations
print(report.sources)       # list of ResearchSource with full content
print(report.response_time) # e.g. 12.34s

All integrations

Framework	Install	Tools available
LangChain	`pip install "agentfetch[langchain] @ git+https://github.com/SID1ART/agentfetch.git"`	`scrape`, `search`, `crawl`, `map`, `status`, `research`
LlamaIndex	`pip install "agentfetch[llamaindex] @ git+https://github.com/SID1ART/agentfetch.git"`	`scrape`, `search`, `crawl`, `map`, `status`, `research`
CrewAI	`pip install "agentfetch[crewai] @ git+https://github.com/SID1ART/agentfetch.git"`	`scrape`, `search`, `crawl`, `map`, `status`, `research`
AutoGen	`pip install git+https://github.com/SID1ART/agentfetch.git`	`scrape`, `search`, `crawl`, `map`, `status`, `research`
OpenAI / Gemini / Groq	`pip install git+https://github.com/SID1ART/agentfetch.git`	`scrape`, `search`, `crawl`, `map`, `status`, `research`
Claude MCP	`pip install git+https://github.com/SID1ART/agentfetch.git`	`scrape`, `search`, `crawl`, `map`, `status`, `research`
Ollama	`pip install git+https://github.com/SID1ART/agentfetch.git`	`extract`
REST	`pip install git+https://github.com/SID1ART/agentfetch.git`	All endpoints + `/agent_research` + streaming

Schema reference

`ScrapeConfig`

Field	Type	Default	Description
`wait_for`	`str`	`None`	CSS selector to wait for before extracting
`include_tags`	`list[str]`	`None`	Only extract these HTML tags
`exclude_tags`	`list[str]`	`None`	Skip these HTML tags during extraction
`viewport`	`dict`	`None`	Browser viewport `{width, height}`
`js_wait_ms`	`int`	`0`	Extra JS wait time in milliseconds
`scrape_links`	`bool`	`True`	Extract links from page
`max_content_length`	`int`	`50000`	Truncate content beyond this length
`citation_links`	`bool`	`False`	Track citation markers `[1]`, `[2]`
`proxy`	`str`	`None`	Proxy URL for this request
`cookies`	`list[dict]`	`None`	Cookies to include in browser session
`headers`	`dict[str,str]`	`None`	Custom HTTP headers
`ja3`	`str`	`None`	JA3 TLS profile for `curl_cffi` bypass (e.g. `"chrome124"`)
`stealth`	`bool`	`True`	Enable browser stealth evasions (playwright-stealth if available)
`actions`	`list[Action]`	`[]`	Action chain to execute before extraction (click, scroll, type, wait, press, select, screenshot, hover, custom_js)
`screenshot`	`bool`	`False`	Capture a full-page final screenshot (PNG, base64-encoded in `screenshot_data`)

`FetchResult`

Field	Type	Description
`url`	`str`	Requested URL
`content`	`str`	Extracted markdown content
`title`	`str`	Page title
`confidence`	`float`	Extraction quality (0.0–1.0)
`content_type`	`str`	Detected type (article, blog, product, etc.)
`word_count`	`int`	Word count of extracted content
`render_mode`	`str`	Renderer used: `static`, `browser`, or `bypass`
`latency_ms`	`int`	Total request time in milliseconds
`cached`	`bool`	Whether result came from cache
`injection_detected`	`bool`	Prompt injection was found and redacted
`links`	`list[str]`	Links extracted from the page
`error`	`str`	Error message if the fetch failed
`duplicate_of`	`str`	URL this content was deduplicated against
`retries`	`int`	Number of retries performed
`citations`	`list[str]`	Citation URLs when `citation_links=True`
`robots_allowed`	`bool`	Whether robots.txt permitted the fetch
`proxy_used`	`str`	Proxy used for this request
`normalized_url`	`str`	Normalized version of the requested URL
`screenshot_data`	`str`	Base64-encoded final PNG screenshot (when `screenshot=True` in ScrapeConfig)
`screenshots`	`list[str]`	Base64-encoded PNG screenshots from mid-flow screenshot actions (when `store_output=True`)

`Action`

Field	Type	Default	Description
Field	Type	Default	Description
-------	------	---------	-------------
`type`	`str`	—	Action type: `click`, `scroll`, `type`, `wait`, `press`, `select`, `screenshot`, `hover`, or `custom_js`
`selector`	`str`	`None`	CSS selector for `click`, `type`, `scroll`, `press`, `select`, `hover` actions
`value`	`str`	`None`	Value: text for `type`, key for `press`, ms for `wait`, pixels for `scroll`, option value for `select`, JS code for `custom_js`
`timeout`	`int`	`5000`	Timeout in ms for selector waits
`store_output`	`bool`	`False`	When `True`, stores `screenshot` action output in `FetchResult.screenshots` list

Action details:

click — clicks a CSS selector, waits for networkidle afterward
scroll — scrolls to selector, "bottom", "top", or by N pixels
type — fills an input field with value
wait — waits N milliseconds (value)
press — presses a key (value, default "Enter"), waits for networkidle
select — selects a <select> option by value
screenshot — captures a full-page PNG; stored in screenshots[] if store_output=True
hover — hovers over a CSS selector
custom_js — runs arbitrary JavaScript from value on the page, waits for networkidle

Examples:

# Hover to reveal dropdown, then click
actions = [
    Action(type="hover", selector="#nav-menu"),
    Action(type="wait", value="500"),
    Action(type="click", selector="#nav-menu .dropdown-item"),
]

# Run custom JS and capture mid-flow screenshot
actions = [
    Action(type="custom_js", value="document.querySelector('.paywall')?.remove()"),
    Action(type="screenshot", store_output=True),
]

# Full-page final screenshot
config = ScrapeConfig(screenshot=True, actions=[...])

`SearchConfig`

Field	Type	Default	Description
`max_results`	`int`	`5`	Max results per engine
`sources`	`list[str]`	`None`	Engines: `duckduckgo`, `google`, `bing`, `searxng`, `brave`, `serpapi`
`scrape_results`	`bool`	`True`	Fetch full content of each result
`searxng_url`	`str`	`""`	Self-hosted SearXNG instance URL
`topic`	`str`	`"general"`	Search topic: `general`, `news`, or `finance`
`time_range`	`str`	`None`	Time filter: `day`, `week`, `month`, or `year`
`country`	`str`	`None`	Boost results from a country (e.g. `"united states"`, `"united kingdom"`)
`include_answer`	`bool`	`False`	Include an LLM-generated answer via Ollama or Anthropic

`SearchResult`

Field	Type	Description
`query`	`str`	Original search query
`results`	`list[FetchResult]`	Search results with extracted content
`source`	`str`	Concatenated engine names used
`sources_used`	`list[str]`	Engines that returned results
`suggestions`	`list[str]`	Search suggestions (if available)
`total_results`	`int`	Total deduplicated result count
`errors`	`dict[str,str]`	Per-engine error messages (e.g. `{"google": "rate limited (429)"}`)
`answer`	`str`	LLM-generated answer when `include_answer=True` in SearchConfig

`MapConfig`

Field	Type	Default	Description
`max_depth`	`int`	`2`	Maximum crawl depth for link discovery
`max_pages`	`int`	`100`	Maximum URLs to discover
`include_patterns`	`list[str]`	`None`	Regex patterns to include only matching paths
`exclude_patterns`	`list[str]`	`None`	Regex patterns to exclude matching paths
`include_domains`	`list[str]`	`None`	Only include URLs from these domains
`exclude_domains`	`list[str]`	`None`	Exclude URLs from these domains
`respect_robots`	`bool`	`True`	Respect robots.txt during crawl discovery

`MapResult`

Field	Type	Description
`base_url`	`str`	The root URL that was mapped
`links`	`list[str]`	Discovered URLs
`total`	`int`	Total number of discovered URLs
`sources`	`list[str]`	Discovery methods used (`sitemap`, `crawl`)

`ResearchConfig`

Field	Type	Default	Description
`prompt`	`str`	—	The research question or topic
`model`	`str`	`"auto"`	Model tier: `"mini"` (fast/cheap), `"pro"` (comprehensive), `"auto"`
`max_sources`	`int`	`20`	Maximum sources to gather
`output_schema`	`dict`	`None`	JSON Schema for structured output in the report
`citation_format`	`str`	`"numbered"`	Citation style: `"numbered"`, `"mla"`, `"apa"`, `"chicago"`
`include_domains`	`list[str]`	`None`	Prioritize results from these domains
`exclude_domains`	`list[str]`	`None`	Exclude results from these domains
`depth`	`str`	`"standard"`	Research depth: `"quick"`, `"standard"`, `"deep"` (deep enables iterative follow-up queries)
`max_iterations`	`int`	`4`	Max follow-up iterations when `depth="deep"`

`ResearchSource`

Field	Type	Description
`url`	`str`	Source URL
`title`	`str`	Page title
`content`	`str`	Extracted text content
`relevance_score`	`float`	Relevance to the research question (0.0–1.0)
`citation`	`str`	Pre-formatted citation string (e.g. `[1]`, `(Author, 2024)`)

`ResearchResult`

Field	Type	Description
`request_id`	`str`	Unique job ID
`query`	`str`	Original research question
`answer`	`str`	Structured markdown report with citations
`sources`	`list[ResearchSource]`	Gathered sources with title, URL, content, and formatted citation
`structured_output`	`dict`	JSON matching `output_schema` if provided
`model_used`	`str`	LLM provider used for synthesis (`ollama`, `anthropic`)
`total_sources`	`int`	Number of sources gathered
`response_time`	`float`	Total research time in seconds
`status`	`str`	`"pending"`, `"running"`, `"complete"`, or `"failed"`
`error`	`str`	Error message if the research failed

Configuration

Environment variables

Variable	Default	Description
`REDIS_URL`	—	Redis connection for caching + job queue
`SEARXNG_URL`	—	SearXNG instance for search (falls back to DuckDuckGo + Google + Bing)
`BRAVE_SEARCH_API_KEY`	—	Brave Search API key (enables `brave` engine, preferred over DuckDuckGo)
`SERPAPI_KEY`	—	SerpAPI key (enables `serpapi` engine, preferred over Google scraping)
`GOOGLE_API_KEY`	—	Google Custom Search API key (used by `google` engine when both key and CX are set)
`GOOGLE_CX`	—	Google Custom Search CX (required with `GOOGLE_API_KEY`)
`ANTHROPIC_API_KEY`	—	For Claude-powered `agent_extract` and research agent synthesis
`ANTHROPIC_MODEL`	`claude-3-haiku-20240307`	Claude model name for extraction and research
`OLLAMA_URL`	—	Ollama endpoint for local LLM extraction and research agent
`OLLAMA_MODEL`	`llama3.2`	Ollama model name for extraction and research
`AGENTFETCH_CACHE_TTL`	`300`	In-memory LRU cache TTL (seconds)
`AGENTFETCH_CACHE_SIZE`	`100`	Max entries in in-memory LRU cache
`AGENTFETCH_STATIC_TIMEOUT`	`15`	HTTP fetch timeout (seconds)
`AGENTFETCH_BROWSER_TIMEOUT`	`30`	Playwright browser timeout (seconds)
`AGENTFETCH_MAX_RETRIES`	`2`	Max retries for failed requests
`AGENTFETCH_DOMAIN_DELAY`	`0.5`	Delay between requests to same domain
`AGENTFETCH_ROBOTS_CHECK`	`false`	Enable robots.txt compliance
`AGENTFETCH_PROXY_LIST`	—	Comma-separated proxy URLs or JSON array
`AGENTFETCH_PROXY_STRATEGY`	`round-robin`	`round-robin` or `random`
`AGENTFETCH_COOKIES_FILE`	—	Path to cookies file (Netscape or JSON)
`AGENTFETCH_PORT`	`8080`	API server port
`AGENTFETCH_JA3_PROFILE`	—	JA3 TLS profile override for `curl_cffi`
`AGENTFETCH_STEALTH`	`true`	Enable browser stealth evasions in Playwright
`AGENTFETCH_STEALTH_BASIC_FALLBACK`	`true`	Fall back to non-stealth browser if stealth fails
`AGENTFETCH_CRAWL_DB`	`agentfetch_crawl.db`	SQLite path for crawl job persistence
`AGENTFETCH_MIN_PROSE_RATIO`	`0.4`	Minimum alpha-char ratio for quality check
`AGENTFETCH_MIN_WORDS`	`10`	Minimum word count for quality check

Self-host

docker-compose up -d
# Starts API (port 8080), MCP SSE (port 8081), Redis
# Optional crawl worker:
docker compose --profile worker up -d

Architecture

                         ┌─────────────┐
                         │   Smart     │
                         │   URL       │
                         │   Router    │
                         └──────┬──────┘
                                │
              ┌─────────────────┼──────────────────┐
              │                 │                   │
              ▼                 ▼                   ▼
      ┌────────────┐   ┌──────────────┐   ┌────────────────┐
      │  Static    │   │  Cloudflare  │   │   Playwright   │
      │  HTTP      │   │  bypass      │   │   Headless     │
      │  (httpx)   │   │  (curl_cffi) │   │   Browser      │
      └─────┬──────┘   └──────┬───────┘   └───────┬────────┘
            │                 │                    │
            └─────────────────┼────────────────────┘
                              │
                              ▼
                    ┌─────────────────┐
                    │  Extraction     │
                    │  Pipeline       │
                    │  trafilatura →  │
                    │  newspaper3k →  │
                    │  readability →  │
                    │  BS4 → plain    │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │  Sanitizer      │
                    │  (13 injection  │
                    │   patterns)     │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │  Post-process   │
                    │  • Citations    │
                    │  • Dedup check  │
                    │  • Max length   │
                    │  • Markdown     │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │   FetchResult   │
                    │   Pydantic      │
                    │   response      │
                    └─────────────────┘

Tests

pip install -e ".[all]"
pytest tests/ -v
# 138 tests passing

License

MIT — free for any use, including commercial.

This server cannot be installed

license - permissive license

quality - not tested

maintenance

How are these scores calculated?

Maintenance

–Maintainers

–Response time

–Release cycle

–Releases (12mo)

Commit activity

Resources

GitHub Repository

Need Help?

Related Servers

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly
Lightport: Open-Sourcing Glama's AI Gateway
By punkpeye on April 27, 2026.
OpenAI
open source

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/SID1ART/agentfetch'

If you have feedback or need assistance with the MCP directory API, please join our Discord server