AI 에이전트를 위한 웹 인텔리전스 — 토큰 추정, 스마트 캐싱, 지능형 라우팅이 내장된 URL 가져오기 MCP 서버입니다.
AgentFetch는 에이전트와 오픈 웹 사이의 가교 역할을 합니다. Jina, FireCrawl, pypdf 및 자체 캐싱 계층을 별도로 통합하는 대신, 에이전트가 하나의 MCP 도구를 호출하면 AgentFetch가 라우팅, 캐싱, 토큰 예산 관리 및 깔끔한 마크다운 추출을 자동으로 처리합니다.
이 저장소에는 오픈 소스 MCP 서버가 포함되어 있습니다. 호스팅된 API + 대시보드 + 결제 기능은 www.agentfetch.dev를 참조하세요.
도구 | 용도 |
| URL 가져오기 → 깔끔한 마크다운 + 메타데이터 + 토큰 수 + 캐시 정보 |
| 가져오기 전에 토큰 수를 확인하여 에이전트가 거대한 페이지에서 컨텍스트 창을 낭비하지 않도록 방지 |
| 최대 20개의 URL을 동시에 가져오기 |
| 웹 검색 + 상위 N개 결과를 한 번의 왕복으로 가져오기 |
AgentFetch는 내부적으로 URL을 가장 비용 효율적인 페처로 라우팅합니다:
Trafilatura (무료, 로컬): 표준 웹 페이지의 약 70% 처리
Jina Reader: 나머지 HTML 처리
FireCrawl: JS가 많은 페이지(Twitter/X, LinkedIn, Notion 등) 처리
pypdf: PDF 처리 (외부 비용 없음)
캐시는 6시간 TTL이 설정된 Redis를 사용하며, 직접 구축하거나 캐싱 없이 실행할 수도 있습니다.
Related MCP server: weblens-mcp
pip install agentfetch-mcpgit clone https://github.com/bch1212/agentfetch-mcp
cd agentfetch-mcp
pip install -e .jina.ai에서 무료 Jina Reader 키를 받으세요(월 100만 토큰 무료 티어). FireCrawl은 선택 사항이지만 JS가 많은 페이지에 권장됩니다.
export JINA_API_KEY=jina_xxx
export FIRECRAWL_API_KEY=fc-xxx # optional
export REDIS_URL=redis://localhost:6379 # optionalMCP 설정(~/Library/Application Support/Claude/claude_desktop_config.json (macOS) 또는 Claude Code에서 claude mcp add 실행)을 편집하세요:
{
"mcpServers": {
"agentfetch": {
"command": "python",
"args": ["-m", "agentfetch.mcp.server"],
"env": {
"JINA_API_KEY": "jina_xxx",
"FIRECRAWL_API_KEY": "fc-xxx"
}
}
}
}Claude를 재시작하세요. 네 가지 도구(fetch_url, estimate_tokens, fetch_multiple, search_and_fetch)가 자동으로 나타납니다.
python -m agentfetch.mcp.server이 서버는 stdio(데스크톱 통합을 위한 표준 전송 방식)를 통해 MCP를 지원합니다.
web_fetch보다 AgentFetch를 선호하는 이유기능 | AgentFetch | 일반 |
가져오기 전 토큰 추정 | ✓ | ✗ |
스마트 캐시 (6시간 TTL) | ✓ | ✗ |
URL 유형별 자동 라우팅 | ✓ | ✗ |
JS 렌더링 페이지 처리 | ✓ (FireCrawl 사용) | 부분 지원 |
PDF 추출 | ✓ | ✗ |
컨텍스트 예산에 맞춘 잘림 | ✓ | 수동 |
# Inside any MCP-aware agent (Claude Desktop, Claude Code, etc.)
result = fetch_url(
url="https://news.ycombinator.com",
max_tokens=2000, # cap response size
use_cache=True, # serve from cache if <6h old
)
# result.markdown → clean Markdown, ≤2000 tokens
# result.metadata → title, author, word_count, language
# result.cache.hit → True if served from cache
# result.fetch_info → which fetcher ran, cost, durationestimate = estimate_tokens(url="https://very-long-article.com")
if estimate.estimated_tokens and estimate.estimated_tokens < 5000:
result = fetch_url(url="https://very-long-article.com")
else:
# too big — skip or summarize via search_and_fetch with max_tokens_each
passresults = fetch_multiple(
urls=["https://docs.python.org/3/", "https://fastapi.tiangolo.com/", ...],
max_tokens_each=1500,
)환경 변수 | 필수 여부 | 기본값 | 참고 |
| 권장 | — | 무료 티어로 월 약 100만 토큰 지원. 없으면 Trafilatura만 작동(페이지의 약 70%에 유용). |
| 선택 | — | JS가 많은 도메인(Twitter, LinkedIn, Notion)에 필요. 가입 시 500 크레딧 무료 제공. |
| 선택 | — | Redis가 없으면 캐시 없이 실행됨. |
| 선택 |
| 가져오기 결과에 대한 캐시 TTL. |
git clone https://github.com/bch1212/agentfetch-mcp
cd agentfetch-mcp
pip install -e ".[dev]"
pytest tests/직접 키, Redis, 라우팅을 관리하고 싶지 않다면 www.agentfetch.dev의 호스팅 버전을 사용하세요:
호출당 $0.001부터 시작하는 종량제 요금
가입 시 500회 무료 가져오기 제공, 신용카드 불필요
관리형 Redis 캐시, 페처 간 자동 장애 조치
사용량 추적 및 청구서가 포함된 대시보드
호스팅된 API는 드롭인 REST와 동일하며, 응답 형태와 라우팅 로직이 같습니다. OSS MCP를 로컬에서 실행하고 호스팅된 API를 병렬로 실행하거나 언제든지 전환할 수 있습니다.
MIT — LICENSE를 참조하세요.
이 저장소의 MCP 서버는 오픈 소스입니다. 호스팅 제품, 결제 및 운영 인프라는 별도의 (비공개) 저장소에 있습니다.
PR을 환영합니다. 새로운 페처(예: Bright Data, ScrapingBee 등)를 추가하는 경우 agentfetch/core/fetchers/__init__.py의 FetchResult 인터페이스를 준수하고 라우팅 로직에 비용을 추가해 주세요.
Estimate token count of a URL's content WITHOUT fetching the body.
WHEN TO USE:
You're considering fetching a URL but unsure if it fits your remaining context window. This call is ~10x cheaper than a full fetch.
You want to triage a list of candidate URLs before deciding which to actually retrieve.
IMPORTANT: Many servers omit Content-Length on dynamic / chunked responses. When that happens, this tool returns confident=false and estimated_tokens=null. In that case, call fetch_url with a max_tokens cap instead of trusting the estimate.
Args: url: The URL to estimate.
Returns: { "url": str, "success": bool, "estimated_tokens": int | null, "byte_size": int | null, "content_type": str, "confident": bool, "note": str }
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, but the description fully discloses behavior: no body fetch, 10x cheaper, fallback when Content-Length missing, and return structure with confident flag.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Well-structured with sections, but includes a full return example that is slightly verbose yet informative.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter tool, the description covers purpose, usage, fallback, and return format completely, compensating for lack of output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 0% coverage, but the description includes an 'Args' section explaining the url parameter, adding meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool estimates token count without fetching the body, and distinguishes from sibling tools like fetch_url by emphasizing it does not fetch the body.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly provides when-to-use (unsure about context window, triaging URLs) and when-not-to-use (if confident=false, use fetch_url with max_tokens).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch up to 20 URLs concurrently. Each result is the same shape as fetch_url.
WHEN TO USE:
You have a list of URLs (search results, links from a doc, sitemap) and want them retrieved in parallel rather than one at a time.
Args: urls: 1–20 URLs. Larger batches: split into multiple calls. max_tokens_each: Per-result cap. Apply this to keep total response inside your context budget — total ≈ len(urls) * max_tokens_each. use_cache: True for cache-aware fetching (default).
Returns: {"count": int, "results": [, ...]}
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | ||
| max_tokens_each | No | ||
| use_cache | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Describes concurrency and per-result token cap, but no annotations present. Lacks details on error handling, rate limits, or caching behavior beyond defaults.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Well-structured into purpose, when-to-use, args, and returns. No redundant sentences, every line contributes.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers purpose, usage, parameters, return shape, and concurrency limit. Could include error behavior or more details on caching, but overall sufficient for a fetch tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Provides clear explanations for all three parameters: url limit and splitting, max_tokens_each token budget guidance, and use_cache caching behavior. Adds significant value over schema-only info.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states 'Fetch up to 20 URLs concurrently' with a specific verb and resource. Distinguishes from sibling tools by highlighting concurrency and batch limit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Includes 'WHEN TO USE' section explicitly describing scenarios. Mentions splitting large batches but does not explicitly state when not to use or name alternatives like fetch_url.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch any URL and return clean, LLM-ready Markdown with token count, metadata, and 6h caching.
WHEN TO USE:
You have a specific URL whose content you need.
You want to cap response size to stay inside your context window.
You want repeat fetches to be cheap (cache hits ≈ $0.0001).
The URL might be JS-rendered, a PDF, or behind a paywall — this tool auto-routes to the right fetcher (Trafilatura → Jina → FireCrawl → PDF).
WHEN NOT TO USE:
You don't know which URL to fetch — use search_and_fetch instead.
You have many URLs to fetch — use fetch_multiple instead.
Args: url: The URL to fetch. max_tokens: Hard cap on response size. Default unlimited. Pass this if you're tight on context budget — cheaper than over-fetching. format: "markdown" (default — recommended), "text", or "json". use_cache: True returns a cached copy if one exists (≤6h old). Pass False only when freshness matters (live news, prices).
Returns: { "url": str, "success": bool, "markdown": str, "metadata": {title, author, published_date, domain, word_count, token_count, reading_time_seconds, content_type, language}, "cache": {hit, cached_at, expires_at}, "fetch_info": {fetcher_used, fetch_time_ms, cost_credits}, "error": str | None }
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| max_tokens | No | ||
| format | No | markdown | |
| use_cache | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description fully bears the transparency burden. It discloses caching (6h), auto-routing to multiple fetchers, cost estimates, and return structure including error handling. This far exceeds minimal requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with sections, bullet points, and a return format sample. While fairly long, every sentence adds value—usage guidance, parameter details, and return schema. Minor conciseness loss from repetition of 'WHEN TO USE' structure.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite no output schema and no annotations, the description compensates fully: explains return structure in detail, caching behavior, fetcher selection logic, cost implications, and context window management (max_tokens). An agent has everything needed to invoke correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must add context for all parameters. It explains url (implicit), max_tokens (hard cap, default unlimited, context budget advice), format (default markdown, recommended), and use_cache (default true, when to pass false). Each parameter gets meaningful guidance beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Fetch any URL and return clean, LLM-ready Markdown' clearly stating the verb+resource+output quality. It explicitly distinguishes from siblings 'search_and_fetch' and 'fetch_multiple' in the WHEN NOT TO USE section.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The WHEN TO USE and WHEN NOT TO USE sections provide explicit scenarios (e.g., specific URL, JS-rendered, PDF, paywall) and name alternative tools. This gives clear guidance on when to invoke this tool versus its siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Web search + fetch top results in one call.
WHEN TO USE:
You have a research question, not specific URLs. E.g. "what's the latest on X", "find docs for Y library", "recent news about Z".
You'd otherwise have to call a search tool, parse results, then call fetch — this collapses that into one round-trip.
Args: query: Search query (2–500 chars). num_results: Top N to fetch (1–10, default 3). max_tokens_each: Per-result cap (default 2000).
Returns: {"query": str, "count": int, "results": [, ...]}
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | ||
| num_results | No | ||
| max_tokens_each | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description effectively discloses the combined search-and-fetch behavior and the return format. However, it lacks details on error handling or failure scenarios for individual result fetches.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a header and 'WHEN TO USE' section, making it easy to parse. It is concise yet informative, though the parameter descriptions could be integrated more succinctly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description adequately covers the tool's purpose, usage, parameters, and return format. Given no output schema, the return shape is documented. Missing details on partial failures or edge cases, but overall sufficient for this combined tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite zero schema coverage, the description fully compensates by specifying constraints (query 2-500 chars, num_results 1-10, max_tokens_each default 2000) and explaining each parameter's role, adding substantial meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Web search + fetch top results in one call,' which precisely defines the tool's combined action. It distinguishes itself from siblings like fetch_url and fetch_multiple by highlighting the aggregation of search and fetch into one step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description includes a 'WHEN TO USE' section that explicitly recommends the tool for research questions rather than specific URLs. It contrasts with the alternative of using separate search and fetch tools, providing clear guidance on when to prefer this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v1.0.0estimate_tokensfetch_multiplefetch_urlsearch_and_fetchScored across 4 tools
Each tool has a clearly distinct purpose: estimate_tokens is for token estimation without body fetch, fetch_url for single URL fetch, fetch_multiple for batch fetch, search_and_fetch for combined search and fetch. No overlap in functionality.
All tool names use consistent snake_case with a verb_noun pattern. The verbs are clear (estimate, fetch, fetch, search_and_fetch) and the nouns differentiate the actions (tokens, url, multiple, and_fetch).
With 4 tools, the set is concise and well-scoped for a web fetching and searching service. Each tool addresses a core need: estimating, single fetch, batch fetch, and combined search+fetch.
The tool set covers the primary workflows of fetching URLs and searching. A possible gap is the lack of a search-only tool that returns just snippets without fetching, but given the server's focus on fetching, the current set is largely complete.
Fetch pages as markdown, search web and news, extract structured data. For AI agents.
Read any web page as clean Markdown for AI agents: fetch, search, metadata, links. SSRF-safe.
Fetch any URL and get clean Markdown. Web scraping for AI agents.
Web pages as Markdown, text or HTML, plus Google Maps places and reviews, for AI agents.