Arachne MCP
Provides a search adapter to query Brave Search API, enabling web searches and optional scraping of results.
Designed as a web-intelligence server for Hermes Agent, providing tools for web crawling, scraping, parsing, search, and browser interaction.
Supports structured JSON extraction via OpenAI-compatible LLM endpoints, allowing extraction of data according to a JSON schema.
Provides a search adapter to query SearXNG self-hosted instances, enabling web searches and optional scraping of results.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Arachne MCPCrawl the homepage of https://example.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Arachne MCP
Arachne MCP adalah web-intelligence server self-hosted untuk Hermes Agent dan klien MCP lain. Proyek ini merupakan penerus web-crawler-mcp versi ringan, dengan arsitektur hybrid HTTP + browser, parser dokumen, crawl budgets, search adapter, structured extraction, browser sessions, dan change detection.
Target proyek ini bukan menjanjikan bahwa semua website pasti dapat diambil. CAPTCHA, login tanpa izin, paywall, anti-bot tingkat lanjut, dan pembatasan hukum tetap harus dihormati. Target yang realistis adalah menjadi crawler self-hosted yang lebih dapat dikendalikan dan lebih cocok untuk agent dibandingkan layanan generik.
Fitur utama
Auto-escalation HTTP → Chromium: mencoba request HTTP yang cepat, kemudian otomatis memakai Playwright bila HTML terlihat kosong atau bergantung pada JavaScript.
13 MCP tools: scrape, crawl, batch, map, parse, search, structured extraction, browser interaction, diff, chunking, dan diagnostics.
Parser multi-format: HTML, plain text, JSON, XML/RSS/Atom, PDF, DOCX, XLSX, CSV, dan TSV.
Main-content extraction: Trafilatura sebagai extractor utama, lalu DOM/BeautifulSoup sebagai fallback untuk dokumentasi dan tabel.
Recursive sitemap discovery: membaca sitemap index bertingkat dan menggabungkannya dengan shallow link discovery.
Strict crawl budgets:
max_pages,max_requests,max_errors,max_depth,max_total_chars, danmax_response_bytesberdiri sendiri.Per-host rate limiter: jeda diterapkan sebelum request berikutnya, termasuk crawl delay dari
robots.txt.Retry yang terkontrol: exponential backoff serta dukungan
Retry-Afteruntuk 429 dan error sementara.Persistent SQLite cache: mengurangi request dan token untuk halaman yang sama.
Change detection: menyimpan snapshot dan mengembalikan unified diff.
Persistent browser session: menyimpan cookie/storage state dengan
session_iduntuk alur login yang memang diotorisasi.Search adapter: SearXNG self-hosted atau Brave Search.
Structured JSON extraction: memakai endpoint LLM OpenAI-compatible, termasuk 9Router/OpenRouter/OpenAI.
Agent safety: hasil web ditandai sebagai untrusted content dan dipindai untuk pola prompt injection umum.
SSRF baseline protection: memblokir localhost, IP privat, link-local, reserved, dan memvalidasi setiap redirect.
Proxy support: satu proxy HTTP/HTTPS dapat dikonfigurasi untuk HTTP dan browser engine.
Content deduplication: halaman duplikat dikenali melalui SHA-256 agar output crawl tidak boros konteks.
Related MCP server: servo-fetch
Tool MCP
Tool | Fungsi |
| Memeriksa engine, parser, search, proxy, dan security policy |
| Mengambil satu URL dengan mode |
| Memproses PDF, DOCX, XLSX, JSON, XML, CSV, dan dokumen lain |
| Menjelajahi website dengan BFS dan crawl budgets keras |
| Mengambil sampai 500 URL dengan concurrency dan shared output budget |
| Menemukan URL melalui sitemap recursive dan shallow crawl |
| Mengambil link yang sudah dinormalisasi |
| Memeriksa izin, crawl delay, request rate, dan sitemap |
| Click, fill, press, select, wait, scroll, goto, dan screenshot |
| Search melalui SearXNG atau Brave, opsional scrape hasil |
| Menghasilkan JSON sesuai JSON Schema melalui LLM compatible |
| Membandingkan halaman dengan snapshot sebelumnya |
| Memecah teks menjadi chunk overlap yang stabil untuk RAG |
Instalasi lokal
Persyaratan:
Python 3.11+
macOS, Linux, atau Windows
Chromium Playwright untuk mode browser
unzip arachne-mcp.zip
cd arachne-mcp
cp .env.example .env
python3 -m venv .venv
source .venv/bin/activate
pip install -e '.[full]'
playwright install chromium
arachne-mcpDefault transport adalah STDIO. Untuk Streamable HTTP:
MCP_TRANSPORT=streamable-http MCP_HOST=0.0.0.0 MCP_PORT=8000 arachne-mcpEndpoint-nya:
http://127.0.0.1:8000/mcpMenjalankan dengan Docker
Buat external network bersama Hermes sekali saja:
docker network create hermes-netKemudian:
cp .env.example .env
docker compose up -d --buildPemeriksaan:
docker compose ps
docker logs arachne-mcp --tail 100Port hanya dipublikasikan ke 127.0.0.1. Container Hermes berkomunikasi melalui network internal menggunakan:
http://arachne-mcp:8000/mcpHubungkan ke Hermes Agent
Tambahkan pada config.yaml Hermes:
mcp_servers:
arachne:
url: "http://arachne-mcp:8000/mcp"
enabled: true
connect_timeout: 30
timeout: 300
supports_parallel_tool_calls: false
tools:
include:
- health
- crawl_url
- parse_url
- crawl_site
- batch_scrape
- map_site
- extract_links
- check_robots
- browser_interact
- search_web
- extract_structured
- crawl_diff
- chunk_content
resources: false
prompts: falseRestart Hermes:
cd /srv/hermes
docker compose up -d
docker logs hermes_container --tail 100Contoh prompt Telegram:
Gunakan Arachne untuk crawl dokumentasi https://example.com/docs.
Gunakan maksimum 20 halaman, 50 request, kedalaman 3, dan total output 80.000 karakter.
Rangkum hasilnya dan sertakan halaman sumber untuk setiap bagian.Untuk halaman JavaScript:
Gunakan crawl_url Arachne dengan render_mode browser untuk membuka URL ini.
Tunggu 2 detik, ambil Markdown dan daftar link internalnya.Browser actions
Contoh input browser_interact:
{
"url": "https://example.com/products",
"actions": [
{"type": "fill", "selector": "input[name=q]", "value": "keyboard"},
{"type": "press", "selector": "input[name=q]", "key": "Enter"},
{"type": "wait", "selector": ".results", "state": "visible"},
{"type": "scroll", "pixels": 1200}
],
"screenshot": false
}Action yang tersedia:
click: membutuhkanselectorfill:selector,valuepress: opsionalselector, sertakeyselect:selector,valuewait:selectorataumsscroll:pixelsgoto:urlevaluate:script, tetapi default dinonaktifkan
Arachne tidak menyediakan action untuk melewati CAPTCHA atau access control.
Persistent browser session
Gunakan session_id dan persist_session=true:
{
"url": "https://portal.example.com/login",
"session_id": "portal-example",
"persist_session": true,
"actions": [
{"type": "fill", "selector": "#email", "value": "user@example.com"},
{"type": "fill", "selector": "#password", "value": "..."},
{"type": "click", "selector": "button[type=submit]"},
{"type": "wait", "selector": ".dashboard"}
]
}Storage state disimpan di data/browser-sessions. Lindungi folder tersebut karena dapat mengandung cookie autentikasi. Jangan mengirim password melalui prompt pada channel yang tidak privat; lebih aman menyediakan session state secara manual.
Search self-hosted
SearXNG
Atur:
SEARXNG_URL=http://searxng:8080Arachne memakai output JSON SearXNG, lalu dapat melakukan scrape terhadap hasil yang dipilih.
Brave Search
BRAVE_SEARCH_API_KEY=your-keySearXNG lebih cocok bila prioritasnya adalah kontrol dan self-hosting penuh.
Structured extraction melalui 9Router
Contoh .env:
LLM_BASE_URL=http://9router:20128/v1
LLM_API_KEY=your-9router-key
LLM_MODEL=kr/qwen3-coder-nextContoh tool input:
{
"url": "https://example.com/product/1",
"instruction": "Ambil informasi produk yang terlihat pada halaman.",
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"price": {"type": ["number", "null"]},
"currency": {"type": ["string", "null"]}
},
"required": ["name", "price", "currency"],
"additionalProperties": false
}
}Model harus mendukung endpoint /chat/completions. Jika model tidak mendukung json_schema, Arachne otomatis mencoba ulang tanpa response_format, lalu memvalidasi JSON yang dihasilkan.
Crawl budgets yang disarankan untuk Hermes
Penggunaan interaktif ringan:
max_pages: 10
max_requests: 25
max_errors: 8
max_depth: 2
max_chars_per_page: 5000
max_total_chars: 40000
concurrency: 3Dokumentasi menengah:
max_pages: 50
max_requests: 120
max_errors: 20
max_depth: 4
max_chars_per_page: 8000
max_total_chars: 200000
concurrency: 6Jangan menaikkan semua batas sekaligus. Output MCP yang terlalu besar dapat mengurangi kemampuan reasoning model walaupun crawler berhasil.
Konfigurasi penting
Environment variable | Default | Keterangan |
|
| Mematuhi robots.txt |
|
| Mengizinkan caller memilih |
|
| Proteksi SSRF |
|
| Batas maksimum request per crawl |
|
| Batas output agregat |
|
| Jumlah context Chromium bersamaan |
|
| TTL cache |
| kosong | Proxy HTTP/HTTPS opsional |
|
| Mengizinkan arbitrary browser JS |
Lihat seluruh opsi di .env.example.
Perbandingan sasaran dengan Firecrawl
Arachne sengaja dioptimalkan untuk self-hosted agent stack:
Area | Arachne | Firecrawl hosted |
Source code dan modifikasi | Sepenuhnya lokal dan modular | Open source core + hosted infrastructure |
Biaya per halaman | Infrastruktur sendiri | Credit-based |
HTTP → browser escalation | Dapat diatur dan diperiksa | Otomatis |
Parser dokumen | Lokal | Tersedia melalui API |
Persistent crawl cache | SQLite lokal | Dikelola layanan |
Change diff | Built-in unified diff | Monitoring tersedia pada layanan |
Prompt-injection wrapping | Built-in untuk respons MCP | Bergantung workflow agent |
Crawl output budget | Hard character budget untuk konteks LLM | Page/concurrency controls |
Search | SearXNG/Brave adapter | Hosted search index |
Anti-bot/proxy reliability | Bergantung proxy/infrastruktur Anda | Infrastruktur proprietary lebih matang |
Arachne dapat unggul dalam privasi, kontrol, extensibility, biaya marginal, dan integrasi Hermes. Firecrawl hosted kemungkinan tetap unggul untuk coverage internet luas, managed proxy rotation, anti-bot, SLA, dan skala besar tanpa operasi sendiri.
Benchmark terhadap Firecrawl
Masukkan URL ke file, satu URL per baris:
cat > urls.txt <<'EOF'
https://example.com
https://docs.python.org/3/
EOFJalankan benchmark Arachne:
PYTHONPATH=src python scripts/benchmark.py --urls urls.txt --output benchmark.jsonBandingkan dengan Firecrawl bila memiliki API key:
export FIRECRAWL_API_KEY=fc-...
PYTHONPATH=src python scripts/benchmark.py \
--urls urls.txt \
--compare-firecrawl \
--output benchmark.jsonMetric yang dicatat:
keberhasilan per URL
latency
panjang konten
engine HTTP/browser
status code
error
Tambahkan dataset website Anda sendiri: docs statis, SPA, blog, PDF, tabel, e-commerce, dan halaman yang sering gagal. Klaim lebih unggul hanya masuk akal setelah hasil pada dataset Anda menunjukkan demikian.
Testing
pytest -qPengujian yang disertakan mencakup:
canonical URL normalization
private-network blocking
HTML metadata dan link extraction
prompt-injection detection/wrapping
content chunking
hard request budget
SQLite cache dan snapshots
local HTTP crawling dan recursive sitemap
Keterbatasan yang disengaja
Arachne tidak secara otomatis:
melewati CAPTCHA
membobol login atau paywall
mengakali access control
memakai akun tanpa izin
menjamin halaman yang memblokir data center IP dapat diambil
mengabaikan
robots.txtkecuali administrator mengaktifkannyamenjamin DNS-rebinding protection sempurna pada semua network stack
Untuk reliability setara layanan hosted pada web yang sangat protektif, Anda tetap memerlukan proxy pool berkualitas, observability, distributed queue, autoscaling browser workers, dan maintenance selector/anti-bot secara berkelanjutan.
Struktur proyek
src/arachne_mcp/
├── cache.py # SQLite cache dan snapshots
├── config.py # Environment settings
├── crawler.py # Orchestration, crawl, map, search, diff
├── errors.py
├── extractors.py # HTML dan document extraction
├── fetchers.py # HTTPX, robots, Playwright, actions
├── models.py
├── rate_limit.py
├── security.py # URL/IP guard
├── server.py # MCP tools
└── url_utils.pyDokumen tambahan:
docs/ARCHITECTURE.mddocs/SECURITY.mddocs/HERMES.md
Lisensi
MIT.
Available Tools
13 toolsbatch_scrapeA
Scrape up to 500 URLs concurrently with a shared output budget.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | ||
| concurrency | No | ||
| render_mode | No | auto | |
| include_links | No | ||
| output_format | No | markdown | |
| max_total_chars | No | ||
| max_chars_per_url | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description must carry the burden. It mentions concurrency and output budget but lacks details on error handling, rate limits, partial results, or other behaviors.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One sentence, front-loaded with essential info, no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 7 parameters and complexity of batch scraping, the description is too brief. It does not cover return format, error behavior, or parameter details, leaving gaps for agent understanding.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description should compensate by explaining key parameters. It only hints at max_total_chars via 'shared output budget', ignoring concurrency, render_mode, include_links, etc.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the tool scrapes up to 500 URLs concurrently with a shared output budget, distinguishing it from single-URL scraping or extraction tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Description implies batch usage but does not explicitly state when to use this tool over alternatives like crawl_url or extract_links, nor does it specify when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
browser_interactA
Operate a browser using safe declarative actions: click, fill, press, select, wait, scroll, and goto.
JavaScript evaluate is disabled unless the server owner explicitly enables it. This tool does not bypass CAPTCHA, authentication, paywalls, or access controls.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| actions | Yes | ||
| headers | No | ||
| max_chars | No | ||
| screenshot | No | ||
| session_id | No | ||
| include_links | No | ||
| output_format | No | markdown | |
| respect_robots | No | ||
| persist_session | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses key behavioral traits: JavaScript eval is disabled by default, and the tool does not bypass CAPTCHA, authentication, paywalls, or access controls. However, it does not mention session handling, potential state changes, or output format implications.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise (3 sentences) and front-loaded with the main purpose. The second and third sentences add important caveats. Could be slightly more structured, but it is efficient without unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (10 parameters, nested actions) and existing output schema, the description covers core purpose and a couple of limitations but omits behavior details like session persistence, output format handling, and action sequence execution rules.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description must compensate, but it only briefly mentions actions and does not explain most parameters (e.g., session_id, persist_session, output_format). The agent gets minimal help understanding parameter semantics beyond their names.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Operate a browser using safe declarative actions' and lists specific actions (click, fill, press, etc.). It differentiates from siblings like crawl_url and extract_links by focusing on interactive browser operations rather than extraction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for interactive browser tasks (clicking, filling forms) but lacks explicit guidance on when to use this tool versus sibling extraction tools such as crawl_url or extract_structured. No 'use this for... not for...' wording is present.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
check_robotsC
Inspect robots.txt permissions, crawl delay, request rate, and sitemap declarations.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description should disclose behavioral traits like read-only nature or rate limits. It only lists what is inspected but does not describe how the tool behaves (e.g., whether it fetches the file, caches results, or requires authentication).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that efficiently conveys the tool's function without extraneous words. It is front-loaded with the key verb 'Inspect' and lists the items inspected.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has an output schema (not shown), but the description does not hint at the output structure. Given the complexity of inspecting multiple items, the description should provide some context about the return format or what constitutes successful retrieval.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description does not explain the 'url' parameter beyond its name. It fails to specify expected format (e.g., full URL with protocol), or that it should point to the root domain's robots.txt.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: inspecting robots.txt permissions, crawl delay, request rate, and sitemap declarations. It distinguishes it from sibling tools like 'crawl_url' or 'extract_links' by focusing on robots.txt analysis.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. For example, it does not specify that it should be used before crawling to check permissions, nor does it mention any prerequisites or limitations.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
chunk_contentB
Split long content into stable, overlapping, hash-addressed chunks for RAG or agents.
| Name | Required | Description | Default |
|---|---|---|---|
| content | Yes | ||
| overlap | No | ||
| chunk_size | No | ||
| max_chunks | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the lack of annotations, the description discloses behavioral traits: 'stable' (implying deterministic, idempotent), 'overlapping' (chunks have shared context), and 'hash-addressed' (content-based addressing). However, it does not explain stability guarantees or the hashing scheme in detail.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no wasted words. It is concise, but it lacks structural elements like bullet points or an example that could improve scannability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description fails to explain the output (despite existence of an output schema) and does not cover parameter semantics. For a tool with four parameters and no annotation support, the description is incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% description coverage for parameters, and the tool description does not mention any parameter meaning, defaults, or usage. All four parameters (content, overlap, chunk_size, max_chunks) are left unexplained, leaving the agent with insufficient information.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'split', the resource 'long content', and the key characteristics 'stable, overlapping, hash-addressed'. It also specifies the intended use case 'for RAG or agents', distinguishing it from sibling tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for splitting content for retrieval contexts, but it does not provide explicit guidance on when to use this tool versus alternatives or when not to use it. The intended context (RAG/agents) is helpful but not sufficient for clear decision-making.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_diffB
Compare a page with its previous persistent snapshot and return a unified diff.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| max_chars | No | ||
| render_mode | No | auto | |
| snapshot_name | No | default | |
| update_snapshot | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description must disclose behavioral traits but fails to mention side effects like snapshot creation or update behavior, or what happens if no snapshot exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, front-loaded sentence with no extraneous words; highly efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 5 parameters and no annotations, the description omits critical context about snapshot management, diff interpretation, and the update_snapshot behavior, leaving the agent underinformed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, but the description adds no parameter explanations beyond what parameter names imply. Does not compensate for the lack of schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool compares a page with its previous persistent snapshot and returns a unified diff, which is specific and distinguishes it from sibling tools like crawl_url or crawl_site.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives; lacks context about prerequisites or typical scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_siteB
Crawl a site with strict request/error/token budgets and per-host rate limiting.
| Name | Required | Description | Default |
|---|---|---|---|
| compact | No | ||
| delay_ms | No | ||
| max_depth | No | ||
| max_pages | No | ||
| start_url | Yes | ||
| max_errors | No | ||
| concurrency | No | ||
| render_mode | No | auto | |
| max_requests | No | ||
| output_format | No | markdown | |
| exclude_pattern | No | ||
| include_pattern | No | ||
| max_total_chars | No | ||
| include_documents | No | ||
| include_subdomains | No | ||
| max_chars_per_page | No | ||
| deduplicate_content | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses key behavioral traits: strict request/error/token budgets and per-host rate limiting. However, with no annotations, the description carries the full burden, and it fails to mention important behaviors like conforming to robots.txt, handling authentication, or how budgets are enforced. Additional context is missing.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise (one sentence, 10 words), which is efficient but underserves a tool with 17 parameters. It front-loads the core action and constraints but lacks the depth that multiple sentences could provide for key details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (17 parameters, 1 required, no schema descriptions, no annotations) and the presence of an output schema (not shown), the description is far from complete. It does not explain return values, effects of parameters, or error scenarios, leaving the agent to guess many operational details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. However, it only mentions budgets and rate limiting, which are not explicit parameter names. None of the 17 parameters (e.g., max_pages, delay_ms, include_pattern) are described or linked to the narrative, leaving the agent to infer from parameter names alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action ('Crawl a site') and adds context about budgets and rate limiting. It implies differentiation from sibling tools like 'crawl_url' which likely handles single URLs, but does not explicitly call out alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides implied usage for controlled crawling with budgets, but no explicit guidance on when to use this tool versus alternatives (e.g., 'crawl_url' for single pages, 'map_site' for structure). No exclusions or conditions are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_urlB
Fetch and extract one URL using fast HTTP, browser rendering, or automatic escalation.
Supports HTML, text, JSON, XML, PDF, DOCX, XLSX, CSV and TSV. Content returned to an agent is marked as untrusted and scanned for common prompt-injection text.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| headers | No | ||
| wait_ms | No | ||
| max_chars | No | ||
| use_cache | No | ||
| screenshot | No | ||
| render_mode | No | auto | |
| include_links | No | ||
| output_format | No | markdown | |
| include_images | No | ||
| respect_robots | No | ||
| remove_selectors | No | ||
| cache_ttl_seconds | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses that content is marked untrusted and scanned for prompt-injection, and lists supported formats. However, no annotations provided, and description fails to mention important behaviors like redirect handling, timeouts, error responses, or mutation. With zero annotation coverage, the description should be more thorough.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is very concise (3 short lines), front-loads purpose. However, it omits parameter details, which would be necessary for completeness. Appropriate length but at expense of utility.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 13 parameters and an output schema, the description is too brief. It doesn't explain caching, output format options, or parameter behaviors. Even with output schema existing, agent needs more context to set parameters correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so description must compensate but does not explain individual parameters. Only 'automatic escalation' relates to render_mode; other 12 parameters are not described. The schema properties' names provide some hint, but description adds little meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states action (fetch and extract), resource (one URL), and methods (HTTP, browser, auto). Distinguishes from siblings like crawl_site and extract_links by specifying single URL extraction and supported formats.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool vs siblings. Although 'one URL' implies single page extraction, no mention of when to prefer this over extract_links, parse_url, or browser_interact.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_linksC
Extract normalized crawlable links from static or JavaScript-rendered pages.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| max_links | No | ||
| render_mode | No | auto | |
| same_host_only | No | ||
| include_subdomains | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It discloses JS rendering support but omits details like side effects (none expected), authentication requirements, rate limits, or what 'normalized' means. The transparency is adequate but not thorough.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence that is direct and front-loaded with the core action. While concise, it sacrifices useful detail. A slightly longer description could improve clarity without significant bloat.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having an output schema (implied) and several parameters, the description is minimal. It does not explain the output format, the normalization process, or how parameters affect behavior. The tool's complexity is higher than the description suggests.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% (only titles provided). The description does not elaborate on any parameter meanings, defaults, or constraints. For a tool with 5 parameters and low coverage, the description should compensate but fails to do so.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the tool extracts normalized crawlable links from static or JS-rendered pages. It specifies the action (extract) and resource (links), and hints at an ability to handle JS rendering. However, it does not explicitly distinguish it from sibling tools like crawl_url or parse_url, which may also extract links.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives (e.g., crawl_url for full crawling, parse_url for URL parsing). The description lacks any when-not-to-use or prerequisite information, leaving the agent without decision context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_structuredC
Extract JSON matching a caller-supplied schema through an optional OpenAI-compatible LLM.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| schema | Yes | ||
| instruction | No | Extract the requested fields accurately. | |
| render_mode | No | auto | |
| max_source_chars | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so the description carries full burden. It mentions 'optional OpenAI-compatible LLM' but does not disclose potential side effects (e.g., API calls, cost, rate limits) or whether the tool is read-only.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The single sentence is too brief; it fails to include important details about parameters, behavior, or output. It is under-specified rather than concise, given the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having an output schema and 5 parameters, the description is inadequate. It does not mention return format, rendering choices, or LLM behavior, making it incomplete for the tool's complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%. The description does not explain any of the 5 parameters (url, schema, instruction, render_mode, max_source_chars), leaving agents without guidance on required or optional fields.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts JSON matching a caller-supplied schema, optionally using an LLM. This distinguishes it from siblings like extract_links (extracts links) or crawl_url (crawls).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool over siblings (e.g., when to use render_mode choices or when LLM is needed). The description only mentions optional LLM but no when-to-use or when-not-to-use context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
healthA
Return enabled engines, integrations, security policy, and parser availability.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden for behavioral disclosure. It accurately describes a read-only operation returning system status. Although it doesn't explicitly state non-destructive behavior, the context strongly implies it, and no contradictions exist.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that directly states the tool's output. It is front-loaded, concise, and contains no filler. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no parameters, the presence of an output schema, and clear differentiation from siblings, the description is fully complete. It provides all necessary context for an agent to correctly invoke and interpret the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, and schema coverage is 100%. The description does not need to add parameter details beyond the schema, as none exist. A baseline of 4 is appropriate since the description contributes no extra semantic value but is not required.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states the tool returns 'enabled engines, integrations, security policy, and parser availability', providing a clear and specific verb-resource combination. It distinguishes itself from sibling tools like crawl_url or search_web, which focus on different actions.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use for checking system health but offers no explicit guidance on when to use this tool versus alternatives. Since siblings are functionally distinct, the context is clear, but there is no direct exclusion criteria or usage scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
map_siteC
Discover site URLs from recursive sitemaps plus optional shallow crawling.
| Name | Required | Description | Default |
|---|---|---|---|
| max_urls | No | ||
| start_url | Yes | ||
| crawl_pages | No | ||
| exclude_pattern | No | ||
| include_pattern | No | ||
| discover_by_crawl | No | ||
| include_subdomains | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description must fully disclose behavioral traits. It only states the method (recursive sitemaps + shallow crawling) without mentioning effects like read-only behavior, rate limits, or robots.txt compliance. This is insufficient for safe invocation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence with no fluff. It is front-loaded with the core purpose. However, it could be expanded slightly to improve completeness without losing conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 7 parameters, no annotations, and an existing output schema, the description is severely incomplete. It omits return value information, parameter behaviors, and handling of edge cases (e.g., exclude/include patterns). The agent lacks context for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for 7 parameters. The description does not explain any parameter (e.g., start_url, max_urls, crawl_pages). It fails to compensate for the lack of schema documentation, leaving the agent without parameter insight.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Discover site URLs from recursive sitemaps plus optional shallow crawling' clearly states the verb, resource, and method. It distinguishes from siblings like crawl_site by mentioning sitemap discovery, but could be more explicit about the contrast.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives (e.g., crawl_site, extract_links). There are no prerequisites, exclusions, or context for selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
parse_urlC
Parse a remote document or non-HTML resource into LLM-ready text/Markdown.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| max_chars | No | ||
| use_cache | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description must disclose behaviors. It does not mention caching behavior, rate limits, supported document types, or what happens with HTML pages despite saying 'non-HTML resource'. The description is vague about the parsing mechanism.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, which is concise, but it lacks structure and fails to provide important details. Every sentence should earn its place; here the single sentence is insufficient for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has three parameters, no annotation coverage, and an output schema not detailed. The description is too brief to be complete, omitting caching, character limits, and parsing scope. Given the complexity, more detail is needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description adds no explanation for parameters. 'max_chars' and 'use_cache' are not described beyond their names, leaving the agent to guess their meaning and defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'Parse' and specifies the resource types ('remote document or non-HTML resource') and output format ('LLM-ready text/Markdown'). It clearly distinguishes from sibling tools like 'search_web' or 'extract_links' which have different purposes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives such as 'extract_structured' or 'browser_interact'. There are no explicit conditions, prerequisites, or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_webC
Search through a configured self-hosted SearXNG or Brave Search provider.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| query | Yes | ||
| scrape_results | No | ||
| max_chars_per_result | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden but only mentions relying on a configured provider. It fails to disclose behavioral traits like rate limits, authentication needs, fallback behavior, or whether results are cached.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, well-front-loaded sentence. While concise, it could incorporate more useful information without becoming verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity (4 parameters, no schema descriptions, no annotations, and an output schema not described), the description is insufficient. It does not explain the effect of scrape_results or max_chars_per_result, nor the structure of search results.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% and the description adds no parameter details. The tool has 4 parameters (limit, query, scrape_results, max_chars_per_result) with no explanations, leaving the AI unable to understand their purpose or valid ranges.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool searches through a configured self-hosted SearXNG or Brave Search provider, distinguishing it from sibling tools like crawl_url or extract_links which perform different operations. However, it could be more specific about the type of search (e.g., general web search) and result format.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives such as crawl_site or map_site. The description lacks any context about prerequisite configurations or typical scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
13 tool updates
v1.0.0- First observed
batch_scrape - First observed
browser_interact - First observed
check_robots - First observed
chunk_content - First observed
crawl_diff - First observed
crawl_site - First observed
crawl_url - First observed
extract_links - First observed
extract_structured - First observed
health - First observed
map_site - First observed
parse_url - First observed
search_web
TDQS
Scored across 13 tools
Each tool serves a distinct purpose in the web scraping pipeline, from health checks to link extraction, crawling, parsing, site mapping, robots inspection, browser interaction, batch scraping, search, structured extraction, diffing, and chunking. No two tools have overlapping functionality.
All tool names follow a consistent verb_noun pattern in snake_case (e.g., extract_links, crawl_url, parse_url). The only exception is 'health', which is a noun but serves as a status probe and fits the convention of a single-word command.
13 tools is appropriate for a comprehensive web scraping and content extraction server. Each tool covers a necessary operation without redundancy, balancing breadth and focus.
The tool surface covers the full lifecycle of web data acquisition: discovery (map_site, search_web), fetching (crawl_url, crawl_site, batch_scrape), parsing (parse_url, extract_links), interaction (browser_interact), compliance (check_robots), processing (chunk_content, extract_structured), and monitoring (health, crawl_diff). No obvious gaps.
Maintenance
Related MCP Connectors
Verified business OSS MCP for search, RSS, crawling, documents, browser, media and transcription.
- fastCRWOAuthio.github.us
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
MCP server for web extraction and rendering via AceDataCloud WebExtrator
Personal knowledge base MCP server with semantic search, auto-categorization, metadata extraction
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceOpen-source web scraper and extraction MCP server with JavaScript rendering, markdown output, PDF/DOCX parsing, structured errors, and validated extraction contract diagnostics for agents.2AGPL 3.0
- AlicenseAqualityAmaintenanceServo-powered MCP server for JS-aware web fetching, content extraction, crawling, and software-rendered screenshots — Chromium-free single binary.6146Apache 2.0
- -licenseNot gradedqualityCmaintenanceSelf-hosted MCP server that provides web scraping and crawling tools, integrating seamlessly with AI frameworks like OpenAI Agents SDK, Cursor, and Claude Code.4-
- AlicenseBqualityBmaintenanceA local MCP server providing web search, page extraction, and safe browser automation tools for Hermes, Claude Code, and other MCP clients.10MIT