Skip to main content
Glama

Arachne MCP

Arachne MCP adalah web-intelligence server self-hosted untuk Hermes Agent dan klien MCP lain. Proyek ini merupakan penerus web-crawler-mcp versi ringan, dengan arsitektur hybrid HTTP + browser, parser dokumen, crawl budgets, search adapter, structured extraction, browser sessions, dan change detection.

Target proyek ini bukan menjanjikan bahwa semua website pasti dapat diambil. CAPTCHA, login tanpa izin, paywall, anti-bot tingkat lanjut, dan pembatasan hukum tetap harus dihormati. Target yang realistis adalah menjadi crawler self-hosted yang lebih dapat dikendalikan dan lebih cocok untuk agent dibandingkan layanan generik.

Fitur utama

  • Auto-escalation HTTP → Chromium: mencoba request HTTP yang cepat, kemudian otomatis memakai Playwright bila HTML terlihat kosong atau bergantung pada JavaScript.

  • 13 MCP tools: scrape, crawl, batch, map, parse, search, structured extraction, browser interaction, diff, chunking, dan diagnostics.

  • Parser multi-format: HTML, plain text, JSON, XML/RSS/Atom, PDF, DOCX, XLSX, CSV, dan TSV.

  • Main-content extraction: Trafilatura sebagai extractor utama, lalu DOM/BeautifulSoup sebagai fallback untuk dokumentasi dan tabel.

  • Recursive sitemap discovery: membaca sitemap index bertingkat dan menggabungkannya dengan shallow link discovery.

  • Strict crawl budgets: max_pages, max_requests, max_errors, max_depth, max_total_chars, dan max_response_bytes berdiri sendiri.

  • Per-host rate limiter: jeda diterapkan sebelum request berikutnya, termasuk crawl delay dari robots.txt.

  • Retry yang terkontrol: exponential backoff serta dukungan Retry-After untuk 429 dan error sementara.

  • Persistent SQLite cache: mengurangi request dan token untuk halaman yang sama.

  • Change detection: menyimpan snapshot dan mengembalikan unified diff.

  • Persistent browser session: menyimpan cookie/storage state dengan session_id untuk alur login yang memang diotorisasi.

  • Search adapter: SearXNG self-hosted atau Brave Search.

  • Structured JSON extraction: memakai endpoint LLM OpenAI-compatible, termasuk 9Router/OpenRouter/OpenAI.

  • Agent safety: hasil web ditandai sebagai untrusted content dan dipindai untuk pola prompt injection umum.

  • SSRF baseline protection: memblokir localhost, IP privat, link-local, reserved, dan memvalidasi setiap redirect.

  • Proxy support: satu proxy HTTP/HTTPS dapat dikonfigurasi untuk HTTP dan browser engine.

  • Content deduplication: halaman duplikat dikenali melalui SHA-256 agar output crawl tidak boros konteks.

Related MCP server: servo-fetch

Tool MCP

Tool

Fungsi

health

Memeriksa engine, parser, search, proxy, dan security policy

crawl_url

Mengambil satu URL dengan mode auto, http, atau browser

parse_url

Memproses PDF, DOCX, XLSX, JSON, XML, CSV, dan dokumen lain

crawl_site

Menjelajahi website dengan BFS dan crawl budgets keras

batch_scrape

Mengambil sampai 500 URL dengan concurrency dan shared output budget

map_site

Menemukan URL melalui sitemap recursive dan shallow crawl

extract_links

Mengambil link yang sudah dinormalisasi

check_robots

Memeriksa izin, crawl delay, request rate, dan sitemap

browser_interact

Click, fill, press, select, wait, scroll, goto, dan screenshot

search_web

Search melalui SearXNG atau Brave, opsional scrape hasil

extract_structured

Menghasilkan JSON sesuai JSON Schema melalui LLM compatible

crawl_diff

Membandingkan halaman dengan snapshot sebelumnya

chunk_content

Memecah teks menjadi chunk overlap yang stabil untuk RAG

Instalasi lokal

Persyaratan:

  • Python 3.11+

  • macOS, Linux, atau Windows

  • Chromium Playwright untuk mode browser

unzip arachne-mcp.zip
cd arachne-mcp

cp .env.example .env
python3 -m venv .venv
source .venv/bin/activate

pip install -e '.[full]'
playwright install chromium

arachne-mcp

Default transport adalah STDIO. Untuk Streamable HTTP:

MCP_TRANSPORT=streamable-http MCP_HOST=0.0.0.0 MCP_PORT=8000 arachne-mcp

Endpoint-nya:

http://127.0.0.1:8000/mcp

Menjalankan dengan Docker

Buat external network bersama Hermes sekali saja:

docker network create hermes-net

Kemudian:

cp .env.example .env
docker compose up -d --build

Pemeriksaan:

docker compose ps
docker logs arachne-mcp --tail 100

Port hanya dipublikasikan ke 127.0.0.1. Container Hermes berkomunikasi melalui network internal menggunakan:

http://arachne-mcp:8000/mcp

Hubungkan ke Hermes Agent

Tambahkan pada config.yaml Hermes:

mcp_servers:
  arachne:
    url: "http://arachne-mcp:8000/mcp"
    enabled: true
    connect_timeout: 30
    timeout: 300
    supports_parallel_tool_calls: false
    tools:
      include:
        - health
        - crawl_url
        - parse_url
        - crawl_site
        - batch_scrape
        - map_site
        - extract_links
        - check_robots
        - browser_interact
        - search_web
        - extract_structured
        - crawl_diff
        - chunk_content
      resources: false
      prompts: false

Restart Hermes:

cd /srv/hermes
docker compose up -d
docker logs hermes_container --tail 100

Contoh prompt Telegram:

Gunakan Arachne untuk crawl dokumentasi https://example.com/docs.
Gunakan maksimum 20 halaman, 50 request, kedalaman 3, dan total output 80.000 karakter.
Rangkum hasilnya dan sertakan halaman sumber untuk setiap bagian.

Untuk halaman JavaScript:

Gunakan crawl_url Arachne dengan render_mode browser untuk membuka URL ini.
Tunggu 2 detik, ambil Markdown dan daftar link internalnya.

Browser actions

Contoh input browser_interact:

{
  "url": "https://example.com/products",
  "actions": [
    {"type": "fill", "selector": "input[name=q]", "value": "keyboard"},
    {"type": "press", "selector": "input[name=q]", "key": "Enter"},
    {"type": "wait", "selector": ".results", "state": "visible"},
    {"type": "scroll", "pixels": 1200}
  ],
  "screenshot": false
}

Action yang tersedia:

  • click: membutuhkan selector

  • fill: selector, value

  • press: opsional selector, serta key

  • select: selector, value

  • wait: selector atau ms

  • scroll: pixels

  • goto: url

  • evaluate: script, tetapi default dinonaktifkan

Arachne tidak menyediakan action untuk melewati CAPTCHA atau access control.

Persistent browser session

Gunakan session_id dan persist_session=true:

{
  "url": "https://portal.example.com/login",
  "session_id": "portal-example",
  "persist_session": true,
  "actions": [
    {"type": "fill", "selector": "#email", "value": "user@example.com"},
    {"type": "fill", "selector": "#password", "value": "..."},
    {"type": "click", "selector": "button[type=submit]"},
    {"type": "wait", "selector": ".dashboard"}
  ]
}

Storage state disimpan di data/browser-sessions. Lindungi folder tersebut karena dapat mengandung cookie autentikasi. Jangan mengirim password melalui prompt pada channel yang tidak privat; lebih aman menyediakan session state secara manual.

Search self-hosted

SearXNG

Atur:

SEARXNG_URL=http://searxng:8080

Arachne memakai output JSON SearXNG, lalu dapat melakukan scrape terhadap hasil yang dipilih.

BRAVE_SEARCH_API_KEY=your-key

SearXNG lebih cocok bila prioritasnya adalah kontrol dan self-hosting penuh.

Structured extraction melalui 9Router

Contoh .env:

LLM_BASE_URL=http://9router:20128/v1
LLM_API_KEY=your-9router-key
LLM_MODEL=kr/qwen3-coder-next

Contoh tool input:

{
  "url": "https://example.com/product/1",
  "instruction": "Ambil informasi produk yang terlihat pada halaman.",
  "schema": {
    "type": "object",
    "properties": {
      "name": {"type": "string"},
      "price": {"type": ["number", "null"]},
      "currency": {"type": ["string", "null"]}
    },
    "required": ["name", "price", "currency"],
    "additionalProperties": false
  }
}

Model harus mendukung endpoint /chat/completions. Jika model tidak mendukung json_schema, Arachne otomatis mencoba ulang tanpa response_format, lalu memvalidasi JSON yang dihasilkan.

Crawl budgets yang disarankan untuk Hermes

Penggunaan interaktif ringan:

max_pages: 10
max_requests: 25
max_errors: 8
max_depth: 2
max_chars_per_page: 5000
max_total_chars: 40000
concurrency: 3

Dokumentasi menengah:

max_pages: 50
max_requests: 120
max_errors: 20
max_depth: 4
max_chars_per_page: 8000
max_total_chars: 200000
concurrency: 6

Jangan menaikkan semua batas sekaligus. Output MCP yang terlalu besar dapat mengurangi kemampuan reasoning model walaupun crawler berhasil.

Konfigurasi penting

Environment variable

Default

Keterangan

ARACHNE_RESPECT_ROBOTS

true

Mematuhi robots.txt

ARACHNE_ALLOW_IGNORE_ROBOTS

false

Mengizinkan caller memilih respect_robots=false

ARACHNE_BLOCK_PRIVATE_NETWORKS

true

Proteksi SSRF

ARACHNE_MAX_REQUESTS

200

Batas maksimum request per crawl

ARACHNE_MAX_TOTAL_CHARS

250000

Batas output agregat

ARACHNE_BROWSER_CONCURRENCY

2

Jumlah context Chromium bersamaan

ARACHNE_CACHE_TTL_SECONDS

3600

TTL cache

ARACHNE_PROXY_URL

kosong

Proxy HTTP/HTTPS opsional

ARACHNE_ALLOW_BROWSER_EVAL

false

Mengizinkan arbitrary browser JS

Lihat seluruh opsi di .env.example.

Perbandingan sasaran dengan Firecrawl

Arachne sengaja dioptimalkan untuk self-hosted agent stack:

Area

Arachne

Firecrawl hosted

Source code dan modifikasi

Sepenuhnya lokal dan modular

Open source core + hosted infrastructure

Biaya per halaman

Infrastruktur sendiri

Credit-based

HTTP → browser escalation

Dapat diatur dan diperiksa

Otomatis

Parser dokumen

Lokal

Tersedia melalui API

Persistent crawl cache

SQLite lokal

Dikelola layanan

Change diff

Built-in unified diff

Monitoring tersedia pada layanan

Prompt-injection wrapping

Built-in untuk respons MCP

Bergantung workflow agent

Crawl output budget

Hard character budget untuk konteks LLM

Page/concurrency controls

Search

SearXNG/Brave adapter

Hosted search index

Anti-bot/proxy reliability

Bergantung proxy/infrastruktur Anda

Infrastruktur proprietary lebih matang

Arachne dapat unggul dalam privasi, kontrol, extensibility, biaya marginal, dan integrasi Hermes. Firecrawl hosted kemungkinan tetap unggul untuk coverage internet luas, managed proxy rotation, anti-bot, SLA, dan skala besar tanpa operasi sendiri.

Benchmark terhadap Firecrawl

Masukkan URL ke file, satu URL per baris:

cat > urls.txt <<'EOF'
https://example.com
https://docs.python.org/3/
EOF

Jalankan benchmark Arachne:

PYTHONPATH=src python scripts/benchmark.py --urls urls.txt --output benchmark.json

Bandingkan dengan Firecrawl bila memiliki API key:

export FIRECRAWL_API_KEY=fc-...
PYTHONPATH=src python scripts/benchmark.py \
  --urls urls.txt \
  --compare-firecrawl \
  --output benchmark.json

Metric yang dicatat:

  • keberhasilan per URL

  • latency

  • panjang konten

  • engine HTTP/browser

  • status code

  • error

Tambahkan dataset website Anda sendiri: docs statis, SPA, blog, PDF, tabel, e-commerce, dan halaman yang sering gagal. Klaim lebih unggul hanya masuk akal setelah hasil pada dataset Anda menunjukkan demikian.

Testing

pytest -q

Pengujian yang disertakan mencakup:

  • canonical URL normalization

  • private-network blocking

  • HTML metadata dan link extraction

  • prompt-injection detection/wrapping

  • content chunking

  • hard request budget

  • SQLite cache dan snapshots

  • local HTTP crawling dan recursive sitemap

Keterbatasan yang disengaja

Arachne tidak secara otomatis:

  • melewati CAPTCHA

  • membobol login atau paywall

  • mengakali access control

  • memakai akun tanpa izin

  • menjamin halaman yang memblokir data center IP dapat diambil

  • mengabaikan robots.txt kecuali administrator mengaktifkannya

  • menjamin DNS-rebinding protection sempurna pada semua network stack

Untuk reliability setara layanan hosted pada web yang sangat protektif, Anda tetap memerlukan proxy pool berkualitas, observability, distributed queue, autoscaling browser workers, dan maintenance selector/anti-bot secara berkelanjutan.

Struktur proyek

src/arachne_mcp/
├── cache.py        # SQLite cache dan snapshots
├── config.py       # Environment settings
├── crawler.py      # Orchestration, crawl, map, search, diff
├── errors.py
├── extractors.py   # HTML dan document extraction
├── fetchers.py     # HTTPX, robots, Playwright, actions
├── models.py
├── rate_limit.py
├── security.py     # URL/IP guard
├── server.py       # MCP tools
└── url_utils.py

Dokumen tambahan:

  • docs/ARCHITECTURE.md

  • docs/SECURITY.md

  • docs/HERMES.md

Lisensi

MIT.

Available Tools

13 tools
batch_scrapeA

Scrape up to 500 URLs concurrently with a shared output budget.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlsYes
concurrencyNo
render_modeNoauto
include_linksNo
output_formatNomarkdown
max_total_charsNo
max_chars_per_urlNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description must carry the burden. It mentions concurrency and output budget but lacks details on error handling, rate limits, partial results, or other behaviors.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One sentence, front-loaded with essential info, no fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 7 parameters and complexity of batch scraping, the description is too brief. It does not cover return format, error behavior, or parameter details, leaving gaps for agent understanding.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description should compensate by explaining key parameters. It only hints at max_total_chars via 'shared output budget', ignoring concurrency, render_mode, include_links, etc.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the tool scrapes up to 500 URLs concurrently with a shared output budget, distinguishing it from single-URL scraping or extraction tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Description implies batch usage but does not explicitly state when to use this tool over alternatives like crawl_url or extract_links, nor does it specify when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

browser_interactA

Operate a browser using safe declarative actions: click, fill, press, select, wait, scroll, and goto.

JavaScript evaluate is disabled unless the server owner explicitly enables it. This tool does not bypass CAPTCHA, authentication, paywalls, or access controls.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
actionsYes
headersNo
max_charsNo
screenshotNo
session_idNo
include_linksNo
output_formatNomarkdown
respect_robotsNo
persist_sessionNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses key behavioral traits: JavaScript eval is disabled by default, and the tool does not bypass CAPTCHA, authentication, paywalls, or access controls. However, it does not mention session handling, potential state changes, or output format implications.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise (3 sentences) and front-loaded with the main purpose. The second and third sentences add important caveats. Could be slightly more structured, but it is efficient without unnecessary words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (10 parameters, nested actions) and existing output schema, the description covers core purpose and a couple of limitations but omits behavior details like session persistence, output format handling, and action sequence execution rules.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description must compensate, but it only briefly mentions actions and does not explain most parameters (e.g., session_id, persist_session, output_format). The agent gets minimal help understanding parameter semantics beyond their names.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Operate a browser using safe declarative actions' and lists specific actions (click, fill, press, etc.). It differentiates from siblings like crawl_url and extract_links by focusing on interactive browser operations rather than extraction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for interactive browser tasks (clicking, filling forms) but lacks explicit guidance on when to use this tool versus sibling extraction tools such as crawl_url or extract_structured. No 'use this for... not for...' wording is present.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

check_robotsC

Inspect robots.txt permissions, crawl delay, request rate, and sitemap declarations.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description should disclose behavioral traits like read-only nature or rate limits. It only lists what is inspected but does not describe how the tool behaves (e.g., whether it fetches the file, caches results, or requires authentication).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence that efficiently conveys the tool's function without extraneous words. It is front-loaded with the key verb 'Inspect' and lists the items inspected.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has an output schema (not shown), but the description does not hint at the output structure. Given the complexity of inspecting multiple items, the description should provide some context about the return format or what constitutes successful retrieval.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description does not explain the 'url' parameter beyond its name. It fails to specify expected format (e.g., full URL with protocol), or that it should point to the root domain's robots.txt.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: inspecting robots.txt permissions, crawl delay, request rate, and sitemap declarations. It distinguishes it from sibling tools like 'crawl_url' or 'extract_links' by focusing on robots.txt analysis.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives. For example, it does not specify that it should be used before crawling to check permissions, nor does it mention any prerequisites or limitations.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

chunk_contentB

Split long content into stable, overlapping, hash-addressed chunks for RAG or agents.

ParametersJSON Schema
NameRequiredDescriptionDefault
contentYes
overlapNo
chunk_sizeNo
max_chunksNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the lack of annotations, the description discloses behavioral traits: 'stable' (implying deterministic, idempotent), 'overlapping' (chunks have shared context), and 'hash-addressed' (content-based addressing). However, it does not explain stability guarantees or the hashing scheme in detail.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence with no wasted words. It is concise, but it lacks structural elements like bullet points or an example that could improve scannability.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description fails to explain the output (despite existence of an output schema) and does not cover parameter semantics. For a tool with four parameters and no annotation support, the description is incomplete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0% description coverage for parameters, and the tool description does not mention any parameter meaning, defaults, or usage. All four parameters (content, overlap, chunk_size, max_chunks) are left unexplained, leaving the agent with insufficient information.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'split', the resource 'long content', and the key characteristics 'stable, overlapping, hash-addressed'. It also specifies the intended use case 'for RAG or agents', distinguishing it from sibling tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for splitting content for retrieval contexts, but it does not provide explicit guidance on when to use this tool versus alternatives or when not to use it. The intended context (RAG/agents) is helpful but not sufficient for clear decision-making.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_diffB

Compare a page with its previous persistent snapshot and return a unified diff.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
max_charsNo
render_modeNoauto
snapshot_nameNodefault
update_snapshotNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Without annotations, the description must disclose behavioral traits but fails to mention side effects like snapshot creation or update behavior, or what happens if no snapshot exists.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, front-loaded sentence with no extraneous words; highly efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 5 parameters and no annotations, the description omits critical context about snapshot management, diff interpretation, and the update_snapshot behavior, leaving the agent underinformed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, but the description adds no parameter explanations beyond what parameter names imply. Does not compensate for the lack of schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool compares a page with its previous persistent snapshot and returns a unified diff, which is specific and distinguishes it from sibling tools like crawl_url or crawl_site.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives; lacks context about prerequisites or typical scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_siteB

Crawl a site with strict request/error/token budgets and per-host rate limiting.

ParametersJSON Schema
NameRequiredDescriptionDefault
compactNo
delay_msNo
max_depthNo
max_pagesNo
start_urlYes
max_errorsNo
concurrencyNo
render_modeNoauto
max_requestsNo
output_formatNomarkdown
exclude_patternNo
include_patternNo
max_total_charsNo
include_documentsNo
include_subdomainsNo
max_chars_per_pageNo
deduplicate_contentNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses key behavioral traits: strict request/error/token budgets and per-host rate limiting. However, with no annotations, the description carries the full burden, and it fails to mention important behaviors like conforming to robots.txt, handling authentication, or how budgets are enforced. Additional context is missing.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise (one sentence, 10 words), which is efficient but underserves a tool with 17 parameters. It front-loads the core action and constraints but lacks the depth that multiple sentences could provide for key details.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (17 parameters, 1 required, no schema descriptions, no annotations) and the presence of an output schema (not shown), the description is far from complete. It does not explain return values, effects of parameters, or error scenarios, leaving the agent to guess many operational details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. However, it only mentions budgets and rate limiting, which are not explicit parameter names. None of the 17 parameters (e.g., max_pages, delay_ms, include_pattern) are described or linked to the narrative, leaving the agent to infer from parameter names alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's action ('Crawl a site') and adds context about budgets and rate limiting. It implies differentiation from sibling tools like 'crawl_url' which likely handles single URLs, but does not explicitly call out alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides implied usage for controlled crawling with budgets, but no explicit guidance on when to use this tool versus alternatives (e.g., 'crawl_url' for single pages, 'map_site' for structure). No exclusions or conditions are mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_urlB

Fetch and extract one URL using fast HTTP, browser rendering, or automatic escalation.

Supports HTML, text, JSON, XML, PDF, DOCX, XLSX, CSV and TSV. Content returned to an agent is marked as untrusted and scanned for common prompt-injection text.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
headersNo
wait_msNo
max_charsNo
use_cacheNo
screenshotNo
render_modeNoauto
include_linksNo
output_formatNomarkdown
include_imagesNo
respect_robotsNo
remove_selectorsNo
cache_ttl_secondsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses that content is marked untrusted and scanned for prompt-injection, and lists supported formats. However, no annotations provided, and description fails to mention important behaviors like redirect handling, timeouts, error responses, or mutation. With zero annotation coverage, the description should be more thorough.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Description is very concise (3 short lines), front-loads purpose. However, it omits parameter details, which would be necessary for completeness. Appropriate length but at expense of utility.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 13 parameters and an output schema, the description is too brief. It doesn't explain caching, output format options, or parameter behaviors. Even with output schema existing, agent needs more context to set parameters correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so description must compensate but does not explain individual parameters. Only 'automatic escalation' relates to render_mode; other 12 parameters are not described. The schema properties' names provide some hint, but description adds little meaning.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states action (fetch and extract), resource (one URL), and methods (HTTP, browser, auto). Distinguishes from siblings like crawl_site and extract_links by specifying single URL extraction and supported formats.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool vs siblings. Although 'one URL' implies single page extraction, no mention of when to prefer this over extract_links, parse_url, or browser_interact.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_structuredC

Extract JSON matching a caller-supplied schema through an optional OpenAI-compatible LLM.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
schemaYes
instructionNoExtract the requested fields accurately.
render_modeNoauto
max_source_charsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so the description carries full burden. It mentions 'optional OpenAI-compatible LLM' but does not disclose potential side effects (e.g., API calls, cost, rate limits) or whether the tool is read-only.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The single sentence is too brief; it fails to include important details about parameters, behavior, or output. It is under-specified rather than concise, given the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having an output schema and 5 parameters, the description is inadequate. It does not mention return format, rendering choices, or LLM behavior, making it incomplete for the tool's complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%. The description does not explain any of the 5 parameters (url, schema, instruction, render_mode, max_source_chars), leaving agents without guidance on required or optional fields.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool extracts JSON matching a caller-supplied schema, optionally using an LLM. This distinguishes it from siblings like extract_links (extracts links) or crawl_url (crawls).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool over siblings (e.g., when to use render_mode choices or when LLM is needed). The description only mentions optional LLM but no when-to-use or when-not-to-use context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

healthA

Return enabled engines, integrations, security policy, and parser availability.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden for behavioral disclosure. It accurately describes a read-only operation returning system status. Although it doesn't explicitly state non-destructive behavior, the context strongly implies it, and no contradictions exist.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence that directly states the tool's output. It is front-loaded, concise, and contains no filler. Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no parameters, the presence of an output schema, and clear differentiation from siblings, the description is fully complete. It provides all necessary context for an agent to correctly invoke and interpret the tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, and schema coverage is 100%. The description does not need to add parameter details beyond the schema, as none exist. A baseline of 4 is appropriate since the description contributes no extra semantic value but is not required.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states the tool returns 'enabled engines, integrations, security policy, and parser availability', providing a clear and specific verb-resource combination. It distinguishes itself from sibling tools like crawl_url or search_web, which focus on different actions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies use for checking system health but offers no explicit guidance on when to use this tool versus alternatives. Since siblings are functionally distinct, the context is clear, but there is no direct exclusion criteria or usage scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

map_siteC

Discover site URLs from recursive sitemaps plus optional shallow crawling.

ParametersJSON Schema
NameRequiredDescriptionDefault
max_urlsNo
start_urlYes
crawl_pagesNo
exclude_patternNo
include_patternNo
discover_by_crawlNo
include_subdomainsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are present, so the description must fully disclose behavioral traits. It only states the method (recursive sitemaps + shallow crawling) without mentioning effects like read-only behavior, rate limits, or robots.txt compliance. This is insufficient for safe invocation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence with no fluff. It is front-loaded with the core purpose. However, it could be expanded slightly to improve completeness without losing conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 7 parameters, no annotations, and an existing output schema, the description is severely incomplete. It omits return value information, parameter behaviors, and handling of edge cases (e.g., exclude/include patterns). The agent lacks context for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% for 7 parameters. The description does not explain any parameter (e.g., start_url, max_urls, crawl_pages). It fails to compensate for the lack of schema documentation, leaving the agent without parameter insight.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'Discover site URLs from recursive sitemaps plus optional shallow crawling' clearly states the verb, resource, and method. It distinguishes from siblings like crawl_site by mentioning sitemap discovery, but could be more explicit about the contrast.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives (e.g., crawl_site, extract_links). There are no prerequisites, exclusions, or context for selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

parse_urlC

Parse a remote document or non-HTML resource into LLM-ready text/Markdown.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
max_charsNo
use_cacheNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.7/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description must disclose behaviors. It does not mention caching behavior, rate limits, supported document types, or what happens with HTML pages despite saying 'non-HTML resource'. The description is vague about the parsing mechanism.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence, which is concise, but it lacks structure and fails to provide important details. Every sentence should earn its place; here the single sentence is insufficient for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has three parameters, no annotation coverage, and an output schema not detailed. The description is too brief to be complete, omitting caching, character limits, and parsing scope. Given the complexity, more detail is needed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description adds no explanation for parameters. 'max_chars' and 'use_cache' are not described beyond their names, leaving the agent to guess their meaning and defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb 'Parse' and specifies the resource types ('remote document or non-HTML resource') and output format ('LLM-ready text/Markdown'). It clearly distinguishes from sibling tools like 'search_web' or 'extract_links' which have different purposes.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives such as 'extract_structured' or 'browser_interact'. There are no explicit conditions, prerequisites, or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_webC

Search through a configured self-hosted SearXNG or Brave Search provider.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo
queryYes
scrape_resultsNo
max_chars_per_resultNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.6/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden but only mentions relying on a configured provider. It fails to disclose behavioral traits like rate limits, authentication needs, fallback behavior, or whether results are cached.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, well-front-loaded sentence. While concise, it could incorporate more useful information without becoming verbose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (4 parameters, no schema descriptions, no annotations, and an output schema not described), the description is insufficient. It does not explain the effect of scrape_results or max_chars_per_result, nor the structure of search results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% and the description adds no parameter details. The tool has 4 parameters (limit, query, scrape_results, max_chars_per_result) with no explanations, leaving the AI unable to understand their purpose or valid ranges.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool searches through a configured self-hosted SearXNG or Brave Search provider, distinguishing it from sibling tools like crawl_url or extract_links which perform different operations. However, it could be more specific about the type of search (e.g., general web search) and result format.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives such as crawl_site or map_site. The description lacks any context about prerequisite configurations or typical scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 13 tool updatesv1.0.0
    • First observedbatch_scrape
    • First observedbrowser_interact
    • First observedcheck_robots
    • First observedchunk_content
    • First observedcrawl_diff
    • First observedcrawl_site
    • First observedcrawl_url
    • First observedextract_links
    • First observedextract_structured
    • First observedhealth
    • First observedmap_site
    • First observedparse_url
    • First observedsearch_web

TDQS

A3.5/5.0

Scored across 13 tools

Disambiguation5/5

Each tool serves a distinct purpose in the web scraping pipeline, from health checks to link extraction, crawling, parsing, site mapping, robots inspection, browser interaction, batch scraping, search, structured extraction, diffing, and chunking. No two tools have overlapping functionality.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern in snake_case (e.g., extract_links, crawl_url, parse_url). The only exception is 'health', which is a noun but serves as a status probe and fits the convention of a single-word command.

Tool Count5/5

13 tools is appropriate for a comprehensive web scraping and content extraction server. Each tool covers a necessary operation without redundancy, balancing breadth and focus.

Completeness5/5

The tool surface covers the full lifecycle of web data acquisition: discovery (map_site, search_web), fetching (crawl_url, crawl_site, batch_scrape), parsing (parse_url, extract_links), interaction (browser_interact), compliance (check_robots), processing (chunk_content, extract_structured), and monitoring (health, crawl_diff). No obvious gaps.

Maintenance

ActivitySlowing
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    B
    maintenance
    Open-source web scraper and extraction MCP server with JavaScript rendering, markdown output, PDF/DOCX parsing, structured errors, and validated extraction contract diagnostics for agents.
    2
    AGPL 3.0
  • A
    license
    A
    quality
    A
    maintenance
    Servo-powered MCP server for JS-aware web fetching, content extraction, crawling, and software-rendered screenshots — Chromium-free single binary.
    6
    146
    Apache 2.0
  • -
    license
    Not graded
    quality
    C
    maintenance
    Self-hosted MCP server that provides web scraping and crawling tools, integrating seamlessly with AI frameworks like OpenAI Agents SDK, Cursor, and Claude Code.
    4
    -
  • A
    license
    B
    quality
    B
    maintenance
    A local MCP server providing web search, page extraction, and safe browser automation tools for Hermes, Claude Code, and other MCP clients.
    10
    MIT