EdgeFetch
Provides an Open Data adapter for querying ArcGIS-hosted datasets through the extractor system.
Provides an adapter for querying arXiv scholarly metadata and papers.
Supports the Brave Search API as a bring-your-own-key backend for the web search tool.
Supports Cloudflare Workers AI as a bring-your-own-LLM backend for the autonomous /v1/agent endpoint.
Provides DOI/Crossref adapters for retrieving scholarly metadata and publication records.
Provides a site adapter for retrieving GitHub repository and related public data.
Provides ATS adapters for querying Greenhouse job boards in US and EU regions.
Provides an adapter for querying Hugging Face model and dataset information.
Provides an adapter for querying npm package metadata.
Supports local Ollama models as a bring-your-own-LLM backend for the autonomous /v1/agent endpoint.
Supports OpenAI as a bring-your-own-LLM backend for the autonomous /v1/agent endpoint, including live SSE streaming and zero data retention.
Exports observability telemetry using OTLP/HTTP and W3C trace context.
Provides an adapter for querying PubMed biomedical literature and metadata.
Supports SearXNG as a self-hosted search backend for the web search tool, enabling zero-cost search.
Provides commerce adapters for polite full-store Shopify catalog scraping, including product and variant data.
Provides an adapter for querying Stack Overflow content and metadata.
Provides commerce adapters for WooCommerce inventory catalogs and variant data.
Provides an adapter for querying Y Combinator company and startup data.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
EdgeFetch
EdgeFetch is a self-hosted web scraping and search server for AI agents — open-source worker, SDKs, and adapters (MIT) with a fair-source hub (FSL-1.1, MIT after two years). You run one ~8 MB Go binary on any box you own (a Raspberry Pi, your Home Assistant, a spare laptop), and your agents get clean Markdown and structured data through MCP — with zero open ports, zero Redis or Postgres, and no per-page bill.
Built for the agent web: user-triggered, never trains, signs its requests with Web Bot Auth, asks for Markdown first, honours
robots.txtand Content Signals, and treats402as a price, not an error.
Try It in 60 Seconds
# Option 1 — Full 15-Tool Hub + BYO-LLM /v1/agent + 50+ Site/ATS/Commerce Adapters + /admin UI + FTS5 in Docker (zero cloud account):
git clone https://github.com/bartrumb/edgefetch.git && cd edgefetch
export CALLER_TOKEN="$(openssl rand -hex 24)" AGENT_TOKEN="$(openssl rand -hex 24)"
docker compose up --build -d
# Open http://127.0.0.1:8787/admin and sign in with $CALLER_TOKEN
# Option 2 — Standalone ~8 MB Local Go Binary for Claude Code, Claude Desktop, Cursor, Windsurf, Gemini CLI & Codex (zero containers/ports):
go install github.com/bartrumb/edgefetch/v2/cmd/edgefetch@latest
edgefetch init && edgefetch doctorRelated MCP server: TheCrawler
What Stands Out
Outbound-only workers with zero open ports (inbound LAN invisibility): Workers on a Raspberry Pi, NAS, or Home Assistant box behind CGNAT/Starlink dial out over TLS WebSockets (
WSS /agent/ws) to your Hub (AGENT_TOKENper site; OAuth 2.1 PKCE + CSRF for MCP//admincallers). Your home router and local LAN remain completely unreachable from the internet (note: outbound fetches originate from your worker node's WAN IP, or from Cloudflare's edge IP when usingtarget: "cloudflare").Dial-time SSRF protection (including the browser):
politeness.RejectNonPublicPeerchecks resolved peer IPs inside Go'snet.Dialer.Controlright beforeconnect()(blocking DNS-rebinding TOCTOU), and headless Chromium subresources are forced through an in-process loopbackCONNECTproxy (--host-resolver-rules=MAP * ~NOTFOUND).Polite by design — with actionable
blocked.suggestionrerouting:robots.txtandContent-Signal: ai-inputare enforced on every hop (including30xredirects),Retry-Afteris honored per host, HTTP402(x402) halts immediately aspayment_requiredwithout auto-paying, anti-bot challenges (Cloudflare, Akamai, DataDome, PerimeterX, AWS WAF, Imperva) halt on the first attempt, and IETF RFC 9421 Web Bot Auth (Ed25519HTTP Message Signatures) is built in. When a path is blocked byrobots.txtor a WAF, EdgeFetch returns a structuredblocked.suggestionsteering your agent to official public JSON boards, registries, or APIs (e.g., Greenhouse/Ashby/Workday CXS instead of LinkedIn, or Overpass/Nominatim instead of OSM/api/).Clean Two-Plane Architecture (Pure Go Edge Engine & Hub + Serverless V8 Hub):
Go Data Plane & Standalone Hub (
cmd/,internal/— 1 external Go modulecoder/websocket,0transitive deps,~8 MBstatic binary): Runs 100% standalone with zero TypeScript/Node (edgefetch scrape,edgefetch mcpoverstdio, oredgefetch --mode=hubwith embedded declarative site adapters!) or as an outbound worker daemon, handling every untrusted socket, dial-time SSRF check, 100% RFC 9309robots.txtcheck, streaming PDF parse, and CDP Chromium session.Cloudflare / Portable Fair-Source Hub Plane (
hub/— vetted runtime packages:re2js,htmlparser2,ws,yaml): Optional full-featured fleet coordinator running natively on Cloudflare Hibernating WebSocket Durable Objects + D1 FTS5 + R2 ($0idle) or self-hostednode:sqlite(dist/portable-server.mjs,0runtimenode_modulesin containers). Combines Google RE2 linear-time Thompson NFA regex safety (re2js), WHATWG HTML5 CSS-selector extraction (parseHtmlDocviahtmlparser2), RFC 6455 WebSocket framing (ws), shared Go+TS declarative YAML manifests (internal/extract/manifests/*.yaml), and 56+ MIT-licensed hot-deployable site/ATS/commerce adapters (hub/src/extractors/).
50+ keyless site, commerce, open-data & ATS adapters + Store
catalog+ Autonomous BYO-LLM/v1/agent: 15 MCP tools (agent,find_extractors,scrape,extract,map,crawl,search,jobs,catalog,search_collection,interact,read_artifact,fetch_url,fetch_batch,list_agents), drop-in Firecrawl/v2/*compatibility, linear64 KiBframework detection, polite full-store Shopify (custom domains +/products.json) & WooCommerce inventory catalogs with variant diffs, and Claude-style dynamic adapter routing across Workable, Greenhouse (US/EU), Ashby, Lever (US/EU), Workday CXS, iCIMS, Rippling, Gem, Pinpoint, Teamtailor, BambooHR, Oracle HCM, SmartRecruiters, SEC EDGAR, Y Combinator, Socrata / CKAN / ArcGIS / Legistar Open Data, DOI/Crossref, OpenAlex, arXiv, ClinicalTrials.gov, PubMed, NIST NVD, GitHub, Stack Overflow, npm, crates.io, Hugging Face, OpenRouter, and 256 MiB streaming PDFs — supporting offline local LLMs (Ollama, LM Studio, vLLM) or online BYO-LLM (OpenAI, Anthropic, Gemini, OpenRouter, Workers AI) with live SSE streaming and$0Zero Data Retention. Add a new adapter in one file vianpm run adapter:new(docs/ADAPTERS.md).
Project status: Young personal project (started November 2026), running daily as the sole web/search layer on the author's 3-node fleet (Home Assistant OS
x86_64, Windows VM, andarm64). Open-source worker, SDKs, and adapters (MIT); fair-source hub (FSL-1.1-MIT, MIT after two years). Issues and PRs are triaged weekly. See docs/LAUNCH-FAQ.md for honest trade-offs.
Head-to-Head vs. Other AI Crawlers & MCP Servers
Full architectural & trade-off breakdown: docs/COMPARISON.md · Common questions: docs/LAUNCH-FAQ.md
Dimension | EdgeFetch ( | Firecrawl (Self-Hosted) | Firecrawl / Tavily (Cloud) | Crawl4AI | Jina Reader |
License | Open-source worker, SDKs & adapters ( | AGPL-3.0 (core) | Proprietary SaaS | Apache-2.0 (+ attribution) | Apache-2.0 |
Minimum Self-Hosted Stack | 1 | Docker Compose + Redis + PostgreSQL + RabbitMQ + Playwright ( | Cloud API key | Python 3.10+ + Playwright ( | Node.js + Puppeteer |
Worker Supply Chain | 1 Go module ( | Dozens of npm + OS deps | Vendor SaaS | 40+ PyPI packages | Dozens of npm deps |
Zero-Port Outbound Home/Edge Fleet ( | Yes — Home Assistant OS, NAS, RPi, Mac/PC behind CGNAT/Starlink | No (needs open ports, VPN, or paid residential proxies) | Paid proxy credits | No (single-host or DIY queue) | No |
Desktop | Yes — | Requires cloud key or full Docker stack | Cloud API key required | Python + Playwright setup | Cloud URL or Node server |
DNS-Rebinding SSRF + Browser Subresource Guard | Dial-time | Application URL blocklist | Vendor-managed | Optional / app-level | App-level |
Web Search ( | Bring your own Serper/Brave/Tavily keys, or SearXNG + | Proprietary / SearXNG | Separate SaaS plans | Bring-your-own |
|
OpenTelemetry ( | Stitched Hub → Go | Internal logs / BullMQ | Vendor dashboard only | Python logs | Internal logs |
Typical Cost (50k scrapes + 5k searches / mo)¹ |
|
|
|
| Credit-metered |
¹ Cost assumptions checked Nov 20, 2026: compares Cloudflare Workers Free ($0) or Paid ($5/mo) within R2 (10 GB) and D1 (5 GB) monthly free tiers plus self-hosted SearXNG ($0, using existing home/VPS compute) against published Firecrawl Standard ($19–$99/mo) + Tavily Researcher ($30+/mo) plans.
Architecture & High-Availability Mesh
flowchart TD
Admin["Browser (/admin Control Center)"] -->|"HMAC Session + CSRF"| HubW["Primary Hub (Cloudflare DO / Fly / Railway / Render)"]
Claude["Claude Code / Cursor (/mcp)"] -->|"OAuth 2.1 PKCE or Bearer"| HubW
Claude -.->|"Automatic HA Failover (baseUrls)"| GoHub["Standby Go Hub (--mode=hub,agent on VPS / Tailscale)"]
HubW -->|"WSS /agent/ws (W3C traceparent)"| S1["Docker / VPS Fat Node (amd64)"]
HubW -->|"WSS /agent/ws (W3C traceparent)"| S2["Mac Mini / RPi5 Node (arm64)"]
HubW -->|"WSS /agent/ws (W3C traceparent)"| S3["Home Assistant OS Add-on"]
GoHub -->|"Active-Active EDGEFETCH_HUB_URLS"| S1
GoHub -->|"Active-Active EDGEFETCH_HUB_URLS"| S2
HubW -.->|"Async OTLP/HTTP JSON"| OTel["Any OTel Collector (Tempo / Jaeger / Honeycomb / Axiom)"]
S1 -.->|"Child httptrace Spans"| OTelDeployment Modes in Detail
Option A: Standalone Worker Container (Linux, macOS, Windows, NAS, RPi, or VPS)
Multi-arch OCI manifests (linux/amd64 + linux/arm64) run as non-root 65532:65532:
# Full "Fat" Agent (Go HTTP/2 + SSRF-proxied Headless Chromium + SwiftShader + Poppler PDF):
docker run -d --name edgefetch-agent --restart=unless-stopped \
--security-opt no-new-privileges:true \
-e EDGEFETCH_HUB_URL=https://<your-hub-domain> \
-e EDGEFETCH_AGENT_TOKEN="<YOUR_AGENT_TOKEN>" \
-e EDGEFETCH_SITE_ID="node-1" \
ghcr.io/bartrumb/edgefetch-agent:latest
# Or Ultra-Light ~9 MB Distroless Agent (HTTP/2 + Markdown + pure-Go PDF, ~15 MB RAM):
docker run -d --name edgefetch-lite --restart=unless-stopped --read-only \
-e EDGEFETCH_HUB_URL=https://<your-hub-domain> \
-e EDGEFETCH_AGENT_TOKEN="<YOUR_AGENT_TOKEN>" \
-e EDGEFETCH_SITE_ID="lite-1" \
ghcr.io/bartrumb/edgefetch-agent:lite --mode=agentOption B: Desktop CLI, stdio MCP, Self-Hosted Go Hub & Bot Auth Keygen
Download a release archive from GitHub Releases or install with go install:
go install github.com/bartrumb/edgefetch/v2/cmd/edgefetch@latest
# Auto-configure zero-daemon, zero-port stdio MCP into Claude Desktop & Cursor:
edgefetch install-mcp
# Verify SSRF guard, loopback CONNECT proxy, OS Chrome/Edge & pdftotext, or scrape from terminal:
edgefetch doctor
edgefetch scrape https://example.com
edgefetch scrape --json --query="installation" https://developers.cloudflare.com/workers/
# Optional: mint an IETF RFC 9421 Ed25519 Web Bot Auth signing keypair:
edgefetch keygen-bot-auth
# Or run a 100% self-hosted Go WebSocket Hub + /v1/scrape + /mcp controller on a small VPS:
EDGEFETCH_CALLER_TOKEN="<32-char-secret>" EDGEFETCH_AGENT_TOKEN="<32-char-secret>" \
edgefetch --mode=hub --addr=0.0.0.0:8080Option C: Cloud Hub (Cloudflare Workers, Render, Railway, Fly.io) & Home Assistant OS
Cloudflare Workers (
$0idle): Click Deploy to Cloudflare above (orcd hub && npm ci && npx wrangler deploy).Render / Railway / Fly.io (Single All-in-One Container + Persistent
/dataVolume): Userender.yaml,railway.json, orfly launch --copy-config(fly.toml).Home Assistant OS (
amd64&aarch64): Click Add Repository to Home Assistant above, install EdgeFetch, sethub_url,agent_token, andsite_id, and click Start.
Feature & Extraction Highlights (v2.6, SCRAPE_PIPELINE_VERSION = 26)
25+ Site & ATS Job-Board Extractors (
hub/src/extractors/registry.ts,hub/src/jobs.ts):ATS Job Boards & Structured
jobsTool: One-call job search & extraction across Greenhouse, Ashby (?includeCompensation=true), Lever, Workday CXS (GET&POST), and SmartRecruiters / Schema.orgJobPostingJSON-LD, normalizing pay ({ min, max, currency, period, basis: "base"|"ote"|"total" }) and filtering out closed postings (contentState: "closed_posting").Developer & Research Sites: 1-hop JSON/raw/patch upgrades for GitHub (raw,
.patch, Issues/PRs, Gist), GitLab, Codeberg, Stack Overflow (✅ Accepted Answer), npm, crates.io, Hugging Face (YAML model cards → GFM tables), Discourse, Lobsters, Reddit, Bluesky (ATProto), Mastodon, DEV.to, pkg.go.dev, PyPI, Wikipedia, IETF RFCs, ArXiv, PubMed/PMC, Hacker News, and YouTube transcripts.
Data-Gathering & SPA Reliability:
Native CDP SPA Engine (
internal/browser/renderer.go): Tracks real CDPNetwork.requestWillBeSent/loadingFinishedXHR/Fetchnetworkidle, triggersIntersectionObserverlazy-load scroll passes, waits on aMutationObserverDOM quiescence check, and flattens open Web ComponentshadowRoottrees.SSR Hydration Recovery (Next.js App Router
self.__next_f.pushRSC,__NEXT_DATA__, Nuxt 3__NUXT_DATA__, Remix, Apollo): Recovers full prose from thin SPA shells in 1 HTTP hop before launching Chromium.Optional Ethical Heavy-SPA Fallback (
Crawl4AI&Firecrawl— Cloud or Self-Hosted): ConfigureCRAWL4AI_API_URLand/orFIRECRAWL_API_URL+FIRECRAWL_API_KEYso heavy SPAs (or Lite-only fleets without local Chromium) fall back automatically after verifyingrobots.txt— while never delegatingrobots.txtblocks,401/403/429, or anti-bot challenges.Conditional
304 Not ModifiedRevalidation: Sends storedETag/Last-Modifiedvalidators fromMETAD1 and serves verified R2 Markdown immediately on304.7-Vendor WAF Catalog + 4 Anti-False-Positive Layers: Accurately classifies Cloudflare, Akamai, DataDome, PerimeterX, AWS WAF, Imperva, and Vercel/Fastly/Kasada challenges while stripping
<pre><code>/<blockquote>and vetoing200 OKarticles (>= 250words, JSON-LD,<nav>) so security/Turnstile docs never false-positive.Opt-in
waybackFallback: Recovers5xx/ DNS-dead link rot from the Internet Archive (never invoked onrobots.txt,403/429, or challenges).Token Efficiency:
-91.6%LLM token reduction on our offline 8-fixture synthetic regression corpus (bench/RESULTS.md), preserving<T>generics (Promise<void>), GFM tables, KaTeX math,.ipynboutputs, and OpenAPI 3.x schemas.
Tested & Supported Platforms Matrix
Status legend:
✅ Daily-Driver + CI Verified: Running daily on the author's 3-node personal fleet (
arm64, Home Assistant OSx86_64, and Windows VM) since Nov 18, 2026 and gated by automated E2E / smoke suites on every PR.✅ Live-Tested + CI Verified: Executed end-to-end on real host OS/runtime during verification and covered by automated CI tests.
🔧 Build & Manifest Verified: Static
CGO_ENABLED=0cross-compilation (go build/ GoReleaser) and deployment manifests verified cleanly; community hardware reports welcome!
Platform / Target | Role(s) | Binary / Image Footprint | Capabilities | Verification Status |
Cloudflare Workers + Hibernating DO + D1 (FTS5) + R2 + Workflows | Cloud Hub ( | Serverless bundle ( | 15-tool MCP, Search, FTS5, R2 Artifacts, RFC 9421, OTLP | ✅ Daily-Driver + CI Verified ( |
Home Assistant OS ( | Outbound Home Worker ( | Multi-arch Alpine + Chromium + SwiftShader + Poppler | HTTP/2, | ✅ Daily-Driver + CI Verified ( |
Windows 11 ( | Local |
| Auto-detects native Windows Chrome / Edge / Brave for CDP JS render | ✅ Daily-Driver + CI Verified (Windows VM node + CLI E2E) |
Linux Docker Worker — Fat ( | Standalone Worker ( | Non-root | Full HTTP/2 + SSRF-proxied Headless Chromium + Poppler PDF | ✅ Live-Tested + CI Verified (live AppArmor + WebGL CI smoke) |
Linux Distroless Worker/Server — Lite ( | Ultra-light Worker / Server / Go Hub ( |
| HTTP/2, Markdown, pure-Go PDF, | ✅ Live-Tested + CI Verified ( |
All-in-One Container ( | Full TypeScript Hub + | Node 26 Alpine + Go + Chromium + Poppler ( |
| ✅ Live-Tested + CI Verified ( |
Linux ( |
|
| Active-active | ✅ Live-Tested + CI Verified (Go 1.27 |
WebAssembly / WASI ( | Sandboxed Offline HTML → Markdown / JSON Extractor |
| Runs inside | ✅ Live-Tested + CI Verified ( |
Chrome / Firefox Manifest V3 Extension | Active-Tab FTS5 Clipper ( | Clips authenticated tabs ethically into private FTS5 collections | ✅ Live-Tested + CI Verified ( | |
Deno Deploy + Turso HTTP + S3 SigV4 | Serverless Hub Adapter | Zero-dep WebCrypto SigV4 (S3/R2/Tigris/MinIO/B2) + Turso | ✅ Live-Tested + CI Verified ( | |
macOS Apple Silicon & Intel ( | Local |
| Auto-detects | 🔧 Build & Manifest Verified ( |
Windows on ARM ( | Desktop CLI, |
| HTTP/2, Markdown, pure-Go PDF, OS Edge/Chrome detection | 🔧 Build & Manifest Verified ( |
Fly.io / Railway / Render / Coolify | 1-Click Cloud Self-Hosted All-in-One Hub + Worker | Persistent | 🔧 Build & Manifest Verified (runs | |
Kubernetes ( | Zero-Trust Fleet Worker Pod | Read-only rootfs, | 🔧 Build & Manifest Verified | |
Unraid, CasaOS, Portainer, Synology, Proxmox LXC & Termux | Homelab / NAS / LXC / Spare-Phone Worker | Outbound | 🔧 Build & Manifest Verified | |
OpenWrt ( | Router / Firewall Appliance Lite Worker |
| HTTP/2, Markdown, pure-Go PDF ( | 🔧 Build & Manifest Verified ( |
Reproducible Go Hot-Path Microbenchmarks
Run go test -bench=. -benchmem ./... locally:
Go Benchmark ( | Latency | Heap Allocations |
|
|
|
|
|
|
|
|
|
|
|
|
Documentation & Contributing
Launch & Architecture FAQ: docs/LAUNCH-FAQ.md
Head-to-Head Comparison & Trade-Offs: docs/COMPARISON.md
Synthetic Extraction Benchmark (
-91.5%tokens on 8 fixtures): bench/RESULTS.mdSite Adapter Authoring Guide (YAML Manifests,
parseHtmlDoc&adapter:test): docs/ADAPTERS.mdOperations, OTLP Tracing, HA Mesh & MCP Guide: docs/USAGE.md
Hub ↔ Go Agent Wire Protocol Spec: docs/PROTOCOL.md
Systems & Go Concurrency Internals: docs/INTERNALS.md
Security Threat Model, Signed Releases & SSRF Controls: SECURITY.md
Bot, WAF & MCP Registry Certification Guide: docs/REGISTRATION.md
Contributing Rules & Code of Conduct: CONTRIBUTING.md · CODE_OF_CONDUCT.md
License & Trademarks
EdgeFetch uses an open-source worker, SDKs, and adapters / fair-source hub license split:
Open-Source Worker, CLI, SDKs & All Site Adapters (
cmd/,internal/,sdk/,extensions/,internal/extract/manifests/,hub/src/extractors/): MIT (andhub/src/extractors/LICENSE).Fair-Source Hub (
hub/, excludinghub/src/extractors/): FSL-1.1-MIT (Functional Source License 1.1, MIT after two years) — free to read, modify, and self-host personally or internally within your organization; third parties may not offer the Hub as a competing commercial hosted/managed service, and each release converts toMITtwo years after publication.Contributions: Inbound =
MIT+ DCO sign-off (git commit -s), per CONTRIBUTING.md.
Trademarks: EdgeFetch™ and the EdgeFetch raven logo are trademarks of Brett Bartrum. Neither the MIT License nor the Functional Source License grants permission to use the EdgeFetch name, edgefetch.dev domain, or project logos to brand, endorse, or market a third-party product or hosted service.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Web scraping, extraction and search for AI agents. Hosted MCP with workspace API keys.
Web scraping, browser automation, screenshots and AI data extraction for AI agents via MCP.
Scrape, crawl and search the web for AI agents via MCP.
Web data and proxies for agents: scrape to Markdown, SERP, crawl/map, unlocker, collectors, proxies.
Related MCP Servers
- AlicenseAqualityAmaintenanceWeb scraping, crawling, and structured data extraction for AI agents. 5 tools: scrape (clean markdown from any URL), crawl (entire sites), map (discover URLs), extract (structured JSON), and search. 833ms avg latency, single binary, self-hostable.91,286AGPL 3.0
- AlicenseNot gradedqualityDmaintenanceOpen-source web scraper and extraction MCP server with JavaScript rendering, markdown output, PDF/DOCX parsing, structured errors, and validated extraction contract diagnostics for agents.2AGPL 3.0
- FlicenseNot gradedqualityAmaintenanceEnables AI agents to search the web through configured providers and crawl individual pages or entire sites into clean Markdown or JSON, including authenticated browsing and deduplicated output. It can run locally over STDIO or HTTP and integrates with MCP clients and command-line workflows.-
- FlicenseNot gradedqualityAmaintenanceEnables AI agents and MCP clients to fetch webpages and crawl sites, extract main content, and convert it into clean Markdown with preserved links for fast, token-efficient scraping.-