open-web-intel-mcp
Searches the Apple Podcasts Search API for shows and parses show metadata, owner email, and recent episodes.
Queries public Greenhouse career APIs to search job postings by company, keywords, location, remote-only status, and result limits.
Queries public Personio career APIs to search job postings by company, keywords, location, remote-only status, and result limits.
Inspects public Shopify storefronts via products and metadata endpoints, returning catalog products and storefront app/pixel signals.
Inspects public WooCommerce storefronts via the Store API products endpoint, returning catalog products and storefront signals.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
open-web-intel-mcp
Zero-Proxy, Zero-Browser Local TypeScript MCP Server & CLI for SSRF-Safe Web Intelligence.
open-web-intel-mcp gives AI coding agents, desktop assistants, and developer pipelines fast, direct access to structured public web data—without headless browsers, paid proxy pools, or API keys. Every outbound HTTP request passes through a shared defense-in-depth SSRF guard with DNS resolution checks, manual redirect hop re-validation, strict timeouts, and streaming byte caps.
Key Capabilities
Zero Headless Browser Overhead: Queries clean public JSON, XML, RSS, DNS, and HTML endpoints directly on Node.js
>=20.18.1(Node 20.18+ / 22).Defense-in-Depth SSRF Protection: Blocks private RFC1918 networks, loopback, CGNAT, cloud metadata (
169.254.169.254), link-local, unique-local IPv6, NAT64/6to4/Teredo, and IPv4-mapped IPv6 addresses across every DNS record andredirect: 'manual'hop.Token-Lean 7-Tool Schema: Concise, read-only tool definitions (~1,460 tokens across all 7 tools combined) designed for minimal LLM context overhead and zero prompt-injection surface.
Dual Entrypoints:
open-web-intel-mcp: Model Context Protocol server overstdio(default) or stateless Streamable HTTP (--http --port 3000).open-web-intel: Standalone terminal CLI with human-readable and--jsonoutput modes.
Related MCP server: crw-mcp
Architecture
flowchart LR
Client["MCP Host / Terminal CLI"] --> Entry["Dual Entrypoints (dist/mcp.js | dist/cli.js)"]
Entry --> Tools["7 Web Intelligence Tools"]
Tools --> Guard["SSRF Guard (src/lib/safe-fetch.ts)"]
Tools --> DNS["DNS Resolver (MX / SPF / DMARC)"]
Guard --> Check["URL & Multi-Family DNS IP Validator"]
Check --> Hop["Manual Redirect Loop (<= 5 Hops, Re-Validated)"]
Hop --> Stream["Streaming Byte Cap & Timeout Reader"]
Stream --> Web["Public JSON / XML / RSS / HTML Endpoints"]Installation & Quick Start
git clone https://github.com/pie527645-cloud/open-web-intel-mcp.git
cd open-web-intel-mcp
npm install
npm run build
# Optional: link `open-web-intel` and `open-web-intel-mcp` binaries globally
npm linkMCP Client Configuration
Point your MCP client directly at the built dist/mcp.js entrypoint:
Claude Desktop (claude_desktop_config.json)
{
"mcpServers": {
"open-web-intel": {
"command": "node",
"args": ["/absolute/path/to/open-web-intel-mcp/dist/mcp.js"]
}
}
}Cursor (.cursor/mcp.json)
{
"mcpServers": {
"open-web-intel": {
"command": "node",
"args": ["/absolute/path/to/open-web-intel-mcp/dist/mcp.js"]
}
}
}Antigravity (mcp_config.json)
{
"mcpServers": {
"open-web-intel": {
"command": "node",
"args": ["/absolute/path/to/open-web-intel-mcp/dist/mcp.js"]
}
}
}Note: On Windows, pass a forward-slash or escaped path (for example
"C:/path/to/open-web-intel-mcp/dist/mcp.js"). Once published to npm, you can also invoke a pinned release via"command": "npx", "args": ["open-web-intel-mcp@1.0.0"].
Stateless Streamable HTTP Mode
Start the server locally bound to 127.0.0.1 (default) with Host and Origin DNS-rebinding protection enabled:
node dist/mcp.js --http --port 3000
# Health check: GET http://127.0.0.1:3000/health
# MCP endpoint: POST http://127.0.0.1:3000/mcp
# Optional custom Host / Origin allowlists for reverse-proxy or container setups:
node dist/mcp.js --http --host 0.0.0.0 --port 3000 --allowed-hosts mcp.internal:3000 --allowed-origins https://app.internalThe 7 MCP Tools
Tool Name | Purpose | Key Inputs |
| Fetches a public URL, strips scripts/styles/navigation boilerplate, converts to clean Markdown with |
|
| Reads |
|
| Queries direct public JSON/XML career APIs for |
|
| Aggregates and deduplicates remote tech listings across |
|
| Queries public Shopify ( |
|
| Extracts public contact emails, social profile links, |
|
| Queries the Apple Podcasts Search API ( |
|
CLI Usage (open-web-intel)
Every tool is available directly from the terminal via open-web-intel (or node dist/cli.js). Pass --json on any subcommand for structured JSON output.
# Show CLI help and all flags
open-web-intel --help
# 1. Fetch clean Markdown + probe /llms.txt
open-web-intel fetch https://modelcontextprotocol.io/introduction --llms-txt --max-chars 20000
# 2. Discover robots.txt and crawl XML sitemaps
open-web-intel sitemap https://modelcontextprotocol.io --max-urls 100 --filter /docs --json
# 3. Search public ATS job boards (greenhouse | lever | ashby | workable | personio | recruitee)
open-web-intel ats greenhouse stripe --keywords "typescript" --remote-only --max-jobs 15 --json
open-web-intel ats lever spotify --region auto --max-jobs 10
# 4. Aggregate & deduplicate remote jobs across 5 public feeds
open-web-intel remote --keywords "backend typescript" --sources remotive,himalayas,weworkremotely --max-jobs 20 --json
# 5. Inspect a Shopify or WooCommerce storefront
open-web-intel store https://allbirds.com --max-products 10 --json
# 6. Enrich a domain with emails, socials, DNS MX/SPF/DMARC, llms.txt & tech stack
open-web-intel enrich cloudflare.com --json
# 7. Search podcasts & parse RSS feeds for host/owner email and recent episodes
open-web-intel podcasts "software engineering" --max-shows 3 --max-episodes 2 --country US --jsonSecurity Design (src/lib/safe-fetch.ts)
Scheme & Credential Enforcement: Permits only
http:andhttps:URLs without embeddedusername:passwordcredentials.Hostname & Full Multi-Family DNS Validation: Rejects
localhost,.localhost,.local,.internal,metadata.google.internal, and IPv6 zone IDs (%). Resolves allAandAAAArecords (dns.lookupwith{ all: true, verbatim: true }) and blocks the request if any returned IP falls in:Loopback (
127.0.0.0/8,::1), unspecified (0.0.0.0/8,::)RFC1918 private (
10.0.0.0/8,172.16.0.0/12,192.168.0.0/16)Carrier-Grade NAT (
100.64.0.0/10)Link-local & Cloud Metadata (
169.254.0.0/16,fe80::/10)Unique local IPv6 (
fc00::/7), site-local (fec0::/10), multicast (224.0.0.0/4,ff00::/8)IPv4-mapped/compatible/translated IPv6 (
::ffff:0:0/96,::/96), NAT64 (64:ff9b::/96), 6to4 (2002::/16), and Teredo (2001:0000::/32)
Socket-Level Connect Pinning (
undici.Agent) & Manual Redirect Re-Validation: Eliminates DNS-rebinding TOCTOU by validating all resolved IPs inside a customundici.Agentconnect.lookupcallback and pinning the TCP/TLS socket directly to the validated public IP. Usesredirect: 'manual'exclusively (<= 5 hops), re-validating everyLocationtarget before connecting and stripping sensitive headers across origins.Resource & Parser Bounds: Enforces per-hop
AbortSignal.timeoutdeadlines, cancels response streams immediately when cumulative chunk bytes exceedmaxBytes, caps HTML parser input at 512 KiB, and uses linearO(n)indexOftag scanning immune to unclosed-tag ReDoS.Streamable HTTP Origin & Host Protection: Validates
HostandOriginheaders on/mcpand/healthbefore instantiating transports (configurable via--allowed-hosts/ALLOWED_HOSTSand--allowed-origins/ALLOWED_ORIGINS).
Docker
docker build -t open-web-intel-mcp .
docker run --rm -p 127.0.0.1:3000:3000 open-web-intel-mcp --http --host 0.0.0.0 --port 3000Development & Testing
npm install
npm run lint # Strict TypeScript check across src/ and tests/
npm run build # Compile ESM output + declarations to dist/
npm test # Run offline SSRF, tool fixture, CLI, and live stdio/HTTP MCP testsNeed Managed Cloud Scale, Scheduling & Pay-Per-Event Apify Actors?
Running high-volume scheduled crawls, webhook pipelines, persistent datasets, or multi-company batch jobs without keeping your local machine online? Check out the companion cloud Actors and automation templates:
Use Case | Managed Cloud Actor / Resource | Highlights |
Multi-Company ATS Career Feeds | Batch-scrape Greenhouse, Lever, Ashby, Workable & SmartRecruiters across dozens of companies on a schedule. | |
Bulk Shopify Store Intelligence | Large-scale catalog, pricing, app, pixel, and storefront intelligence exported to JSON/CSV/Webhook. | |
B2B Domain & Tech Stack Enrichment | Enrich hundreds of domains with public contacts, DNS MX/SPF/DMARC posture, | |
Newsletter & Content Archives | Full newsletter archive extraction with clean Markdown conversion and outbound link graphs. | |
Workflow & n8n Templates Hub | Ready-to-import automation blueprints, scheduling guides, and integration recipes. |
License
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
Web data for agents: YouTube transcripts, screenshots, Google News, WHOIS, jobs, tech stack, more.
Web tools for AI agents: URL to clean Markdown, site crawl, screenshots, PDF/Office to text, OCR.
Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.
Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record.
Related MCP Servers
- AlicenseAqualityAmaintenanceWeb content extraction for AI agents. 10 tools: scrape, crawl, map, batch, extract, summarize, diff, brand, search, research. Uses TLS fingerprinting to bypass anti-bot without a headless browser. Outputs LLM-optimized markdown with 67% fewer tokens than raw HTML.102,443AGPL 3.0
- AlicenseAqualityAmaintenanceWeb scraping, crawling, and structured data extraction for AI agents. 5 tools: scrape (clean markdown from any URL), crawl (entire sites), map (discover URLs), extract (structured JSON), and search. 833ms avg latency, single binary, self-hostable.91,277AGPL 3.0
- AlicenseAqualityCmaintenanceEnables AI agents to fetch one or many web pages concurrently and receive clean, LLM-ready Markdown with navigation, ads and boilerplate stripped away, plus extracted metadata such as title, description, author, Open Graph tags and canonical URL. It runs locally over stdio with no API key or account, requiring only a single command.3MIT
- FlicenseNot gradedqualityAmaintenanceExposes 17 scraping tools over stdio or SSE so an LLM agent can fetch pages, extract structured data, run full crawls, and take screenshots entirely through conversation. Works with Claude Code, Claude Desktop, Cursor, Windsurf, ChatGPT, LangChain, and the OpenAI Agents SDK.-