"Web crawling techniques and tools" matching MCP connectors:

500+ pay-per-call tools + skill packs. USDC + USDG on 12 chains, or free via proof-of-work.

Give AI assistants secure access to your organization's structured business data. Search records, create and update records, retrieve schema information, and manage workflow states using natural language. You need two values for every request: x-api-key — your Web Data Forms API Key x-group-id — your Web Data Forms Group ID You can find these in your Web Data Forms accounts group->information page. Preferred method: request header When possible, pass the credentials as HTTP headers: x-api-key: <your-api-key> x-group-id: <your-group-id> This is the preferred option because it keeps credentials out of the URL and is more secure. Fallback method: query parameters If your MCP client does not support custom headers, the server also accepts the credentials as URL query parameters. Example: https://mcp.webdataforms.com?x-api-key=abc123&x-group-id=xyz456 Detailed information here: https://github.com/Web-Data-Forms/mcp-server-docs/blob/main/README.md

Scrape and extract application reviews from both the Google Play Store and the Apple App Store.

Build, edit, host, and publish websites from AI assistants. Setup: https://mcp.orivox.org/

Drive real Android & iOS devices and web browsers from natural language for mobile + web QA. 145+ tools across device control, app management, automation sessions, browser automation, and flow recording / replay. Bearer-auth — get a token at robotactions.com → Profile → API Tokens.

Scan public websites for privacy, cookie, tracker, consent, policy, and disclosure risk signals. Start instantly with zero-auth Light mode—no account or API key required.

Checks whether a website is readable and citable by AI systems (ChatGPT, Claude, Perplexity, etc.)

Pay-per-call web scraping for AI agents via x402 on Base USDC. Six tools, no signup.

Pay-per-call web scraping for AI agents — no signup, no API keys, just USDC micropayments via the x402 protocol on Base. Six tools: scrape webpages, extract structured data, capture screenshots, parse metadata, manage browser sessions, and run workflows. Runs on Cloudflare Workers with Browser Rendering. Free tier: 10 calls per wallet per 30 days.

Reliable web access for AI agents: smart HTTP, rotating proxies, and full-browser rendering.

Read any web page as clean Markdown for AI agents: fetch, search, metadata, links. SSRF-safe.

Turn HTML or Markdown into a clean, styled PDF and get a download link.

AgentReady.market audit: can an AI shopping agent find, understand and BUY on this store? /100.

Scrape, crawl and search the web for AI agents via MCP.

Give your agent live data from Twitter, Reddit, the web and GitHub. No API keys, no scraping stack.

Deterministic public-web change observation with evidence-bound commercial interpretation.

AI-powered website change monitoring - manage monitors and alerts via MCP.

Web tools for agents: fetch URL as markdown (free MCP) + x402 scrape, links, AI JSON, snapshot.

Domain/IP intelligence, web page capture and search APIs

Check if a link on a live page is followable: rel tokens, meta robots and X-Robots-Tag