"Web crawling techniques and tools" matching MCP servers:

Pay-per-call web page extraction, on-page SEO audits and text analysis for AI agents. Paid in USDC on Base via x402; no API keys, failed calls are not charged.

Five general-purpose stdio MCP servers for SEO: access to SERP, Wordstat, Google Search Console, Yandex.Webmaster and Yandex.Metrica straight from Claude Code (or any MCP client). All tools are read-only, output is strict JSON. Not tied to a specific site: defaults (GSC property, Webmaster host, Metrica counter) are configured on the fly.

An MCP server that lets agents run free, keyless lead-leak audits on UK local service businesses — detecting form platforms and their outreach implications, checking whether phone numbers are tappable tel: links, comparing phone numbers across a site and free directories, screening website hygiene, and locating a business's own site. It also bundles these into a single full audit that returns a prioritised list of fixable issues alongside top local competitors.

Enables Claude to query Bing Webmaster Tools performance, crawl, sitemap, backlink, and keyword data in plain language through the official Bing Webmaster API. It is read-only and runs locally using your own API key.

Provides 13 offline SEO and marketing tools to any MCP client, covering ad metric calculators, text and heading analysis, keyword density, and generators for meta tags, Open Graph, hreflang, schema markup, XML sitemaps, and robots.txt. Everything runs locally with no account, API key, or network calls.

A streamlined MCP server for XMP metadata embedding with beautiful formatting and smart filename indicators, enabling metadata embedding, reading, validation, and report generation for lifestyle, product, and orbit schemas.

Provides honest A/B test verdicts using interactive cards, with statistical calculations for conversion rates, sample sizes, and Bayesian probability-to-beat.

Provides tools to extract the first or all sitemap URLs from a website's robots.txt file.

Enables keyword research, organic rankings, competitor and backlink gap analysis, historical/mobile research, and AI visibility lookups by driving Semrush's own read endpoints through an authenticated Chromium session, with no developer API key or CAPTCHA service required. Every operation is also exposed as a native CLI command.

Enables coding agents to search and retrieve 500+ production-ready React and Tailwind UI sections, page recipes, and brand-styled components over MCP, along with imagery and social post templates.

An MCP server for converting App Store Optimization (ASO) data from platforms like App Store Connect into Web SEO data. It enables the creation of synced websites, keyword research, and automated blog content generation based on app store metadata.

Get access to real-time SEO data, including: keyword insights, backlink data, traffic estimates and more. Allow AI tools and Large Language Models (LLMs) to tap into the real-time SEO Review Tools API with natural language commands.

Enables AI agents to check a list of websites by running an Apify Actor through one tool, returning per-site tech stack, on-page SEO issues and score, robots.txt rules for AI crawlers, sitemap, llms.txt presence, and an AI-readiness score. Supports local stdio operation with an Apify token for tool calls.

A TypeScript implementation of a Model Context Protocol server that provides marketing tools based on Open Strategy Partners' methodologies, enabling content creation, optimization, and product positioning through tools like value map generation, meta information creation, and content editing.

Enables domain investors, webmasters, and autonomous agents to perform comprehensive domain audits including brand scoring, TLD tier analysis, and DNS connectivity checks, and to generate strategic brand suffixes and industry compound variations for portfolio expansion.

A utility MCP server providing 10 specialized tools for newsletter content preparation and optimization, including subject line generation, HTML-to-text extraction, read time estimation, and email validation. Enables newsletter operators, developers, and content teams to automate pre-send workflows and audit newsletter issues through natural language interactions.

MCP server for MarkItUp's AI image-annotation pipeline. Generate polished marketing-visual variations of any screenshot, regenerate, AI outpaint, and remove backgrounds — powered by Claude analysis + Gemini rendering.

Enables AI assistants to manage Meta Ads (Facebook, Instagram) end-to-end through natural conversation, including launching campaigns, uploading creatives, updating budgets, and analyzing performance.

AI visibility and SEO data for AI agents. Check whether ChatGPT, Claude, Perplexity and Gemini mention your brand and which sources they cite, plus keyword volume and Google Trends.

An SEO MCP server providing free SEO tools including on-page audits, broken link checks, sitemap analysis, and keyword ideas via Google Suggest, all without requiring an API key.