Provides an AMP checker that verifies AMP implementation quality and discoverability on web pages.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Web Crawler MCP Server run a full SEO audit on example.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Comprehensive SEO auditing and web crawling through AI with MCP.
Part of the Insightful Pipe MCP Server Collection — The Web Crawler MCP server enables Claude, ChatGPT, Cursor, and other AI assistants to perform SEO audits, extract data, and analyze websites. Run 25+ specialized SEO checks, generate sitemaps, and extract brand information.
https://crawler.insightfulmcp.com/Related MCP server: seo-audit-mcp
Web Crawler MCP is a remote Model Context Protocol server that provides AI assistants with comprehensive SEO auditing and web crawling capabilities. This integration allows you to:
Run 25+ specialized SEO checkers on any URL
Generate XML sitemaps and robots.txt files
Extract brand design elements from websites
Audit technical SEO, content quality, and security
Analyze mobile responsiveness and page speed
Copy the MCP Server URL: https://crawler.insightfulmcp.com/
Scroll to the bottom and click Add custom connector
Paste the URL and click Add
Click Connect and sign in to InsightfulPipe if asked. Claude then lists the connector as connected.
Custom MCP servers are added through ChatGPT's Developer mode. Availability depends on your ChatGPT plan, and workspace admins may need to allow it.
Turn on Developer mode in ChatGPT settings
Create a new app for a remote MCP server and paste the URL: https://crawler.insightfulmcp.com/
Sign in to InsightfulPipe if asked. The connection finishes as soon as you are signed in.
See OpenAI's guide: Developer mode and MCP apps in ChatGPT
claude mcp add --transport http crawler https://crawler.insightfulmcp.com/Add the server to ~/.cursor/mcp.json (all projects) or .cursor/mcp.json (one project):
{
"mcpServers": {
"crawler": {
"url": "https://crawler.insightfulmcp.com/"
}
}
}When Cursor shows Needs authentication, click Connect and sign in to InsightfulPipe if asked.
33 actions: 33 read, 0 write.
Action | Description |
| Check permissions for AI training bots and crawlers |
| Verify AMP implementation quality and discoverability |
| Extract brand design details from a website (colors, fonts, buttons, logos) |
| Validate canonical URLs and conflicting hints |
| Validate declared charset and encoding consistency |
| Analyze on-page content quality and keyword density |
| Score readability, word count, and duplication risk |
| Detect obsolete tags or attributes still in use |
| Confirm favicon/icon discovery across devices |
| Review heading hierarchy (H1-H6) for structure gaps |
| Review international hreflang annotations |
| Check alt text, formats, and compression levels |
| Review robots/meta directives affecting indexation |
| Map internal links and surface opportunities |
| Verify html[lang] attributes and language targeting |
| Detect broken or low-quality links on page |
| Extract all links from a page and categorize them (internal and external) |
| Parse llms.txt instructions for AI crawlers |
| Inspect meta title/description tags across the page |
| Audit responsive layout rules and breakpoints |
| Ensure viewport meta tags and responsive meta configs are in place |
| Measure page weight and render-blocking assets |
| Measure load times and highlight slow resources |
| Parse robots.txt directives and highlight blocking issues |
| Generate robots.txt content with optional AI bot restrictions |
| Check CSP/HSTS and other security headers |
| Comprehensive SEO audit covering technical and content issues |
| Crawl the target domain and produce an XML sitemap |
| Validate Open Graph and Twitter Card metadata |
| Inspect certificate chain, expiry, and HTTPS enforcement |
| Trace HTTP status/redirect chains for a URL |
| Validate Schema.org markup and structured data coverage |
| Evaluate keyword usage and cleanliness of URLs |
You decide what AI agents can do with each connected account:
Turn individual actions on or off for every connected account, so agents only see the actions you allow.
Connect as Read-only or Read & Write. A read-only connection can only enable read actions.
Destructive actions stay off by default. Actions such as deletes are disabled until an admin enables them.
Team access per account. Restricted team members only use the accounts they are granted, with the read actions enabled on them.
"Run a comprehensive SEO audit on https://example.com""Check the robots.txt and indexability of https://example.com""Map all the internal links on this page""Generate an XML sitemap for https://example.com""Extract the brand colors, fonts, and logos from https://company.com""Check the SSL certificate and security headers for this site"Comprehensive audits - 25+ specialized checks
Technical SEO - Robots, canonical, structured data
Content analysis - Quality scoring and optimization
Debug issues - Find broken links, redirect chains
Performance - Page speed and resource analysis
Security - SSL and header validation
Client audits - Quick site assessments
Competitive analysis - Analyze competitor sites
Reporting - Generate findings conversationally
Secure requests - HTTPS support
The Web Crawler MCP server is included in every InsightfulPipe plan, together with all other MCP servers and the CLI. Plans start at $29.99/month, and you can try it for 7 days. See insightfulpipe.com/pricing for current plans.
Claude skills for SEO and marketing — ready-made skills that run on your connected data
Visit insightfulpipe.com/mcp-servers to discover our full collection of MCP servers for marketing and analytics.
PageSpeed MCP - Performance analysis
Google Search Console MCP - Search analytics
Screenshot MCP - Visual capture
Google My Business MCP - Local SEO
Google Analytics MCP - Website analytics
Enrichment MCP - Data enrichment
Documentation: insightfulpipe.com/docs
All MCP Servers: insightfulpipe.com/mcp-servers
Email: support@insightfulpipe.com
Insightful Pipe — AI-powered marketing analytics through MCP servers. Explore all integrations →
MIT License - see LICENSE for details.
This server cannot be deployed
Audit any site's AI visibility from your assistant: crawler access, rendering, and schema.
Run SEO + AI-visibility (GEO) audits from Claude, Cursor & other AI clients.
Run full technical SEO audits of any website from your AI agent and get structured findings back.
SEO audits, Search Console, GA4, keyword and backlink data for Claude, ChatGPT and Cursor.
