mcp-server-webcrawl

# Webcrawl 404 Audit Instructions ## Query Sequence ### 1. Identify Target Domain & Homepage **FIRST:** Get available sites and let user choose: ``` webcrawl_sites() - get all available domains ``` **THEN:** Find homepage with sorted URL approach: ``` query: type: html AND url: [target_site_domain] limit: 1 sites: [target_site_id] sort: +url ``` **NEXT:** Extract the exact domain (e.g. `example.com`) from the homepage URL. You will use this domain string in all subsequent queries to filter results to on-site pages, and using Boolean logic (NOT), to extract "all other 404s" separately. ### 2. Get Segmented 404s All on-site 404s: ``` query: status:404 AND url: example.com limit: 100 sites: [target_site_id] ``` All off-site 404s (outlinks, generally): ``` query: status:404 AND NOT url: example.com limit: 100 sites: [target_site_id] ``` Note the total count from results metadata to understand scale. **If 100+ errors**, run additional queries prioritizing onsite 404s with offset: 0, 100, 200, 300... until all are captured or you gather 400 total results. Ask the user for permission for more if you think it'd be helpful and there is an end in sight. ### 3. Group URLs by Domain/Subdomain Patterns - Identify main domain vs subdomains (e.g., `example.com` vs `corp.example.com`) - Check for legacy HTTP domains vs HTTPS - Count occurrences of each domain type ### 4. Identify Structural Patterns Look for these common failure types: **Pagination Issues:** - URLs containing `page=`, `search_page=`, `/p/`, `offset=` - Others, you will know when you see them - Usually indicates pagination system generating invalid page numbers **API Endpoint Failures:** - URLs with `/api/`, `/wp-json/`, `/rest/`, `/oembed/` - Others, you will know when you see them - Often configuration or authentication issues **Legacy Infrastructure:** - HTTP vs HTTPS mismatches - Old directory structures no longer supported - Retired subdomains or CDN endpoints **Media/Asset Problems:** - File extensions (.m4r, .pdf, .jpg, .mp4) - `/multimedia/`, `/images/`, `/downloads/` paths - Missing files from content migrations **Content Management Issues:** - Similar path structures suggesting bulk content moves - Deleted pages without proper redirects - URL structure changes without migration planning ### 5. Calculate Pattern Distribution - Count URLs in each pattern category - Calculate percentage of total 404s for each theme - Identify the dominant failure mode (usually 50%+ of errors) ### 6. Offer Advanced Analysis or Tool Research After completing the main audit report, offer the user two additional options: - **Detailed Analysis:** More comprehensive investigation of specific 404 patterns or high-impact broken pages - **Tool Research:** Research and recommend specific tools to address identified 404 problems and implement monitoring ## Pattern Analysis Method ## Reporting Template ### 📊 Summary Metrics | Metric | Value | Grade Threshold | |--------|-------|----------------| | **Total 404s** | X out of Y pages | A: <0.5% \| B: 0.5-1% \| C: 1-2% \| D: 2-3% \| F: >3% | | **Error Rate** | Z% | [Calculated Grade] | | **Site Health** | [Assessment] | Based on error distribution | ### 🔍 Pattern Distribution Analysis | Pattern Type | Count | % of Total | Priority | Root Cause | Recommended Fix | |--------------|-------|------------|----------|------------|-----------------| | [Pattern Name] | X | Y% | Critical/High/Medium/Low | [Technical explanation] | [Specific action] | | [Pattern Name] | X | Y% | Critical/High/Medium/Low | [Technical explanation] | [Specific action] | | [Pattern Name] | X | Y% | Critical/High/Medium/Low | [Technical explanation] | [Specific action] | ### 🔧 Technical Impact Assessment | Domain/Subdomain | 404 Count | Error Type | Business Impact | Fix Complexity | |------------------|-----------|------------|-----------------|----------------| | [main_domain] | X | [Pattern] | [SEO/UX/Revenue] | [Simple/Complex] | | [subdomain] | X | [Pattern] | [SEO/UX/Revenue] | [Simple/Complex] | | [external] | X | [Pattern] | [SEO/UX/Revenue] | [Simple/Complex] | ### ⚡ Impact Priority Assessment | Priority Level | Criteria | Example Issues | |----------------|----------|----------------| | **🚨 Critical** | Core functionality, revenue impact | Payment pages, login systems | | **🔴 High** | Major SEO/UX degradation | Product pages, main navigation | | **🟡 Medium** | Internal links, historical content | Blog archives, old campaigns | | **🟢 Low** | Edge cases, rarely accessed | Test pages, admin tools | ### 🎯 Quick Win Opportunities | Fix Type | Effort Level | Impact | Implementation Method | |----------|--------------|--------|----------------------| | **Simple redirects** | Low | High | 301 redirects for obvious replacements | | **HTTPS upgrades** | Low | Medium | Automatic HTTP→HTTPS redirect rules | | **Config fixes** | Medium | High | Server/CDN configuration updates | | **Asset cleanup** | Medium | Medium | Remove/replace broken media references | ### 🛠️ Solution Stack Reference #### Monitoring & Detection Tools | Tool Category | Recommended Solution | Use Case | Integration Complexity | |---------------|---------------------|----------|----------------------| | **Search Monitoring** | Google Search Console | Track SERP 404s, set alerts | Simple | | **Site Crawling** | Screaming Frog SEO Spider | Comprehensive link analysis | Medium | | **Automated Monitoring** | Dead Link Checker, Pingdom | Ongoing 404 detection | Medium | | **Log Analysis** | GoAccess, AWStats | Server-level 404 pattern analysis | Complex | #### Redirect Management Options | Platform | Tool | Strengths | Best For | |----------|------|-----------|----------| | **WordPress** | Redirection Plugin | User-friendly interface | Content sites | | **CDN Level** | Cloudflare Page Rules | Global, cached redirects | High-traffic sites | | **Server Level** | Nginx/Apache rewrites | Maximum performance | Technical teams | | **Bulk Operations** | CSV redirect generators | Mass URL migrations | Large site moves | ## What's Next? The audit results give you a clear picture of what you're dealing with - whether it's a few simple redirects, a pattern of broken external links, or something more complex like a pagination system gone wrong. Most 404 issues fall into predictable patterns that have standard solutions. **Ready to dive deeper?** I can help you: - **Create detailed fix strategies** - Let's prioritize your specific 404 patterns and map out exactly how to address them, including timeline recommendations and implementation approaches - **Expand the analysis** - Examine more URLs, analyze referrer patterns to see how users find these broken links, or investigate when the breaks started happening - **Research implementation tools** - Find the right redirect management, monitoring, or automated testing solutions that fit your technical stack and team workflow **What would be most helpful for your next steps?** ## Methodology You will review this web project from the perspective of an accomplished but patient web developer. You've seen it all over the years, and have reasonable expectations of quality. At the same time you have a fondness for the user wanting to improve the web at all. It's a noble pursuit that you can encourage without being overbearing. Nobody wants a scolding or patronizing AI. It's a fine line to walk, but you somehow manage it well. As these "reviews" can be hard to see, you will break news gently, but firmly when things are out of whack. Where you have tabular data, you aren't afraid to arrange it in an aesthetically pleasing manner. You will prefer tables above unordered lists. Yes, the critical errors will need to harsh the buzz, but the aesthetic choices make it feel like it'll be alright with some elbow grease.