diogenes-web-scraping-engine-mcp
Server Details
Production web scraping and extraction engine with markdown extraction, DOM pruning, and anti-bot mitigation.
- Status
- Healthy
- Uptime
- 100.0% over the last 24 hours
- Last Tested
- Transport
- Streamable HTTP · MCP 2024-11-05
- URL
- ChatGPT
- –
- Claude
- –
TDQS
Scored across 12 tools
Each tool targets a distinct extraction/audit concern (articles, links, metadata, tables, contacts, CSS selectors, SEO, paywall, sitemap, RSS, sanitization) and descriptions explicitly point to the alternative tool when overlap exists. The main soft spots are scrape_readable_article vs extract_clean_markdown (both can fetch and output readable content) and the family of extract_* tools, but the cross-references keep misselection unlikely.
All 12 tools follow a clean verb_noun snake_case pattern (compile_rss_feed, crawl_sitemap_urls, extract_clean_markdown, parse_html_tables, sanitize_html_content). No camelCase mixing, vague verbs like 'process', or version suffixes.
12 tools sits squarely in the well-scoped 3-15 range, and each tool maps to a distinct scraping/extraction operation rather than padding. The suite feels like a deliberate pipeline (fetch -> extract -> convert -> audit) rather than a grab bag.
Coverage is broad: fetching, article extraction, links, metadata, tables, contacts, CSS selectors, markdown/RSS conversion, sanitization, SEO and paywall auditing, and sitemap parsing. Minor gaps remain around JS-rendered page handling, robots.txt/crawl-politeness checks, and no orchestrated multi-page crawl or job/cache management, but agents can chain the existing tools to work around these.
Available Tools
12 toolscompile_rss_feedWeb Content to RSS 2.0 & Atom Feed CompilerARead-onlyIdempotentInspect
Transforms JSON item collections or article card arrays into standardized RSS 2.0, Atom 1.0, or JSON Feed 1.1 syndication feeds. Use when syndicating blog articles, product announcements, or changelog cards to RSS readers and AI agents; for compiling structured sitemap listings, use crawl_sitemap_urls instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'items_json' and 'channel_title' ('feed_standard' defaults to 'rss_2_0'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.
Inputs: items_json (required), channel_title (required), feed_standard. Returns: Deterministic execution response schema for compile_rss_feed.
crawl_sitemap_urlsXML Sitemap & Index Crawler Tree DecompilerARead-onlyIdempotentInspect
Parses XML sitemaps and sitemap index files into structured URL trees with change frequency and last-modified filters. Use when discovering domain architectures, mapping crawl frontiers, or indexing blog/product catalogues; for extracting on-page hyperlinks from raw HTML, use extract_page_hyperlinks instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'sitemap_xml' ('filter_criteria' defaults to 'all_entries'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.
Inputs: sitemap_xml (required), max_results, filter_criteria. Returns: Deterministic execution response schema for crawl_sitemap_urls.
detect_paywall_and_bypassDOM Paywall, Modal Gate & Blurred Content Recovery AuditorARead-onlyIdempotentInspect
Audits web page DOM hierarchies to detect soft paywalls, blur overlays, modal gates, and recovers concealed body text. Use when diagnosing paywall obstructions, diagnosing subscriber gating scripts, or auditing content access barriers; for assessing general SEO hygiene, use evaluate_seo_diagnostics instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('detection_profile' defaults to 'heuristic_dom_overlay'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.
Inputs: html_content (required), target_domain, detection_profile. Returns: Deterministic execution response schema for detect_paywall_and_bypass.
evaluate_seo_diagnosticsWeb SEO Diagnostics & Heading Hierarchy AuditorARead-onlyIdempotentInspect
Audits HTML heading structures, title lengths, meta descriptions, image alt tags, and OpenGraph compliance into a health score. Use when auditing landing pages, verifying SEO compliance before publishing, or evaluating crawl friendliness; for extracting article bodies, use scrape_readable_article instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('audit_level' defaults to 'comprehensive_audit'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: html_content (required), audit_level, target_keyword. Returns: Deterministic execution response schema for evaluate_seo_diagnostics.
extract_clean_markdownWeb HTML to Clean LLM-Optimized Markdown EngineARead-onlyIdempotentInspect
Converts raw HTML pages or live web URLs into clean, boilerplate-free, LLM-optimized Markdown with preserved hyperlinks and headings. Accepts either 'url' to fetch directly or raw 'html_content' from upstream scrape_readable_article; for isolating tabular data into structured datasets, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'url' or 'html_content' ('conversion_mode' defaults to 'clean_semantic_article'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: url, html_content, preserve_links, conversion_mode. Returns: Deterministic execution response schema for extract_clean_markdown.
extract_contact_infoLead Contact, Email, Phone & Social Handle ExtractorARead-onlyIdempotentInspect
Extracts email addresses, telephone numbers, physical addresses, and social network handles from web text and DOM footers. Use when researching sales leads, aggregating business contact directories, or extracting customer service info; for parsing structured tabular data, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'content' ('extraction_focus' defaults to 'all_contacts'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: content (required), country_hint, extraction_focus. Returns: Deterministic execution response schema for extract_contact_info.
extract_page_hyperlinksWeb Hyperlink, Anchor & Asset Extraction FilterARead-onlyIdempotentInspect
Extracts, normalizes, and filters internal and external hyperlinks, anchor texts, and downloadable asset targets from HTML. Use when building site crawlers, auditing broken links, or extracting reference citations; for isolating primary readable article narrative text, use scrape_readable_article instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('link_scope' defaults to 'all_links'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: html_content (required), base_url, link_scope. Returns: Deterministic execution response schema for extract_page_hyperlinks.
extract_structured_metadataOpenGraph, Twitter Cards & JSON-LD Microdata ExtractorARead-onlyIdempotentInspect
Extracts OpenGraph tags, Twitter Card metadata, canonical URLs, and Schema.org JSON-LD microdata scripts from live URLs or raw HTML. Accepts either 'url' to fetch directly or raw 'html_content' from upstream scrape_readable_article; for extracting contact emails and phone numbers, use extract_contact_info instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'url' or 'html_content' ('metadata_targets' defaults to 'all_protocols'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: url, html_content, include_favicons, metadata_targets. Returns: Deterministic execution response schema for extract_structured_metadata.
isolate_css_selectorsDeterministic Edge DOM CSS Selector Scraper & IsolatorARead-onlyIdempotentInspect
Evaluates CSS selector queries against live URLs or raw HTML DOM trees to extract targeted inner text, outer HTML, or element attributes. Accepts either 'url' to fetch directly or raw 'html_content' from upstream scrape_readable_article; for parsing complete table matrices into JSON, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'url' or 'html_content' plus 'selector_query' ('extraction_mode' defaults to 'inner_text'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: selector_query (required), url, html_content, extraction_mode. Returns: Deterministic execution response schema for isolate_css_selectors.
parse_html_tablesHTML Table to Structured JSON & Markdown ParserARead-onlyIdempotentInspect
Detects and parses HTML table elements from live URLs or raw HTML into structured JSON record arrays and GitHub-Flavored Markdown tabular grids. Accepts either 'url' to fetch directly or raw 'html_content' from upstream scrape_readable_article; for extracting full article body text, use scrape_readable_article instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'url' or 'html_content' ('output_format' defaults to 'markdown_and_json'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: url, min_rows, html_content, output_format. Returns: Deterministic execution response schema for parse_html_tables.
sanitize_html_contentEdge HTML Sanitizer, Script Stripper & XSS NeutralizerARead-onlyIdempotentInspect
Sanitizes raw HTML or live URLs by removing malicious script tags, iframes, inline event handlers, and data URIs into safe markup. Accepts either 'url' to fetch directly or raw 'html_content' from upstream scrape_readable_article; for converting clean HTML into markdown, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'url' or 'html_content' ('sanitization_strictness' defaults to 'strip_scripts_and_iframes'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: url, html_content, preserve_images, sanitization_strictness. Returns: Deterministic execution response schema for sanitize_html_content.
scrape_readable_articleWeb Page Fetcher, Readability Extractor & HTML ParserARead-onlyIdempotentInspect
Fetches any public HTTP/HTTPS URL and extracts article text, headings, author bylines, and raw html_content. Use as the universal URL fetcher before downstream HTML parsing tools (parse_html_tables, isolate_css_selectors, extract_clean_markdown); for converting full raw HTML without fetching, pass html_content directly. Stateless edge call (max 32 KB, 60 RPM): accepts 'url' or 'html_content' ('readability_profile' defaults to 'article_body_strict'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: url, html_content, include_byline, readability_profile. Returns: Deterministic execution response schema for scrape_readable_article.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceRemote MCP server that lets AI assistants read tracked competitive intelligence — competitor news, reviews, website changes, social posts and ads — and act on it by managing projects, competitors, battle cards, reports and canvases. Every answer carries a source and a date, and write actions only publish when explicitly shared.MIT
- AlicenseCqualityCmaintenanceCompetitor Monitor AI - MCP server providing AI-powered tools and automation by MEOK AI Labs112 npm46 PyPIMIT
- AlicenseAqualityDmaintenanceEnables brand visibility monitoring across major AI platforms like ChatGPT, Claude, Gemini, and Perplexity. It allows users to track visibility scores, analyze competitor data, and receive actionable insights to improve AI-generated brand recommendations.1611 npm1MIT

industrylens-mcpofficial
AlicenseNot gradedqualityCmaintenanceBrowse IndustryLens's published competitive-intelligence reports and head-to-head competitor comparisons from any AI agent — real, source-backed data.MIT
Glama MCP Gateway
Add one secure layer between your agents and this server.