Diogenes Web Extraction, Readability & Markdown Engine MCP
This MCP server exposes 12 stateless, read-only web extraction/scraping tools for turning web content into LLM-ready text, structured data, diagnostics, and feeds.
Convert raw HTML into clean LLM-optimized Markdown with headings and links preserved.
Extract readable article bodies, bylines, excerpts, and estimated reading time.
Parse OpenGraph, Twitter Card, canonical URL, and Schema.org JSON-LD metadata.
Convert HTML tables into JSON records, Markdown tables, or CSV strings.
Query HTML DOM with CSS selectors to extract text, HTML, or attributes.
Detect paywalls, overlays, and blurred/modal gates and recover concealed text.
Parse XML sitemaps and sitemap indexes into structured URL trees.
Extract, normalize, and filter page hyperlinks, anchors, and downloadable assets.
Sanitize untrusted HTML by removing scripts, iframes, event handlers, and XSS vectors.
Extract emails, phone numbers, physical addresses, and social profiles.
Audit SEO headings, titles, meta descriptions, alt tags, OpenGraph, and health score.
Compile JSON item collections into RSS 2.0, Atom 1.0, or JSON Feed 1.1.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Diogenes Web Extraction, Readability & Markdown Engine MCP Extract clean markdown from https://example.com/article"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Diogenes Web Extraction, Readability & Markdown Engine MCP (diogenes-web-scraping-engine-mcp)
Diogenes Web Extraction, Readability & Markdown Engine MCP is a focused, zero-bloat Model Context Protocol (streamable-http) package powered by the Diogenes Microservice Gateway.
Instead of loading all 121 gateway endpoints into your LLM's context window, installing this package exposes only the 12 specialized web-scraping-engine tools needed for Web Extraction & Scraping:
12 high-performance web extraction, readability & markdown tools: clean LLM markdown compiler, readable article extractor, OpenGraph & JSON-LD metadata parser, HTML table to JSON/Markdown converter, CSS selector isolator, paywall & overlay bypass auditor, XML sitemap crawler, hyperlink extractor, HTML XSS sanitizer, contact & lead info parser, SEO heading diagnostics, and RSS/Atom feed compiler.
⚡ 1-Click Install for Cursor & Claude Desktop
No API keys, user accounts, or local dependencies required. Zero setup — paste directly into your client config:
🖱️ Cursor IDE (.cursor/mcp.json)
Add to .cursor/mcp.json in your project root or global Cursor settings:
{
"mcpServers": {
"diogenes-web-scraping-engine": {
"transport": "streamable-http",
"url": "https://diogenesgateway.com/mcp/web-scraping-engine"
}
}
}🧠 Claude Desktop (claude_desktop_config.json)
Add to claude_desktop_config.json (Mac: ~/Library/Application Support/Claude/claude_desktop_config.json | Windows: %APPDATA%\Claude\claude_desktop_config.json):
{
"mcpServers": {
"diogenes-web-scraping-engine": {
"transport": "streamable-http",
"url": "https://diogenesgateway.com/mcp/web-scraping-engine"
}
}
}Prefer local stdio? Run via npx:
npx -y @stand-out-of-my-sun/diogenes-web-scraping-engine-mcp
Related MCP server: Website to Markdown MCP
Included Tools in web-scraping-engine (12 Tools)
MCP Tool Name | Underlying REST Route | Settlement | Capability |
|
| Free Developer Tier (0 sats) | Converts raw HTML pages into clean, boilerplate-free, LLM-optimized Markdown with preserved hyperlinks and headings. Use when ingesting raw web documents or blog posts into agent context windows; for isolating tabular data into structured datasets, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('conversion_mode' defaults to 'clean_semantic_article'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. |
|
| Free Developer Tier (0 sats) | Extracts primary article body, author bylines, publication timestamps, and estimated reading duration from web pages. Use when stripping navigation bars, advertisement containers, and cookie banners to isolate core editorial content; for converting full HTML to markdown, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('readability_profile' defaults to 'article_body_strict'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. |
|
| 10 sats (L402) | Extracts OpenGraph tags, Twitter Card metadata, canonical URLs, and Schema.org JSON-LD microdata scripts from HTML. Use when indexing social share cards, author identity, breadcrumbs, and rich snippet schemas for AI search agents; for scanning page hyperlinks and outbound references, use extract_page_hyperlinks instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('metadata_targets' defaults to 'all_protocols'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. |
|
| 10 sats (L402) | Detects and parses HTML table elements into structured JSON record arrays and GitHub-Flavored Markdown tabular grids. Use when converting financial statements, product comparison grids, or sports statistics into structured data tables; for extracting raw clean markdown prose, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('output_format' defaults to 'markdown_and_json'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. |
|
| 10 sats (L402) | Evaluates CSS selector queries against HTML DOM trees to extract targeted inner text, outer HTML, or element attributes. Use when targeting specific elements like pricing badges, product titles, review cards, or navigation breadcrumbs; for parsing unstructured contact information, use extract_contact_info instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' and 'selector_query' ('extraction_mode' defaults to 'inner_text'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. |
|
| 15 sats (L402) | Audits web page DOM hierarchies to detect soft paywalls, blur overlays, modal gates, and recovers concealed body text. Use when diagnosing paywall obstructions, diagnosing subscriber gating scripts, or auditing content access barriers; for assessing general SEO hygiene, use evaluate_seo_diagnostics instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('detection_profile' defaults to 'heuristic_dom_overlay'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid. |
|
| 15 sats (L402) | Parses XML sitemaps and sitemap index files into structured URL trees with change frequency and last-modified filters. Use when discovering domain architectures, mapping crawl frontiers, or indexing blog/product catalogues; for extracting on-page hyperlinks from raw HTML, use extract_page_hyperlinks instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'sitemap_xml' ('filter_criteria' defaults to 'all_entries'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid. |
|
| 10 sats (L402) | Extracts, normalizes, and filters internal and external hyperlinks, anchor texts, and downloadable asset targets from HTML. Use when building site crawlers, auditing broken links, or extracting reference citations; for isolating primary readable article narrative text, use scrape_readable_article instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('link_scope' defaults to 'all_links'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. |
|
| 10 sats (L402) | Sanitizes raw HTML by removing malicious script tags, iframes, inline event handlers, and data URIs into safe markup. Use when preparing untrusted third-party web content for rendering in agent dashboards or email clients; for converting clean HTML into markdown, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('sanitization_strictness' defaults to 'strip_scripts_and_iframes'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. |
|
| 10 sats (L402) | Extracts email addresses, telephone numbers, physical addresses, and social network handles from web text and DOM footers. Use when researching sales leads, aggregating business contact directories, or extracting customer service info; for parsing structured tabular data, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'content' ('extraction_focus' defaults to 'all_contacts'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. |
|
| 10 sats (L402) | Audits HTML heading structures, title lengths, meta descriptions, image alt tags, and OpenGraph compliance into a health score. Use when auditing landing pages, verifying SEO compliance before publishing, or evaluating crawl friendliness; for extracting article bodies, use scrape_readable_article instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('audit_level' defaults to 'comprehensive_audit'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. |
|
| 15 sats (L402) | Transforms JSON item collections or article card arrays into standardized RSS 2.0, Atom 1.0, or JSON Feed 1.1 syndication feeds. Use when syndicating blog articles, product announcements, or changelog cards to RSS readers and AI agents; for compiling structured sitemap listings, use crawl_sitemap_urls instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'items_json' and 'channel_title' ('feed_standard' defaults to 'rss_2_0'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid. |
Free Account-Less Demo Mode: Append
?demo=trueto any REST endpoint above for an instant HTTP 200 schema preview at zero cost.
🚀 Upgrade to Frontier AI Models & Autonomous Agent Pipelines
Looking for multi-model reasoning or live web retrieval? Pair web-scraping-engine with our paid Gemini Suite & Frontier Reasoning packages (45 models including Gemini 2.5 Pro, Antigravity Agent, Deep Research Pro, and Web Scraping) starting at only 10 sats / 1 GRLC / $0.01 with account-less pay-per-call micropayments:
👉 Explore Full Gateway & Frontier AI Suite
Other Modular Diogenes MCP Packages
Install only the domain packages relevant to your workflow:
Package | Scoped MCP Endpoint | Spoke Repository |
1. EVE: Frontier Sui Move Suite (14 Tools) |
| |
2. Agent Guardrails & Sandbox (12 Tools) |
| |
3. Web3, DeFi & 402 Security (15 Tools) |
| |
4. Gemini Suite & AI Agents (15 Tools) |
| |
5. Context, Code & Solvers (15 Tools) |
| |
6. Dev, Data & Workflow Utilities (14 Tools) |
| |
7. Web Extraction, Readability & Markdown Engine (12 Tools) |
| |
8. Neural Search, Grounded Research & Deep Intelligence (12 Tools) |
| |
9. Database Intelligence, SQL Engine & Schema Guard (12 Tools) |
| |
10. Full Master Gateway (All 121 Services) |
|
TensorBlock & MCP Index Profile Metadata
Field | Value |
Category |
|
Transport |
|
Endpoint |
|
Manifest |
|
Health Check |
|
Auth |
|
Install Confidence |
|
License |
|
Verification |
|
Maintainer |
|
Docs |
|
Support |
|
Available Tools
12 toolscompile_rss_feedWeb Content to RSS 2.0 & Atom Feed CompilerARead-onlyIdempotent
Transforms JSON item collections or article card arrays into standardized RSS 2.0, Atom 1.0, or JSON Feed 1.1 syndication feeds. Use when syndicating blog articles, product announcements, or changelog cards to RSS readers and AI agents; for compiling structured sitemap listings, use crawl_sitemap_urls instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'items_json' and 'channel_title' ('feed_standard' defaults to 'rss_2_0'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.
Inputs: items_json (required), channel_title (required), feed_standard. Returns: Deterministic execution response schema for compile_rss_feed.
crawl_sitemap_urlsXML Sitemap & Index Crawler Tree DecompilerARead-onlyIdempotent
Parses XML sitemaps and sitemap index files into structured URL trees with change frequency and last-modified filters. Use when discovering domain architectures, mapping crawl frontiers, or indexing blog/product catalogues; for extracting on-page hyperlinks from raw HTML, use extract_page_hyperlinks instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'sitemap_xml' ('filter_criteria' defaults to 'all_entries'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.
Inputs: sitemap_xml (required), max_results, filter_criteria. Returns: Deterministic execution response schema for crawl_sitemap_urls.
detect_paywall_and_bypassDOM Paywall, Modal Gate & Blurred Content Recovery AuditorARead-onlyIdempotent
Audits web page DOM hierarchies to detect soft paywalls, blur overlays, modal gates, and recovers concealed body text. Use when diagnosing paywall obstructions, diagnosing subscriber gating scripts, or auditing content access barriers; for assessing general SEO hygiene, use evaluate_seo_diagnostics instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('detection_profile' defaults to 'heuristic_dom_overlay'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.
Inputs: html_content (required), target_domain, detection_profile. Returns: Deterministic execution response schema for detect_paywall_and_bypass.
evaluate_seo_diagnosticsWeb SEO Diagnostics & Heading Hierarchy AuditorARead-onlyIdempotent
Audits HTML heading structures, title lengths, meta descriptions, image alt tags, and OpenGraph compliance into a health score. Use when auditing landing pages, verifying SEO compliance before publishing, or evaluating crawl friendliness; for extracting article bodies, use scrape_readable_article instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('audit_level' defaults to 'comprehensive_audit'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: html_content (required), audit_level, target_keyword. Returns: Deterministic execution response schema for evaluate_seo_diagnostics.
extract_clean_markdownWeb HTML to Clean LLM-Optimized Markdown EngineARead-onlyIdempotent
Converts raw HTML pages into clean, boilerplate-free, LLM-optimized Markdown with preserved hyperlinks and headings. Use when ingesting raw web documents or blog posts into agent context windows; for isolating tabular data into structured datasets, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('conversion_mode' defaults to 'clean_semantic_article'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: html_content (required), preserve_links, conversion_mode. Returns: Deterministic execution response schema for extract_clean_markdown.
extract_contact_infoLead Contact, Email, Phone & Social Handle ExtractorARead-onlyIdempotent
Extracts email addresses, telephone numbers, physical addresses, and social network handles from web text and DOM footers. Use when researching sales leads, aggregating business contact directories, or extracting customer service info; for parsing structured tabular data, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'content' ('extraction_focus' defaults to 'all_contacts'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: content (required), country_hint, extraction_focus. Returns: Deterministic execution response schema for extract_contact_info.
extract_page_hyperlinksWeb Hyperlink, Anchor & Asset Extraction FilterARead-onlyIdempotent
Extracts, normalizes, and filters internal and external hyperlinks, anchor texts, and downloadable asset targets from HTML. Use when building site crawlers, auditing broken links, or extracting reference citations; for isolating primary readable article narrative text, use scrape_readable_article instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('link_scope' defaults to 'all_links'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: html_content (required), base_url, link_scope. Returns: Deterministic execution response schema for extract_page_hyperlinks.
extract_structured_metadataOpenGraph, Twitter Cards & JSON-LD Microdata ExtractorARead-onlyIdempotent
Extracts OpenGraph tags, Twitter Card metadata, canonical URLs, and Schema.org JSON-LD microdata scripts from HTML. Use when indexing social share cards, author identity, breadcrumbs, and rich snippet schemas for AI search agents; for scanning page hyperlinks and outbound references, use extract_page_hyperlinks instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('metadata_targets' defaults to 'all_protocols'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: html_content (required), include_favicons, metadata_targets. Returns: Deterministic execution response schema for extract_structured_metadata.
isolate_css_selectorsDeterministic Edge DOM CSS Selector Scraper & IsolatorARead-onlyIdempotent
Evaluates CSS selector queries against HTML DOM trees to extract targeted inner text, outer HTML, or element attributes. Use when targeting specific elements like pricing badges, product titles, review cards, or navigation breadcrumbs; for parsing unstructured contact information, use extract_contact_info instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' and 'selector_query' ('extraction_mode' defaults to 'inner_text'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: html_content (required), selector_query (required), extraction_mode. Returns: Deterministic execution response schema for isolate_css_selectors.
parse_html_tablesHTML Table to Structured JSON & Markdown ParserARead-onlyIdempotent
Detects and parses HTML table elements into structured JSON record arrays and GitHub-Flavored Markdown tabular grids. Use when converting financial statements, product comparison grids, or sports statistics into structured data tables; for extracting raw clean markdown prose, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('output_format' defaults to 'markdown_and_json'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: html_content (required), min_rows, output_format. Returns: Deterministic execution response schema for parse_html_tables.
sanitize_html_contentEdge HTML Sanitizer, Script Stripper & XSS NeutralizerARead-onlyIdempotent
Sanitizes raw HTML by removing malicious script tags, iframes, inline event handlers, and data URIs into safe markup. Use when preparing untrusted third-party web content for rendering in agent dashboards or email clients; for converting clean HTML into markdown, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('sanitization_strictness' defaults to 'strip_scripts_and_iframes'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: html_content (required), preserve_images, sanitization_strictness. Returns: Deterministic execution response schema for sanitize_html_content.
scrape_readable_articleReadability Article Body & Byline ExtractorARead-onlyIdempotent
Extracts primary article body, author bylines, publication timestamps, and estimated reading duration from web pages. Use when stripping navigation bars, advertisement containers, and cookie banners to isolate core editorial content; for converting full HTML to markdown, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('readability_profile' defaults to 'article_body_strict'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.
Inputs: html_content (required), include_byline, readability_profile. Returns: Deterministic execution response schema for scrape_readable_article.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 12 tools
Most tools target distinct extraction facets, and descriptions explicitly cross-reference alternatives such as extract_clean_markdown vs scrape_readable_article and extract_structured_metadata vs evaluate_seo_diagnostics. Some overlap remains between readable-article extraction, markdown conversion, paywall recovery, and CSS-selector extraction, but the boundaries are mostly clear.
All tool names use consistent snake_case verb_noun-style patterns with clear action verbs such as extract, scrape, parse, isolate, detect, crawl, sanitize, evaluate, and compile. The convention is predictable throughout.
Twelve tools is well within a reasonable range for a web extraction and readability engine, and each tool corresponds to a recognizable operation rather than redundant variants. The surface feels deliberately scoped rather than bloated.
The set covers a broad extraction surface: markdown conversion, article extraction, metadata, tables, CSS selectors, links, sitemaps, sanitization, contact info, SEO diagnostics, and feed generation. A notable gap is the lack of a URL fetch/render or multi-page crawl tool, since most tools require raw HTML as input.
Maintenance
Related MCP Connectors
Clean Markdown and AI-readability scoring for any URL. Built for AI agents.
Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record.
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Turn public web pages into clean Markdown for LLMs, RAG and AI agents
Related MCP Servers
- AlicenseAqualityCmaintenanceEnables AI agents to read web pages reliably, returning clean markdown content, hyperlinks, and metadata without navigation or ad noise.34 npmMIT
- AlicenseAqualityCmaintenanceEnables AI agents to fetch one or many web pages concurrently and receive clean, LLM-ready Markdown with navigation, ads and boilerplate stripped away, plus extracted metadata such as title, description, author, Open Graph tags and canonical URL. It runs locally over stdio with no API key or account, requiring only a single command.3MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to read any web page — including pages behind a signed-in session, localhost, and JavaScript-heavy sites — as clean, focus-ranked, token-budgeted markdown using a real browser running locally, with no cloud relay or API key. Also supports structured extraction, link listing, screenshots, and policy-gated interaction.13 npm1MIT
- FlicenseNot gradedqualityAmaintenanceEnables any LLM agent to fetch, extract, crawl, and screenshot websites through conversation by exposing 17 scraping tools over stdio or SSE.-