Skip to main content
Glama
Diogenes Web Extraction, Readability & Markdown Engine MCP

Diogenes Web Extraction, Readability & Markdown Engine MCP

Diogenes Web Extraction, Readability & Markdown Engine MCP (diogenes-web-scraping-engine-mcp)

Indexed on TensorBlock MCP Index Glama MCP Server M8ven Score Master Gateway

Diogenes Web Extraction, Readability & Markdown Engine MCP is a focused, zero-bloat Model Context Protocol (streamable-http) package powered by the Diogenes Microservice Gateway.

Instead of loading all 121 gateway endpoints into your LLM's context window, installing this package exposes only the 12 specialized web-scraping-engine tools needed for Web Extraction & Scraping:

12 high-performance web extraction, readability & markdown tools: clean LLM markdown compiler, readable article extractor, OpenGraph & JSON-LD metadata parser, HTML table to JSON/Markdown converter, CSS selector isolator, paywall & overlay bypass auditor, XML sitemap crawler, hyperlink extractor, HTML XSS sanitizer, contact & lead info parser, SEO heading diagnostics, and RSS/Atom feed compiler.


⚡ 1-Click Install for Cursor & Claude Desktop

No API keys, user accounts, or local dependencies required. Zero setup — paste directly into your client config:

🖱️ Cursor IDE (.cursor/mcp.json)

Add to .cursor/mcp.json in your project root or global Cursor settings:

{
  "mcpServers": {
    "diogenes-web-scraping-engine": {
      "transport": "streamable-http",
      "url": "https://diogenesgateway.com/mcp/web-scraping-engine"
    }
  }
}

🧠 Claude Desktop (claude_desktop_config.json)

Add to claude_desktop_config.json (Mac: ~/Library/Application Support/Claude/claude_desktop_config.json | Windows: %APPDATA%\Claude\claude_desktop_config.json):

{
  "mcpServers": {
    "diogenes-web-scraping-engine": {
      "transport": "streamable-http",
      "url": "https://diogenesgateway.com/mcp/web-scraping-engine"
    }
  }
}

Prefer local stdio? Run via npx:

npx -y @stand-out-of-my-sun/diogenes-web-scraping-engine-mcp

Related MCP server: Website to Markdown MCP

Included Tools in web-scraping-engine (12 Tools)

MCP Tool Name

Underlying REST Route

Settlement

Capability

extract_clean_markdown

POST /api/v1/extract/clean-markdown

Free Developer Tier (0 sats)

Converts raw HTML pages into clean, boilerplate-free, LLM-optimized Markdown with preserved hyperlinks and headings. Use when ingesting raw web documents or blog posts into agent context windows; for isolating tabular data into structured datasets, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('conversion_mode' defaults to 'clean_semantic_article'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

scrape_readable_article

POST /api/v1/extract/readable-article

Free Developer Tier (0 sats)

Extracts primary article body, author bylines, publication timestamps, and estimated reading duration from web pages. Use when stripping navigation bars, advertisement containers, and cookie banners to isolate core editorial content; for converting full HTML to markdown, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('readability_profile' defaults to 'article_body_strict'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

extract_structured_metadata

POST /api/v1/extract/structured-metadata

10 sats (L402)

Extracts OpenGraph tags, Twitter Card metadata, canonical URLs, and Schema.org JSON-LD microdata scripts from HTML. Use when indexing social share cards, author identity, breadcrumbs, and rich snippet schemas for AI search agents; for scanning page hyperlinks and outbound references, use extract_page_hyperlinks instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('metadata_targets' defaults to 'all_protocols'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

parse_html_tables

POST /api/v1/extract/html-tables

10 sats (L402)

Detects and parses HTML table elements into structured JSON record arrays and GitHub-Flavored Markdown tabular grids. Use when converting financial statements, product comparison grids, or sports statistics into structured data tables; for extracting raw clean markdown prose, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('output_format' defaults to 'markdown_and_json'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

isolate_css_selectors

POST /api/v1/extract/css-selectors

10 sats (L402)

Evaluates CSS selector queries against HTML DOM trees to extract targeted inner text, outer HTML, or element attributes. Use when targeting specific elements like pricing badges, product titles, review cards, or navigation breadcrumbs; for parsing unstructured contact information, use extract_contact_info instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' and 'selector_query' ('extraction_mode' defaults to 'inner_text'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

detect_paywall_and_bypass

POST /api/v1/extract/paywall-detect

15 sats (L402)

Audits web page DOM hierarchies to detect soft paywalls, blur overlays, modal gates, and recovers concealed body text. Use when diagnosing paywall obstructions, diagnosing subscriber gating scripts, or auditing content access barriers; for assessing general SEO hygiene, use evaluate_seo_diagnostics instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('detection_profile' defaults to 'heuristic_dom_overlay'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.

crawl_sitemap_urls

POST /api/v1/extract/sitemap-crawler

15 sats (L402)

Parses XML sitemaps and sitemap index files into structured URL trees with change frequency and last-modified filters. Use when discovering domain architectures, mapping crawl frontiers, or indexing blog/product catalogues; for extracting on-page hyperlinks from raw HTML, use extract_page_hyperlinks instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'sitemap_xml' ('filter_criteria' defaults to 'all_entries'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.

extract_page_hyperlinks

POST /api/v1/extract/page-hyperlinks

10 sats (L402)

Extracts, normalizes, and filters internal and external hyperlinks, anchor texts, and downloadable asset targets from HTML. Use when building site crawlers, auditing broken links, or extracting reference citations; for isolating primary readable article narrative text, use scrape_readable_article instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('link_scope' defaults to 'all_links'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

sanitize_html_content

POST /api/v1/extract/sanitize-html

10 sats (L402)

Sanitizes raw HTML by removing malicious script tags, iframes, inline event handlers, and data URIs into safe markup. Use when preparing untrusted third-party web content for rendering in agent dashboards or email clients; for converting clean HTML into markdown, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('sanitization_strictness' defaults to 'strip_scripts_and_iframes'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

extract_contact_info

POST /api/v1/extract/contact-info

10 sats (L402)

Extracts email addresses, telephone numbers, physical addresses, and social network handles from web text and DOM footers. Use when researching sales leads, aggregating business contact directories, or extracting customer service info; for parsing structured tabular data, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'content' ('extraction_focus' defaults to 'all_contacts'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

evaluate_seo_diagnostics

POST /api/v1/extract/seo-diagnostics

10 sats (L402)

Audits HTML heading structures, title lengths, meta descriptions, image alt tags, and OpenGraph compliance into a health score. Use when auditing landing pages, verifying SEO compliance before publishing, or evaluating crawl friendliness; for extracting article bodies, use scrape_readable_article instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('audit_level' defaults to 'comprehensive_audit'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

compile_rss_feed

POST /api/v1/extract/rss-feed-compiler

15 sats (L402)

Transforms JSON item collections or article card arrays into standardized RSS 2.0, Atom 1.0, or JSON Feed 1.1 syndication feeds. Use when syndicating blog articles, product announcements, or changelog cards to RSS readers and AI agents; for compiling structured sitemap listings, use crawl_sitemap_urls instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'items_json' and 'channel_title' ('feed_standard' defaults to 'rss_2_0'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.

Free Account-Less Demo Mode: Append ?demo=true to any REST endpoint above for an instant HTTP 200 schema preview at zero cost.


🚀 Upgrade to Frontier AI Models & Autonomous Agent Pipelines

Looking for multi-model reasoning or live web retrieval? Pair web-scraping-engine with our paid Gemini Suite & Frontier Reasoning packages (45 models including Gemini 2.5 Pro, Antigravity Agent, Deep Research Pro, and Web Scraping) starting at only 10 sats / 1 GRLC / $0.01 with account-less pay-per-call micropayments: 👉 Explore Full Gateway & Frontier AI Suite


Other Modular Diogenes MCP Packages

Install only the domain packages relevant to your workflow:

Package

Scoped MCP Endpoint

Spoke Repository

1. EVE: Frontier Sui Move Suite (14 Tools)

https://diogenesgateway.com/mcp/eve-frontier

stand-out-of-my-sun/diogenes-eve-frontier-mcp

2. Agent Guardrails & Sandbox (12 Tools)

https://diogenesgateway.com/mcp/agent-guardrails

stand-out-of-my-sun/diogenes-agent-guardrails-mcp

3. Web3, DeFi & 402 Security (15 Tools)

https://diogenesgateway.com/mcp/web3-security

stand-out-of-my-sun/diogenes-web3-security-mcp

4. Gemini Suite & AI Agents (15 Tools)

https://diogenesgateway.com/mcp/gemini-suite

stand-out-of-my-sun/diogenes-gemini-suite-mcp

5. Context, Code & Solvers (15 Tools)

https://diogenesgateway.com/mcp/context-code-solver

stand-out-of-my-sun/diogenes-context-code-solver-mcp

6. Dev, Data & Workflow Utilities (14 Tools)

https://diogenesgateway.com/mcp/dev-utilities

stand-out-of-my-sun/diogenes-dev-utilities-mcp

7. Web Extraction, Readability & Markdown Engine (12 Tools)

https://diogenesgateway.com/mcp/web-scraping-engine

stand-out-of-my-sun/diogenes-web-scraping-engine-mcp

8. Neural Search, Grounded Research & Deep Intelligence (12 Tools)

https://diogenesgateway.com/mcp/neural-search-engine

stand-out-of-my-sun/diogenes-neural-search-engine-mcp

9. Database Intelligence, SQL Engine & Schema Guard (12 Tools)

https://diogenesgateway.com/mcp/database-intelligence

stand-out-of-my-sun/diogenes-database-intelligence-mcp

10. Full Master Gateway (All 121 Services)

https://diogenesgateway.com/mcp

stand-out-of-my-sun/diogenes-micro-gateway


TensorBlock & MCP Index Profile Metadata

Field

Value

Category

Web Extraction & Scraping

Transport

streamable-http

Endpoint

https://diogenesgateway.com/mcp/web-scraping-engine

Manifest

https://diogenesgateway.com/.well-known/mcp/web-scraping-engine.json

Health Check

https://diogenesgateway.com/api/v1/health

Auth

none (100% account-less free tier, MCP discovery & ?demo=true previews; optional 6 flexible crypto payment rails for production calls: L402, x402, Base MPP, Garlicoin 0-Conf, BSC WGRLC, or Prepaid GRLC session API keys)

Install Confidence

high (Zero-install remote Streamable HTTP endpoint + root package.json, smithery.yaml, server.json, glama.json, and mcp.json)

License

MIT

Verification

verified (@stand-out-of-my-sun verified via https://diogenesgateway.com/.well-known/tensorblock.json & https://diogenesgateway.com/.well-known/mcp/web-scraping-engine.json)

Maintainer

@stand-out-of-my-sun

Docs

https://diogenesgateway.com/llms.txt

Support

https://github.com/stand-out-of-my-sun/diogenes-web-scraping-engine-mcp/issues

Available Tools

12 tools
compile_rss_feedWeb Content to RSS 2.0 & Atom Feed CompilerA
Read-onlyIdempotent

Transforms JSON item collections or article card arrays into standardized RSS 2.0, Atom 1.0, or JSON Feed 1.1 syndication feeds. Use when syndicating blog articles, product announcements, or changelog cards to RSS readers and AI agents; for compiling structured sitemap listings, use crawl_sitemap_urls instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'items_json' and 'channel_title' ('feed_standard' defaults to 'rss_2_0'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.

Inputs: items_json (required), channel_title (required), feed_standard. Returns: Deterministic execution response schema for compile_rss_feed.

crawl_sitemap_urlsXML Sitemap & Index Crawler Tree DecompilerA
Read-onlyIdempotent

Parses XML sitemaps and sitemap index files into structured URL trees with change frequency and last-modified filters. Use when discovering domain architectures, mapping crawl frontiers, or indexing blog/product catalogues; for extracting on-page hyperlinks from raw HTML, use extract_page_hyperlinks instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'sitemap_xml' ('filter_criteria' defaults to 'all_entries'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.

Inputs: sitemap_xml (required), max_results, filter_criteria. Returns: Deterministic execution response schema for crawl_sitemap_urls.

detect_paywall_and_bypassDOM Paywall, Modal Gate & Blurred Content Recovery AuditorA
Read-onlyIdempotent

Audits web page DOM hierarchies to detect soft paywalls, blur overlays, modal gates, and recovers concealed body text. Use when diagnosing paywall obstructions, diagnosing subscriber gating scripts, or auditing content access barriers; for assessing general SEO hygiene, use evaluate_seo_diagnostics instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('detection_profile' defaults to 'heuristic_dom_overlay'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.

Inputs: html_content (required), target_domain, detection_profile. Returns: Deterministic execution response schema for detect_paywall_and_bypass.

evaluate_seo_diagnosticsWeb SEO Diagnostics & Heading Hierarchy AuditorA
Read-onlyIdempotent

Audits HTML heading structures, title lengths, meta descriptions, image alt tags, and OpenGraph compliance into a health score. Use when auditing landing pages, verifying SEO compliance before publishing, or evaluating crawl friendliness; for extracting article bodies, use scrape_readable_article instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('audit_level' defaults to 'comprehensive_audit'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

Inputs: html_content (required), audit_level, target_keyword. Returns: Deterministic execution response schema for evaluate_seo_diagnostics.

extract_clean_markdownWeb HTML to Clean LLM-Optimized Markdown EngineA
Read-onlyIdempotent

Converts raw HTML pages into clean, boilerplate-free, LLM-optimized Markdown with preserved hyperlinks and headings. Use when ingesting raw web documents or blog posts into agent context windows; for isolating tabular data into structured datasets, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('conversion_mode' defaults to 'clean_semantic_article'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

Inputs: html_content (required), preserve_links, conversion_mode. Returns: Deterministic execution response schema for extract_clean_markdown.

extract_contact_infoLead Contact, Email, Phone & Social Handle ExtractorA
Read-onlyIdempotent

Extracts email addresses, telephone numbers, physical addresses, and social network handles from web text and DOM footers. Use when researching sales leads, aggregating business contact directories, or extracting customer service info; for parsing structured tabular data, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'content' ('extraction_focus' defaults to 'all_contacts'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

Inputs: content (required), country_hint, extraction_focus. Returns: Deterministic execution response schema for extract_contact_info.

extract_structured_metadataOpenGraph, Twitter Cards & JSON-LD Microdata ExtractorA
Read-onlyIdempotent

Extracts OpenGraph tags, Twitter Card metadata, canonical URLs, and Schema.org JSON-LD microdata scripts from HTML. Use when indexing social share cards, author identity, breadcrumbs, and rich snippet schemas for AI search agents; for scanning page hyperlinks and outbound references, use extract_page_hyperlinks instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('metadata_targets' defaults to 'all_protocols'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

Inputs: html_content (required), include_favicons, metadata_targets. Returns: Deterministic execution response schema for extract_structured_metadata.

isolate_css_selectorsDeterministic Edge DOM CSS Selector Scraper & IsolatorA
Read-onlyIdempotent

Evaluates CSS selector queries against HTML DOM trees to extract targeted inner text, outer HTML, or element attributes. Use when targeting specific elements like pricing badges, product titles, review cards, or navigation breadcrumbs; for parsing unstructured contact information, use extract_contact_info instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' and 'selector_query' ('extraction_mode' defaults to 'inner_text'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

Inputs: html_content (required), selector_query (required), extraction_mode. Returns: Deterministic execution response schema for isolate_css_selectors.

parse_html_tablesHTML Table to Structured JSON & Markdown ParserA
Read-onlyIdempotent

Detects and parses HTML table elements into structured JSON record arrays and GitHub-Flavored Markdown tabular grids. Use when converting financial statements, product comparison grids, or sports statistics into structured data tables; for extracting raw clean markdown prose, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('output_format' defaults to 'markdown_and_json'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

Inputs: html_content (required), min_rows, output_format. Returns: Deterministic execution response schema for parse_html_tables.

sanitize_html_contentEdge HTML Sanitizer, Script Stripper & XSS NeutralizerA
Read-onlyIdempotent

Sanitizes raw HTML by removing malicious script tags, iframes, inline event handlers, and data URIs into safe markup. Use when preparing untrusted third-party web content for rendering in agent dashboards or email clients; for converting clean HTML into markdown, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('sanitization_strictness' defaults to 'strip_scripts_and_iframes'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

Inputs: html_content (required), preserve_images, sanitization_strictness. Returns: Deterministic execution response schema for sanitize_html_content.

scrape_readable_articleReadability Article Body & Byline ExtractorA
Read-onlyIdempotent

Extracts primary article body, author bylines, publication timestamps, and estimated reading duration from web pages. Use when stripping navigation bars, advertisement containers, and cookie banners to isolate core editorial content; for converting full HTML to markdown, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('readability_profile' defaults to 'article_body_strict'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid.

Inputs: html_content (required), include_byline, readability_profile. Returns: Deterministic execution response schema for scrape_readable_article.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.6/5.0

    Scored across 12 tools

    Disambiguation4/5

    Most tools target distinct extraction facets, and descriptions explicitly cross-reference alternatives such as extract_clean_markdown vs scrape_readable_article and extract_structured_metadata vs evaluate_seo_diagnostics. Some overlap remains between readable-article extraction, markdown conversion, paywall recovery, and CSS-selector extraction, but the boundaries are mostly clear.

    Naming Consistency5/5

    All tool names use consistent snake_case verb_noun-style patterns with clear action verbs such as extract, scrape, parse, isolate, detect, crawl, sanitize, evaluate, and compile. The convention is predictable throughout.

    Tool Count5/5

    Twelve tools is well within a reasonable range for a web extraction and readability engine, and each tool corresponds to a recognizable operation rather than redundant variants. The surface feels deliberately scoped rather than bloated.

    Completeness4/5

    The set covers a broad extraction surface: markdown conversion, article extraction, metadata, tables, CSS selectors, links, sitemaps, sanitization, contact info, SEO diagnostics, and feed generation. A notable gap is the lack of a URL fetch/render or multi-page crawl tool, since most tools require raw HTML as input.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      C
      maintenance
      Enables AI agents to read web pages reliably, returning clean markdown content, hyperlinks, and metadata without navigation or ad noise.
      3
      4 npm
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Enables AI agents to fetch one or many web pages concurrently and receive clean, LLM-ready Markdown with navigation, ads and boilerplate stripped away, plus extracted metadata such as title, description, author, Open Graph tags and canonical URL. It runs locally over stdio with no API key or account, requiring only a single command.
      3
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to read any web page — including pages behind a signed-in session, localhost, and JavaScript-heavy sites — as clean, focus-ranked, token-budgeted markdown using a real browser running locally, with no cloud relay or API key. Also supports structured extraction, link listing, screenshots, and policy-gated interaction.
      13 npm
      1
      MIT