webdatatools-rag-mcp
An MCP server for web search, crawling, extraction, monitoring, and RAG preparation, billed through your own Apify token.
Search Google and fetch top results (or direct URLs) as Markdown/text.
Crawl sites to Markdown/text with depth, page, sitemap, robots, and path controls.
Extract article/news text and metadata (title, author, date, tags, images).
Extract Schema.org JSON-LD and Open Graph/Twitter Card data.
Scrape Google News by query/topic/date and resolve publisher URLs.
Monitor PR Newswire, Business Wire, and GlobeNewswire releases by keyword/time.
README also lists PDF-to-Markdown, llms.txt generation, RAG chunking, Medium scraping, Google AI Overviews scraping, and RSS reading.
Provides AI web search capabilities using Google results, returning clean page text and Markdown for LLM context and RAG pipelines.
Provides Google News scraping and monitoring via RSS search by keyword, topic, or site, returning news articles for AI agents.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@webdatatools-rag-mcp crawl example.com and convert it to Markdown for my RAG pipeline"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
WebDataTools Web content for AI & RAG MCP server
webdatatools-rag-mcp
An MCP server with 12 web content for ai & rag tools for AI agents — Claude Desktop, Cursor, Cline or any MCP client. Web search with page text, website-to-Markdown crawling, clean article extraction, JSON-LD/structured data, Google News and press-release monitoring — ready for LLM context and RAG pipelines.
This server uses your own Apify API token. Every tool call runs a WebDataTools Actor under your Apify account and is billed to your Apify credit — pay per result, the price is in each tool description. Your token is only sent to Apify's API.
Quick start
Requires Node.js 18+.
APIFY_TOKEN=apify_api_... npx -y github:paulet4a-commits/webdatatools-rag-mcpGet a free token (the free plan includes monthly credit): https://console.apify.com/settings/integrations
Related MCP server: Firecrawl MCP Server
Claude Desktop / Cursor
Add this to claude_desktop_config.json (Claude Desktop) or .cursor/mcp.json (Cursor):
{
"mcpServers": {
"webdatatools-rag": {
"command": "npx",
"args": [
"-y",
"github:paulet4a-commits/webdatatools-rag-mcp"
],
"env": {
"APIFY_TOKEN": "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
}
}
}Tools (12)
Tool | What it does | Price (free plan) | Backing Actor |
| AI Web Search & Read: Google results as clean Markdown | $0.005 / result | |
| Website to Markdown — Content Crawler for LLM & RAG | $0.001 / result | |
| llms.txt Generator (Website to llms.txt & llms-full.txt) | $0.001 / page | |
| PDF to Markdown Converter (Text, Headings, Metadata) | $0.002 / pdf | |
| RAG Text Chunker (Markdown to Chunks, Token Counts) | $0.0002 / chunk | |
| Article & News Extractor (clean text, author, date, markdown) | $0.002 / result | |
| Structured Data & JSON-LD Extractor (Schema.org, Open Graph) | $0.002 / result | |
| Google News Scraper (Keyword, Topic, Site, Date Range) | $0.0005 / article | |
| Press Release Monitor: PR Newswire, BusinessWire, GlobeNewswire | $0.001 / result | |
| Medium Scraper (Articles, Claps, Tags, Reading Time) | $0.004 / article | |
| Google AI Overviews Scraper (AI Overview Answers and Sources) | $0.0125 / AI Overview | |
| RSS Feed Reader (RSS and Atom Items) | $0.0005 / item |
More WebDataTools MCP servers
webdatatools-mcp-server — the 10 most popular tools in one server
webdatatools-domain-mcp — Domain & website intelligence
webdatatools-social-mcp — Search, video & social data
webdatatools-leads-mcp — Leads, jobs & company data
webdatatools-dev-mcp — Developer, app & research data
License
MIT
Available Tools
6 toolsai_web_searchA
AI Web Search runs a Google search, fetches the top organic results and returns clean Markdown per result — one call turns a question into LLM-ready context for agents, RAG and MCP. Billed to your own Apify account: ~$0.005 per result (Apify free-plan price, lower on paid plans).
Inputs: urls, query, queries, maxResults, +5 more.
article_extractorA
Article & News Extractor returns clean article text, title, author(s), publish/modified date, tags and images from any news or blog URL — one row per URL, as Markdown, plain text or HTML. Billed to your own Apify account: ~$0.002 per result (Apify free-plan price, lower on paid plans).
Inputs: urls (required), outputFormat, includeImages.
google_news_scraperA
Google News Scraper returns headlines from Google News RSS search and topic feeds for any keyword, site: or when: query, and resolves each article's real publisher URL — one row per article. Billed to your own Apify account: ~$0.0005 per article (Apify free-plan price, lower on paid plans).
Inputs: queries (required), topics, country, language, +3 more.
press_release_monitorA
Press Release Monitor returns keyword-matched press releases from PR Newswire, Business Wire and GlobeNewswire RSS feeds — company, publish date, category and summary, one row per release. Billed to your own Apify account: ~$0.001 per result (Apify free-plan price, lower on paid plans).
Inputs: sources, keywords, sinceHours, includeBody, +1 more.
structured_data_extractorA
Structured Data & JSON-LD Extractor reads every Schema.org JSON-LD block and Open Graph tag on a page and returns one clean row per URL — product, price, rating, article, job posting, event, recipe, FAQ and breadcrumb data, ready for RAG pipelines and SEO rich-result audits. Billed to your own Apify account: ~$0.002 per result (Apify free-plan price, lower on paid plans).
Inputs: urls (required), followRedirects, includeOpenGraph, includeRawJsonLd.
website_to_markdownB
Content Crawler turns any site into clean Markdown per page for LLMs, RAG pipelines and vector DBs — no headless browser, $1 per 1,000 pages. Billed to your own Apify account: ~$0.001 per result (Apify free-plan price, lower on paid plans).
Inputs: startUrls (required), maxDepth, maxPages, useSitemap, +6 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 6 tools
Most tools are clearly differentiated by source or output type: general web search, site-to-Markdown crawl, article extraction, structured data, Google News, and press releases. The main overlap is between website_to_markdown and article_extractor when processing a single news/blog article, but their broader purposes are still distinguishable.
All names use snake_case and are descriptive, which makes them readable and predictable. The patterns vary slightly (action-oriented names like ai_web_search vs. noun-extractor names like article_extractor), but the set remains consistent overall.
Six tools is a well-scoped size for a web-data RAG server. Each tool has a distinct source or extraction focus, so the surface does not feel bloated or thin.
The set covers core web-data acquisition workflows: search, crawling, article extraction, structured data, news, and press releases. Minor gaps such as PDF extraction, social media sources, or advanced crawl controls exist, but the core RAG pipeline needs are well represented.
Maintenance
Related MCP Connectors
Scrape, crawl and search the web for AI agents via MCP.
Web scraping, extraction and search for AI agents. Hosted MCP with workspace API keys.
Web scraping, browser automation, screenshots and AI data extraction for AI agents via MCP.
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
Related MCP Servers
- AlicenseAqualityBmaintenanceEnables AI agents to retrieve Google search results, YouTube data, podcasts, Bluesky, Telegram, and Substack content through MCP tools, using the user's own Apify API token and billed per result.10MIT
- AlicenseAqualityBmaintenanceEnables MCP-compatible AI agents to search, scrape, crawl, map, and interact with live web pages, returning clean, structured, agent-ready content. It also supports recurring page monitoring, autonomous deep research, and credit-usage checks.26181,410 npm1MIT
- FlicenseNot gradedqualityAmaintenanceEnables AI agents to run web searches across configured providers and crawl individual pages or whole sites into clean Markdown or JSON, including authenticated crawls and local or HTTP MCP deployment.-
- FlicenseNot gradedqualityAmaintenanceEnables AI agents to search the web through configured providers and crawl single pages or whole sites into clean Markdown or JSON, including content behind logins, over MCP STDIO or HTTP.-