What can you do with this server?

The MCP WebAnalyzer server enables intelligent web content analysis and processing with AI-powered capabilities: • Content Extraction & Conversion - Extract web pages to clean, structured markdown while removing ads, navigation, and clutter • AI-Powered Q&A - Answer questions about web content using OpenAI GPT models (GPT-3.5, GPT-4, GPT-4 Turbo, GPT-5) • Web Crawling - Discover subpages from URLs with configurable depth and page limits • Content Summarization - Generate concise one-line summaries and RAG-optimized extracts with optional question focusing • Smart Content Ranking - Analyze and rank content importance using custom algorithms • IDE Integration - Works with Claude Desktop, Cursor, VS Code, and PyCharm

Which integrations are available for this server?

Uses Celery for distributed asynchronous task processing and background job management Supports containerized deployment and orchestration of the web analyzer services Uses environment variable configuration for server settings, security keys, and connection information Built on FastAPI to provide high-performance web analysis endpoints and API documentation Integration with OpenAI's API for AI-powered web content analysis and summarization Exposes metrics endpoints for monitoring server performance and usage statistics Uses Redis for caching, session management, and as a message broker for task processing

How do I use MCP WebAnalyzer?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@MCP WebAnalyzer summarize the key points from https://example.com/blog/post" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

MCP WebAnalyzer

by kimdonghwi94

Overview Schema Source Related Servers Discussions Admin

Python

Hybrid

🔍 Web Analyzer MCP

A powerful MCP (Model Context Protocol) server for intelligent web content analysis and summarization. Built with FastMCP, this server provides smart web scraping, content extraction, and AI-powered question-answering capabilities.

✨ Features

🎯 Core Tools

url_to_markdown - Extract and summarize key web page content
- Analyzes content importance using custom algorithms
- Removes ads, navigation, and irrelevant content
- Keeps only essential information (tables, images, key text)
- Outputs structured markdown optimized for analysis
web_content_qna - AI-powered Q&A about web content
- Extracts relevant content sections from web pages
- Uses intelligent chunking and relevance matching
- Answers questions using OpenAI GPT models

🚀 Key Features

Smart Content Ranking: Algorithm-based content importance scoring
Essential Content Only: Removes clutter, keeps what matters
Multi-IDE Support: Works with Claude Desktop, Cursor, VS Code, PyCharm
Flexible Models: Choose from GPT-3.5, GPT-4, GPT-4 Turbo, or GPT-5

Related MCP server: ScrapeGraph MCP Server

📦 Installation

Prerequisites

uv (Python package manager)
Chrome/Chromium browser (for Selenium)
OpenAI API key (for Q&A functionality)

🚀 Quick Start with uv (Recommended)

# Clone the repository
git clone https://github.com/kimdonghwi94/web-analyzer-mcp.git
cd web-analyzer-mcp

# Run directly with uv (auto-installs dependencies)
uv run mcp-webanalyzer

Installing via Smithery

To install web-analyzer-mcp for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @kimdonghwi94/web-analyzer-mcp --client claude

IDE/Editor Integration

Add to your Claude Desktop_config.json file. See Claude Desktop MCP documentation for more details.

{
  "mcpServers": {
    "web-analyzer": {
      "command": "uv",
      "args": [
        "--directory",
        "/path/to/web-analyzer-mcp",
        "run", 
        "mcp-webanalyzer"
      ],
      "env": {
        "OPENAI_API_KEY": "your_openai_api_key_here",
        "OPENAI_MODEL": "gpt-4"
      }
    }
  }
}

Add the server using Claude Code CLI:

claude mcp add web-analyzer -e OPENAI_API_KEY=your_api_key_here -e OPENAI_MODEL=gpt-4 -- uv --directory /path/to/web-analyzer-mcp run mcp-webanalyzer

Add to your Cursor settings (File > Preferences > Settings > Extensions > MCP):

{
  "mcpServers": {
    "web-analyzer": {
      "command": "uv",
      "args": [
        "--directory",
        "/path/to/web-analyzer-mcp",
        "run", 
        "mcp-webanalyzer"
      ],
      "env": {
        "OPENAI_API_KEY": "your_openai_api_key_here",
        "OPENAI_MODEL": "gpt-4"
      }
    }
  }
}

See JetBrains AI Assistant Documentation for more details.

In JetBrains IDEs go to Settings → Tools → AI Assistant → Model Context Protocol (MCP)
Click + Add
Click on Command in the top-left corner of the dialog and select the As JSON option from the list
Add this configuration and click OK:

{
  "mcpServers": {
    "web-analyzer": {
      "command": "uv",
      "args": [
        "--directory",
        "/path/to/web-analyzer-mcp",
        "run", 
        "mcp-webanalyzer"
      ],
      "env": {
        "OPENAI_API_KEY": "your_openai_api_key_here",
        "OPENAI_MODEL": "gpt-4"
      }
    }
  }
}

🎛️ Tool Descriptions

`url_to_markdown`

Converts web pages to clean markdown format with essential content extraction.

Parameters:

url (string): The web page URL to analyze

Returns: Clean markdown content with structured data preservation

`web_content_qna`

Answers questions about web page content using intelligent content analysis.

Parameters:

url (string): The web page URL to analyze
question (string): Question about the page content

Returns: AI-generated answer based on page content

🏗️ Architecture

Content Extraction Pipeline

URL Validation - Ensures proper URL format
HTML Fetching - Uses Selenium for dynamic content
Content Parsing - BeautifulSoup for HTML processing
Element Scoring - Custom algorithm ranks content importance
Content Filtering - Removes duplicates and low-value content
Markdown Conversion - Structured output generation

Q&A Processing Pipeline

Content Chunking - Intelligent text segmentation
Relevance Scoring - Matches content to questions
Context Selection - Picks most relevant chunks
Answer Generation - OpenAI GPT integration

🏗️ Project Structure

web-analyzer-mcp/
├── web_analyzer_mcp/          # Main Python package
│   ├── __init__.py           # Package initialization
│   ├── server.py             # FastMCP server with tools
│   ├── web_extractor.py      # Web content extraction engine
│   └── rag_processor.py      # RAG-based Q&A processor
├── scripts/                   # Build and utility scripts
│   └── build.js              # Node.js build script
├── README.md                 # English documentation
├── README.ko.md              # Korean documentation
├── package.json              # npm configuration and scripts
├── pyproject.toml            # Python package configuration
├── .env.example              # Environment variables template
└── dist-info.json            # Build information (generated)

🛠️ Development

Modern Development with uv

# Clone repository
git clone https://github.com/kimdonghwi94/web-analyzer-mcp.git
cd web-analyzer-mcp

# Development commands
uv run mcp-webanalyzer     # Start development server
uv run python -m pytest   # Run tests
uv run ruff check .        # Lint code
uv run ruff format .       # Format code
uv sync                    # Sync dependencies

# Install development dependencies
uv add --dev pytest ruff mypy

# Create production build
npm run build

Alternative: Traditional Python Development

# Setup Python environment (if not using uv)
pip install -e .[dev]

# Development commands
python -m web_analyzer_mcp.server  # Start server
python -m pytest tests/            # Run tests
python -m ruff check .             # Lint code
python -m ruff format .            # Format code
python -m mypy web_analyzer_mcp/   # Type checking

🤝 Contributing

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

📋 Roadmap

Support for more content types (PDFs, videos)
Multi-language content extraction
Custom extraction rules
Caching for frequently accessed content
Webhook support for real-time updates

⚠️ Limitations

Requires Chrome/Chromium for JavaScript-heavy sites
OpenAI API key needed for Q&A functionality
Rate limited to prevent abuse
Some sites may block automated access

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙋‍♂️ Support

Create an issue for bug reports or feature requests
Contribute to discussions in the GitHub repository
Check the documentation for detailed guides

🌟 Acknowledgments

Built with FastMCP framework
Inspired by HTMLRAG techniques for web content processing
Thanks to the MCP community for feedback and contributions

Made with ❤️ for the MCP community

Available Tools

2 tools

url_to_markdown_toolA

Extract and convert web page content to markdown format.

This tool scrapes a web page, removes unnecessary elements, ranks content by importance using a custom algorithm, and returns clean markdown. Perfect for RAG applications.

Args: url: The web page URL to analyze and convert

Returns: str: Clean markdown representation of the web page content

ParametersJSON Schema

Name	Required	Description	Default
`url`	Yes

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

A4.5/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden and does well by disclosing key behavioral traits: it scrapes web pages, removes unnecessary elements, ranks content by importance using a custom algorithm, and returns clean markdown. This covers the transformation process and output characteristics beyond basic functionality.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is efficiently structured with a clear purpose statement, elaboration of the process, usage context, and separate Args/Returns sections. Every sentence adds value without redundancy, and the information is appropriately front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity (web scraping with algorithmic ranking), no annotations, and the presence of an output schema (which handles return value documentation), the description provides complete context. It explains the transformation process, use case, parameter meaning, and output format adequately.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage and only one parameter, the description compensates well by explaining the 'url' parameter as 'The web page URL to analyze and convert,' adding meaningful context about its purpose. However, it doesn't specify URL format requirements or constraints, preventing a perfect score.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('Extract and convert web page content to markdown format') and distinguishes it from the sibling tool 'web_content_qna' by focusing on conversion rather than Q&A. It provides a complete verb+resource+output specification.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly states 'Perfect for RAG applications,' providing clear context for when to use this tool. However, it doesn't specify when NOT to use it or explicitly contrast with the sibling 'web_content_qna' tool, which would be needed for a score of 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_content_qnaA

Answer questions about web page content using RAG.

This tool combines web scraping with RAG (Retrieval Augmented Generation) to answer specific questions about web page content. It extracts relevant content sections and uses AI to provide accurate answers.

Args: url: The web page URL to analyze question: The question to answer based on the page content

Returns: str: AI-generated answer based on the web page content

ParametersJSON Schema

Name	Required	Description	Default
`url`	Yes
`question`	Yes

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

A4.2/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses key behavioral traits: it performs web scraping and uses AI (RAG) to generate answers, which implies external API calls and potential rate limits or latency. However, it lacks details on error handling, authentication needs, or specific limitations (e.g., website compatibility, content size). The description does not contradict any annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately sized and front-loaded: the first sentence states the core purpose, followed by brief elaboration on the method, and ends with clear Arg/Return sections. Every sentence adds value without redundancy, and the structure is logical and efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (involving web scraping and AI), no annotations, and an output schema exists (returns a string), the description is mostly complete. It covers purpose, method, parameters, and return type, but could benefit from more behavioral context (e.g., limitations, errors). The output schema reduces the need to explain return values, but the description still lacks some operational details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It adds meaningful semantics beyond the schema by explaining that 'url' is for 'the web page URL to analyze' and 'question' is 'the question to answer based on the page content', clarifying their roles. However, it does not provide format examples or constraints (e.g., URL validation, question phrasing). With 0% coverage and 2 parameters, this is above baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose as 'Answer questions about web page content using RAG' with specific verbs ('answer questions', 'combines web scraping with RAG', 'extracts relevant content sections') and distinguishes it from the sibling tool 'url_to_markdown_tool' by focusing on Q&A rather than conversion. It explicitly mentions the resource (web page content) and method (RAG).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use this tool ('to answer specific questions about web page content'), but does not explicitly state when not to use it or name alternatives. It implies usage for Q&A tasks involving web content, though lacks explicit exclusions or comparisons to the sibling tool beyond their different functions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

2 tool updatesv1.0.0Jan 3, 2026
- Changedurl_to_markdown_tool3 fields changed
  - removedInput schema / properties / url / title
```
Removed value: -"Url"
```
  - removedOutput schema / properties / result / title
```
Removed value: -"Result"
```
  - removedOutput schema / title
```
Removed value: -"_WrappedResult"
```
- Changedweb_content_qna4 fields changed
  - removedInput schema / properties / question / title
```
Removed value: -"Question"
```
  - removedInput schema / properties / url / title
```
Removed value: -"Url"
```
  - removedOutput schema / properties / result / title
```
Removed value: -"Result"
```
  - removedOutput schema / title
```
Removed value: -"_WrappedResult"
```
2 tool updatesAug 15, 2025
- First observedurl_to_markdown_tool
- First observedweb_content_qna

Deploy Server

RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.1/5.0

Scored 2026-04-18 09:20 across 2 tools

Disambiguation5/5

The two tools have clearly distinct purposes: one converts web pages to markdown for general content extraction, while the other answers specific questions about web page content using RAG. There is no overlap in functionality or ambiguity about when to use each tool.

Naming Consistency5/5

Both tools follow a consistent snake_case naming pattern with descriptive names that clearly indicate their function: url_to_markdown_tool and web_content_qna. The naming convention is uniform and predictable across the tool set.

Tool Count2/5

With only 2 tools, the server feels thin for a web analysis domain. While the tools cover basic extraction and Q&A, there are likely missing operations like content summarization, metadata extraction, or batch processing that would make the set more complete and useful for agents.

Completeness2/5

For a web analysis server, the tool surface has significant gaps. It lacks essential operations such as summarizing content, extracting structured data (e.g., tables, links), analyzing page structure, or handling multiple URLs. Agents will struggle with common web analysis tasks beyond the two provided tools.

Maintenance

ActivityInactive

ResponsivenessNo issues

Resources

GitHub Repository

Need Help?

Related Servers

Related MCP Connectors

mcp-webextratorOAuth
io.github.AceDataCloud
MCP server for web extraction and rendering via AceDataCloud WebExtrator
Web Scraping Browser Automation
2
RampifyOAuth
dev.rampify
SEO MCP server: crawl your site, find AI-visibility gaps, and ship the fix from your coding agent.
Marketing Web Scraping Developer Tools
23
hithereiamaliff-mcp-nextcloudOAuth
ai.smithery
A comprehensive Model Context Protocol (MCP) server that enables AI assistants to interact with yo…
Cloud Storage File Systems Workplace & Productivity
237
Jina Reader A
io.github.pipeworx-io
Jina AI Reader/Search MCP — turn any URL into clean LLM-ready markdown, plus web search.
Web Scraping Search
332

Related MCP Servers

MyAIServ MCP Server
RAG Systems Vector Databases
eagurin
A
license
Not graded
quality
D
maintenance
A high-performance FastAPI server supporting Model Context Protocol (MCP) for seamless integration with Large Language Models, featuring REST, GraphQL, and WebSocket APIs, along with real-time monitoring and vector search capabilities.
Updated a year ago (2025-03-14 21:51 UTC)
8
MIT
ScrapeGraph MCP Serverofficial
Web Scraping RAG Systems Browser Automation
ScrapeGraphAI
A
license
A
quality
B
maintenance
A production-ready Model Context Protocol server that enables language models to leverage AI-powered web scraping capabilities, offering tools for transforming webpages to markdown, extracting structured data, and executing AI-powered web searches.
Updated 2 months ago (2026-07-17 08:36 UTC)
8
106
MIT
Better Fetch
Browser Automation Web Scraping RAG Systems
flutterninja9
A
license
B
quality
D
maintenance
A Model Context Protocol server that intelligently fetches and processes web content, transforming websites and documentation into clean, structured markdown with nested URL crawling capabilities.
Updated 6 months ago (2026-03-05 03:16 UTC)
2
10
9
MIT
URL Agent
Browser Automation Web Scraping
rossja
A
license
Not graded
quality
D
maintenance
MCP server for intelligent web crawling using a ReAct agent. Adaptively explores websites and returns structured analysis.
Updated 7 months ago (2026-02-18 06:02 UTC)
Apache 2.0

🔍 Web Analyzer MCP

✨ Features

🎯 Core Tools

🚀 Key Features

📦 Installation

Prerequisites

🚀 Quick Start with uv (Recommended)

Installing via Smithery

IDE/Editor Integration

🎛️ Tool Descriptions

url_to_markdown

web_content_qna

🏗️ Architecture

Content Extraction Pipeline

Q&A Processing Pipeline

🏗️ Project Structure

🛠️ Development

Modern Development with uv

Alternative: Traditional Python Development

🤝 Contributing

📋 Roadmap

⚠️ Limitations

📄 License

🙋‍♂️ Support

🌟 Acknowledgments

Available Tools

Output Schema

TDQS

Output Schema

TDQS

Tool Schema Changelog

TDQS

Maintenance

Resources

Related MCP Connectors

Related MCP Servers

MyAIServ MCP Server

ScrapeGraph MCP Serverofficial

Better Fetch

URL Agent

`url_to_markdown`

`web_content_qna`