Skip to main content
Glama
himly0302

Web Analysis MCP

by himly0302

Web Analysis MCP

MCP 服务 - 使用 SearXNG 搜索 + Creeper 爬虫,通过 LLM 总结网页内容后返回,避免 token 超限问题。

特性

  • 🔍 SearXNG 智能搜索 + 域名过滤

  • 🧠 LLM 智能过滤 + 主题分类

  • 🕷️ Creeper 高效爬虫 + 并发处理

  • 🧠 DeepSeek LLM 智能总结

  • ⚡ 缓存优化 + 批量处理

Related MCP server: DuckDuckGo MCP Server

快速开始

1. 安装依赖

git clone <repository-url>
cd web-analysis-mcp
npm install

2. 部署外部服务

# SearXNG 搜索引擎
docker run -d -p 8086:8080 searxng/searxng

# Creeper 爬虫
git clone https://github.com/ihub-tech/Creeper.git /home/lyf/workspaces/creeper
cd /home/lyf/workspaces/creeper && pip install -r requirements.txt

3. 配置环境变量

cp .env.example .env
# 编辑 .env 文件,配置以下必需项:
SEARXNG_BASE_URL=http://127.0.0.1:8086
CREEPER_PATH=/home/lyf/workspaces/creeper
SUMMARY_API_KEY=sk-your-deepseek-key-here

# 可选配置
FILTER_MAX_RESULTS=8
DOMAIN_BLACKLIST=pinterest.com,facebook.com,instagram.com
DOMAIN_WHITELIST=github.com,stackoverflow.com

# LLM 智能过滤(可选)
FILTER_LLM_ENABLED=true
FILTER_LLM_API_KEY=sk-your-filter-api-key
FILTER_LLM_BASE_URL=https://api.deepseek.com
FILTER_LLM_MODEL=glm-4.5-air

4. 获取 API Key

5. 运行

# 开发模式
npm run dev

# 生产模式
npm run build && npm start

6. Claude Desktop 配置

{
  "mcpServers": {
    "web-analysis": {
      "command": "node",
      "args": ["/path/to/web-analysis-mcp/dist/index.js"],
      "env": {
        "SEARXNG_BASE_URL": "http://127.0.0.1:8086",
        "CREEPER_PATH": "/home/lyf/workspaces/creeper",
        "SUMMARY_API_KEY": "sk-your-key"
      }
    }
  }
}

可用工具

工具

描述

主要参数

web_search

智能搜索 + 爬取 + 总结

query, max_results, include_domains, exclude_domains

智能过滤机制

规则过滤

  • 黑名单过滤: 排除低质量域名

  • 白名单优先: 白名单结果排在前面

  • 数量限制: 保留指定数量的高质量结果

  • 去重机制: 移除重复链接

LLM 智能过滤(可选)

启用后使用 LLM 进行主题分类和关联性过滤:

  • 识别 10 种主题类型:编程、军事、政治、历史、科学、科技、娱乐、体育、金融、其他

  • 单次调用完成主题分类和结果筛选

  • 大幅降低后续总结成本

# 规则过滤配置
FILTER_MAX_RESULTS=8
DOMAIN_BLACKLIST=pinterest.com,facebook.com
DOMAIN_WHITELIST=github.com,stackoverflow.com

# LLM 过滤配置
FILTER_LLM_ENABLED=true
FILTER_LLM_MODEL=glm-4.5-air

测试

# 安装测试工具
npm install -g @modelcontextprotocol/inspector

# 开发模式测试(推荐)
npm run test:dev

# 生产模式测试
npm run test:inspector

故障排除

常见问题

SearXNG 连接失败

curl http://127.0.0.1:8086/search?q=test&format=json

Creeper 错误

cd /path/to/creeper && python creeper.py --help

调试模式

LOG_LEVEL=debug npm run dev

文档链接

License

MIT

Available Tools

1 tool

TDQS

B3.2/5.0
Disambiguation5/5

With only one tool, there is no possibility of ambiguity or overlap between tools. The tool's purpose is clearly distinct as it is the only one available.

Naming Consistency5/5

A single tool inherently has perfect naming consistency, as there are no other tools to compare against. The name 'web_search' follows a clear verb_noun pattern.

Tool Count2/5

A single tool for a server named 'Web Analysis MCP' feels thin and under-scoped. The tool performs web search and summarization, but the server name suggests broader analysis capabilities (e.g., link analysis, content extraction, performance metrics) that are not covered.

Completeness2/5

The server's domain appears to be web analysis, but with only a search and summarization tool, there are significant gaps. Missing operations might include analyzing page structure, extracting specific data, checking SEO metrics, or monitoring web changes, making the surface severely incomplete for the implied purpose.

Maintenance

ActivityInactive
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/himly0302/web-analysis-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server