MCP Webscan Server
MCP 网络扫描服务器
用于 Web 内容扫描和分析的模型上下文协议 (MCP) 服务器。该服务器提供从网页获取、分析和提取信息的工具。
特征
页面抓取:将网页转换为 Markdown 格式,方便分析
链接提取:从网页中提取并分析链接
网站爬取:递归爬取网站以发现内容
链接检查:识别网页上的断开链接
模式匹配:查找与特定模式匹配的 URL
站点地图生成:为网站生成 XML 站点地图
Related MCP server: Open Crawler MCP Server
安装
通过 Smithery 安装
要通过Smithery自动安装 Claude Desktop 的 Webscan:
npx -y @smithery/cli install mcp-server-webscan --client claude手动安装
# Clone the repository
git clone <repository-url>
cd mcp-server-webscan
# Install dependencies
npm install
# Build the project
npm run build用法
启动服务器
npm start该服务器在 stdio 传输上运行,使其与 Claude Desktop 等 MCP 客户端兼容。
可用工具
fetch-page获取网页并将其转换为 Markdown。
参数:
url(必需):要获取的页面的 URL。selector(可选):CSS 选择器用于定位特定内容。
extract-links从网页中提取所有链接及其文本。
参数:
url(必需):要分析的页面的 URL。baseUrl(可选):过滤链接的基本 URL。limit(可选,默认值:100):返回的最大链接数。
crawl-site递归地爬取网站直至指定深度。
参数:
url(必填):要抓取的起始 URL。maxDepth(可选,默认值:2):最大爬行深度(0-5)。
check-links检查页面上的断开的链接。
参数:
url(必需):检查链接的 URL。
find-patterns查找与特定模式匹配的 URL。
参数:
url(必填):要搜索的 URL。pattern(必需):与 JavaScript 兼容的正则表达式模式,用于匹配 URL。
generate-site-map通过爬取生成一个简单的 XML 站点地图。
参数:
url(必需):站点地图抓取的根 URL。maxDepth(可选,默认值:2):发现 URL 的最大抓取深度(0-5)。limit(可选,默认值:1000):站点地图中包含的最大 URL 数量。
Claude Desktop 的使用示例
在 Claude Desktop 设置中配置服务器:
{
"mcpServers": {
"webscan": {
"command": "node",
"args": ["path/to/mcp-server-webscan/build/index.js"], // Corrected path
"env": {
"NODE_ENV": "development",
"LOG_LEVEL": "info" // Example: Set log level via env var
}
}
}
}在对话中使用这些工具:
Could you fetch the content from https://example.com and convert it to Markdown?发展
先决条件
Node.js >= 18
npm
项目结构(重构后)
mcp-server-webscan/
├── src/
│ ├── config/
│ │ └── ConfigurationManager.ts
│ ├── services/
│ │ ├── CheckLinksService.ts
│ │ ├── CrawlSiteService.ts
│ │ ├── ExtractLinksService.ts
│ │ ├── FetchPageService.ts
│ │ ├── FindPatternsService.ts
│ │ ├── GenerateSitemapService.ts
│ │ └── index.ts
│ ├── tools/
│ │ ├── checkLinksTool.ts
│ │ ├── checkLinksToolParams.ts
│ │ ├── crawlSiteTool.ts
│ │ ├── crawlSiteToolParams.ts
│ │ ├── extractLinksTool.ts
│ │ ├── extractLinksToolParams.ts
│ │ ├── fetchPageTool.ts
│ │ ├── fetchPageToolParams.ts
│ │ ├── findPatterns.ts
│ │ ├── findPatternsToolParams.ts
│ │ ├── generateSitemapTool.ts
│ │ ├── generateSitemapToolParams.ts
│ │ └── index.ts
│ ├── types/
│ │ ├── checkLinksTypes.ts
│ │ ├── crawlSiteTypes.ts
│ │ ├── extractLinksTypes.ts
│ │ ├── fetchPageTypes.ts
│ │ ├── findPatternsTypes.ts
│ │ ├── generateSitemapTypes.ts
│ │ └── index.ts
│ ├── utils/
│ │ ├── errors.ts
│ │ ├── index.ts
│ │ ├── logger.ts
│ │ ├── markdownConverter.ts
│ │ └── webUtils.ts
│ ├── initialize.ts
│ └── index.ts # Main server entry point
├── build/ # Compiled JavaScript (Corrected)
├── node_modules/
├── .clinerules
├── .gitignore
├── Dockerfile
├── LICENSE
├── mcp-consistant-servers-guide.md
├── package.json
├── package-lock.json
├── README.md
├── RFC-2025-001-Refactor.md
├── smithery.yaml
└── tsconfig.json建筑
npm run build开发模式
npm run dev错误处理
服务器实现了全面的错误处理:
参数无效
网络错误
内容解析错误
URL 验证
所有错误均根据 MCP 规范正确格式化。
贡献
分叉存储库
创建你的功能分支(
git checkout -b feature/amazing-feature)提交您的更改(
git commit -m 'Add some amazing feature')推送到分支(
git push origin feature/amazing-feature)打开拉取请求
执照
MIT 许可证 - 详情请参阅许可证文件
Available Tools
6 toolscheck-linksA
Fetches a given URL, extracts all anchor ('a') links, and checks each linked URL for validity (reachability via HEAD request). Returns a list of checked links with their status ('valid', 'broken', or 'invalid_url' if the href attribute couldn't be resolved to a full URL).
Inputs: url (required).
crawl-siteA
Recursively crawls a website starting from a given URL up to a specified maximum depth. It follows links within the same origin and returns a list of all unique URLs found during the crawl.
Inputs: url (required), maxDepth.
extract-linksA
Extract and analyze all hyperlinks from a web page, organizing them into a structured format with URLs, anchor text, and contextual information. Performance-optimized with stream processing and worker threads for efficient handling of large pages. Works with either a direct URL or raw HTML content. Handles relative and absolute URLs properly by supporting an optional base URL parameter. Results can be limited to prevent overwhelming output for link-dense pages. Returns a comprehensive link inventory that includes destination URLs, link text, titles (if available), and whether links are internal or external to the source domain. Useful for site mapping, content analysis, broken link checking, SEO analysis, and as a preparatory step for targeted crawling operations.
Inputs: url (required), baseUrl, limit.
fetch-pageA
Fetches the HTML content of a given URL. Optionally, it can select a specific part of the HTML using a CSS selector and convert only that part (or the whole body if no selector is provided) to Markdown format. Returns the resulting Markdown text.
Inputs: url (required), selector.
find-patternsA
Fetches a web page, extracts all anchor ('a') links, resolves their absolute URLs, and returns a list of links whose URLs match a given JavaScript-compatible regular expression pattern. Includes the URL and anchor text for each match.
Inputs: pattern (required), url (required).
generate-site-mapA
Crawls a website starting from a given URL up to a specified depth and generates an XML sitemap containing the discovered URLs (up to a specified limit).
Inputs: url (required), limit, maxDepth.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 6 tools
There is significant overlap between tools, particularly check-links, extract-links, and find-patterns, which all involve extracting and analyzing links from web pages, leading to potential confusion. However, crawl-site and generate-site-map have more distinct purposes for recursive crawling and sitemap generation, respectively, and fetch-page stands out for content retrieval and conversion.
All tool names follow a consistent kebab-case pattern with clear verb-noun structures (e.g., check-links, crawl-site, extract-links). This uniformity makes the tool set predictable and easy to navigate, with no deviations in naming style.
With 6 tools, the count is well-scoped for a web scanning server, covering key operations like link checking, crawling, content fetching, and sitemap generation. Each tool has a clear role, avoiding bloat while providing comprehensive coverage for the domain.
The tool set covers core web scanning tasks such as link extraction, validation, crawling, and sitemap generation, with fetch-page adding content retrieval and conversion. A minor gap exists in lacking tools for security scanning (e.g., vulnerability checks) or performance analysis, but the provided tools support most common workflows effectively.
Maintenance
Related MCP Connectors
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
On-page crawl, tech-stack detection, DNS, SSL and WHOIS as agent tools.
Web extraction MCP: page content, metadata, articles, contacts, email checks, URL probes.
Fetch web/social data, screenshots, scrape/extract content, generate images, audit SEO & links.
Related MCP Servers
- FlicenseNot gradedqualityNot gradedmaintenanceEnables querying and retrieving content from webpages by parsing sitemap.xml files and fetching HTML content from specified URLs. Includes rate limiting for abuse protection.-
- AlicenseNot gradedqualityCmaintenanceEnables web crawling and content extraction from web pages, supporting multiple output formats like text, markdown, XML, and JSON, with robots.txt compliance and rate limiting.14 npm1MIT
- AlicenseAqualityDmaintenanceEnables web scraping of URLs, including extracting text, links, images, metadata, taking screenshots, and using CSS selectors for structured data extraction.610 npmMIT
- FlicenseAqualityCmaintenanceEnables fetching, crawling, and analyzing web pages with 29 tools for SEO audits, content extraction, and more.29-