# 网页抓取服务器
一个功能强大的MCP服务器，提供不受robots.txt限制的网页抓取功能，支持多种HTTP方法和自定义请求设置。
A powerful MCP server that offers web scraping capabilities unrestricted by robots.txt, supports multiple HTTP methods and custom request Settings.## 工具列表 Tool List
本MCP服务封装下列工具，可让模型通过标准化接口调用以下功能。 本MCP服务封装下列工具，可让模型通过标准化接口调用以下功能。
| 工具 Tool | 描述 Description |
|-------|--------------------|
| fetch | Fetch content from a URL without following robots.txt restrictions. Returns the response body as text. |
## 检查服务 ## Inspector
工具在线测试： [https://mcp.xiaobenyang.com/inspector/1777316659329027](https://mcp.xiaobenyang.com/inspector/1777316659329027)
Online Tool test [https://mcp.xiaobenyang.com/inspector/1777316659329027](https://mcp.xiaobenyang.com/inspector/1777316659329027)
## 服务配置 MCP Server Config
> #### 如何获取 XBY-APIKEY ？ How to get XBY-APIKEY ?
> 访问小笨羊科技网站 [https://xiaobenyang.com](https://xiaobenyang.com)，注册用户即可获得APIKEY
> Visit XiaoBenYang website [https://xiaobenyang.com](https://xiaobenyang.com), register and get the APIKEY.
### SSE
```json
{
"mcpServers": {
"网页抓取服务器": {
"headers": {
"XBY-APIKEY": "<YOUR_XBY_APIKEY>"
},
"type": "sse",
"url": "https://mcp.xiaobenyang.com/1777316659329027/sse"
}
}
}
```
### STREAMABLE HTTP
```json
{
"mcpServers": {
"网页抓取服务器": {
"headers": {
"XBY-APIKEY": "<YOUR_XBY_APIKEY>"
},
"type": "streamable_http",
"url": "https://mcp.xiaobenyang.com/1777316659329027/mcp"
}
}
}
```
### STDIO
```json
{
"mcpServers": {
"网页抓取服务器": {
"command": "npx",
"args": [
"-y",
"xiaobenyang-mcp"
],
"env": {
"XBY_APIKEY": "<YOUR_XBY_APIKEY>",
"mcpId": "1777316659329027",
},
"transport": "stdio"
}
}
}
```
