web2kb
Allows Hermes AI to fetch web pages and search the knowledge base for relevant information.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@web2kbsearch for 'deployment process' in my knowledge base"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
web2kb
把网页变成大模型可检索的知识库。
Turn any webpage into a searchable knowledge base for LLMs.
Zero external dependencies — pure Python stdlib. Works on Linux, macOS, and Windows.
零外部依赖 — 纯 Python 标准库实现,全平台通用。
为什么用 web2kb?
内网机器不能联网?想把公司 Wiki / 文档站 / 技术博客变成 AI 助手的知识库?
web2kb 帮你:
抓取网页(支持内网地址)→ 自动转 Markdown → 按标题分块
索引到 SQLite FTS5 → 全文检索,支持中文
通过 CLI 或 MCP 协议查询 → 任何 AI 助手都能用
Related MCP server: Crawl4AI MCP
快速开始
# 抓取一个页面
python cli.py fetch https://example.com/docs
# 搜索
python cli.py search "部署流程"
# 列出所有来源
python cli.py list安装
Linux / macOS
git clone https://github.com/2578D/web2kb.git
cd web2kb
bash install.shWindows
git clone https://github.com/2578D/web2kb.git
cd web2kb
.\install.ps1 # 以管理员身份运行或者直接用 Python 调用:
python cli.py fetch http://内网地址/page功能
功能 | 说明 |
HTML → Markdown | 纯 Python 标准库实现,无需 |
按标题分块 | 根据 h1-h6 自动分割文档,保持语义完整性 |
FTS5 全文检索 | SQLite FTS5 引擎,支持中文(自动回退 LIKE 搜索) |
CLI 工具 | 终端下直接使用,零配置 |
MCP 服务器 | 通过 stdio 协议向任何 MCP 兼容的 AI 助手提供服务 |
导入本地文件 | 支持导入本地 HTML 文件 |
跨平台 | Linux / macOS / Windows,Python 3.9+ 即可运行 |
零外部依赖 | 只用 Python 标准库,无需 pip install |
使用指南
CLI 命令
# 抓取网页并索引
web2kb fetch https://example.com/docs
web2kb fetch https://example.com/docs --source my-docs
# 搜索知识库
web2kb search "数据库连接失败"
web2kb search "配置项" --source my-docs --limit 10
# 导入本地 HTML 文件
web2kb import ./page.html --source my-docs
# 列出所有来源
web2kb list
# 删除来源
web2kb delete my-docs
# 统计信息
web2kb statsMCP 服务器
启动 MCP 服务器(供 Claude Code、Cursor、OpenClaw、Hermes 等 AI 助手连接):
python server.pyMCP 工具
工具 | 参数 | 说明 |
|
| 抓取网页并索引 |
|
| 搜索知识库 |
| — | 列出所有已索引来源 |
|
| 删除某个来源 |
|
| 导入本地 HTML 文件 |
MCP 配置示例
Claude Code / Cursor / OpenClaw 配置:
{
"mcpServers": {
"web2kb": {
"command": "python3",
"args": ["/path/to/web2kb/server.py"]
}
}
}Windows 路径:
{
"mcpServers": {
"web2kb": {
"command": "python",
"args": ["D:\\web2kb\\server.py"]
}
}
}数据迁移
把知识库从一台机器搬到另一台机器:
# 只需搬运一个文件
scp ~/.web2kb/knowledge.db user@内网机器:~/.web2kb/数据库文件在 ~/.web2kb/knowledge.db,SQLite 格式,单文件,拿走即用。
工作原理
┌─────────────┐ ┌──────────────┐ ┌────────────────┐
│ 网页 URL │───>│ HTML → MD │───>│ SQLite FTS5 │
│ 本地 HTML │ │ 按标题分块 │ │ 全文索引 │
└─────────────┘ └──────────────┘ └───────┬────────┘
│
┌────────────────────┘
▼
┌──────────────────┐
│ MCP Server │ ← 任何 MCP 兼容的 AI 助手
│ CLI 工具 │ ← 终端直接使用
└──────────────────┘项目结构
web2kb/
├── core.py # 核心引擎:HTML 解析、分块、FTS5 索引、检索
├── server.py # MCP 服务器(JSON-RPC over stdio)
├── cli.py # CLI 入口
├── web2kb.bat # Windows 批处理包装
├── install.sh # Linux/macOS 安装脚本
├── install.ps1 # PowerShell 安装脚本
├── DESIGN.md # 设计文档
└── README.md # 本文件许可证
MIT
This server cannot be installed
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Servers
- Alicense-qualityDmaintenanceEnables AI agents to search, deep-read, and build knowledge bases from Markdown, PDF, DOCX, and PPTX documents via MCP tools for retrieval, document navigation, and ingestion.Last updated73612MIT
- FlicenseBqualityAmaintenanceEnables AI assistants to crawl websites, extract dynamic content, navigate links, and save structured Markdown files via the MCP protocol, with support for anti-bot bypass, CSS selectors, and custom JavaScript execution.Last updated143
- Flicense-qualityDmaintenanceCompiles web sources into structured, backlinked wiki pages in an Obsidian-compatible vault and enables querying the knowledge base via MCP.Last updated
- Flicense-qualityDmaintenanceEnables LLMs to fetch and process web page contents by providing tools to retrieve raw HTML, extract clean text, and convert web pages to markdown format.Last updated
Related MCP Connectors
Read a URL as clean markdown, screenshot a website, url to PDF. Web access for agents, no signup.
Jina AI Reader/Search MCP — turn any URL into clean LLM-ready markdown, plus web search.
Your memory, everywhere AI goes. Build knowledge once, access it via MCP anywhere.
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/2578D/web2kb'
If you have feedback or need assistance with the MCP directory API, please join our Discord server