🦈 DocShark

DocShark is a powerful MCP (Model Context Protocol) server designed to scrape, index, and search any documentation website. It creates a local, highly-searchable knowledge base from public documentation pages using FTS5 (Full-Text Search) and BM25 ranking, allowing AI assistants to query the latest docs effortlessly.

🚀 Features

• Automated Crawling: Discovers pages via sitemap.xml with fallback to BFS link crawling.

• Smart Extraction: Uses Readability and Turndown to extract main content and convert it to clean Markdown, filtering out navbars and sidebars.

• Semantic Chunking: Splits content based on headings, preserving contextual headers for better AI understanding.

• High-Performance Search: Built-in SQLite + FTS5 indexing with BM25 ranking for accurate and lightning-fast search results.

• JS-Rendered Site Support: Tiered fetching strategy automatically detects React/Vue SPAs (empty shells) and upgrades to puppeteer-core if you have it installed (zero-config, auto-fallback).

• Polite Crawling: Respects robots.txt and implements rate limiting to prevent overloading documentation servers.

• Standard MCP Tooling: Connect perfectly with Desktop Claude, VS Code, Cursor, and any other MCP-compatible clients via standard stdio or http/sse transports.

Related MCP server: Whoosh RAG MCP

📦 What We Have Done (Phase 1)

Phase 1: Core Engine is fully implemented and tested.

• ✅ Custom SQLite Database with FTS5 virtual tables and auto-sync triggers.

• ✅ Web scraping engine supporting standard fetch() and puppeteer-core.

• ✅ Markdown processor utilizing Readability + Turndown.

• ✅ Heading-based semantic chunker (500-1200 tokens per chunk).

• ✅ Asynchronous job manager and queue system.

• ✅ Complete HTTP API (REST endpoints + SSE event streams).

• ✅ Seamless integration of 4 MCP tools: manage_library, search_docs, list_libraries, and get_doc_page.

• ✅ Robust CLI interface (start, add, rename, search, list).

🏗️ What We Are Doing

We are actively polishing the integration between the core engine and external MCP clients (like VS Code Agents and Claude Desktop).

🔮 What We Plan To Do (Phase 2 & Beyond)

• Web Dashboard: An intuitive SvelteKit dashboard to manage your synced libraries, view crawl progress in real-time (via SSE), and test searches manually.

• Incremental Crawling: Smarter refresh jobs that compare ETag and Last-Modified headers to only re-scrape updated pages.

• Vector Search (RAG): Integration of lightweight vector embeddings for semantic similarity search alongside the existing FTS5 keyword search.

• Advanced Scraping Setup: Support for custom CSS selectors to define exactly where content lives in non-standard documentation websites.

🛠️ Usage

Quick Start (from npm)

You can run DocShark directly without installing it globally using bunx:

# Add a documentation library to the index
bunx docshark add https://valibot.dev/guides/ --depth 2

# Search your indexed docs
bunx docshark search "schema validation"

Installation

To install DocShark globally as a CLI tool:

DocShark is intended to be installed and run with Bun.

# Global Bun installation
bun add -g docshark

After installation, you can use the docshark command:

docshark list

# List libraries not crawled in 14+ days and refresh them interactively
docshark stale

# Update the global Bun installation when a new release is published
docshark update

# Script-friendly update check
docshark update --check --quiet

Interactive CLI runs will also let you know when a newer version is available. Update notices are intentionally skipped for MCP stdio mode so they never interfere with protocol output.

Interactive docshark list and docshark stale runs also check library freshness: anything not crawled in the last 14 days (configurable via DOCSHARK_STALE_DAYS) is listed with its age, followed by Refresh N stale libraries now? [y/N] — answering y re-crawls them all. Skip it with docshark list --no-stale-check, disable it entirely with DOCSHARK_DISABLE_STALE_CHECK=1, and note that non-interactive runs (pipes, CI) never prompt. AIs see the same signal over MCP via the Age/⚠️ columns of list_libraries and manage_library action=stale, so your assistant can tell you when your docs are stale too.

Icon styles

DocShark's CLI defaults to plain icons — common-monospace Unicode (✓ ✗ ⚠ ↻) that renders in any terminal font; when a style lacks a glyph (the shark, for example) it falls back to the emoji automatically. Change it from the CLI:

docshark icons               # show the effective style and where it comes from
docshark icons nerd          # persist a style to ~/.docshark/config.json
docshark --icons emoji list  # one-off override for a single command

Precedence: --icons flag → DOCSHARK_ICONS env → config file → plain. Valid styles: emoji | nerd | plain | none.

The nerd style uses Material Design Icons glyphs from Nerd Fonts; if your main font lacks them, install the Symbols Nerd Font Mono font from nerdfonts.com/font-downloads as a fallback (or just use plain). MCP tool output always keeps emoji, since AI chat clients render those reliably.

Responsive output

All CLI output fits your terminal width automatically: tables shrink their flexible columns (with … ellipses) instead of breaking, help text re-flows to a dynamic label column, and long messages wrap at word boundaries. The width is re-read from stdout.columns on every command, so after a resize the next command always lays out correctly. Force a fixed width with DOCSHARK_WIDTH=<20–500> (useful in scripts and CI, where the default becomes 80). Piped search/get output stays raw Markdown so scripts can consume it unchanged.

For scripts, docshark update --check exits 0 when current, 10 when a newer version is available, and 1 when the version check could not be completed.

🧠 Agent Skills

DocShark includes official Agent Skills available on the skills.sh registry. These skills teach AI assistants exactly how to set up, use, and troubleshoot the DocShark MCP server.

To install a skill directly into your AI coding assistant:

# Add the 'docshark' skill for using the MCP tools
npx skills add Michael-Obele/docshark --skill docshark

# Add the 'using-docshark' skill for setup and configuration help
npx skills add Michael-Obele/docshark --skill using-docshark

Skill Setup by Code Editor

The npx skills add CLI automatically configures skills for most editors, but here is how they integrate:

• Cursor: Skills are added to .cursor/rules/

• Windsurf: Skills are added to .windsurfrules

• VS Code (Cline / Roo Code): Skills are added to .clinerules or .roomodes

• Trae: Skills are added to .trae/skills/

• GitHub Copilot: Skills are appended to .github/copilot-instructions.md

Check out the skills/README.md for detailed workflows on how these skills optimize your AI coding experience.

🔌 MCP Integration

VS Code (GitHub Copilot / MCP Extension)

Add DocShark to your .vscode/settings.json or global MCP configuration:

{
  "mcpServers": {
    "docshark": {
      "command": "bunx",
      "args": ["-y", "docshark", "start", "--stdio"]
    }
  }
}

Cursor

1. Open Cursor Settings > Models > MCP.

2. Click + Add New MCP Server.

3. Name: docshark

4. Type: command

5. Command: bunx -y docshark start --stdio

Claude Desktop

Edit your Claude Desktop configuration file:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

• Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "docshark": {
      "command": "bunx",
      "args": ["-y", "docshark", "start", "--stdio"]
    }
  }
}

🛠️ Development

Local Setup

Ensure you have Bun installed.

# Clone the repository
git clone https://github.com/Michael-Obele/docshark.git
cd docshark

# Install dependencies
bun install

# (Optional) Enable auto-detection & scraping of Javascript React/Vue single-page apps
bun add puppeteer-core

# Start the DocShark MCP server in HTTP mode for local testing
bun run src/cli.ts start --port 6380

Local CLI Debugging

# Run CLI directly while developing
bun run src/cli.ts list

Tests

Run the core regression suite before merging or publishing changes:

# From the repo root
pnpm test:core

# Or from packages/core
bun test scripts/*.test.ts

The suite covers the current core engine surfaces: SQLite storage and migrations, library management, extraction, chunking, search, crawl helpers, API routes, and MCP tool wrappers.

🔄 Versioning & Changelog

This project uses Google's Release Please to automate versioning and changelog generation.

• Semantic Versioning: Our versions automatically bump (e.g. 0.0.1 -> 0.0.2 or 0.1.0) based on standard Conventional Commits (feat:, fix:, chore:, etc.).

• Automated: A PR is automatically created on master when standard commits are merged, generating a standard CHANGELOG.md.

📜 License

This project is open-source and available under the MIT License.

Built to empower AI agents with the latest knowledge.