Spider MCP Server

Name: Spider MCP Server
Author: oeo

An MCP server for crawling and spidering documentation websites, extracting clean text content, and using LLM-powered analysis to provide intelligent summaries and context through the Model Context Protocol.

Features

Crawl entire documentation websites with configurable depth limits
Extract clean text content from HTML pages using multiple extraction methods
Intelligent content parsing that removes navigation, ads, and other noise
LLM-powered content analysis using Anthropic Claude (Haiku/Sonnet)
Automatic content summarization and key point extraction
Intelligent code example extraction and categorization
Code example detection across multiple programming languages
Intelligent link discovery and relevance ranking
Content type classification (tutorial, reference, API docs, etc.)
Respect robots.txt and implement proper crawling etiquette
File-based caching with TTL and compression
Search through cached documentation content with LLM enhancement
Concurrent crawling with rate limiting
Support for various documentation layouts and formats

Installation

git clone <repository-url> cd spider-mcp bun install cp .env.example .env

Usage

As MCP Server

Run the server to expose MCP tools:

bun run dev

MCP Client Integration

This server implements the Model Context Protocol (MCP) specification and communicates via stdio transport. MCP enables AI assistants to securely access external tools and data sources. To use this server, you need an MCP-compatible client like Claude Desktop, or you can integrate it programmatically using the MCP SDK.

Claude Desktop Integration

Add to your Claude Desktop MCP configuration:

{ "mcpServers": { "spider-mcp": { "command": "bun", "args": ["run", "/path/to/spider-mcp/src/index.ts"], "env": { "ANTHROPIC_API_KEY": "your_api_key_here" } } } }

Programmatic Usage Examples

// Example using @modelcontextprotocol/sdk import { Client } from '@modelcontextprotocol/sdk/client/index.js'; import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js'; const client = new Client({ name: "spider-client", version: "1.0.0" }, { capabilities: {} }); // Connect to the spider MCP server const transport = new StdioClientTransport({ command: "bun", args: ["run", "/path/to/spider-mcp/src/index.ts"] }); await client.connect(transport); // Extract code examples from documentation const result = await client.request({ method: "tools/call", params: { name: "spider_docs", arguments: { url: "https://docs.anthropic.com/en/docs/quickstart", enable_llm_analysis: true, llm_analysis_type: "code_examples", max_depth: 2 } } });

Available Tools

spider_docs

Crawl a documentation website and cache the content with optional LLM analysis.

Parameters:

url (required): Base URL of the documentation site
max_depth (optional): Maximum crawl depth (default: 3)
include_patterns (optional): URL patterns to include
exclude_patterns (optional): URL patterns to exclude
enable_llm_analysis (optional): Enable LLM-powered content analysis
llm_analysis_type (optional): Type of analysis:
- full - Complete analysis with summary, key points, links, and code examples
- summary - Content summarization only
- links - Link analysis and relevance ranking
- classification - Content type classification
- code_examples - Extract and categorize code examples only

Example with code extraction:

{ "url": "https://docs.example.com/api", "max_depth": 2, "enable_llm_analysis": true, "llm_analysis_type": "code_examples" }

get_page

Retrieve a specific page from the cache.

Parameters:

url (required): URL of the page to retrieve

search_docs

Search through cached documentation content.

Parameters:

query (required): Search query
limit (optional): Maximum results to return (default: 10)

list_pages

List all cached pages with optional filtering and sorting.

Parameters:

filter (optional): Filter pages by URL pattern
sort (optional): Sort field (url, title, timestamp)
order (optional): Sort order (asc, desc)

clear_cache

Clear cached pages matching a pattern.

Parameters:

url_pattern (optional): Pattern to match for clearing

analyze_content

Perform LLM-powered analysis on a specific cached page.

Parameters:

url (required): URL of the page to analyze
analysis_type (optional): Same options as spider_docs above

Example for code extraction:

{ "url": "https://docs.example.com/api/users", "analysis_type": "code_examples" }

get_summary

Get an intelligent summary of a cached page.

Parameters:

url (required): URL of the page to summarize
summary_length (optional): Length of summary (short, medium, long)
focus_areas (optional): Specific areas to focus on in the summary

Configuration

Configuration can be provided through:

Environment variables (see .env.example)
JSON configuration files in config/ directory
Runtime parameters passed to tools

LLM Integration

To enable LLM-powered content analysis, set your Anthropic API key:

export ANTHROPIC_API_KEY=your_api_key_here

The server will automatically detect the API key and enable LLM features. Without an API key, the server operates in basic mode with programmatic content extraction only.

Example Configuration

{ "maxDepth": 3, "maxPages": 100, "concurrency": 5, "userAgent": "SpiderMCP/1.0 Documentation Crawler", "timeout": 10000, "retryAttempts": 3, "cacheTTL": 86400000, "respectRobotsTxt": true, "includePatterns": ["/docs/*", "/api/*"], "excludePatterns": ["/blog/*", "*.pdf"] }

Development

bun test bun run typecheck bun run build

Testing the Spider

Use the included test script to try out the functionality locally:

# Test with code example extraction on a documentation site bun run test-spider.ts https://docs.anthropic.com/en/docs/quickstart # Test with a specific site focusing on code examples ANTHROPIC_API_KEY=your_key bun run test-spider.ts https://docs.example.com # Test code-only extraction bun run test-spider.ts https://docs.python.org/3/tutorial/

The test script demonstrates the spider functionality outside of MCP and will show you:

Crawled pages with metadata
Extracted code examples with languages and categories
LLM analysis results including summaries and classifications
Cache statistics and performance metrics

Note: The test script uses the spider functionality directly, not through MCP protocol.

Architecture

src/spider/ - Core crawling and parsing logic
src/mcp/ - MCP server implementation and tool handlers
src/extractors/ - Content extraction strategies
src/llm/ - LLM integration and content analysis
src/utils/ - Utility functions and configuration
cache/ - File system cache storage
config/ - Configuration files

Extraction Methods

The server supports multiple content extraction methods:

Readability - Mozilla Readability algorithm for article extraction
Cheerio - Custom CSS selector-based extraction
Markdown - HTML to Markdown conversion with Turndown
LLM Analysis - Anthropic Claude for intelligent content understanding and code extraction

Code Example Extraction

The LLM analysis can intelligently extract and categorize code examples from documentation:

Language Detection: Automatically detects programming languages (JavaScript, Python, bash, JSON, etc.)
Code Categories:
- api_call - API requests, SDK calls, HTTP requests
- configuration - Config files, settings, environment setup
- implementation - Complete functions, classes, modules
- usage_example - How to use a feature or library
- snippet - Small code fragments or utilities
- complete_example - Full working examples or applications
Format Preservation: Maintains exact formatting, indentation, and syntax
Context Awareness: Provides meaningful descriptions for each code example

Example Output Format

When using code example extraction, the output includes structured code examples:

{ "codeExamples": [ { "language": "javascript", "code": "const response = await fetch('/api/users', {\n method: 'GET',\n headers: {\n 'Authorization': 'Bearer ' + apiKey\n }\n});", "description": "Fetch users from API with authentication", "category": "api_call" }, { "language": "python", "code": "import requests\n\nresponse = requests.get('/api/users', headers={\n 'Authorization': f'Bearer {api_key}'\n})", "description": "Python example for API authentication", "category": "api_call" }, { "language": "json", "code": "{\n \"database\": {\n \"host\": \"localhost\",\n \"port\": 5432\n }\n}", "description": "Database configuration file", "category": "configuration" } ] }

Caching

Two-tier caching: in-memory + file system
Configurable TTL and cache size limits
Automatic cleanup of expired entries
Domain-based cache organization

Robots.txt Compliance

Fetches and parses robots.txt automatically
Respects crawl-delay directives
Honors disallow/allow rules
Discovers sitemaps

Rate Limiting

Configurable concurrent request limits
Exponential backoff for failed requests
Per-domain crawl delays
Timeout handling

License

MIT

Spider MCP Server

Spider MCP Server

Features

Installation

Usage

As MCP Server

MCP Client Integration

Claude Desktop Integration

Programmatic Usage Examples

Available Tools

spider_docs

get_page

search_docs

list_pages

clear_cache

analyze_content

get_summary

Configuration

LLM Integration

Example Configuration

Development

Testing the Spider

Architecture

Extraction Methods

Code Example Extraction

Example Output Format

Caching

Robots.txt Compliance

Rate Limiting

License

Resources

New MCP Servers

Latest Blog Posts

MCP directory API