# Spider MCP Server
An MCP server for crawling and spidering documentation websites, extracting clean text content, and using LLM-powered analysis to provide intelligent summaries and context through the Model Context Protocol.
## Features
- Crawl entire documentation websites with configurable depth limits
- Extract clean text content from HTML pages using multiple extraction methods
- Intelligent content parsing that removes navigation, ads, and other noise
- LLM-powered content analysis using Anthropic Claude (Haiku/Sonnet)
- Automatic content summarization and key point extraction
- Intelligent code example extraction and categorization
- Code example detection across multiple programming languages
- Intelligent link discovery and relevance ranking
- Content type classification (tutorial, reference, API docs, etc.)
- Respect robots.txt and implement proper crawling etiquette
- File-based caching with TTL and compression
- Search through cached documentation content with LLM enhancement
- Concurrent crawling with rate limiting
- Support for various documentation layouts and formats
## Installation
```bash
git clone <repository-url>
cd spider-mcp
bun install
cp .env.example .env
```
## Usage
### As MCP Server
Run the server to expose MCP tools:
```bash
bun run dev
```
### MCP Client Integration
This server implements the Model Context Protocol (MCP) specification and communicates via stdio transport. MCP enables AI assistants to securely access external tools and data sources. To use this server, you need an MCP-compatible client like Claude Desktop, or you can integrate it programmatically using the MCP SDK.
#### Claude Desktop Integration
Add to your Claude Desktop MCP configuration:
```json
{
"mcpServers": {
"spider-mcp": {
"command": "bun",
"args": ["run", "/path/to/spider-mcp/src/index.ts"],
"env": {
"ANTHROPIC_API_KEY": "your_api_key_here"
}
}
}
}
```
#### Programmatic Usage Examples
```javascript
// Example using @modelcontextprotocol/sdk
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
const client = new Client({
name: "spider-client",
version: "1.0.0"
}, {
capabilities: {}
});
// Connect to the spider MCP server
const transport = new StdioClientTransport({
command: "bun",
args: ["run", "/path/to/spider-mcp/src/index.ts"]
});
await client.connect(transport);
// Extract code examples from documentation
const result = await client.request({
method: "tools/call",
params: {
name: "spider_docs",
arguments: {
url: "https://docs.anthropic.com/en/docs/quickstart",
enable_llm_analysis: true,
llm_analysis_type: "code_examples",
max_depth: 2
}
}
});
```
### Available Tools
#### spider_docs
Crawl a documentation website and cache the content with optional LLM analysis.
Parameters:
- `url` (required): Base URL of the documentation site
- `max_depth` (optional): Maximum crawl depth (default: 3)
- `include_patterns` (optional): URL patterns to include
- `exclude_patterns` (optional): URL patterns to exclude
- `enable_llm_analysis` (optional): Enable LLM-powered content analysis
- `llm_analysis_type` (optional): Type of analysis:
- `full` - Complete analysis with summary, key points, links, and code examples
- `summary` - Content summarization only
- `links` - Link analysis and relevance ranking
- `classification` - Content type classification
- `code_examples` - Extract and categorize code examples only
**Example with code extraction:**
```json
{
"url": "https://docs.example.com/api",
"max_depth": 2,
"enable_llm_analysis": true,
"llm_analysis_type": "code_examples"
}
```
#### get_page
Retrieve a specific page from the cache.
Parameters:
- `url` (required): URL of the page to retrieve
#### search_docs
Search through cached documentation content.
Parameters:
- `query` (required): Search query
- `limit` (optional): Maximum results to return (default: 10)
#### list_pages
List all cached pages with optional filtering and sorting.
Parameters:
- `filter` (optional): Filter pages by URL pattern
- `sort` (optional): Sort field (url, title, timestamp)
- `order` (optional): Sort order (asc, desc)
#### clear_cache
Clear cached pages matching a pattern.
Parameters:
- `url_pattern` (optional): Pattern to match for clearing
#### analyze_content
Perform LLM-powered analysis on a specific cached page.
Parameters:
- `url` (required): URL of the page to analyze
- `analysis_type` (optional): Same options as `spider_docs` above
**Example for code extraction:**
```json
{
"url": "https://docs.example.com/api/users",
"analysis_type": "code_examples"
}
```
#### get_summary
Get an intelligent summary of a cached page.
Parameters:
- `url` (required): URL of the page to summarize
- `summary_length` (optional): Length of summary (short, medium, long)
- `focus_areas` (optional): Specific areas to focus on in the summary
## Configuration
Configuration can be provided through:
1. Environment variables (see `.env.example`)
2. JSON configuration files in `config/` directory
3. Runtime parameters passed to tools
### LLM Integration
To enable LLM-powered content analysis, set your Anthropic API key:
```bash
export ANTHROPIC_API_KEY=your_api_key_here
```
The server will automatically detect the API key and enable LLM features. Without an API key, the server operates in basic mode with programmatic content extraction only.
### Example Configuration
```json
{
"maxDepth": 3,
"maxPages": 100,
"concurrency": 5,
"userAgent": "SpiderMCP/1.0 Documentation Crawler",
"timeout": 10000,
"retryAttempts": 3,
"cacheTTL": 86400000,
"respectRobotsTxt": true,
"includePatterns": ["/docs/*", "/api/*"],
"excludePatterns": ["/blog/*", "*.pdf"]
}
```
## Development
```bash
bun test
bun run typecheck
bun run build
```
### Testing the Spider
Use the included test script to try out the functionality locally:
```bash
# Test with code example extraction on a documentation site
bun run test-spider.ts https://docs.anthropic.com/en/docs/quickstart
# Test with a specific site focusing on code examples
ANTHROPIC_API_KEY=your_key bun run test-spider.ts https://docs.example.com
# Test code-only extraction
bun run test-spider.ts https://docs.python.org/3/tutorial/
```
The test script demonstrates the spider functionality outside of MCP and will show you:
- Crawled pages with metadata
- Extracted code examples with languages and categories
- LLM analysis results including summaries and classifications
- Cache statistics and performance metrics
**Note**: The test script uses the spider functionality directly, not through MCP protocol.
## Architecture
- `src/spider/` - Core crawling and parsing logic
- `src/mcp/` - MCP server implementation and tool handlers
- `src/extractors/` - Content extraction strategies
- `src/llm/` - LLM integration and content analysis
- `src/utils/` - Utility functions and configuration
- `cache/` - File system cache storage
- `config/` - Configuration files
## Extraction Methods
The server supports multiple content extraction methods:
1. **Readability** - Mozilla Readability algorithm for article extraction
2. **Cheerio** - Custom CSS selector-based extraction
3. **Markdown** - HTML to Markdown conversion with Turndown
4. **LLM Analysis** - Anthropic Claude for intelligent content understanding and code extraction
## Code Example Extraction
The LLM analysis can intelligently extract and categorize code examples from documentation:
- **Language Detection**: Automatically detects programming languages (JavaScript, Python, bash, JSON, etc.)
- **Code Categories**:
- `api_call` - API requests, SDK calls, HTTP requests
- `configuration` - Config files, settings, environment setup
- `implementation` - Complete functions, classes, modules
- `usage_example` - How to use a feature or library
- `snippet` - Small code fragments or utilities
- `complete_example` - Full working examples or applications
- **Format Preservation**: Maintains exact formatting, indentation, and syntax
- **Context Awareness**: Provides meaningful descriptions for each code example
### Example Output Format
When using code example extraction, the output includes structured code examples:
```json
{
"codeExamples": [
{
"language": "javascript",
"code": "const response = await fetch('/api/users', {\n method: 'GET',\n headers: {\n 'Authorization': 'Bearer ' + apiKey\n }\n});",
"description": "Fetch users from API with authentication",
"category": "api_call"
},
{
"language": "python",
"code": "import requests\n\nresponse = requests.get('/api/users', headers={\n 'Authorization': f'Bearer {api_key}'\n})",
"description": "Python example for API authentication",
"category": "api_call"
},
{
"language": "json",
"code": "{\n \"database\": {\n \"host\": \"localhost\",\n \"port\": 5432\n }\n}",
"description": "Database configuration file",
"category": "configuration"
}
]
}
```
## Caching
- Two-tier caching: in-memory + file system
- Configurable TTL and cache size limits
- Automatic cleanup of expired entries
- Domain-based cache organization
## Robots.txt Compliance
- Fetches and parses robots.txt automatically
- Respects crawl-delay directives
- Honors disallow/allow rules
- Discovers sitemaps
## Rate Limiting
- Configurable concurrent request limits
- Exponential backoff for failed requests
- Per-domain crawl delays
- Timeout handling
## License
MIT
