# 🍓 BerryRAG: Local Vector Database with Playwright MCP Integration

A complete local RAG (Retrieval-Augmented Generation) system that integrates Playwright MCP web scraping with vector database storage for Claude.

## ✨ Features

- **Zero-cost self-hosted** vector database
- **Playwright MCP integration** for automated web scraping
- **Multiple embedding providers** (sentence-transformers, OpenAI, fallback)
- **Smart content processing** with quality filters
- **Claude-optimized** context formatting
- **MCP server** for direct Claude integration
- **Command-line tools** for manual operation

## 🚀 Quick Start

### 1. Installation

```bash
git clone https://github.com/berrydev-ai/berry-rag.git
cd berry-rag

# Install dependencies
npm run install-deps

# Setup directories and instructions
npm run setup
```

### 2. Configure Claude Desktop

Add to your `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest"]
    },
    "berry-rag": {
      "command": "node",
      "args": ["mcp_servers/vector_db_server.js"],
      "cwd": "/Users/eberry/BerryDev/berry-rag"
    }
  }
}
```

### 3. Start Using

```bash
# Example workflow:
# 1. Scrape with Playwright MCP through Claude
# 2. Process into vector DB
npm run process-scraped

# 3. Search your knowledge base
npm run search "React hooks"
```

## 📁 Project Structure

```
berry-rag/
├── src/                          # Python source code
│   ├── rag_system.py            # Core vector database system
│   └── playwright_integration.py # Playwright MCP integration
├── mcp_servers/                  # MCP server implementations
│   └── vector_db_server.ts      # TypeScript MCP server
├── storage/                      # Vector database storage
│   ├── documents.db             # SQLite metadata
│   └── vectors/                 # NumPy embedding files
├── scraped_content/             # Playwright saves content here
└── dist/                        # Compiled TypeScript
```

## 🔧 Commands

### Streamlit Web Interface

Launch the web interface for easy interaction with your RAG system:

```bash
# Start the Streamlit web interface
python run_streamlit.py

# Or directly with streamlit
streamlit run streamlit_app.py
```

The web interface provides:

- **🔍 Search**: Interactive document search with similarity controls
- **📄 Context**: Generate formatted context for AI assistants
- **➕ Add Document**: Upload files or paste content directly
- **📚 List Documents**: Browse your document library
- **📊 Statistics**: System health and performance metrics

### NPM Scripts

| Command                   | Description                             |
| ------------------------- | --------------------------------------- |
| `npm run install-deps`    | Install all dependencies                |
| `npm run setup`           | Initialize directories and instructions |
| `npm run build`           | Compile TypeScript MCP server           |
| `npm run process-scraped` | Process scraped files into vector DB    |
| `npm run search`          | Search the knowledge base               |
| `npm run list-docs`       | List all documents                      |

### Python CLI

```bash
# RAG System Operations
python src/rag_system.py search "query"
python src/rag_system.py context "query"  # Claude-formatted
python src/rag_system.py add <url> <title> <file>
python src/rag_system.py list
python src/rag_system.py stats

# Playwright Integration
python src/playwright_integration.py process
python src/playwright_integration.py setup
python src/playwright_integration.py stats
```

## 🤖 Usage with Claude

### 1. Scraping Documentation

```
"Use Playwright to scrape the React hooks documentation from https://react.dev/reference/react and save it to the scraped_content directory"
```

### 2. Processing into Vector Database

```
"Process all new scraped files and add them to the BerryRAG vector database"
```

### 3. Querying Knowledge Base

```
"Search the BerryRAG database for information about React useState best practices"

"Get context from the vector database about implementing custom hooks"
```

## 🔌 MCP Tools Available to Claude

BerryRAG provides two powerful MCP servers for Claude integration:

### Vector DB Server Tools

- `add_document` - Add content directly to vector DB
- `search_documents` - Search for similar content
- `get_context` - Get formatted context for queries
- `list_documents` - List all stored documents
- `get_stats` - Vector database statistics
- `process_scraped_files` - Process Playwright scraped content
- `save_scraped_content` - Save content for later processing

### BerryExa Server Tools

- `crawl_content` - Advanced web content extraction with subpage support
- `extract_links` - Extract internal links for subpage discovery
- `get_content_preview` - Quick content preview without full processing

📖 **For complete MCP setup and usage guide, see [BERRY_MCP.md](BERRY_MCP.md)**

## 🧠 Embedding Providers

The system supports multiple embedding providers with automatic fallback:

1. **sentence-transformers** (recommended, free, local)
2. **OpenAI embeddings** (requires API key, set `OPENAI_API_KEY`)
3. **Simple hash-based** (fallback, not recommended for production)

## ⚙️ Configuration

### Environment Variables

```bash
# Optional: for OpenAI embeddings
export OPENAI_API_KEY=your_key_here
```

### Content Quality Filters

The system automatically filters out:

- Content shorter than 100 characters
- Navigation-only content
- Repetitive/duplicate content
- Files larger than 500KB

### Chunking Strategy

- Default chunk size: 500 characters
- Overlap: 50 characters
- Smart boundary detection (sentences, paragraphs)

## 📊 Monitoring

### Check System Status

```bash
# Vector database statistics
python src/rag_system.py stats

# Processing status
python src/playwright_integration.py stats

# View recent documents
python src/rag_system.py list
```

### Storage Information

- **Database**: `storage/documents.db` (SQLite metadata)
- **Vectors**: `storage/vectors/` (NumPy arrays)
- **Scraped Content**: `scraped_content/` (Markdown files)

## 🔍 Example Workflows

### Academic Research

1. Scrape research papers with Playwright
2. Process into vector database
3. Query for specific concepts across all papers

### Documentation Management

1. Scrape API documentation from multiple sources
2. Build unified searchable knowledge base
3. Get contextual answers about implementation details

### Content Aggregation

1. Scrape blog posts and articles
2. Create topic-based knowledge clusters
3. Find related content across sources

## 🛠️ Development

### Building the MCP Server

```bash
npm run build
```

### Running in Development Mode

```bash
npm run dev  # TypeScript watch mode
```

### Testing

```bash
# Test RAG system
python src/rag_system.py stats

# Test integration
python src/playwright_integration.py setup

# Test MCP server
node mcp_servers/vector_db_server.js
```

## 🚨 Troubleshooting

### Common Issues

**Python dependencies missing:**

```bash
pip install -r requirements.txt
```

**TypeScript compilation errors:**

```bash
npm install
npm run build
```

**Embedding model download slow:**
The first run downloads sentence-transformers model (~90MB). This is normal.

**No results from search:**

- Check if documents were processed: `python src/rag_system.py list`
- Verify content quality filters aren't too strict
- Try broader search terms

### Logs and Debugging

- Python logs: Check console output
- MCP server logs: Stderr output
- Processing status: `scraped_content/.processed_files.json`

## 📝 License

MIT License - feel free to modify and extend for your needs.

## 🤝 Contributing

This is a personal project for Eric Berry, but feel free to fork and adapt for your own use cases.

---

**Happy scraping and searching!** 🕷️🔍✨