# MCP Log Collector - Comprehensive Guide

A production-ready client-server implementation for remote log collection across networks. The server runs on Linux with direct file access, and clients can connect from anywhere using HTTP/HTTPS.

## 📋 Table of Contents

- [Overview](#overview)
- [Quick Start (Automated)](#quick-start-automated)
- [Architecture](#architecture)
- [Installation](#installation)
  - [Automated Setup](#automated-setup)
  - [Manual Setup](#manual-setup)
- [Network Options](#network-options)
- [Running the System](#running-the-system)
- [Configuration](#configuration)
- [Usage](#usage)
- [Troubleshooting](#troubleshooting)
- [Advanced Topics](#advanced-topics)

---

## Overview

### Key Features

- ✅ **No SSH Required** - Direct HTTP/HTTPS communication
- ✅ **Cross-Network** - Works across different networks via Cloudflare Tunnel
- ✅ **Production Ready** - Systemd services with auto-restart
- ✅ **Secure** - HTTPS, API keys, DDoS protection
- ✅ **Fast** - HTTP keep-alive vs SSH handshake overhead
- ✅ **Automated** - One-command setup and start scripts
- ✅ **Multiple Clients** - Single server, many clients

### Use Cases

- Remote log monitoring for Linux servers
- Security log analysis with Claude Desktop
- Centralized log collection from multiple systems
- Real-time log access across networks

---

## Quick Start (Automated)

**Want to get started in 5 minutes? Use the automated scripts!**

### Linux Server (5 minutes)

```bash
cd server
sudo bash setup_linux.sh
# Choose option 2 for Cloudflare Tunnel (recommended)
# Enter your domain (e.g., logs.yourdomain.com)
# Authenticate in browser
# Done!
```

### Windows Client (2 minutes)

```cmd
setup_windows.bat
# Enter server URL from Linux output
# Test connection
# Done!
```

### Start Services

**Linux:**
```bash
sudo bash start_server.sh start
```

**Windows:**
```cmd
start_client.bat
```

That's it! Jump to [Running the System](#running-the-system) for daily usage.

---

## Architecture

### System Design

```
┌─────────────────────┐         HTTP/HTTPS        ┌─────────────────────┐
│  Windows Machine    │ ──────────────────────> │  Linux System       │
│  (Client)           │                            │  (Server)           │
│  - MCP Client       │ <────────────────────── │  - FastAPI Server   │
│  - Claude Desktop   │                            │  - Direct Log       │
│                     │                            │    File Access      │
└─────────────────────┘                            └─────────────────────┘
```

### With Cloudflare Tunnel (Production)

```
[Windows Client] ──HTTPS──> [Cloudflare CDN] ──Tunnel──> [Linux Server]
    (Anywhere)              (DDoS Protected)   (No ports)   (localhost:8000)
```

### Directory Structure

```
mcp-2/
├── server/                    # Deploy on Linux system
│   ├── server.py              # FastAPI server with MCP endpoints
│   ├── config.yaml            # Log files configuration
│   ├── .env                   # Server configuration
│   ├── requirements.txt       # Server dependencies
│   ├── log-collector.service  # Systemd service file
│   ├── setup_linux.sh         # Automated setup script
│   ├── start_server.sh        # Start/manage services
│   ├── stop_server.sh         # Stop services
│   └── restart_server.sh      # Restart services
│
├── client.py                  # MCP client (runs on Windows)
├── test_client.py             # Test script for client
├── .env                       # Client configuration (server URL)
├── requirements.txt           # Client dependencies
├── setup_windows.ps1          # Automated Windows setup
├── setup_windows.bat          # Batch wrapper for setup
├── start_client.ps1           # Start client with checks
├── start_client.bat           # Batch wrapper for starting
└── README.md                  # This file
```

---

## Installation

### Automated Setup

**Recommended for production deployments.**

#### Linux Server Setup

```bash
cd server
chmod +x setup_linux.sh
sudo bash setup_linux.sh
```

**Interactive options:**
1. **Local network only** - Server accessible on LAN (port 8000)
2. **Production with Cloudflare Tunnel** - Accessible from anywhere via HTTPS

**What it does:**
- ✅ Installs system dependencies (Python, curl, jq)
- ✅ Installs MCP server to `/opt/mcp-log-collector`
- ✅ Configures systemd service for auto-start
- ✅ Sets up Cloudflare Tunnel (if selected)
- ✅ Configures firewall (if needed)
- ✅ Starts and tests services

**Time:** 3-7 minutes (depending on Cloudflare authentication)

#### Windows Client Setup

```cmd
setup_windows.bat
```

Or with PowerShell directly:
```powershell
.\setup_windows.ps1 -ServerURL "https://logs.yourdomain.com" -ApiKey "optional-key"
```

**What it does:**
- ✅ Checks Python and pip installation
- ✅ Installs Python dependencies
- ✅ Configures `.env` file interactively
- ✅ Tests server connection
- ✅ Creates helper scripts

**Time:** 2-3 minutes

---

### Manual Setup

**For advanced users or custom configurations.**

#### Linux Server (Manual)

#### Linux Server (Manual)

**1. Copy Server Files**

Transfer the `server/` directory to your Linux system:

```bash
# On Linux system
sudo mkdir -p /opt/mcp-log-collector
cd /opt/mcp-log-collector

# Copy files (server.py, config.yaml, .env, requirements.txt)
```

**2. Install Dependencies**

```bash
cd /opt/mcp-log-collector
pip3 install -r requirements.txt
```

**3. Configure Server**

Edit `.env` to set server binding (use 0.0.0.0 for network access or 127.0.0.1 for localhost-only):

```bash
nano .env
```

```env
SERVER_HOST=0.0.0.0
SERVER_PORT=8000
# Optional: API_KEY=your-secret-key
```

**4. Configure Log Files**

Edit `config.yaml` to specify which log files to collect:

```bash
nano config.yaml
```

```yaml
log_files:
  - path: /var/log/audit/audit.log
    description: "Audit logs"
    requires_sudo: true
  
  - path: /var/log/secure
    description: "Security logs"
    requires_sudo: true
  
  - path: /var/log/syslog
    description: "System logs"
    requires_sudo: false
```

**5. Setup Sudo (if needed)**

For logs requiring sudo, configure passwordless sudo:

```bash
sudo visudo

# Add this line (replace 'username' with actual user)
username ALL=(ALL) NOPASSWD: /usr/bin/tail
```

**6. Test Server Manually**

```bash
python3 server.py
```

Server should start on `http://0.0.0.0:8000`

**7. Setup as Systemd Service**

```bash
# Copy service file
sudo cp log-collector.service /etc/systemd/system/

# Enable and start service
sudo systemctl enable log-collector.service
sudo systemctl start log-collector.service

# Check status
sudo systemctl status log-collector.service
```

**8. Configure Firewall**

Allow incoming connections on port 8000 (skip if using Cloudflare Tunnel):

```bash
# For UFW (Ubuntu/Debian)
sudo ufw allow 8000/tcp

# For firewalld (RHEL/CentOS)
sudo firewall-cmd --permanent --add-port=8000/tcp
sudo firewall-cmd --reload
```

#### Windows Client (Manual)

**1. Install Dependencies**

```bash
cd mcp-2
pip install -r requirements.txt
```

**2. Configure Client**

Edit `.env` with your Linux server URL:

Edit `.env` with your Linux server URL:

```env
REMOTE_SERVER_URL=http://192.168.1.100:8000
# or for Cloudflare Tunnel:
# REMOTE_SERVER_URL=https://logs.yourdomain.com
```

**3. Test Connection**

```bash
python test_client.py
```

**4. Run MCP Client**

```bash
python client.py
```

---

## Network Options

Choose the deployment method that fits your needs:

### Option 1: Local Network Only

**Best for:** Same LAN or VPN connections

**Setup:**
- Server binds to `0.0.0.0:8000`
- Open firewall port 8000
- Client uses `http://<server-ip>:8000`

**Pros:** Simple, fast setup  
**Cons:** Only works on same network

### Option 2: Cloudflare Tunnel (Recommended)

**Best for:** Internet access, production deployments

**Setup:**
```bash
# Automated (included in setup_linux.sh option 2)
sudo bash setup_linux.sh

# Or manual - see CLOUDFLARE_TUNNEL.md
```

**Pros:** 
- ✅ No firewall configuration
- ✅ Free HTTPS
- ✅ DDoS protection
- ✅ Access from anywhere

**Cons:** Requires Cloudflare account and domain

### Option 3: Port Forwarding

**Best for:** Quick testing without domain

**Setup:**
- Forward port 8000 on router to Linux server IP
- Client uses `http://<public-ip>:8000`

**Pros:** Simple  
**Cons:** Exposes port to internet, no HTTPS

### Option 4: ngrok (Testing Only)

**Best for:** Quick demos

```bash
# On Linux
ngrok http 8000
# Use provided URL in Windows client
```

---

## Running the System

### Daily Operations

#### Start Services

**Linux Server:**
```bash
# Automated start/stop/restart
sudo bash start_server.sh start
sudo bash start_server.sh stop
sudo bash start_server.sh restart

# Check status (no sudo needed)
bash status_server.sh

# View live logs
sudo bash start_server.sh logs
```

**Windows Client:**
```cmd
start_client.bat
```

Or with PowerShell:
```powershell
.\start_client.ps1
```

#### Check Status

**Linux:**
```bash
sudo bash start_server.sh status
```

Output:
```
Service Status:

  MCP Server:         ● RUNNING
  Cloudflare Tunnel:  ● RUNNING

  Health Check:       ● HEALTHY
  Server Response:    1.0.0

Access URLs:
  Local:     http://localhost:8000
  Network:   http://192.168.1.100:8000
  Public:    https://logs.yourdomain.com
```

**Windows:**
```powershell
.\test_connection.ps1
```

#### Stop Services

**Linux:**
```bash
sudo bash stop_server.sh
```

**Windows:**
Press `Ctrl+C` in client terminal

### Auto-Start Configuration

**Linux** - Services auto-start on boot (configured by setup script):
```bash
# Verify auto-start is enabled
sudo systemctl is-enabled log-collector.service
sudo systemctl is-enabled cloudflared  # if using Cloudflare

# To disable auto-start
sudo systemctl disable log-collector.service
```

**Windows** - Optional auto-start:

1. **Task Scheduler:**
   - Create Basic Task
   - Trigger: "When I log on"
   - Action: `powershell.exe -ExecutionPolicy Bypass -File "C:\path\to\start_client.ps1"`

2. **Startup Folder:**
   - Press `Win+R`, type `shell:startup`
   - Create shortcut to `start_client.bat`

---

## Configuration

### Server Configuration

**Location:** `/opt/mcp-log-collector/`

#### Environment Variables (`.env`)

```env
# Server binding (0.0.0.0 for network, 127.0.0.1 for localhost only)
SERVER_HOST=0.0.0.0
SERVER_PORT=8000

# Optional API authentication
API_KEY=your-secret-key-here
```

#### Log Files (`config.yaml`)

```yaml
log_files:
  - path: /var/log/audit/audit.log
    description: "Linux audit system logs"
    requires_sudo: true
  
  - path: /var/log/secure
    description: "Security/authentication logs"
    requires_sudo: true
  
  - path: /var/log/syslog
    description: "System logs"
    requires_sudo: false
  
  - path: /var/log/apache2/access.log
    description: "Apache access logs"
    requires_sudo: false
```

**After editing:**
```bash
sudo systemctl restart log-collector.service
```

### Client Configuration

**Location:** `mcp-2/.env`

```env
# Remote server URL
REMOTE_SERVER_URL=https://logs.yourdomain.com

# Number of log lines to collect (default)
NUM_LINES=10

# Optional API key (must match server)
API_KEY=your-secret-key-here
```

---

## Usage

### Available MCP Tools

When running the client, these tools are available to Claude Desktop:

#### 1. `collect_logs`

Collects the last N lines from all configured log files.

**Parameters:**
- `num_lines` (optional): Number of lines to collect (default: 10)

**Example:**
```json
{
  "tool": "collect_logs",
  "arguments": {
    "num_lines": 50
  }
}
```

**Response:**
```
=== Log Collection Results ===

File: /var/log/audit/audit.log (Audit logs)
[log entries...]

File: /var/log/secure (Security logs)
[log entries...]
```

#### 2. `list_configured_logs`

Lists all log files configured on the server.

**Parameters:** None

**Example:**
```json
{
  "tool": "list_configured_logs",
  "arguments": {}
}
```

**Response:**
```
Configured Log Files:
====================

Path: /var/log/audit/audit.log
Description: Audit logs
Sudo Required: True

Path: /var/log/secure
Description: Security logs
Sudo Required: True
```

### Testing the System

**Test Connection:**
```bash
python test_client.py
```bash
python test_client.py
```

Expected output:
```
MCP Log Collector Client - Test Script
================================================================================

Testing connection to remote server...
Server URL: https://logs.yourdomain.com
================================================================================
✓ Server is reachable and healthy!

Testing list_configured_logs...
================================================================================
Configured Log Files:

Path: /var/log/audit/audit.log
Description: Linux audit system logs
Sudo Required: True
--------------------------------------------------------------------------------

Testing collect_logs...
================================================================================
✓ Successfully collected logs from 3 files
```

### Integrating with Claude Desktop

Add to your MCP configuration (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "log-collector": {
      "command": "python",
      "args": ["C:/path/to/mcp-2/client.py"]
    }
  }
}
```

Replace `C:/path/to/mcp-2/` with your actual path.

**Usage in Claude:**
- "Show me the last 50 lines from the audit logs"
- "What log files are configured?"
- "Collect recent security logs"

---

## Troubleshooting

### Linux Server Issues

#### Service Won't Start

```bash
# Check status
sudo systemctl status log-collector.service

# View recent logs
sudo journalctl -u log-collector.service -n 50

# Check if port is in use
sudo netstat -tulpn | grep 8000

# Test manually
python3 /opt/mcp-log-collector/server.py
```

#### Permission Denied on Log Files

```bash
# Check file permissions
ls -l /var/log/audit/audit.log

# Test sudo access
sudo tail -n 10 /var/log/audit/audit.log

# Verify sudoers configuration
sudo visudo -c
```

#### Cloudflare Tunnel Issues

```bash
# Check tunnel status
sudo systemctl status cloudflared
sudo journalctl -u cloudflared -n 50

# Verify tunnel
cloudflared tunnel info log-collector

# Test DNS
nslookup logs.yourdomain.com

# Restart tunnel
sudo systemctl restart cloudflared
```

### Windows Client Issues

#### Cannot Connect to Server

**Checklist:**
1. ✅ Server is running: `curl http://server-ip:8000/health`
2. ✅ Firewall allows connections
3. ✅ REMOTE_SERVER_URL in `.env` is correct
4. ✅ Network connectivity: `ping server-ip`

**Test connection:**
```powershell
.\test_connection.ps1
```

#### Python Not Found

```powershell
# Check Python installation
python --version

# If not found, install from:
# https://www.python.org/downloads/

# Ensure "Add to PATH" is checked during installation
```

#### Module Import Errors

```powershell
# Reinstall dependencies
pip install -r requirements.txt

# Or install specific package
pip install httpx mcp
```

#### Timeout Errors

- Server may be overloaded
- Network latency too high
- Increase timeout in client code if needed

### Common Issues

#### Port 8000 Already in Use

```bash
# Find process using port
sudo netstat -tulpn | grep 8000

# Kill process (replace PID)
sudo kill -9 <PID>

# Or change port in .env
```

#### Logs Not Updating

```bash
# Verify log files exist
ls -l /var/log/audit/audit.log

# Check log rotation
sudo logrotate -f /etc/logrotate.conf

# Restart server
sudo systemctl restart log-collector.service
```

---

## Advanced Topics

### API Key Authentication

**Enable on Server:**

Edit `/opt/mcp-log-collector/.env`:
```env
API_KEY=your-super-secret-key-here
```

Update `server.py` to require API key (see CLOUDFLARE_TUNNEL.md for code).

**Enable on Client:**

Edit `mcp-2/.env`:
```env
API_KEY=your-super-secret-key-here
```

Restart both services.

### Multiple Servers

**Client can connect to multiple servers:**

1. Create separate client directories
2. Configure different `.env` files
3. Add multiple entries to Claude Desktop config:

```json
{
  "mcpServers": {
    "logs-prod": {
      "command": "python",
      "args": ["C:/path/to/mcp-2-prod/client.py"]
    },
    "logs-staging": {
      "command": "python",
      "args": ["C:/path/to/mcp-2-staging/client.py"]
    }
  }
}
```

### Log Filtering

**Server-side:** Modify `server.py` to add filtering by pattern, date, etc.

**Client-side:** Use Claude to analyze and filter logs after collection.

### Performance Optimization

**Server:**
- Limit log file sizes with `logrotate`
- Use `grep` for pre-filtering large logs
- Add caching for frequently accessed logs

**Client:**
- Adjust `NUM_LINES` to balance detail vs performance
- Use connection pooling for multiple requests

### Monitoring

**Server Metrics:**
```bash
# View service status
sudo bash start_server.sh status

# Monitor resource usage
htop
sudo systemctl status log-collector.service

# Watch logs in real-time
sudo bash start_server.sh logs
```

**Cloudflare Analytics:**
- Go to Cloudflare Dashboard
- Zero Trust > Access > Analytics

### Backup and Disaster Recovery

**Backup Configuration:**
```bash
# Backup server config
sudo tar -czf mcp-backup.tar.gz /opt/mcp-log-collector

# Backup Cloudflare config
tar -czf cloudflare-backup.tar.gz ~/.cloudflared
```

**Restore:**
```bash
# Restore server
sudo tar -xzf mcp-backup.tar.gz -C /

# Restart service
sudo systemctl restart log-collector.service
```

---

## Advantages Over Alternatives

### vs SSH-Based Approach

| Feature | SSH-Based | MCP Log Collector |
|---------|-----------|-------------------|
| Connection Speed | ⚠️ Slower (handshake overhead) | ✅ Fast (HTTP keep-alive) |
| File Access | ⚠️ Remote execution | ✅ Direct local access |
| Multiple Clients | ⚠️ One session per client | ✅ Single server, many clients |
| Production Ready | ⚠️ Manual execution | ✅ Systemd service |
| Real-time | ❌ Not supported | ✅ Can add WebSocket |
| Cross-Network | ⚠️ VPN required | ✅ Cloudflare Tunnel |
| Authentication | SSH keys | API keys + HTTPS |

### vs Manual Log Collection

| Feature | Manual | Automated |
|---------|--------|-----------|
| Time to collect | Minutes | Seconds |
| Error prone | High | Low |
| Multiple sources | Tedious | Automatic |
| Integration | None | Claude Desktop |

---

## Security Best Practices

### Essential Security

1. ✅ **Use HTTPS** - Cloudflare Tunnel provides free HTTPS
2. ✅ **API Keys** - Enable API key authentication in production
3. ✅ **Firewall** - Only open port 8000 if not using Cloudflare
4. ✅ **Sudo** - Configure passwordless sudo only for specific commands
5. ✅ **Log Rotation** - Prevent disk space issues

### Advanced Security

6. ✅ **Cloudflare Access** - Add authentication layer (email/Google/GitHub)
7. ✅ **Rate Limiting** - Configure in Cloudflare Dashboard
8. ✅ **IP Whitelist** - Restrict client IPs if possible
9. ✅ **Audit Logging** - Log all access attempts
10. ✅ **Regular Updates** - Keep dependencies updated

### Security Checklist

- [ ] HTTPS enabled (Cloudflare Tunnel or reverse proxy)
- [ ] API keys configured and secured
- [ ] Firewall rules configured
- [ ] Sudo access restricted to specific commands
- [ ] Log files contain no sensitive credentials
- [ ] Services run with minimal privileges
- [ ] Regular security updates applied
- [ ] Access logs monitored

---

## Maintenance

### Regular Tasks

**Weekly:**
- Check service status
- Review access logs
- Monitor disk space

**Monthly:**
- Update Python packages: `pip install --upgrade -r requirements.txt`
- Review and rotate large log files
- Check for security updates

**Quarterly:**
- Review log file configuration
- Update Cloudflare Tunnel if used
- Test disaster recovery procedures

### Updating

**Server Update:**
```bash
cd /opt/mcp-log-collector
sudo systemctl stop log-collector.service

# Update files (server.py, etc.)
# Update dependencies
pip3 install --upgrade -r requirements.txt

sudo systemctl start log-collector.service
```

**Client Update:**
```powershell
# Update files (client.py, etc.)
# Update dependencies
pip install --upgrade -r requirements.txt
```

### Uninstallation

**Linux Server:**
```bash
# Stop and disable services
sudo systemctl stop log-collector.service cloudflared
sudo systemctl disable log-collector.service cloudflared

# Remove files
sudo rm -rf /opt/mcp-log-collector
sudo rm /etc/systemd/system/log-collector.service

# Remove Cloudflare Tunnel (if used)
cloudflared service uninstall
cloudflared tunnel delete log-collector
```

**Windows Client:**
```powershell
# Just delete the directory
# No system-wide changes were made
```

---

## Additional Resources

### Documentation Files

- **CLOUDFLARE_TUNNEL.md** - Detailed Cloudflare Tunnel setup
- **SETUP_GUIDE.md** - Automated setup documentation
- **QUICK_START.md** - Quick reference for daily operations

### External Links

- [Cloudflare Tunnel Docs](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/)
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
- [MCP Protocol](https://modelcontextprotocol.io/)

### Support

For issues or questions:
1. Check this README and additional documentation
2. Review service logs: `sudo journalctl -u log-collector.service`
3. Test with `test_client.py` for connection issues
4. Verify configuration files (`.env`, `config.yaml`)

---

## Future Enhancements

Potential additions for future versions:

- [ ] WebSocket streaming for real-time log monitoring
- [ ] Web dashboard for log visualization
- [ ] Advanced filtering and search capabilities
- [ ] Multiple log format support (JSON, CSV, etc.)
- [ ] Log aggregation from multiple servers
- [ ] Built-in log analysis and anomaly detection
- [ ] Automated alerting for critical events
- [ ] Database storage for historical logs

---

## License

This code is provided as-is for log collection purposes.

---

## Quick Command Reference

### Linux Server Commands

```bash
# Setup (one-time)
sudo bash setup_linux.sh

# Daily operations
sudo bash start_server.sh start      # Start services
sudo bash start_server.sh stop       # Stop services
sudo bash start_server.sh restart    # Restart services
sudo bash start_server.sh status     # Check status
sudo bash start_server.sh logs       # View logs

# Configuration
sudo nano /opt/mcp-log-collector/config.yaml  # Edit log files
sudo nano /opt/mcp-log-collector/.env         # Edit settings
sudo systemctl restart log-collector.service  # Apply changes
```

### Windows Client Commands

```cmd
REM Setup (one-time)
setup_windows.bat

REM Daily operations
start_client.bat          REM Start client
test_connection.ps1       REM Test connection

REM Configuration
notepad .env              REM Edit settings
```

### Troubleshooting Commands

```bash
# Linux
sudo systemctl status log-collector.service
sudo journalctl -u log-collector.service -n 50
curl http://localhost:8000/health
sudo netstat -tulpn | grep 8000

# Windows
python test_client.py
python --version
pip list | grep -E "httpx|mcp"
```

---

**🎉 You're all set! Start with the [Quick Start](#quick-start-automated) section to get running in minutes.**