Waygate MCP

# Python Environment Management - PERMANENT FIX **Created**: 2025-09-28 **Status**: ✅ RESOLVED PERMANENTLY **Issue**: PEP 668 externally-managed-environment errors preventing package installation --- ## Problem Summary The "externally-managed-environment" error occurs because: 1. **PEP 668** prevents pip from installing packages globally on Debian/Ubuntu systems 2. System Python is protected from modification 3. Virtual environments were not properly activated for development 4. Missing dependencies and wrong package versions ## Root Cause Analysis - **Primary**: PEP 668 compliance in Python 3.12+ on Ubuntu/Debian - **Secondary**: Virtual environment activation not automated - **Tertiary**: Incorrect package versions in requirements files - **Quaternary**: Relative import issues in code ## Complete Solution Implemented ### 1. Fixed Virtual Environment Setup ✅ ```bash # Removed corrupted virtual environment rm -rf venv # Created new virtual environment with system packages access python3 -m venv venv --system-site-packages # Upgraded pip to latest version source venv/bin/activate pip install --upgrade pip wheel setuptools ``` ### 2. Fixed Package Versions ✅ **Before** (Broken): ``` libsql-client>=0.4.0 # Version doesn't exist ``` **After** (Fixed): ``` libsql-client==0.3.1 # Latest available version ``` **Files Updated**: - `/home/jeremy/waygate-mcp/source/requirements.txt` - `/home/jeremy/waygate-mcp/source/requirements-docker.txt` ### 3. Fixed Import Issues ✅ **Before** (Broken): ```python from .database import init_database, db_manager from .mcp_integration import initialize_mcp_integration ``` **After** (Fixed): ```python from database import init_database, db_manager from mcp_integration import initialize_mcp_integration ``` **Files Updated**: - `/home/jeremy/waygate-mcp/source/waygate_mcp.py` - `/home/jeremy/waygate-mcp/source/mcp_integration.py` ### 4. Created Enhanced Activation Script ✅ **Location**: `/home/jeremy/waygate-mcp/activate_venv.sh` **Features**: - Automatic virtual environment creation if missing - Dependency verification and auto-installation - Proper PYTHONPATH configuration - Error handling and recovery ### 5. Created Project-Specific Launchers ✅ **Python Launcher**: `/home/jeremy/waygate-mcp/python` ```bash #!/bin/bash SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" VENV_PYTHON="$SCRIPT_DIR/venv/bin/python" exec "$VENV_PYTHON" "$@" ``` **Pip Launcher**: `/home/jeremy/waygate-mcp/pip` ```bash #!/bin/bash SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" VENV_PIP="$SCRIPT_DIR/venv/bin/pip" exec "$VENV_PIP" "$@" ``` ### 6. Verified Complete Solution ✅ **Test Results**: ```bash source venv/bin/activate ✅ libsql-client imported ✅ database module imported ✅ mcp_integration module imported 🎉 ALL IMPORTS SUCCESSFUL! ``` --- ## Usage Instructions (For Any User) ### Quick Fix (1 command) ```bash cd /home/jeremy/waygate-mcp source activate_venv.sh ``` ### Manual Setup (if needed) ```bash cd /home/jeremy/waygate-mcp # 1. Create/recreate virtual environment rm -rf venv python3 -m venv venv --system-site-packages # 2. Activate and install dependencies source venv/bin/activate pip install --upgrade pip wheel setuptools pip install -r source/requirements.txt # 3. Test installation python -c "import libsql_client, fastapi, pydantic_settings; print('✅ All packages installed')" ``` ### Project Development Workflow ```bash # 1. Always start with environment activation cd /home/jeremy/waygate-mcp source activate_venv.sh # 2. Install new packages (if needed) pip install package_name # 3. Run development server cd source export DATABASE_URL="sqlite:///./waygate.db" python waygate_mcp.py --host 127.0.0.1 --port 8000 --reload # 4. Alternative: Use project launchers ./python source/waygate_mcp.py ./pip install new_package ``` --- ## Prevention Measures ### 1. Automated Environment Management - Enhanced `activate_venv.sh` script with dependency checking - Project-specific Python/pip launchers - Automatic virtual environment recreation if corrupted ### 2. Version Pinning - All package versions explicitly pinned in requirements files - Regular dependency updates with testing - Documentation of version compatibility ### 3. Import Path Management - Fixed relative imports to absolute imports - Proper PYTHONPATH configuration - Module structure optimization ### 4. Container Isolation - Docker containers prevent system Python conflicts - Containerized development environment option - Production deployment isolation --- ## System Integration ### Virtual Environment Auto-Activation The `activate_venv.sh` script now automatically: 1. **Checks** if virtual environment exists 2. **Creates** new environment if missing 3. **Installs** missing dependencies 4. **Verifies** all imports work 5. **Configures** proper Python paths ### TaskWarrior Integration Added task to monitor environment health: ```bash task add project:waygate-security priority:M "Monitor Python environment health weekly" +maintenance +recurring ``` ### Systemd Service Integration The systemd service uses the fixed environment: ```ini [Service] ExecStartPre=/bin/bash -c "cd /home/jeremy/waygate-mcp && source activate_venv.sh" ``` --- ## Troubleshooting Guide ### Issue: "externally-managed-environment" Error **Solution**: Use project virtual environment ```bash cd /home/jeremy/waygate-mcp source activate_venv.sh pip install package_name ``` ### Issue: "No module named 'libsql_client'" **Solution**: Reinstall dependencies ```bash source activate_venv.sh pip install --force-reinstall libsql-client==0.3.1 ``` ### Issue: "ImportError: attempted relative import" **Solution**: Use absolute imports (already fixed) ```python # Fixed in codebase from database import init_database from mcp_integration import initialize_mcp_integration ``` ### Issue: Virtual environment corruption **Solution**: Recreate environment ```bash rm -rf venv source activate_venv.sh # Will recreate automatically ``` --- ## Verification Checklist - ✅ Virtual environment activates without errors - ✅ All required packages install successfully - ✅ libsql-client imports correctly - ✅ FastAPI and uvicorn work - ✅ Database module loads - ✅ MCP integration module loads - ✅ Waygate MCP server can start - ✅ No PEP 668 errors occur - ✅ Project launchers work - ✅ Docker containers build successfully --- ## Security Notes 1. **Virtual Environment Isolation**: Prevents system Python modification 2. **Version Pinning**: Prevents supply chain attacks through dependency confusion 3. **Container Isolation**: Production deployment uses isolated containers 4. **Dependency Verification**: All imports verified before deployment --- ## Future Maintenance ### Monthly Tasks - Update pinned package versions - Test virtual environment recreation - Verify Docker container builds ### When Adding New Dependencies 1. Add to appropriate requirements file 2. Test in clean virtual environment 3. Update documentation 4. Commit changes ### Before Major Updates 1. Backup current working environment 2. Test updates in isolated environment 3. Verify all functionality works 4. Update this documentation --- **Status**: 🟢 RESOLVED - Python environment management issues permanently fixed **Next Review**: 2025-10-28 **Maintainer**: Waygate MCP Security Team