MCP Dataset Onboarding Server

# 🤖 MCP Auto Dataset Processor - Complete Guide ## 🎯 **What We Built** You now have a **fully automated dataset onboarding system** that watches your Google Drive folder and processes new files automatically - no manual file IDs needed! ## 🚀 **Quick Start (3 Steps)** ### **Step 1: Start the Auto-Processor** ```bash python start_auto_processor.py ``` ### **Step 2: Upload Files** - Upload any CSV or Excel file to your `MCP_server` Google Drive folder - That's it! No file IDs, no manual commands ### **Step 3: Watch the Magic** - Files are automatically detected within 30 seconds - Complete processing pipeline runs automatically - All artifacts saved in organized folders ## 📊 **Monitoring & Management** ### **Real-Time Dashboard** ```bash # View current status python processor_dashboard.py # Live monitoring (auto-refreshing) python processor_dashboard.py --live # Detailed analytics python processor_dashboard.py --stats ``` ### **Manual Controls** ```bash # Run single check python auto_processor.py --once # List processed files python auto_processor.py --list # Custom check interval python auto_processor.py --interval 60 # Reset processed files log python auto_processor.py --reset ``` ## 🔧 **How It Works** ### **Intelligent File Detection** - ✅ Monitors Google Drive folder continuously - ✅ Only processes supported formats (CSV, Excel) - ✅ Ignores already processed files - ✅ Waits for upload completion before processing - ✅ Handles multiple files efficiently ### **Automatic Processing Pipeline** 1. **File Detection** → New file uploaded to Google Drive 2. **Download** → File retrieved automatically 3. **Analysis** → Metadata extraction and statistics 4. **Quality Rules** → Intelligent DQ rule generation 5. **Documentation** → Excel contracts and reports 6. **Organization** → Structured folder creation 7. **Tracking** → Processing log updated ### **Smart Features** - **Duplicate Prevention**: Won't process the same file twice - **Error Recovery**: Handles failures gracefully - **Batch Processing**: Can handle multiple files at once - **Progress Tracking**: Maintains detailed logs - **Resource Efficient**: Minimal system impact ## 📁 **Output Structure** Each processed dataset gets its own organized folder: ``` processed_datasets/ └── your_dataset_name/ ├── original_file.csv # Original dataset ├── dataset_metadata.json # Column info & stats ├── dataset_contract.xlsx # Professional contract ├── dataset_dq_report.json # Quality assessment └── README.md # Human-readable summary ``` ## 🎛️ **Configuration** ### **Default Settings** - **Check Interval**: 30 seconds - **File Age Threshold**: 1 minute (prevents processing during upload) - **Supported Formats**: CSV, Excel (.xlsx, .xls) - **Max Files Per Cycle**: 5 ### **Customization** Edit `auto_config.py` to adjust: - Check frequency - File age requirements - Supported formats - Logging levels - Output folders ## 🔍 **Troubleshooting** ### **No Files Being Processed?** 1. Check Google Drive folder permissions 2. Verify service account has access 3. Ensure files are supported formats 4. Check `processed_files.json` for duplicates ### **Processing Errors?** 1. Check Google Drive connectivity 2. Verify file formats are valid 3. Check disk space for output folders 4. Review error logs in console ### **Dashboard Not Showing Data?** 1. Ensure `processed_files.json` exists 2. Check Google Drive API access 3. Verify folder IDs in `.env` file ## 🎉 **Benefits** ### **Before (Manual)** - ❌ Find file ID manually - ❌ Run commands for each file - ❌ Track processed files yourself - ❌ Organize outputs manually - ❌ Monitor progress constantly ### **After (Automated)** - ✅ Just upload files to Google Drive - ✅ Everything happens automatically - ✅ Smart duplicate detection - ✅ Organized output structure - ✅ Real-time monitoring dashboard ## 🚀 **Production Deployment** ### **Run as Service (Linux)** ```bash # Create systemd service sudo nano /etc/systemd/system/mcp-auto-processor.service [Unit] Description=MCP Auto Dataset Processor After=network.target [Service] Type=simple User=your-user WorkingDirectory=/path/to/mcp ExecStart=/usr/bin/python3 start_auto_processor.py Restart=always [Install] WantedBy=multi-user.target # Enable and start sudo systemctl enable mcp-auto-processor sudo systemctl start mcp-auto-processor ``` ### **Run as Service (Windows)** Use Task Scheduler or Windows Service Wrapper to run `start_auto_processor.py` automatically. ### **Docker Deployment** ```bash # Build image docker build -t mcp-auto-processor . # Run with auto-processor docker run -d \ -v /path/to/service-account.json:/app/keys/service-account.json \ -v /path/to/processed_datasets:/app/processed_datasets \ -e GOOGLE_SERVICE_ACCOUNT_KEY_PATH=/app/keys/service-account.json \ -e MCP_SERVER_FOLDER_ID=your_server_folder_id \ -e MCP_CLIENT_FOLDER_ID=your_client_folder_id \ mcp-auto-processor python start_auto_processor.py ``` ## 🎯 **Use Cases** ### **Data Teams** - Automatic ingestion of daily reports - Continuous data quality monitoring - Self-service data onboarding ### **Business Users** - Upload spreadsheets for instant analysis - Automated documentation generation - Quality-checked data delivery ### **Data Engineers** - Hands-off data pipeline integration - Automated metadata cataloging - Quality rule enforcement ## 🏆 **You Now Have** A **production-ready, fully automated dataset onboarding system** that: - ✅ Requires zero manual intervention - ✅ Processes files within 30 seconds of upload - ✅ Generates professional documentation - ✅ Maintains organized data catalogs - ✅ Provides real-time monitoring - ✅ Scales to handle multiple files - ✅ Recovers from errors gracefully **Just upload files to Google Drive and walk away!** 🚀