Umbrella MCP Server

{ "masterTestRun": { "startTime": "2025-09-24T09:46:55.043Z", "endTime": "2025-09-24T09:46:59.264Z", "duration": 4221, "testSuites": 6, "successful": 3 }, "frameworkCapabilities": { "endpointsCovered": 20, "accountTypes": 2, "mspCustomers": 2, "testVariations": "Multiple parameter combinations per endpoint", "reportFormats": [ "JSON", "CSV", "Console" ], "authenticationMethods": [ "Direct API", "MCP Protocol" ], "categories": [ "Cost Analysis", "Budget Management", "Recommendations", "Anomaly Detection", "User Management", "MSP Management", "Resource Explorer", "Kubernetes", "Commitment Analysis" ] }, "testSuiteResults": { "directApi": null, "mcpTest": null, "demo": null, "demoframework": { "code": 0, "stdout": "🧪 Comprehensive Endpoint Testing Framework Demo\n================================================\n\n2025-09-24T09:46:55 📋 🚀 COMPREHENSIVE ENDPOINT TESTING DEMONSTRATION\n2025-09-24T09:46:55 📋 This demo shows what the full test would produce with real credentials\n\n2025-09-24T09:46:55 📋 📊 TEST EXECUTION SUMMARY:\n2025-09-24T09:46:55 📋 Total Tests Run: 9\n2025-09-24T09:46:55 ✅ Successful: 7 (77.8%)\n2025-09-24T09:46:55 ❌ Failed: 2 (22.2%)\n2025-09-24T09:46:55 📋 With Data: 7 (77.8%)\n2025-09-24T09:46:55 📋 Total Duration: 45.0s\n2025-09-24T09:46:55 📋 \n🏢 ACCOUNT PERFORMANCE:\n2025-09-24T09:46:55 📋 SAOLA:\n2025-09-24T09:46:55 📋 Tests: 4\n2025-09-24T09:46:55 📋 Success Rate: 3/4 (75.0%)\n2025-09-24T09:46:55 📋 Data Rate: 3/4 (75.0%)\n2025-09-24T09:46:55 📋 Avg Response Time: 1435ms\n2025-09-24T09:46:55 📋 Total Data Retrieved: 35.9KB\n2025-09-24T09:46:55 📋 AllCloud:\n2025-09-24T09:46:55 📋 Tests: 5\n2025-09-24T09:46:55 📋 Success Rate: 4/5 (80.0%)\n2025-09-24T09:46:55 📋 Data Rate: 4/5 (80.0%)\n2025-09-24T09:46:55 📋 Avg Response Time: 1950ms\n2025-09-24T09:46:55 📋 Total Data Retrieved: 69.0KB\n2025-09-24T09:46:55 📋 \n📂 ENDPOINT CATEGORY ANALYSIS:\n2025-09-24T09:46:55 📋 Cost Analysis: 3/3 working (3 with data)\n2025-09-24T09:46:55 📋 Budget Management: 1/1 working (1 with data)\n2025-09-24T09:46:55 📋 Recommendations: 3/3 working (3 with data)\n2025-09-24T09:46:55 📋 Kubernetes: 0/1 working (0 with data)\n2025-09-24T09:46:55 📋 Anomaly Detection: 0/1 working (0 with data)\n2025-09-24T09:46:55 ✅ \n✅ WORKING ENDPOINTS (with data):\n2025-09-24T09:46:55 📋 /invoices/caui - Basic unblended costs - 1250ms\n2025-09-24T09:46:55 📋 /budgets/v2/i/ - AWS budgets only - 890ms\n2025-09-24T09:46:55 📋 /recommendationsNew/heatmap/summary - All recommendations summary - 2100ms\n2025-09-24T09:46:55 📋 /invoices/caui - Service breakdown - amortized (Bank Leumi) - 1850ms\n2025-09-24T09:46:55 📋 /recommendationsNew/heatmap/summary - All recommendations summary (Bank Leumi) - 3200ms\n2025-09-24T09:46:55 📋 /invoices/caui - EC2 costs only (Bank Hapoalim) - 1100ms\n2025-09-24T09:46:55 📋 /recommendationsNew/heatmap/summary - All recommendations summary (Bank Hapoalim) - 2800ms\n2025-09-24T09:46:55 ❌ \n❌ FAILED ENDPOINTS:\n2025-09-24T09:46:55 ⚠️ /kubernetes - 404: No Kubernetes clusters found for this account\n2025-09-24T09:46:55 ⚠️ /anomaly-detection (Bank Leumi) - 403: Insufficient permissions for anomaly detection data\n2025-09-24T09:46:55 📋 \n🏦 MSP CUSTOMER COMPARISON:\n2025-09-24T09:46:55 📋 Bank Leumi:\n2025-09-24T09:46:55 📋 Endpoint Tests: 2/3 successful\n2025-09-24T09:46:55 📋 Data Available: 2/3 endpoints\n2025-09-24T09:46:55 📋 Potential Annual Savings: $120,000\n2025-09-24T09:46:55 📋 Bank Hapoalim:\n2025-09-24T09:46:55 📋 Endpoint Tests: 2/2 successful\n2025-09-24T09:46:55 📋 Data Available: 2/2 endpoints\n2025-09-24T09:46:55 📋 Potential Annual Savings: $85,000\n2025-09-24T09:46:55 📋 \n⚡ PERFORMANCE METRICS:\n2025-09-24T09:46:55 📋 Average Response Time: 1721ms\n2025-09-24T09:46:55 📋 Fastest Response: 800ms\n2025-09-24T09:46:55 📋 Slowest Response: 3200ms\n2025-09-24T09:46:55 📋 Total Data Retrieved: 0.10MB\n2025-09-24T09:46:55 📋 \n📋 GENERATING COMPREHENSIVE REPORTS:\n2025-09-24T09:46:55 ✅ Summary Report: /Users/david/Downloads/MCP/UmbrellaMCP/scripts/debug/demo-test-summary-1758707215093.json\n2025-09-24T09:46:55 ✅ Detailed Results: /Users/david/Downloads/MCP/UmbrellaMCP/scripts/debug/demo-test-results-1758707215093.json\n2025-09-24T09:46:55 ✅ CSV Export: /Users/david/Downloads/MCP/UmbrellaMCP/scripts/debug/demo-test-results-1758707215093.csv\n2025-09-24T09:46:55 📋 \n📊 WHAT THE REAL TEST PROVIDES:\n2025-09-24T09:46:55 📋 • Comprehensive endpoint coverage across all API categories\n2025-09-24T09:46:55 📋 • Multi-account testing (Direct customers vs MSP customers)\n2025-09-24T09:46:55 📋 • Parameter variation testing for each endpoint\n2025-09-24T09:46:55 📋 • Performance metrics and response time analysis\n2025-09-24T09:46:55 📋 • Data availability and content verification\n2025-09-24T09:46:55 📋 • Error detection and categorization\n2025-09-24T09:46:55 📋 • Customer-specific testing for MSP accounts\n2025-09-24T09:46:55 📋 • Detailed reporting in multiple formats (JSON, CSV)\n2025-09-24T09:46:55 📋 • Automated test execution without human intervention\n2025-09-24T09:46:55 ⚠️ \n🎯 TO RUN THE FULL TEST:\n2025-09-24T09:46:55 📋 1. Set environment variables:\n2025-09-24T09:46:55 📋 export SAOLA_PASSWORD=\"your_saola_password\"\n2025-09-24T09:46:55 📋 export ALLCLOUD_PASSWORD=\"your_allcloud_password\"\n2025-09-24T09:46:55 📋 2. Run the comprehensive test:\n2025-09-24T09:46:55 📋 node scripts/debug/direct-api-comprehensive-test.cjs\n2025-09-24T09:46:55 📋 # OR\n2025-09-24T09:46:55 📋 node scripts/debug/mcp-endpoint-test.cjs\n\n🎉 Demo completed! This shows what the full automated test provides.\nThe actual test will run all endpoints with real authentication and data.\n", "stderr": "", "success": true }, "directapitest": { "code": 0, "stdout": "🧪 Comprehensive Direct API Tester\n===================================\n⚠️ Missing environment variables:\n export SAOLA_PASSWORD=\"your_password_here\"\n export ALLCLOUD_PASSWORD=\"your_password_here\"\n\nYou can still run the test by setting passwords inline or modifying the script.\n2025-09-24T09:46:56.157Z 📋 🚀 Starting Comprehensive Direct API Test\n2025-09-24T09:46:56.158Z 📋 Testing 20 endpoints with multiple variations\n2025-09-24T09:46:56.158Z 📋 \n🏢 Testing SAOLA (Direct Customer)...\n2025-09-24T09:46:56.158Z 📋 Authenticating SAOLA (Direct Customer)...\n2025-09-24T09:46:56.316Z ❌ SAOLA (Direct Customer) authentication error: getaddrinfo ENOTFOUND saola.umbrellacost.com\n2025-09-24T09:46:56.316Z ⚠️ Skipping SAOLA (Direct Customer) due to authentication failure\n2025-09-24T09:46:56.316Z 📋 \n🏢 Testing AllCloud (MSP Customer)...\n2025-09-24T09:46:56.316Z 📋 Authenticating AllCloud (MSP Customer)...\n2025-09-24T09:46:57.192Z ❌ AllCloud (MSP Customer) authentication failed: 301 - \n2025-09-24T09:46:57.192Z ⚠️ Skipping AllCloud (MSP Customer) due to authentication failure\n2025-09-24T09:46:57.193Z 📋 \n📊 COMPREHENSIVE API TEST RESULTS\n2025-09-24T09:46:57.193Z 📋 Total Tests: 0\n2025-09-24T09:46:57.193Z ✅ Successful: 0 (NaN%)\n2025-09-24T09:46:57.193Z ❌ Failed: 0 (NaN%)\n2025-09-24T09:46:57.193Z 📋 Total Duration: 1.0s\n2025-09-24T09:46:57.194Z ✅ Summary report saved: /Users/david/Downloads/MCP/UmbrellaMCP/scripts/debug/api-test-summary-1758707217193.json\n2025-09-24T09:46:57.195Z ✅ Detailed results saved: /Users/david/Downloads/MCP/UmbrellaMCP/scripts/debug/api-test-results-detailed-1758707217194.json\n2025-09-24T09:46:57.195Z ✅ CSV results saved: /Users/david/Downloads/MCP/UmbrellaMCP/scripts/debug/api-test-results-1758707217195.csv\n2025-09-24T09:46:57.195Z 📋 \n🔄 ACCOUNT COMPARISON\n2025-09-24T09:46:57.195Z 📋 SAOLA (Direct Customer):\n2025-09-24T09:46:57.195Z 📋 Tests: 0\n2025-09-24T09:46:57.195Z 📋 Success: 0/0 (NaN%)\n2025-09-24T09:46:57.195Z 📋 With Data: 0/0 (NaN%)\n2025-09-24T09:46:57.195Z 📋 Avg Response Time: NaNms\n2025-09-24T09:46:57.195Z 📋 Status Codes: \n2025-09-24T09:46:57.195Z 📋 AllCloud (MSP Customer):\n2025-09-24T09:46:57.195Z 📋 Tests: 0\n2025-09-24T09:46:57.195Z 📋 Success: 0/0 (NaN%)\n2025-09-24T09:46:57.195Z 📋 With Data: 0/0 (NaN%)\n2025-09-24T09:46:57.195Z 📋 Avg Response Time: NaNms\n2025-09-24T09:46:57.195Z 📋 Status Codes: \n2025-09-24T09:46:57.195Z 📋 \n🔍 ENDPOINT ANALYSIS\n2025-09-24T09:46:57.195Z ✅ Top performing endpoints:\n2025-09-24T09:46:57.195Z ❌ \nProblematic endpoints:\n2025-09-24T09:46:57.195Z ✅ \nEndpoint analysis saved: /Users/david/Downloads/MCP/UmbrellaMCP/scripts/debug/api-endpoint-analysis-1758707217195.json\n2025-09-24T09:46:57.195Z 📋 \n🚨 ERROR ANALYSIS\n2025-09-24T09:46:57.195Z ❌ Total Errors: 0\n2025-09-24T09:46:57.195Z ⚠️ \nMost common error codes:\n2025-09-24T09:46:57.195Z ⚠️ \nMost problematic endpoints:\n\n🎉 Comprehensive direct API testing completed!\nCheck the generated reports for detailed analysis.\n", "stderr": "", "success": true }, "mcpservertest": { "code": 0, "stdout": "🧪 MCP Endpoint Comprehensive Tester\n====================================\n2025-09-24T09:46:58 📋 🚀 Starting MCP Endpoint Tests\n2025-09-24T09:46:58 ⚠️ Skipping SAOLA - no password set\n2025-09-24T09:46:58 ⚠️ Skipping AllCloud - no password set\n2025-09-24T09:46:58 ⚠️ No passwords provided. Set environment variables:\n2025-09-24T09:46:58 📋 export SAOLA_PASSWORD=\"your_password\"\n2025-09-24T09:46:58 📋 export ALLCLOUD_PASSWORD=\"your_password\"\n2025-09-24T09:46:58 📋 \n📊 TEST SUMMARY\n2025-09-24T09:46:58 📋 Total Tests: 0\n2025-09-24T09:46:58 ✅ Successful: 0 (NaN%)\n2025-09-24T09:46:58 ❌ Failed: 0 (NaN%)\n2025-09-24T09:46:58 📋 Duration: 0.0s\n2025-09-24T09:46:58 📋 \n📋 ACCOUNT RESULTS:\n2025-09-24T09:46:58 ✅ \nDetailed results: /Users/david/Downloads/MCP/UmbrellaMCP/scripts/debug/mcp-test-results-1758707218259.json\n2025-09-24T09:46:58 ✅ CSV results: /Users/david/Downloads/MCP/UmbrellaMCP/scripts/debug/mcp-test-results-1758707218260.csv\n\n🎉 Testing completed!\n", "stderr": "", "success": true } }, "nextSteps": { "forFullTesting": "Set SAOLA_PASSWORD and ALLCLOUD_PASSWORD environment variables", "forDevelopment": "Modify test scripts in scripts/debug/ directory", "forReporting": "Check generated report files in scripts/debug/ directory", "forIntegration": "Use these scripts in CI/CD pipelines or monitoring systems" } }