MaverickMCP

# Integration Tests for MaverickMCP Orchestration This directory contains comprehensive integration tests for the multi-agent orchestration system implemented in MaverickMCP. ## Files - **`test_orchestration_complete.py`** - Comprehensive integration test suite with 18+ test scenarios - **`run_integration_tests.sh`** - Test runner script with environment setup - **`README.md`** - This documentation file ## Test Coverage ### Orchestration Tools Tested 1. **`agents_orchestrated_analysis`** - Multi-agent financial analysis with intelligent routing 2. **`agents_deep_research_financial`** - Web search-powered research with AI analysis 3. **`agents_compare_multi_agent_analysis`** - Multi-perspective agent comparison ### Test Categories - **Functional Tests** (12 scenarios) - Core functionality validation - **Error Handling** (3 scenarios) - Graceful error response testing - **Concurrent Execution** (3 scenarios) - Performance under concurrent load - **Performance Benchmarks** - Execution time and memory usage monitoring ## Usage ### Quick Test Run ```bash # From tests/integration directory ./run_integration_tests.sh # Or run directly python test_orchestration_complete.py ``` ### From Project Root ```bash # Run integration tests cd tests/integration && ./run_integration_tests.sh # Run specific test with uv uv run python tests/integration/test_orchestration_complete.py ``` ## Test Scenarios ### Orchestrated Analysis (4 tests) - Conservative LLM-powered routing - Aggressive rule-based routing - Moderate hybrid routing - Day trader fast execution ### Deep Research Financial (4 tests) - Basic company research - Standard sector research - Comprehensive market research - Exhaustive crypto research ### Multi-Agent Comparison (3 tests) - Market vs supervisor stock analysis - Conservative multi-agent portfolio - Aggressive growth strategy ### Error Handling (3 tests) - Invalid parameters for each tool - Graceful error response validation - Exception handling verification ### Performance Tests (3 tests) - Concurrent execution stress test - Memory usage monitoring - Response time benchmarking ## Expected Results **Successful Test Run:** ``` 🎉 ALL TESTS PASSED! (18/18) Total Execution Time: 45.67s Average execution time: 2340ms ``` **Performance Benchmarks:** - Orchestrated Analysis: ~1500ms average - Deep Research: ~25000ms average (includes web search) - Multi-Agent Comparison: ~3000ms average ## Test Configuration Tests are configured in `test_orchestration_complete.py`: ```python TEST_CONFIG = { "timeout_seconds": 300, # 5 minutes max per test "concurrent_limit": 3, # Maximum concurrent tests "performance_monitoring": True, "detailed_validation": True, "save_results": True, } ``` ## Output Files - **Log file**: `integration_test_YYYYMMDD_HHMMSS.log` - **Results file**: `integration_test_results_YYYYMMDD_HHMMSS.json` ## Requirements - Python 3.12+ - All MaverickMCP dependencies installed - Optional: OPENAI_API_KEY for real LLM calls - Optional: EXA_API_KEY, TAVILY_API_KEY for web search testing ## Troubleshooting **Import Errors:** - Ensure you're running from the correct directory - Check that all dependencies are installed: `uv sync` **API Errors:** - Tests use mock responses if API keys aren't configured - Add API keys to `.env` file for full functionality testing **Timeout Errors:** - Increase `timeout_seconds` in TEST_CONFIG for slower systems - Some tests (especially deep research) take 20-30 seconds ## Integration with CI/CD The test suite returns proper exit codes: - `0` - All tests passed - `1` - Some tests failed - `2` - Test suite execution failed - `130` - Interrupted by user (SIGINT) Example CI usage: ```yaml - name: Run Integration Tests run: | cd tests/integration ./run_integration_tests.sh ```