Code-Index-MCP

# MCP vs Native Performance - Interim Analysis Report ## Executive Summary After completing 11 out of 48 tests (22.9%), we have gathered sufficient data to identify clear performance patterns. **MCP tools are 1.9x faster on average** but use **3.5x more tokens**. Most critically, we discovered that **native tests are failing because test directories only contain index files, not source code**. ## Key Findings ### 1. Speed Performance 🚀 - **MCP Average**: 2,264ms per query - **Native Average**: 4,312ms per query - **Winner**: MCP is 1.9x faster ### 2. Token Efficiency 💰 - **MCP Average**: 9,686 tokens per query - **Native Average**: 2,762 tokens per query - **Winner**: Native uses 71% fewer tokens ### 3. Success Rates ✅ - **MCP**: 43% success rate (3/7 tests) - **Native**: 0% success rate (0/4 tests) - **Issue**: Native tests fail due to missing source files in test directories ## Critical Discovery The native tests are systematically failing because the test directories (`/workspaces/Code-Index-MCP/test_indexes/`) only contain index database files, not the actual source code. This makes native tool searches impossible. Example error: ``` "No source code files found in the directory - only index database files present" ``` ## Performance by Repository ### Go (Gin) - 7/12 tests complete - MCP: 50% success rate, avg 1,838ms - Native: 0% success rate (no source files) ### Python (Django) - 2/12 tests complete - MCP: 100% success rate, found BaseHandler successfully - Native: Failed due to missing source files ### JavaScript (React) - 1/12 tests complete - MCP: Failed to find React-specific results - Native: Not tested yet ### Rust (Tokio) - 1/12 tests complete - MCP: Failed to find tokio-specific results - Native: Not tested yet ## Key Insights 1. **MCP Cross-Repository Issue**: MCP searches are returning results from other repositories instead of the target repository, suggesting the index might not be properly scoped. 2. **Native Test Setup Issue**: The test configuration assumes source code is present, but only index files exist in the test directories. 3. **Symbol Searches**: MCP excels at symbol searches when properly indexed (e.g., Django BaseHandler found successfully). 4. **Content Searches**: Both approaches struggle with content searches (TODO, FIXME) in the test repositories. ## Recommendations ### Immediate Actions 1. **Fix Test Setup**: Either: - Point native tests to actual source code directories - Copy source files to test directories - Adjust test expectations 2. **Fix MCP Repository Scoping**: Ensure MCP searches are properly filtered to target repositories ### Performance Guidance (Preliminary) - **Use MCP for**: Symbol lookups, cross-file navigation, semantic searches - **Use Native for**: Simple pattern matching (if source access is available) - **Token Budget**: Consider 3.5x token cost when choosing MCP ## Next Steps 1. Continue testing remaining 37 configurations 2. Address the native test directory issue 3. Investigate MCP cross-repository result filtering 4. Complete full analysis once all tests are done --- **Note**: These are interim results based on 22.9% completion. Final recommendations may change with complete data.