# Real MCP vs Native Performance Analysis Report
Generated: 2025-06-24
## Executive Summary
Based on actual Claude Code agent testing, we found significant differences between MCP and native retrieval methods:
- **Native search succeeded** where MCP failed to find the target
- **Native was faster** (2.5s vs 4.5s)
- **Native used fewer tokens** (850 vs 1,700)
- **MCP tools returned incomplete/incorrect results**
## Test Case: Finding BM25Indexer Class Definition
### MCP Performance
```json
{
"query": "Find the BM25Indexer class definition",
"mode": "mcp",
"tools_used": ["mcp__code-index-mcp__get_status", "mcp__code-index-mcp__symbol_lookup", "mcp__code-index-mcp__search_code"],
"results_found": 2,
"execution_time": 4.5,
"token_estimate": 1700,
"success": false
}
```
**Issues Identified:**
1. MCP tools couldn't find the actual class definition
2. `symbol_lookup` returned no results for "BM25Indexer"
3. `search_code` found only references, not the definition
4. Semantic search returned many irrelevant results
5. Index appears incomplete or improperly configured
### Native Performance
```json
{
"query": "Find the BM25Indexer class definition",
"mode": "native",
"tools_used": ["Grep", "Read"],
"results_found": 1,
"execution_time": 2.5,
"token_estimate": 850,
"success": true
}
```
**Success Factors:**
1. Grep immediately found 8 relevant files
2. Correctly identified the definition file
3. Read tool efficiently examined the specific file
4. Found exact class definition at line 36
## Detailed Comparison
| Metric | MCP | Native | Winner |
|--------|-----|--------|--------|
| Success | ❌ Failed | ✅ Success | Native |
| Time | 4.5s | 2.5s | Native (1.8x faster) |
| Tokens | 1,700 | 850 | Native (50% fewer) |
| Tool Calls | 6 | 2 | Native (more efficient) |
| Accuracy | Found references only | Found exact definition | Native |
## Root Cause Analysis
### Why MCP Failed
1. **Index Issues**: The MCP index appears incomplete or corrupted
2. **Plugin Problems**: Dispatcher timeout issues may have prevented proper indexing
3. **Schema Mismatch**: BM25 indexes have different schema than plugins expect
4. **Configuration**: MCP tools not properly connected to the index
### Why Native Succeeded
1. **Direct Access**: Grep searches actual files, not indexes
2. **Simple Pattern Matching**: "class BM25Indexer" is easy to find
3. **No Dependencies**: Doesn't rely on complex plugin system
4. **Predictable Behavior**: Standard Unix tools work reliably
## Implications
### When to Use Native Tools
- Simple pattern searches
- When MCP index is unavailable/incomplete
- Quick file location tasks
- Debugging index issues
### When MCP Should Excel (if working)
- Complex symbol relationships
- Cross-file dependencies
- Semantic understanding
- Large codebases where grep would be slow
## Recommendations
### Immediate Actions
1. **Fix MCP Index**: Rebuild index with proper schema
2. **Verify Plugin Loading**: Ensure plugins load without timeout
3. **Test Index Completeness**: Verify all files are indexed
4. **Add Fallback**: When MCP fails, automatically try native
### Long-term Improvements
1. **Hybrid Approach**: Use MCP for complex queries, native for simple
2. **Index Validation**: Add checks to ensure index completeness
3. **Performance Monitoring**: Track success rates for both methods
4. **User Guidance**: Provide clear indicators when to use each method
## Conclusion
This real-world test revealed critical issues with the MCP implementation:
- The theoretical 401x performance improvement is not realized in practice
- Native tools remain more reliable for basic searches
- MCP infrastructure needs significant fixes before it can deliver promised benefits
The test validates the importance of:
1. Maintaining native tool access as fallback
2. Properly testing index completeness
3. Real-world validation vs theoretical benchmarks
4. Understanding tool limitations
Until MCP issues are resolved, developers should prefer native tools for critical searches while MCP undergoes necessary improvements.
