Semantica Search MCP

🔍 Semantic code search for Claude Code - Index and search codebases using natural language with AI embeddings

Why Semantica?

Problem: Finding code with grep or regex is slow, requires exact syntax, and misses semantic relationships.

Solution: Semantica indexes your codebase using AI embeddings, enabling natural language search:

❌ Traditional: grep -r "def authenticate" app/
✅ Semantica: "Find authentication logic"
   → Returns auth functions, middleware, login flows across all files

Real examples:

• "Where is the database connection configured?" → Returns DB setup and connection code

• "Show error handling patterns" → Returns try/catch blocks, error classes, rescue blocks

• "Find user validation logic" → Returns validators, service methods, model validations

Related MCP server: RAGex

✨ Key Features

🚀 Production-Ready (Phases 1-3 Complete)

• ✅ 100% indexing success rate - AST split-merge chunking eliminates errors

• ✅ 2x faster than local - OpenAI provider outperforms Ollama

• ✅ Automatic re-indexing - Git hooks keep index fresh (<10s updates)

• ✅ Multiple providers - Ollama (local, free) or OpenAI (cloud, fast)

• ✅ Enhanced UX - Pre-flight estimates, progress tracking, clear guidance

🌳 AST-Based Indexing

• Smart code chunking preserves function/class boundaries

• Uses tree-sitter for language-aware parsing

• 50% chunk reduction vs naive splitting

• Supports TypeScript, JavaScript, Ruby

🎯 Hybrid Search

• Combines vector similarity (semantic) + TF-IDF (keywords)

• 40% more efficient than vector-only search

• Query expansion with code-specific synonyms

• Dynamic weight adjustment per query type

⚡ Auto Re-Indexing

• Git hooks detect changes automatically

• Incremental updates in <10 seconds (42x faster!)

• Merkle tree-based change detection

• Background processing (non-blocking)

🚀 Quick Start

Option 1: Local Setup (Free, Private)

Prerequisites: Docker

# 1. Start services
docker run -d -p 19530:19530 milvusdb/milvus:latest
docker run -d -p 11434:11434 ollama/ollama:latest
docker exec ollama ollama pull nomic-embed-text

# 2. Install Semantica
git clone <your-repo-url>
cd semantica-search-mcp
npm install && npm run build

# 3. Configure Claude Code
# Add to ~/.config/claude/claude_desktop_config.json (Linux)
# Or ~/Library/Application Support/Claude/claude_desktop_config.json (macOS)
{
  "mcpServers": {
    "semantica-search": {
      "command": "/absolute/path/to/semantica-search-mcp/build/index.js"
    }
  }
}

# 4. Index your first project
# In Claude Code:
"Index the codebase at /path/to/your-project"

Option 2: Cloud Setup (Fast, Scalable)

Prerequisites: OpenAI API key

# 1. Install Semantica (same as Option 1, steps 2-3)

# 2. Set API key
export OPENAI_API_KEY="sk-..."

# 3. Create project config
# In your project: .semantica/config.json
{
  "embedding": {
    "provider": "openai",
    "model": "text-embedding-3-small",
    "dimensions": 1536,
    "batchSize": 128,
    "concurrency": 3,
    "openai": {
      "apiKey": "${OPENAI_API_KEY}",
      "timeout": 30000
    }
  },
  "vectordb": {
    "provider": "milvus",
    "collectionName": "my_project"
  }
}

# 4. Index your project (same as Option 1)

⚙️ Configuration Guide

Configuration File Location

.semantica/config.json in your project root

Complete Configuration Reference

{
  "version": "1.0.0",

  "project": {
    "name": "my-project",
    "root": "/path/to/project",
    "languages": ["typescript", "javascript", "ruby"]
  },

  "indexing": {
    "granularity": "hybrid",
    "chunkingStrategy": "ast-split-merge",
    "maxChunkSize": 250,
    "overlap": 50,
    "include": ["src/**/*", "lib/**/*"],
    "exclude": ["node_modules/**", "**/*.test.*"],
    "languageConfig": {
      "typescript": {
        "extensions": [".ts", ".tsx"],
        "chunkTypes": ["function", "class", "interface", "type"]
      },
      "ruby": {
        "extensions": [".rb"],
        "chunkTypes": ["def", "class", "module"]
      }
    }
  },

  "embedding": {
    "provider": "openai",
    "model": "text-embedding-3-small",
    "dimensions": 1536,
    "batchSize": 128,
    "concurrency": 3,
    "openai": {
      "apiKey": "${OPENAI_API_KEY}",
      "timeout": 30000
    }
  },

  "vectordb": {
    "provider": "milvus",
    "collectionName": "my_project",
    "milvus": {
      "host": "localhost",
      "port": 19530,
      "indexType": "IVF_FLAT",
      "metricType": "COSINE"
    }
  },

  "search": {
    "strategy": "hybrid",
    "maxResults": 10,
    "minScore": 0.5,
    "hybrid": {
      "vectorWeight": 0.7,
      "keywordWeight": 0.3
    }
  }
}

Configuration Options Explained

indexing - What Files to Index

Best Practice:

{
  "include": ["src/**/*", "lib/**/*"], // Core code only
  "exclude": [
    "node_modules/**", // Dependencies
    "**/*.test.*", // Tests
    "**/*.spec.*", // Specs
    "dist/**", // Build output
    "coverage/**" // Test coverage
  ]
}

embedding - How to Generate Embeddings

Ollama Settings (Local, Free):

{
  "provider": "ollama",
  "model": "nomic-embed-text",
  "dimensions": 768,
  "batchSize": 64,
  "concurrency": 5,
  "ollama": {
    "host": "http://localhost:11434",
    "timeout": 30000
  }
}

OpenAI Settings (Cloud, Fast):

{
  "provider": "openai",
  "model": "text-embedding-3-small",
  "dimensions": 1536,
  "batchSize": 128,
  "concurrency": 3,
  "openai": {
    "apiKey": "${OPENAI_API_KEY}",
    "timeout": 30000
  }
}

vectordb - Where to Store Vectors

search - How to Search

🎯 Best Practices

For Small Projects (<500 files)

{
  "indexing": {
    "include": ["src/**/*"],
    "exclude": ["**/*.test.*"]
  },
  "embedding": {
    "provider": "ollama", // Free, fast enough
    "batchSize": 32,
    "concurrency": 3
  }
}

Time: <1 minute Cost: FREE

For Medium Projects (500-5K files)

{
  "indexing": {
    "include": ["src/**/*", "lib/**/*"],
    "exclude": ["node_modules/**", "**/*.test.*", "dist/**"]
  },
  "embedding": {
    "provider": "openai", // Faster, worth the cost
    "batchSize": 128,
    "concurrency": 3
  }
}

Time: 2-5 minutes Cost: $0.05-$0.15

For Large Projects (5K-10K files)

{
  "indexing": {
    "include": [
      "app/models/**/*", // Focus on core business logic
      "app/services/**/*",
      "app/queries/**/*"
    ],
    "exclude": [
      "**/*.test.*",
      "app/controllers/**", // Exclude less critical code
      "app/views/**"
    ]
  },
  "embedding": {
    "provider": "openai",
    "batchSize": 128,
    "concurrency": 3 // Safe for Tier 1
  }
}

Time: 10-15 minutes Cost: $0.10-$0.25

For CI/CD Integration

{
  "embedding": {
    "provider": "openai", // No Docker needed!
    "concurrency": 2, // Conservative for CI
    "openai": {
      "apiKey": "${OPENAI_API_KEY}" // From CI secrets
    }
  }
}

Advantage: No local infrastructure, easy setup

📊 Provider Comparison

Embedding Providers

OpenAI Models

Cost Examples (OpenAI text-embedding-3-small)

Daily incremental updates: <$0.10/day (practically free!)

🧪 Test Results & Validation

Unit Tests: 47/47 Passing ✅

npm test

# Results:
Test Suites: 3 passed
Tests:       47 passed (21 Ollama + 26 OpenAI)
Coverage:    100% (providers)
Time:        ~25s

Integration Tests - Real Codebases

Tested with real OpenAI and Ollama APIs:

*Smaller test set (352 files) for Ollama baseline

Key Findings:

• ✅ OpenAI is 39-43% faster for large repos

• ✅ 97-98% success rate with optimal settings (concurrency: 3)

• ✅ Cost is negligible ($0.001-$0.12 per project)

• ✅ Incremental re-indexing: <10 seconds (both providers)

Performance Benchmarks

Indexing Speed

Speed Comparison (OpenAI vs Ollama)

Large Ruby Project (8,367 files, 34,761 chunks):

OpenAI saves 8.5 minutes (39% faster) 🚀

📖 Usage Examples

Index a Codebase

"Index the codebase at /Users/me/Projects/my-app"

Output:

📊 Pre-flight check for my-app
────────────────────────────────────────────────────────

📁 Scope:
   • Files to index: 2,367
   • Estimated chunks: 8,474
   • Provider: openai

⏱️  Estimated time: ~2-3 minutes
   (This is a one-time operation)

💰 Estimated cost: ~$0.0297

🔍 System checks:
   ✅ Configuration file
   ✅ Vector database connection
   ✅ Embedding provider
   ✅ Disk space

✅ Ready to index!
   Indexing will run in background - you can continue working.

────────────────────────────────────────────────────────

🚀 Indexing started in background!

Job ID: index_1707445123
Estimated time: ~2-3 minutes
Estimated cost: ~$0.0297

💡 You can continue using Claude Code normally.
   Check progress: "Get index status"
   I'll show a summary when indexing completes!

📝 This is a one-time operation. Future updates via git hooks are <10s.

Search Code

"Search for authentication logic in my-app"

Returns:

🔍 Found 8 results (0.7s):

1. src/services/auth.service.ts:45-67 (score: 0.92)
   export class AuthService {
     async authenticate(credentials: Credentials) {
       // JWT-based authentication
     }
   }

2. src/middleware/auth.middleware.ts:12-28 (score: 0.87)
   export function requireAuth(req, res, next) {
     // Check JWT token
   }

Check Index Status

"Get index status for my-app"

While indexing:

📊 Indexing in progress (Job #index_1707445123)

Phase: Embedding
Progress: 67.3% (5,700/8,474 chunks)
Speed: 52 chunks/s
ETA: 2.1 minutes

After completion:

✅ Index Status for my-app

Collection: my_app
Status: Ready
Vectors: 8,346
Dimensions: 1536
Last updated: 2 minutes ago

🏆 What We've Achieved

Phase 2 Improvements (Complete)

• ✅ 100% indexing success (was 94%)

• ✅ 8-10x faster (5.9s vs 42s for small repos)

• ✅ Auto re-indexing via git hooks

• ✅ Background operations (non-blocking)

• ✅ Enhanced search quality (TF-IDF + query expansion)

• ✅ JavaScript support added

Phase 3.1 Improvements (Complete)

• ✅ OpenAI provider (2x faster for large repos)

• ✅ Pre-flight estimates (time/cost upfront)

• ✅ Better UX (clear guidance, suggestions)

• ✅ Language filtering (only index supported types)

• ✅ 26 unit tests (100% coverage on providers)

🛠️ MCP Tools

index_codebase - Index a project

Parameters:

• path (required): Project root directory

• background (optional): Run in background (default: true)

Features:

• Pre-flight estimates (files, time, cost)

• Health checks before starting

• Background mode by default

• Progress tracking

• Beautiful completion summary

search_code - Semantic search

Parameters:

• query (required): Natural language search query

• maxResults (optional): Number of results (default: 10)

• minScore (optional): Similarity threshold 0-1 (default: 0.7)

• language (optional): Filter by language

• pathPattern (optional): Filter by path regex

Features:

• Hybrid search (vector + keyword)

• Query expansion (synonyms)

• TF-IDF keyword extraction

• Ranked results with scores

get_index_status - Check status

Features:

• Live progress if indexing

• Collection statistics if idle

• Vector count and dimensions

• Last update timestamp

Additional Tools

• reindex_changed_files - Incremental update (<10s)

• enable_git_hooks - Auto re-index on git operations

• onboard_project - One-command setup

• reset_state - Emergency cleanup

• clear_index - Delete all data

⚡ Performance Tips

Optimize for Speed

1. Use OpenAI (2x faster for large repos)

{ "embedding": { "provider": "openai", "concurrency": 3 } }

2. Increase concurrency (if Tier 2+)

{ "embedding": { "concurrency": 5 } } // For Tier 2+ (5,000 RPM)

3. Selective indexing (index only core code)

{
  "indexing": {
    "include": ["app/models/**", "app/services/**"]
  }
}

Optimize for Cost

1. Use Ollama (completely free)

{ "embedding": { "provider": "ollama" } }

2. Selective indexing (fewer files = lower cost)

3. Use incremental updates (git hooks, automatic!)

Optimize for Reliability

1. Lower concurrency (97-98% success)

{ "embedding": { "concurrency": 3 } } // vs 5: more reliable

2. Use Ollama (100% success, no rate limits)

🔧 Troubleshooting

"No files found to index"

Cause: Include patterns don't match any files

Solution:

{
  "indexing": {
    "include": ["**/*.ts", "**/*.rb"], // Match all supported files
    "exclude": ["node_modules/**"]
  }
}

"Vector database not accessible"

Cause: Milvus not running

Solution:

# Check if running
curl http://localhost:19530/healthz

# Start if needed
docker run -d -p 19530:19530 milvusdb/milvus:latest

"Embedding provider not accessible"

For Ollama:

# Check if running
curl http://localhost:11434/api/tags

# Start if needed
ollama serve

For OpenAI:

# Check API key is set
echo $OPENAI_API_KEY

# Set if missing
export OPENAI_API_KEY="sk-..."

Rate Limiting (OpenAI)

Symptom: Many retry messages, <95% success rate

Solution: Reduce concurrency

{
  "embedding": {
    "concurrency": 2, // Down from 3 or 5
    "batchSize": 64 // Down from 128
  }
}

Slow Indexing

Cause: Large file count or conservative settings

Solutions:

1. Selective indexing - index only core directories

2. Increase concurrency - if no rate limits

3. Use OpenAI - 2x faster than Ollama

4. Exclude more - skip tests, docs, generated code

🎓 Advanced Usage

Incremental Re-Indexing

Automatic (Recommended):

"Enable git hooks for my-project"

Git hooks auto-update index on:

• Branch switches (<10s)

• Pull/merge operations (<10s)

• New commits (<10s)

Manual:

"Re-index changed files in my-project"

Multi-Project Setup

Index multiple projects independently:

# Project 1
cd /path/to/project1
# Create .semantica/config.json with collectionName: "project1"

# Project 2
cd /path/to/project2
# Create .semantica/config.json with collectionName: "project2"

# Index both
"Index the codebase at /path/to/project1"
"Index the codebase at /path/to/project2"

# Search specific project
"Search for auth in project1"

Provider Switching

Switch from Ollama to OpenAI:

1. Update config:

{
  "embedding": {
    "provider": "openai",
    "dimensions": 1536 // Changed from 768!
  }
}

2. Clear old index (dimension changed):

"Clear index for my-project"

3. Re-index:

"Index the codebase at /path/to/my-project"

📚 Documentation

All configuration options are documented in this README. For development guidance, see CLAUDE.md.

🤝 Contributing

Development Setup

git clone <repo-url>
cd semantica-search-mcp
npm install
npm run build

Development Workflow

npm run watch          # Auto-rebuild on changes
npm test              # Run all tests
npm run test:watch    # Watch mode
npm run test:coverage # Coverage report
npm run inspector     # MCP debugging

Code Quality

• TypeScript: Strict mode enabled

• Tests: Jest with 80%+ coverage target

• Linting: Automatic formatting

• Architecture: Provider pattern for extensibility

📈 Performance Metrics

Indexing Performance (Phase 2 → Phase 3)

Search Quality

🔒 Security & Privacy

Data Handling

Ollama (Local):

• ✅ 100% local processing

• ✅ No data leaves your machine

• ✅ Complete privacy

OpenAI (Cloud):

• ⚠️ Code chunks sent to OpenAI API

• ⚠️ Embeddings only (not searchable by OpenAI)

• ⚠️ Use environment variables for API keys (never commit!)

API Key Management

Never commit API keys:

{
  "openai": {
    "apiKey": "${OPENAI_API_KEY}" // ✅ Environment variable
  }
}

Not this:

{
  "openai": {
    "apiKey": "sk-proj-..." // ❌ NEVER hardcode!
  }
}

🎯 FAQ

Q: How long does indexing take? A: 3s-15 min depending on size. Small projects (<100 files): <30s. Large projects (5K+ files): 10-15 min. This is one-time - incremental updates are <10s!

Q: How much does OpenAI cost? A: $0.001-$0.20 per project for initial index. Daily updates: <$0.10. Most projects cost less than a coffee! ☕

Q: Can I switch between Ollama and OpenAI? A: Yes! Just update config and re-index (dimension change requires clearing old index).

Q: What happens if indexing is interrupted? A: Just re-run. It's a one-time operation, no checkpointing needed for simplicity.

Q: Does it work offline? A: With Ollama: YES (100% local). With OpenAI: NO (requires internet).

Q: How does this compare to Cursor or GitHub Copilot? A: Cursor indexes ~500-2K files in 1-3 min (with caching). We index ALL files (8K+) in 12-13 min. More complete, comparable speed when accounting for coverage.

🚀 What's Next

Completed ✅

• Phase 1: TypeScript/Ruby, Ollama, Milvus, AST chunking

• Phase 2: 100% success, auto re-index, JavaScript, performance

• Phase 3.1: OpenAI provider, UX improvements, testing

In Progress 🔄

• Phase 3.2: Qdrant vector DB provider (lighter alternative)

• Phase 3.3: Professional documentation

• Phase 3.4: Release v2.1.0

Future 🔮

• Python, Go, Java language support

• Embedding cache (50-70% faster re-indexing)

• BM25 keyword search

• Web dashboard UI

📄 License

Private (for now)

🙏 Acknowledgments

Built with research from:

• cAST: AST-Aware Code Chunking

• GitHub Copilot Semantic Indexing

• Cursor Code Indexing

• Claude Context MCP

Questions or issues? Check the documentation or create an issue.

Ready to get started? Follow the Quick Start guide above! 🚀