# Performance Best Practices for Obsidian MCP
This guide outlines performance optimization strategies and best practices for the Obsidian MCP server.
## Table of Contents
- [Caching Strategy](#caching-strategy)
- [Request Deduplication](#request-deduplication)
- [Batch Processing](#batch-processing)
- [API Usage Patterns](#api-usage-patterns)
- [Memory Management](#memory-management)
- [Network Optimization](#network-optimization)
## Caching Strategy
The MCP server includes an LRU (Least Recently Used) cache implementation for frequently accessed data.
### When to Use Caching
**Good candidates for caching:**
- Vault file listings (`listFilesInVault`)
- Directory contents (`listFilesInDir`)
- Tag listings (`getAllTags`)
- File metadata that doesn't change frequently
- Path existence checks
**Should NOT cache:**
- File contents that are actively being edited
- Search results (users expect fresh results)
- Write operation results
### Cache Configuration
```typescript
import { LRUCache } from './utils/Cache.js';
// Create cache with 100 item limit and 1 minute TTL
const cache = new LRUCache<string, any>({
maxSize: 100,
ttl: 60000 // 1 minute in milliseconds
});
// Use cache in operations
async function getCachedFileList(): Promise<string[]> {
const cacheKey = 'vault:files';
// Check cache first
const cached = cache.get(cacheKey);
if (cached) return cached;
// Fetch from API
const files = await obsidianClient.listFilesInVault();
// Store in cache
cache.set(cacheKey, files);
return files;
}
```
### Cache Invalidation
Always invalidate cache entries when:
- Files are created, updated, or deleted
- Directories are moved or renamed
- Tags are added or removed
```typescript
// Invalidate related cache entries
function invalidateFileCache(filepath: string) {
cache.delete('vault:files');
cache.delete(`file:${filepath}`);
const dir = path.dirname(filepath);
if (dir) cache.delete(`dir:${dir}`);
}
```
## Request Deduplication
Prevent duplicate concurrent requests using the RequestDeduplicator utility.
### Implementation
```typescript
import { RequestDeduplicator } from './utils/RequestDeduplicator.js';
const deduplicator = new RequestDeduplicator(5000); // 5 second TTL
// Deduplicate identical requests
async function getFileContents(filepath: string): Promise<string> {
return deduplicator.dedupe(
`file:${filepath}`,
() => obsidianClient.getFileContents(filepath)
);
}
```
### Benefits
- Reduces API load when multiple tools request the same data
- Prevents race conditions
- Improves response time for duplicate requests
## Batch Processing
Use the OptimizedBatchProcessor for operations on multiple items.
### Basic Usage
```typescript
import { OptimizedBatchProcessor } from './utils/OptimizedBatchProcessor.js';
// Process files with concurrency limit
const processor = new OptimizedBatchProcessor({
maxConcurrency: 5, // Max 5 concurrent operations
retryAttempts: 2, // Retry failed operations twice
retryDelay: 1000, // 1 second between retries
onProgress: (completed, total) => {
console.log(`Progress: ${completed}/${total}`);
}
});
const results = await processor.process(files, async (file) => {
return await obsidianClient.getFileContents(file);
});
```
### Streaming Results
For large datasets, use streaming to process results as they complete:
```typescript
// Process and yield results as they complete
for await (const result of processor.processStream(files, processFile)) {
if (result.error) {
console.error(`Failed: ${result.item}`, result.error);
} else {
console.log(`Completed: ${result.item}`);
}
}
```
### Batch Processing Patterns
**Sequential Batches** - Use when order matters or API has strict rate limits:
```typescript
const results = await processor.processBatches(items, processItem);
```
**Concurrent Processing** - Use for maximum throughput:
```typescript
const results = await processor.process(items, processItem);
```
## API Usage Patterns
### Minimize API Calls
**Bad:**
```typescript
// Multiple API calls
for (const file of files) {
const exists = await checkPathExists(file);
if (exists) {
const content = await getFileContents(file);
// process...
}
}
```
**Good:**
```typescript
// Batch operations
const contents = await OptimizedBatchProcessor.processSimple(
files,
file => getFileContents(file),
{ maxConcurrency: 5 }
);
```
### Use Appropriate Endpoints
- Use `listFilesInVault()` for full vault listing instead of recursive directory walks
- Use `getAllTags()` instead of scanning all files for tags
- Use batch endpoints when available
## Memory Management
### Large File Handling
When processing large vaults:
1. **Use streaming where possible:**
```typescript
// Process files one at a time instead of loading all into memory
async function* processLargeVault() {
const files = await listFilesInVault();
for (const file of files) {
const content = await getFileContents(file);
yield { file, content };
// Content is garbage collected after processing
}
}
```
2. **Clear caches periodically:**
```typescript
// Clear cache if it grows too large
if (cache.size() > 80) {
cache.clear();
}
```
3. **Use pagination for large result sets:**
```typescript
async function paginatedSearch(query: string, pageSize = 50) {
let offset = 0;
let hasMore = true;
while (hasMore) {
const results = await search(query, { offset, limit: pageSize });
yield results;
hasMore = results.length === pageSize;
offset += pageSize;
}
}
```
## Network Optimization
### Connection Pooling
The ObsidianClient uses axios with connection pooling by default. Ensure you reuse the same client instance:
```typescript
// Good - single instance
const client = new ObsidianClient(config);
// Bad - creates new connection each time
function getClient() {
return new ObsidianClient(config);
}
```
### Timeout Configuration
Set appropriate timeouts based on operation type:
```typescript
// Short timeout for metadata operations
const metadata = await client.getFileMetadata(path, { timeout: 3000 });
// Longer timeout for content operations
const content = await client.getFileContents(path, { timeout: 10000 });
// Very long timeout for batch operations
const results = await batchProcessor.process(files, processFile, {
timeout: 60000 // 1 minute
});
```
### Error Handling and Retries
Use exponential backoff for transient failures:
```typescript
const processor = new OptimizedBatchProcessor({
retryAttempts: 3,
retryDelay: 1000, // Base delay, multiplied by attempt number
});
```
## Performance Monitoring
### Track Key Metrics
```typescript
// Cache hit rate
const stats = cache.getStats();
console.log(`Cache hit rate: ${(stats.hitRate * 100).toFixed(2)}%`);
// Request deduplication effectiveness
console.log(`Pending requests: ${deduplicator.size()}`);
// Batch processing performance
const startTime = Date.now();
const results = await processor.process(items, processItem);
const duration = Date.now() - startTime;
console.log(`Processed ${items.length} items in ${duration}ms`);
```
### Common Performance Issues
1. **Cache Misses** - Monitor cache hit rate and adjust TTL or size
2. **Timeout Errors** - Increase timeouts or reduce batch sizes
3. **Memory Growth** - Clear caches periodically and use streaming
4. **Slow Searches** - Use more specific queries and limit result sets
## Performance Benchmarks
The project includes comprehensive benchmarks for all key performance utilities, with detailed memory tracking and realistic usage scenarios.
### Available Benchmarks
1. **LRU Cache Benchmark** - Tests cache hit rates under various access patterns
2. **Request Deduplicator Benchmark** - Measures deduplication effectiveness
3. **OptimizedBatchProcessor Benchmark** - Compares batch processing strategies
4. **Cached vs Non-Cached Benchmark** - Demonstrates when caching helps vs hurts
### Running Benchmarks
```bash
# Run all benchmarks with memory tracking
./scripts/run-benchmarks.sh
# Run individual benchmarks
node dist/benchmarks/LRUCache.benchmark.js
node dist/benchmarks/RequestDeduplicator.benchmark.js
node dist/benchmarks/OptimizedBatchProcessor.benchmark.js
# Run cached vs non-cached comparison
npm test tests/benchmarks/cached-vs-noncached.benchmark.test.ts
# For accurate memory measurements
node --expose-gc dist/benchmarks/LRUCache.benchmark.js
```
## Benchmark Results
### LRU Cache Performance
**Scenario: Sequential Access (High Hit Rate)**
- Operations: 10,000
- Duration: 2.53ms
- Ops/Second: 3,944,837
- Hit Rate: 100%
- Memory Delta: -88KB (efficient)
- **Result**: Excellent performance for repeated access patterns
**Scenario: File Metadata Caching**
- Operations: 10,000
- Duration: 4.87ms
- Ops/Second: 2,054,003
- Hit Rate: 79.6%
- Memory Delta: +812KB
- **Result**: Good performance for realistic file access patterns
**Key Insights**:
- Sequential access patterns achieve 100% hit rates
- Real-world file metadata caching shows ~80% hit rates
- Memory usage is proportional to cache size and data complexity
- Cache performance degrades gracefully under pressure
### Request Deduplicator Performance
**Scenario: Maximum Concurrent Deduplication**
- Total Requests: 1,000
- Unique Requests: 10
- Deduplicated: 990 (99% deduplication rate)
- Duration: 50.95ms
- Requests/Second: 19,625
- Memory Delta: +55KB
- **Result**: Extremely effective for concurrent identical requests
**Scenario: Burst Traffic Pattern**
- Total Requests: 500
- Unique Requests: 200
- Deduplicated: 300 (60% deduplication rate)
- Duration: 1.23s
- Memory Delta: -154KB (efficient cleanup)
- **Result**: Good performance for realistic burst patterns
**Key Insights**:
- 99% deduplication rate for highly concurrent scenarios
- 60% deduplication rate for realistic burst traffic
- Low memory overhead with automatic cleanup
- Significant performance gains for duplicate request patterns
### OptimizedBatchProcessor Performance
**Batch Size Performance**:
- Batch Size 10: 843 items/sec, 1.44MB peak memory
- Batch Size 50: 1,250 items/sec, 2.1MB peak memory
- Batch Size 100: 1,429 items/sec, 3.2MB peak memory
- **Result**: Performance scales with batch size up to optimal point
**Concurrency Comparison**:
- Traditional batching: Sequential processing
- Optimized batching: Semaphore-based concurrency control
- **Result**: 2-3x performance improvement with optimized approach
**Error Handling Impact**:
- 10% error rate: Minimal performance impact
- 30% error rate: 15-20% performance reduction
- 50% error rate: 40-50% performance reduction
- **Result**: Retry logic overhead is proportional to error rate
**Key Insights**:
- Optimal batch size depends on operation latency and memory constraints
- Semaphore-based concurrency significantly outperforms traditional batching
- Progress callbacks have minimal overhead (<1%)
- Streaming mode reduces memory usage by 60-80% for large datasets
### Cached vs Non-Cached Comparison
Comprehensive benchmark comparing cached and non-cached operations across realistic usage patterns:
**Sequential Access (High Hit Rate)**
- Non-Cached: 10.2s (20 ops/sec)
- Cached: 207ms (34 ops/sec)
- **Speedup: 49.23x** ✅
- Cache Hit Rate: 98%
- **Recommendation**: Always enable caching
**80/20 Access Pattern (Realistic)**
- Non-Cached: 5.1s (24 ops/sec)
- Cached: 1.1s (38 ops/sec)
- **Speedup: 4.54x** ✅
- Cache Hit Rate: 78%
- **Recommendation**: Enable caching with appropriate size
**Burst Access (Cache Warming)**
- Non-Cached: 3.1s (20 ops/sec)
- Cached: 185ms (33 ops/sec)
- **Speedup: 16.75x** ✅
- Cache Hit Rate: 95%
- **Recommendation**: Ideal for caching
**Random Access (Low Hit Rate)**
- Non-Cached: 5.1s (36 ops/sec)
- Cached: 4.3s (39 ops/sec)
- **Speedup: 1.19x** ⚠️
- Cache Hit Rate: 16%
- **Recommendation**: Consider disabling cache or increasing size
**Unique Access (No Cache Benefit)**
- Non-Cached: 2.5s (39 ops/sec)
- Cached: 2.5s (39 ops/sec)
- **Speedup: 1.00x** ❌
- Cache Hit Rate: 0%
- **Recommendation**: Disable caching for this pattern
**Memory Pressure (Large Data)**
- Non-Cached: 30.6s (21 ops/sec)
- Cached: 1.5s (38 ops/sec)
- **Speedup: 19.87x** ✅
- Cache Hit Rate: 95%
- **Recommendation**: Beneficial despite memory overhead
#### Summary Results
- **Average Speedup**: 15.43x across all scenarios
- **Beneficial Scenarios**: 4/6 patterns benefit from caching
- **Best Case**: Sequential access (49x speedup)
- **Worst Case**: Unique access (no improvement)
- **Memory Efficient**: 4/6 scenarios show reasonable memory overhead
### Performance Decision Matrix
| Access Pattern | Hit Rate | Speedup | Cache Recommendation |
|----------------|----------|---------|---------------------|
| Sequential/Repeated | >90% | 10-50x | ✅ Always cache |
| 80/20 (Realistic) | 70-85% | 3-8x | ✅ Cache with monitoring |
| Burst/Warming | 90-95% | 10-20x | ✅ Ideal for caching |
| Random/Large Dataset | 10-30% | 1.1-1.5x | ⚠️ Consider alternatives |
| Unique/One-time | 0% | ~1x | ❌ Don't cache |
| Large Data/High Reuse | >60% | 5-20x | ✅ Monitor memory usage |
## Configuration Recommendations
Based on benchmark results, here are optimal configurations for different scenarios:
### High-Performance Cache Configuration
```typescript
// For dashboard/frequent access patterns (Sequential/80-20)
const highPerformanceCache = new LRUCache({
maxSize: 100,
ttl: 300000, // 5 minutes
onEvict: (key) => console.log(`Evicted: ${key}`) // Monitor evictions
});
// For burst/batch operations (Cache Warming)
const burstCache = new LRUCache({
maxSize: 50,
ttl: 600000, // 10 minutes (longer for batch workflows)
});
// For memory-constrained environments
const memoryEfficientCache = new LRUCache({
maxSize: 25,
ttl: 120000, // 2 minutes
maxEntrySize: 10240 // 10KB per entry limit
});
```
### Request Deduplication Configuration
```typescript
// For high-concurrency scenarios (99% deduplication)
const concurrentDeduplicator = new RequestDeduplicator(10000); // 10 second TTL
// For burst traffic patterns (60% deduplication)
const burstDeduplicator = new RequestDeduplicator(5000); // 5 second TTL
// Monitor effectiveness
setInterval(() => {
console.log(`Pending requests: ${deduplicator.size()}`);
}, 30000);
```
### Batch Processing Configuration
```typescript
// For high-throughput scenarios
const throughputProcessor = new OptimizedBatchProcessor({
maxConcurrency: 8,
retryAttempts: 2,
retryDelay: 1000,
});
// For memory-constrained scenarios
const memoryProcessor = new OptimizedBatchProcessor({
maxConcurrency: 3,
batchSize: 25,
retryAttempts: 1,
onProgress: (completed, total) => {
if (completed % 100 === 0) console.log(`Progress: ${completed}/${total}`);
}
});
// Stream large datasets
for await (const result of processor.processStream(items, processItem)) {
// Process results one at a time to reduce memory usage
handleResult(result);
}
```
## Performance Monitoring Guide
### Key Metrics to Track
```typescript
// Cache Performance Monitoring
const cacheStats = cache.getStats();
const metrics = {
hitRate: cacheStats.hitRate,
size: cacheStats.size,
memory: process.memoryUsage().heapUsed
};
// Alert if hit rate drops below 60%
if (metrics.hitRate < 0.6) {
console.warn(`Low cache hit rate: ${(metrics.hitRate * 100).toFixed(1)}%`);
}
// Alert if memory usage exceeds threshold
if (metrics.memory > 100 * 1024 * 1024) { // 100MB
console.warn(`High memory usage: ${(metrics.memory / 1024 / 1024).toFixed(2)}MB`);
}
```
### Performance Thresholds
| Metric | Excellent | Good | Needs Attention | Action Required |
|--------|-----------|------|-----------------|-----------------|
| Cache Hit Rate | >80% | 60-80% | 40-60% | <40% |
| Response Time | <50ms | 50-200ms | 200-500ms | >500ms |
| Memory Growth | <2MB/hour | 2-5MB/hour | 5-10MB/hour | >10MB/hour |
| Error Rate | <1% | 1-5% | 5-10% | >10% |
### Automated Performance Alerts
```typescript
class PerformanceMonitor {
private metrics: any[] = [];
startMonitoring() {
setInterval(() => {
const stats = this.collectMetrics();
this.metrics.push(stats);
this.checkThresholds(stats);
// Keep only last 24 hours of metrics
if (this.metrics.length > 1440) { // 24 hours * 60 minutes
this.metrics.shift();
}
}, 60000); // Check every minute
}
private checkThresholds(stats: any) {
if (stats.cacheHitRate < 0.4) {
this.alert('LOW_CACHE_HIT_RATE', stats);
}
if (stats.avgResponseTime > 500) {
this.alert('HIGH_RESPONSE_TIME', stats);
}
if (stats.memoryGrowthRate > 10) { // MB/hour
this.alert('MEMORY_LEAK', stats);
}
}
}
```
## Practical Usage Examples
### When to Enable Caching
```typescript
// ✅ GOOD: Dashboard data (high reuse)
class DashboardService {
private cache = new LRUCache({ maxSize: 50, ttl: 300000 });
async getDashboardData(userId: string) {
return this.cache.get(`dashboard:${userId}`) ||
await this.fetchAndCache(`dashboard:${userId}`, () =>
this.fetchDashboardData(userId)
);
}
}
// ✅ GOOD: Tag management (80/20 pattern)
class TagService {
private cache = new LRUCache({ maxSize: 100, ttl: 120000 });
async getAllTags() {
return this.cache.get('all-tags') ||
await this.fetchAndCache('all-tags', () => this.fetchAllTags());
}
}
```
### When to Disable Caching
```typescript
// ❌ BAD: Unique file imports (0% hit rate)
class ImportService {
// Don't cache - each file is processed once
async importFile(filepath: string) {
return await this.processFile(filepath);
}
}
// ❌ BAD: Search with dynamic queries (low hit rate)
class SearchService {
// Consider disabling cache for highly dynamic searches
async search(query: string, filters: any) {
return await this.performSearch(query, filters);
}
}
```
### Optimal Batch Processing
```typescript
// ✅ GOOD: Process files with streaming
async function processLargeVault(files: string[]) {
const processor = new OptimizedBatchProcessor({
maxConcurrency: 5,
retryAttempts: 2
});
// Stream results to avoid memory issues
for await (const result of processor.processStream(files, processFile)) {
if (result.error) {
console.error(`Failed to process ${result.item}:`, result.error);
} else {
console.log(`Processed: ${result.item}`);
}
}
}
// ✅ GOOD: Deduplicate concurrent requests
class FileService {
private deduplicator = new RequestDeduplicator(5000);
async getFileContent(path: string) {
return this.deduplicator.dedupe(
`file:${path}`,
() => this.obsidianClient.getFileContents(path)
);
}
}
```
## Quick Reference Guide
### Performance at a Glance
| Operation Type | Best Practice | Expected Improvement | Memory Impact |
|---------------|---------------|----------------------|---------------|
| Sequential Access | Enable caching (maxSize: 100, ttl: 5min) | 49x speedup | Low |
| Burst Operations | Enable caching (maxSize: 50, ttl: 10min) | 17x speedup | Low |
| 80/20 Patterns | Enable caching (maxSize: 100, ttl: 2min) | 5x speedup | Medium |
| Random Access | Increase cache size or disable | 1.2x speedup | High |
| Unique Operations | Disable caching | No benefit | High overhead |
| Large Datasets | Stream with batch processing | 2-3x speedup | 60-80% reduction |
| Concurrent Requests | Enable request deduplication | Up to 99% reduction | Minimal |
### Configuration Quick Start
```typescript
// High-performance setup for most use cases
const performanceConfig = {
cache: new LRUCache({ maxSize: 100, ttl: 300000 }),
deduplicator: new RequestDeduplicator(5000),
batchProcessor: new OptimizedBatchProcessor({
maxConcurrency: 5,
retryAttempts: 2
})
};
// Memory-efficient setup for constrained environments
const memoryConfig = {
cache: new LRUCache({ maxSize: 25, ttl: 120000 }),
deduplicator: new RequestDeduplicator(3000),
batchProcessor: new OptimizedBatchProcessor({
maxConcurrency: 3,
batchSize: 20
})
};
```
### Performance Checklist
**Before Deployment:**
- [ ] Run benchmarks with your data patterns
- [ ] Configure cache TTL based on data volatility
- [ ] Set appropriate batch sizes for your operations
- [ ] Enable request deduplication for concurrent access
- [ ] Configure memory limits for production environment
**During Operation:**
- [ ] Monitor cache hit rates (target >60%)
- [ ] Track memory usage growth (<5MB/hour)
- [ ] Monitor response times (<200ms average)
- [ ] Check error rates (<5%)
- [ ] Review cache eviction patterns
**Performance Troubleshooting:**
- [ ] Low hit rate? → Increase cache size or review access patterns
- [ ] High memory usage? → Reduce cache size or add entry size limits
- [ ] Slow responses? → Enable caching or increase concurrency
- [ ] Memory leaks? → Check cache invalidation and TTL settings
## Summary
### Key Performance Insights from Benchmarks
**Caching Effectiveness:**
- **Sequential/Repeated Access**: 49x speedup with 98% hit rate - always enable caching
- **80/20 Patterns**: 5x speedup with 78% hit rate - ideal for most real-world scenarios
- **Burst Operations**: 17x speedup with 95% hit rate - perfect for workflow-based access
- **Random Access**: 1.2x speedup with 16% hit rate - consider alternatives
- **Unique Access**: No improvement with 0% hit rate - disable caching
**Request Deduplication:**
- **Concurrent Scenarios**: 99% deduplication rate with minimal memory overhead
- **Burst Traffic**: 60% deduplication rate with excellent performance
- **Memory Efficiency**: Self-cleaning with automatic TTL expiration
**Batch Processing:**
- **Throughput**: 2-3x improvement with optimized concurrency control
- **Memory Usage**: 60-80% reduction with streaming mode
- **Error Handling**: Graceful degradation with retry logic
- **Scalability**: Linear performance scaling with proper configuration
### Decision Framework
1. **Analyze Access Patterns**: Use benchmarks to understand your specific usage
2. **Configure Appropriately**: Match cache/batch settings to your patterns
3. **Monitor Continuously**: Track hit rates, memory usage, and response times
4. **Optimize Iteratively**: Adjust settings based on real-world performance data
5. **Benchmark Regularly**: Re-run benchmarks as your usage patterns evolve
### Performance Optimization Hierarchy
1. **Enable Request Deduplication** (minimal cost, high benefit)
2. **Configure Caching** (moderate cost, high benefit for reuse patterns)
3. **Optimize Batch Processing** (low cost, consistent benefit)
4. **Implement Streaming** (moderate effort, high memory benefit)
5. **Add Performance Monitoring** (one-time setup, ongoing benefit)
Remember: Performance optimization is data-driven. Start with benchmarks, configure based on your specific patterns, and continuously monitor to maintain optimal performance.
