# Automated Testing Guide
## Table of Contents
1. [Overview](#overview)
2. [Test Organization](#test-organization)
3. [Running Tests](#running-tests)
4. [Writing Tests](#writing-tests)
5. [Test Types](#test-types)
6. [Best Practices](#best-practices)
7. [Coverage](#coverage)
8. [Performance Testing](#performance-testing)
9. [Troubleshooting](#troubleshooting)
---
## Overview
This project uses **Vitest** as the testing framework with comprehensive test coverage across multiple dimensions:
- **Framework**: Vitest 4.0.14
- **Test Runner**: Node environment with forked pool
- **Coverage**: V8 provider with text and HTML reports
- **Property Testing**: fast-check for fuzz testing
- **Timeouts**: 60s for tests, 120s for hooks
### Key Features
- Single-fork execution to reduce memory overhead
- Sequential test execution for stability
- Comprehensive test categories (unit, integration, performance, etc.)
- Real data testing capabilities
- Coverage reporting
---
## Test Organization
Tests are organized in the `tests/` directory with the following structure:
```
tests/
├── unit/ # Unit tests for isolated functions
├── integration/ # Integration tests for end-to-end flows
├── indexing/ # Indexing-specific tests
│ ├── unit/
│ ├── integration/
│ ├── accuracy/
│ ├── performance/
│ ├── resource/
│ ├── edge-cases/
│ ├── security/
│ ├── contacts/
│ ├── caching/
│ └── negative/
├── perf/ # Performance benchmarks
├── contract/ # MCP protocol contract tests
├── fuzz/ # Fuzz and property-based tests
├── chaos/ # Chaos engineering tests
├── concurrency/ # Concurrency and parallelism tests
├── recovery/ # Error recovery and resilience tests
├── stress/ # Stress and load tests
├── timezone/ # Timezone handling tests
├── snapshots/ # Snapshot tests
└── personal/ # Personal/manual verification tests
```
---
## Running Tests
### Quick Start
```bash
# Run all tests
npm test
# Run with watch mode
npm run test:watch
# Run with coverage
npm run test:coverage
```
### By Category
#### Unit Tests
```bash
npm run test:unit
```
#### Integration Tests
```bash
npm run test:integration
# Specific integration test
npm run test:integration:periodic
```
#### Indexing Tests
```bash
# All indexing tests
npm run test:idx
# Specific indexing test categories
npm run test:idx:unit # Unit tests
npm run test:idx:integration # Integration tests
npm run test:idx:accuracy # Accuracy tests
npm run test:idx:perf # Performance tests
npm run test:idx:resource # Resource usage tests
npm run test:idx:edge # Edge cases
npm run test:idx:security # Security tests
npm run test:idx:contacts # Contacts tests
npm run test:idx:cache # Caching tests
npm run test:idx:negative # Negative test cases
```
#### Performance Tests
```bash
# All performance tests (with real data)
npm run perf
# Mock data only
npm run perf:mock
# Specific performance tests
npm run perf:indexing # Indexing performance
npm run perf:search # Search performance
npm run perf:tools # Tool performance
npm run perf:server # MCP server performance
npm run perf:embedding # Embedding performance
npm run perf:datasources # Data source performance
npm run perf:memory # Memory usage
npm run perf:stress # Stress tests (120s timeout)
# Quick performance check
npm run perf:quick
# Data source specific tests
npm run perf:mail
npm run perf:messages
npm run perf:calendar
npm run perf:contacts
# Advanced performance tests
npm run perf:negative # Negative cases
npm run perf:edge-cases # Edge cases
npm run perf:regression # Regression tests
npm run perf:lancedb # LanceDB specific
npm run perf:background # Background indexing
npm run perf:dates # Date parsing
# Full performance suite (120s timeout)
npm run perf:full
```
#### Other Test Types
```bash
npm run test:fuzz # Fuzz testing
npm run test:chaos # Chaos engineering
npm run test:contract # MCP protocol contracts
npm run test:concurrency # Concurrency tests
npm run test:recovery # Recovery tests
npm run test:timezone # Timezone tests
npm run test:stress # Stress tests
npm run test:snapshot # Snapshot tests
```
### Environment Variables
- `USE_REAL_DATA=1` - Use real data instead of mocks for performance tests
```bash
USE_REAL_DATA=1 npm run perf:search
```
---
## Writing Tests
### Basic Test Structure
```javascript
import { describe, it, expect, beforeAll, afterAll } from 'vitest'
describe('Feature Name', () => {
beforeAll(async () => {
// Setup code (runs once before all tests)
})
afterAll(async () => {
// Cleanup code (runs once after all tests)
})
it('should do something specific', async () => {
// Arrange
const input = 'test data'
// Act
const result = await functionUnderTest(input)
// Assert
expect(result).toBe('expected output')
})
})
```
### Unit Test Example
```javascript
import { describe, it, expect } from 'vitest'
import { parseDate } from '../lib/date-parser.js'
describe('Date Parser', () => {
it('should parse ISO date strings', () => {
const result = parseDate('2025-01-15')
expect(result).toBeInstanceOf(Date)
expect(result.getFullYear()).toBe(2025)
})
it('should handle invalid dates', () => {
const result = parseDate('invalid')
expect(result).toBeNull()
})
})
```
### Integration Test Example
```javascript
import { describe, it, expect, beforeAll } from 'vitest'
import { buildIndex, search } from '../lib/index.js'
describe('Search Integration', () => {
beforeAll(async () => {
await buildIndex()
}, 120000) // 2 minute timeout for setup
it('should find relevant results', async () => {
const results = await search('test query')
expect(results).toHaveLength(5)
expect(results[0].score).toBeGreaterThan(0.8)
})
})
```
### Performance Test Example
```javascript
import { describe, it, expect } from 'vitest'
describe('Performance', () => {
it('should complete search within time limit', async () => {
const start = Date.now()
await search('query')
const duration = Date.now() - start
expect(duration).toBeLessThan(1000) // Must complete in 1 second
})
})
```
### Fuzz Test Example
```javascript
import { describe, it } from 'vitest'
import fc from 'fast-check'
describe('Fuzz Tests', () => {
it('should handle arbitrary strings', () => {
fc.assert(
fc.property(fc.string(), (input) => {
// Function should not throw for any string
const result = processInput(input)
return result !== undefined
})
)
})
})
```
---
## Test Types
### 1. Unit Tests (`tests/unit/`)
Test individual functions and modules in isolation.
**Characteristics:**
- Fast execution
- No external dependencies
- Pure functions preferred
- Mock external calls
**Example Files:**
- `parsing.test.js` - Date and query parsing
- `search-logic.test.js` - Search algorithms
- `validators.test.js` - Input validation
- `error-handling.test.js` - Error handling
### 2. Integration Tests (`tests/integration/`)
Test how components work together.
**Characteristics:**
- May use real data sources
- Test full workflows
- Longer execution time
- End-to-end scenarios
**Example Files:**
- `tools.test.js` - Tool integration
- `advanced-features.test.js` - Complex features
- `periodic-indexing-e2e.test.js` - Background indexing
### 3. Performance Tests (`tests/perf/`)
Measure and benchmark performance.
**Characteristics:**
- Time-based assertions
- Memory usage monitoring
- Throughput testing
- Regression detection
**Example Files:**
- `search.perf.test.js` - Search performance
- `indexing.perf.test.js` - Indexing speed
- `memory.perf.test.js` - Memory profiling
- `stress.perf.test.js` - Stress testing
### 4. Indexing Tests (`tests/indexing/`)
Comprehensive testing of indexing functionality.
**Subcategories:**
- **Unit** - Core indexing functions
- **Integration** - Full indexing pipeline
- **Accuracy** - Result quality
- **Performance** - Speed and efficiency
- **Resource** - Memory and disk usage
- **Edge Cases** - Unusual inputs
- **Security** - Security boundaries
- **Contacts** - Contact indexing
- **Caching** - Cache behavior
- **Negative** - Error cases
### 5. Fuzz Tests (`tests/fuzz/`)
Property-based testing with random inputs using fast-check.
**Characteristics:**
- Random input generation
- Property verification
- Edge case discovery
- Regression prevention
### 6. Contract Tests (`tests/contract/`)
Verify MCP protocol compliance.
**Characteristics:**
- Protocol validation
- Schema verification
- API contracts
- Compatibility testing
### 7. Concurrency Tests (`tests/concurrency/`)
Test parallel execution and race conditions.
**Characteristics:**
- Thread safety
- Race condition detection
- Resource locking
- Parallel execution
### 8. Recovery Tests (`tests/recovery/`)
Test error recovery and resilience.
**Characteristics:**
- Failure scenarios
- Recovery mechanisms
- Data integrity
- Graceful degradation
### 9. Chaos Tests (`tests/chaos/`)
Introduce random failures to test resilience.
**Characteristics:**
- Random failures
- Resource exhaustion
- Network errors
- System instability
### 10. Stress Tests (`tests/stress/`)
Test system limits and breaking points.
**Characteristics:**
- High load
- Resource saturation
- Performance degradation
- Stability under pressure
---
## Best Practices
### 1. Test Structure
✅ **DO:**
- Use descriptive test names
- Follow AAA pattern (Arrange, Act, Assert)
- One assertion per test when possible
- Group related tests with `describe`
❌ **DON'T:**
- Write tests that depend on execution order
- Share state between tests
- Test implementation details
- Ignore failing tests
### 2. Async Testing
```javascript
// ✅ DO: Use async/await
it('should fetch data', async () => {
const data = await fetchData()
expect(data).toBeDefined()
})
// ❌ DON'T: Forget to await
it('should fetch data', () => {
const data = fetchData() // Missing await!
expect(data).toBeDefined() // Will fail
})
```
### 3. Test Isolation
```javascript
// ✅ DO: Clean up after tests
describe('Database Tests', () => {
afterEach(async () => {
await cleanupDatabase()
})
it('should create record', async () => {
await createRecord()
// Test will clean up
})
})
// ❌ DON'T: Leave test artifacts
it('should create record', async () => {
await createRecord()
// Database is now dirty for next test
})
```
### 4. Timeouts
```javascript
// ✅ DO: Set appropriate timeouts for slow operations
it('should build index', async () => {
await buildIndex()
}, 120000) // 2 minute timeout
// Default timeout is 60s - increase for slow tests
```
### 5. Mocking
```javascript
// ✅ DO: Mock external dependencies
import { vi } from 'vitest'
it('should call external API', async () => {
const mockFetch = vi.fn().mockResolvedValue({ data: 'test' })
global.fetch = mockFetch
await functionThatFetches()
expect(mockFetch).toHaveBeenCalled()
})
```
### 6. Real Data Testing
```javascript
// Use environment variable to control real data usage
const useRealData = process.env.USE_REAL_DATA === '1'
it('should search emails', async () => {
if (useRealData) {
const results = await searchRealEmails('query')
expect(results.length).toBeGreaterThan(0)
} else {
const results = await searchMockEmails('query')
expect(results).toEqual([/* mock data */])
}
})
```
### 7. Performance Assertions
```javascript
it('should complete quickly', async () => {
const start = Date.now()
await performOperation()
const duration = Date.now() - start
expect(duration).toBeLessThan(1000)
console.log(`Operation took ${duration}ms`)
})
```
---
## Coverage
### Generating Coverage Reports
```bash
npm run test:coverage
```
Coverage reports are generated in:
- **Text**: Console output
- **HTML**: `coverage/index.html`
### Coverage Configuration
Defined in [vitest.config.js](vitest.config.js:8-13):
- Provider: V8
- Includes: Root `.js` files and `lib/**/*.js`
- Excludes: `node_modules`, `tests`, `vitest.config.js`
### Coverage Goals
- **Overall**: Aim for 80%+ coverage
- **Critical paths**: 100% coverage for core functionality
- **Error handling**: Cover all error branches
- **Edge cases**: Test boundary conditions
### Viewing Coverage
```bash
# Generate and open HTML report
npm run test:coverage
open coverage/index.html
```
---
## Performance Testing
### Performance Test Structure
Performance tests should:
1. Measure execution time
2. Track memory usage
3. Set acceptable thresholds
4. Report metrics
5. Detect regressions
### Example Performance Test
```javascript
import { describe, it, expect } from 'vitest'
describe('Search Performance', () => {
const TIMEOUT_MS = 1000
const MAX_MEMORY_MB = 100
it('should search within time limit', async () => {
const start = Date.now()
const startMem = process.memoryUsage().heapUsed
const results = await search('test query')
const duration = Date.now() - start
const memUsed = (process.memoryUsage().heapUsed - startMem) / 1024 / 1024
console.log(`Search took ${duration}ms, used ${memUsed.toFixed(2)}MB`)
expect(duration).toBeLessThan(TIMEOUT_MS)
expect(memUsed).toBeLessThan(MAX_MEMORY_MB)
expect(results.length).toBeGreaterThan(0)
})
})
```
### Benchmarking
```javascript
const iterations = 1000
const start = Date.now()
for (let i = 0; i < iterations; i++) {
await operation()
}
const total = Date.now() - start
const avg = total / iterations
console.log(`Average: ${avg.toFixed(2)}ms`)
console.log(`Total: ${total}ms`)
console.log(`Throughput: ${(iterations / (total / 1000)).toFixed(2)} ops/sec`)
```
### Performance Test Best Practices
1. **Warm up**: Run operations once before measuring
2. **Multiple iterations**: Average over many runs
3. **Stable environment**: Close other applications
4. **Real data**: Use `USE_REAL_DATA=1` for realistic tests
5. **Baseline**: Establish baseline metrics
6. **Regression detection**: Alert on significant slowdowns
---
## Troubleshooting
### Common Issues
#### Tests Timeout
**Problem**: Tests exceed the timeout limit
**Solutions:**
```javascript
// Increase timeout for specific test
it('slow test', async () => {
// test code
}, 120000) // 2 minutes
// Or increase globally in vitest.config.js
```
#### Memory Issues
**Problem**: Tests consume too much memory
**Solutions:**
- Run tests sequentially (already configured)
- Clear caches between tests
- Use smaller datasets
- Check for memory leaks
```javascript
afterEach(() => {
// Clear caches
cache.clear()
// Force garbage collection (if available)
if (global.gc) global.gc()
})
```
#### Flaky Tests
**Problem**: Tests pass/fail inconsistently
**Solutions:**
- Avoid timing dependencies
- Ensure proper cleanup
- Remove test interdependencies
- Use deterministic data
- Add retry logic for network tests
```javascript
it.retry(3)('flaky network test', async () => {
// Test that might fail due to network issues
})
```
#### Real Data Tests Fail
**Problem**: Tests fail when using real data
**Solutions:**
```bash
# Build test index first
npm run test:idx:build
# Clean and rebuild if needed
npm run test:idx:clean
npm run test:idx:build
# Check environment
USE_REAL_DATA=1 npm run perf:search
```
#### Process Doesn't Exit
**Problem**: Test process hangs after completion
**Solution:**
- Already configured with `forceExit: true` in [vitest.config.js](vitest.config.js:33)
- Check for unclosed connections
- Ensure proper cleanup in `afterAll` hooks
```javascript
afterAll(async () => {
await closeConnections()
await cleanupResources()
})
```
### Debug Mode
Run tests with verbose output:
```bash
# Verbose reporter
npm run test:integration:periodic
# Node inspector
node --inspect-brk ./node_modules/vitest/vitest.js run
# Log level
DEBUG=* npm test
```
### Test Isolation
If tests are affecting each other:
```javascript
// Use separate describe blocks
describe.concurrent('isolated tests', () => {
it('test 1', async () => {
// Runs in isolation
})
it('test 2', async () => {
// Runs in isolation
})
})
```
---
## Additional Resources
- [Vitest Documentation](https://vitest.dev/)
- [fast-check Documentation](https://fast-check.dev/)
- [V8 Coverage Documentation](https://v8.dev/blog/javascript-code-coverage)
- Project configuration: [vitest.config.js](vitest.config.js)
- Test scripts: [package.json](package.json:7-62)
---
## Quick Reference
### Common Commands
```bash
# Development workflow
npm run test:watch # Watch mode
npm run test:unit # Quick unit tests
npm run test:integration # Integration tests
# Before commit
npm run test:coverage # Check coverage
npm test # Run all tests
# Performance
npm run perf:quick # Quick perf check
npm run perf:regression # Regression tests
# Indexing
npm run test:idx:unit # Index unit tests
npm run test:idx:accuracy # Index accuracy
# Troubleshooting
npm run test:idx:clean # Clean test index
npm run test:idx:build # Rebuild test index
```
### Test File Naming
- Unit tests: `*.test.js`
- Performance tests: `*.perf.test.js`
- Location: `tests/<category>/<name>.test.js`
### Configuration Files
- [vitest.config.js](vitest.config.js) - Vitest configuration
- [package.json](package.json) - Test scripts and dependencies