M.I.M.I.R - Multi-agent Intelligent Memory & Insight Repository

# Scaling

**Scale NornicDB for high availability and performance.**

## Scaling Options

| Strategy | Use Case | Complexity |
|----------|----------|------------|
| Vertical | Quick wins | Low |
| Read Replicas | Read-heavy workloads | Medium |
| Sharding | Large datasets | High |

## Vertical Scaling

### Increase Resources

```yaml
# Docker Compose
services:
  nornicdb:
    deploy:
      resources:
        limits:
          memory: 8G
          cpus: '4'
```

### Memory Optimization

```bash
nornicdb serve \
  --memory-limit=4GB \
  --gc-percent=50 \
  --pool-enabled=true
```

### Query Optimization

```bash
nornicdb serve \
  --query-cache-size=5000 \
  --query-cache-ttl=10m \
  --parallel=true \
  --parallel-workers=4
```

## Read Replicas

### Hot Standby Architecture

```
┌─────────────┐     ┌─────────────┐
│   Primary   │────▶│   Replica   │
│   (Write)   │     │   (Read)    │
└─────────────┘     └─────────────┘
        │                  │
        ▼                  ▼
   Write Requests    Read Requests
```

### Configuration

```yaml
# Primary server
replication:
  role: primary
  replicas:
    - host: replica-1.nornicdb.local
      port: 7687
    - host: replica-2.nornicdb.local
      port: 7687

# Replica server
replication:
  role: replica
  primary:
    host: primary.nornicdb.local
    port: 7687
```

### Load Balancing

```nginx
# nginx.conf
upstream nornicdb_read {
    server replica-1:7474;
    server replica-2:7474;
}

upstream nornicdb_write {
    server primary:7474;
}

server {
    location /db/neo4j/tx/commit {
        # Route writes to primary
        proxy_pass http://nornicdb_write;
    }
    
    location /nornicdb/search {
        # Route reads to replicas
        proxy_pass http://nornicdb_read;
    }
}
```

## High Availability

### Raft Consensus

For automatic failover:

```yaml
cluster:
  enabled: true
  mode: raft
  nodes:
    - id: node-1
      host: node-1.nornicdb.local
      port: 7687
    - id: node-2
      host: node-2.nornicdb.local
      port: 7687
    - id: node-3
      host: node-3.nornicdb.local
      port: 7687
```

### Kubernetes StatefulSet

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nornicdb
spec:
  serviceName: nornicdb
  replicas: 3
  selector:
    matchLabels:
      app: nornicdb
  template:
    metadata:
      labels:
        app: nornicdb
    spec:
      containers:
      - name: nornicdb
        image: timothyswt/nornicdb-arm64-metal:latest
        env:
        - name: NORNICDB_CLUSTER_MODE
          value: "raft"
        - name: NORNICDB_NODE_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        ports:
        - containerPort: 7474
        - containerPort: 7687
        - containerPort: 7688  # Raft port
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi
```

## Caching

### Query Cache

```yaml
cache:
  query:
    enabled: true
    size: 5000
    ttl: 5m
  
  embedding:
    enabled: true
    size: 10000
```

### External Cache (Redis)

```yaml
cache:
  external:
    enabled: true
    type: redis
    address: redis:6379
```

## Performance Tuning

### Parallel Query Execution

```bash
nornicdb serve \
  --parallel=true \
  --parallel-workers=0 \  # Auto-detect CPUs
  --parallel-batch-size=1000
```

### Connection Pooling

```yaml
pool:
  enabled: true
  max_connections: 100
  idle_timeout: 5m
```

### Object Pooling

```bash
# Reduce memory allocations
nornicdb serve --pool-enabled=true
```

## Monitoring at Scale

### Key Metrics

- Request rate per node
- Replication lag
- Query latency percentiles
- Memory usage per node
- Disk I/O

### Prometheus Alerts

```yaml
groups:
  - name: nornicdb-scaling
    rules:
      - alert: HighLoad
        expr: nornicdb_http_requests_total > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High request rate - consider scaling"
      
      - alert: ReplicationLag
        expr: nornicdb_replication_lag_seconds > 10
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Replication lag detected"
```

## Capacity Planning

### Sizing Guidelines

| Nodes | Edges | RAM | Storage |
|-------|-------|-----|---------|
| 1M | 5M | 4GB | 10GB |
| 10M | 50M | 16GB | 100GB |
| 100M | 500M | 64GB | 1TB |

### Growth Projections

```bash
# Monitor growth
curl http://localhost:7474/metrics | grep nornicdb_nodes_total
curl http://localhost:7474/metrics | grep nornicdb_storage_bytes
```

## See Also

- **[Deployment](deployment.md)** - Deployment guide
- **[Monitoring](monitoring.md)** - Performance monitoring
- **[Clustering](../user-guides/clustering.md)** - HA clustering guide
- **[Cluster Security](cluster-security.md)** - Authentication for clusters
- **[Clustering Roadmap](../architecture/clustering-roadmap.md)** - Future sharding plans