# V14.1.0 - Storage Engine Revolution & Performance Hardening
> **Release Date**: 2026-01-17
> **Codename**: "Async Phoenix" & "World-Class Architecture"
Boring V14.1.0 represents a fundamental architectural evolution, transforming the core storage and resilience infrastructure to achieve **world-class performance, reliability, and observability**.
---
## 🚀 Storage Engine Revolution (Phases 1-5)
### Phase 1: Core Stabilization
- **Concurrency Optimization**: Refactored `BrainManager.learn_pattern` to compute embeddings outside the global write lock, reducing contention during CPU-intensive operations.
- **Fast-Path Reconciler**: Optimized `SystemReconciler.check_integrity` to skip O(N) ledger scans when the checkpoint matches the current ledger state (sequence-based validation).
- **Forward-Compatible Locking**: Hardened `RobustLock` to gracefully handle future lock file formats (`len(parts) >= 2`), preventing accidental deletion of valid locks from newer versions.
### Phase 2: SQLite WAL Migration
- **Event Ledger Upgrade**: Migrated from JSONL file-based storage to **SQLite with Write-Ahead Logging (WAL)** for enhanced concurrency, ACID compliance, and durability.
- **Auto-Migration**: Seamless one-way migration from legacy `events.jsonl` to `events.db` with automatic state backup (`.jsonl` → `.migrated`, `state.json` → `.bak`).
- **Sequence-Based Integrity**: Replaced file byte offsets with integer sequence IDs for precise state tracking and reconciliation.
- **Hash Chain Preservation**: Maintained `prev_hash` and `checksum` fields for cryptographic integrity verification.
### Phase 3: Async & High Performance
- **Background Writer**: Implemented `BackgroundWriter` thread with batched SQLite commits, decoupling API response times from disk I/O.
- **43x Throughput Speedup**: Achieved **4373 RPS** in async mode vs **102 RPS** in sync mode (measured via `tests/stress/ledger_spam.py`).
- **User-Facing Benchmark**: New `boring doctor stress` command for production-ready performance validation.
- **Configurable Async Mode**: Added `ASYNC_LEDGER_WRITE` setting (default: `True`) for user control over background commit behavior.
### Phase 4: Resilience & Self-Healing
- **Circuit Breaker Pattern**: Protected critical paths (LLM calls, Brain Retrieval) from cascading failures using resilient circuit breakers with automatic recovery.
- **System Supervisor**: Centralized component health monitoring for background threads, database integrity, and circuit states.
- **Integrated Diagnostics**: `boring doctor check` now includes operational health reports from the Supervisor.
### Phase 5: World-Class Observability
- **Correlation IDs**: Implemented end-to-end request tracing across logs and telemetry using `contextvars.ContextVar`.
- **Structured JSON Logging**: Enhanced `BoringJSONFormatter` to include correlation IDs in every log entry for unified trace correlation.
- **Resource Monitoring**: Automated CPU and RAM usage tracking via `TelemetryManager.record_resource_usage()`.
- **Kernel Integration**: All flow executions now generate unique correlation IDs for complete observability.
---
## 📊 Performance Benchmarks
**Storage Engine Throughput** (measured via `boring doctor stress --count 1000`):
| Mode | Throughput (RPS) | Latency (1k events) | Speedup |
| :--- | :--- | :--- | :--- |
| **Sync (Legacy)** | 102.7 RPS | 9.83s | Baseline |
| **Async (New)** | **4373.4 RPS** | **0.23s** | **42.9x faster** |
**User-Facing Benchmark**: `boring doctor stress --count 100` achieved **2149 RPS** (100 events in 0.047s).
---
## 🔐 Data Integrity
- **Hash Chain Validation**: All events maintain cryptographic integrity via `prev_hash` and `checksum` fields.
- **Sequence Continuity**: Automated verification ensures no gaps or duplicates in event sequences.
- **Safe Migration**: Legacy JSONL data is preserved as `.migrated` backup after successful SQLite import.
- **Verified Integrity**: `EventStore.verify_integrity()` performs full hash-chain validation across all events.
---
## 🔬 New CLI Commands
| Command | Description |
|---------|-------------|
| `boring doctor stress [--count N] [--async/--sync]` | High-performance stress test for storage engine (benchmarks async vs sync throughput) |
---
## 🛠️ Technical Improvements
### Configuration
- **Async Storage Engine**: Added `ASYNC_LEDGER_WRITE: bool = True` to `config.py` for user control over background commit mode.
- **Restored Legacy Settings**: Fixed regression by restoring `MAX_HOURLY_CALLS`, `PROMPT_FILE`, and notification settings.
### Resilience
- **Circuit Breaker**: New `src/boring/utils/resilience.py` module with `CircuitBreaker` class for protecting external dependencies.
- **Supervisor**: New `src/boring/core/supervisor.py` module for centralized component health monitoring.
### Observability
- **Correlation Context**: Added `CORRELATION_ID` context variable to `telemetry.py` for request tracing.
- **Enhanced Logging**: Updated `logging.py` to inject correlation IDs into all JSON log entries.
---
## 🔧 Fixes
- **Syntax Errors**: Fixed duplicate `else` block in `reconciler.py` (line 136) causing `SyntaxError`.
- **Configuration Regression**: Restored missing `MAX_HOURLY_CALLS`, `PROMPT_FILE`, and notification settings to `config.py`.
- **Forward Compatibility**: Modified `RobustLock._is_lock_valid` to accept lock formats with `len(parts) >= 2`, preventing accidental deletion of future lock file versions.
- **Import Errors**: Added missing `uuid`, `time`, `json`, and `hashlib` imports to `events.py`.
---
## ⚠️ Breaking Changes
None. All changes are backward compatible with automatic migration.
---
## 📚 Documentation
- **Walkthrough**: Added comprehensive `walkthrough.md` documenting all Phase 1-5 improvements with benchmark results.
- **Task Tracking**: Updated `task.md` with completion status for all architectural upgrade phases.
- **Changelog**: Enhanced `CHANGELOG.md` with detailed v14.1.0 entry covering all improvements.
---
## 👥 Contributors
Thanks to the Boring Team for enabling this Storage Engine Revolution!
