# Phase 4 Refactoring Plan - Large Core Files
**Start Date**: January 2025
**Goal**: Target large core files (900+ lines) for modularization and refactoring
**Estimated Impact**: 2,000-3,500 line reduction
## Phase 4 Overview
After successfully refactoring tool files in Phase 3 (758 lines eliminated), Phase 4 targets the largest files in the codebase for modularization, extraction of reusable components, and architectural improvements.
## Target Files Analysis
### Top 10 Largest Files
```
1. crackerjack_integration.py 1,632 lines
2. crackerjack_tools.py 1,340 lines
3. serverless_mode.py 1,285 lines
4. quality_engine.py 1,256 lines
5. llm_providers.py 1,254 lines
6. advanced_search.py 1,023 lines
7. server_core.py 983 lines
8. natural_scheduler.py 964 lines
9. session_manager.py 947 lines
10. team_knowledge.py 924 lines
```
**Total lines in top 10**: 11,608 lines (30.4% of estimated 38k codebase)
## Phase 4 Strategy
### Approach
1. **Analyze each large file** for extraction opportunities
1. **Identify common patterns** across files
1. **Extract modules** where appropriate
1. **Create specialized utilities** for specific domains
1. **Maintain 100% functionality** throughout
### Focus Areas
1. **Modularization** - Break large files into logical modules
1. **Extraction** - Pull reusable components into utilities
1. **Simplification** - Reduce complexity and duplication
1. **Organization** - Better file structure and separation of concerns
## Priority Targets (Days 1-5)
### Day 1: Crackerjack Integration (1,632 lines)
**File**: `crackerjack_integration.py`
**Estimated Reduction**: 400-600 lines (25-37%)
**Analysis Needed**:
- Command execution patterns
- Output parsing logic
- Progress tracking
- Result aggregation
- Quality metrics calculation
**Extraction Opportunities**:
- `utils/crackerjack/command_execution.py` - Command runners
- `utils/crackerjack/output_parser.py` - Parse tool output
- `utils/crackerjack/progress_tracker.py` - Progress tracking
- `utils/crackerjack/metrics_aggregator.py` - Metrics collection
**Expected Structure**:
```
crackerjack_integration.py (1,032 lines after)
└─ utils/crackerjack/
├─ command_execution.py (150 lines)
├─ output_parser.py (200 lines)
├─ progress_tracker.py (150 lines)
└─ metrics_aggregator.py (100 lines)
```
### Day 2: Quality Engine (1,256 lines)
**File**: `quality_engine.py`
**Estimated Reduction**: 300-500 lines (24-40%)
**Analysis Needed**:
- Quality scoring algorithms
- Project health checks
- Code analysis integration
- Trend analysis
- Recommendation generation
**Extraction Opportunities**:
- `utils/quality/scoring.py` - Quality score algorithms
- `utils/quality/health_checks.py` - System health checks
- `utils/quality/trend_analysis.py` - Historical trends
- `utils/quality/recommendations.py` - Recommendation engine
**Expected Structure**:
```
quality_engine.py (756-956 lines after)
└─ utils/quality/
├─ scoring.py (150 lines)
├─ health_checks.py (150 lines)
├─ trend_analysis.py (100 lines)
└─ recommendations.py (100 lines)
```
### Day 3: Serverless Mode (1,285 lines)
**File**: `serverless_mode.py`
**Estimated Reduction**: 300-450 lines (23-35%)
**Analysis Needed**:
- Storage backend implementations
- Session serialization
- State management
- Backend-specific adapters
- Configuration handling
**Extraction Opportunities**:
- `backends/redis_backend.py` - Redis storage implementation
- `backends/s3_backend.py` - S3-compatible storage
- `backends/local_backend.py` - Local file storage
- `utils/session_serialization.py` - Serialization utilities
**Expected Structure**:
```
serverless_mode.py (835-985 lines after)
└─ backends/
├─ redis_backend.py (150 lines)
├─ s3_backend.py (150 lines)
├─ local_backend.py (100 lines)
└─ base_backend.py (100 lines)
```
### Day 4: Session Manager (947 lines)
**File**: `core/session_manager.py`
**Estimated Reduction**: 250-400 lines (26-42%)
**Analysis Needed**:
- Session lifecycle operations
- Quality assessment integration
- Git operations
- Context management
- State tracking
**Extraction Opportunities**:
- `core/lifecycle.py` - Session lifecycle operations
- `core/quality_assessment.py` - Quality checks
- `core/context_manager.py` - Context handling
- `core/state_tracker.py` - State management
**Expected Structure**:
```
session_manager.py (547-697 lines after)
└─ core/
├─ lifecycle.py (150 lines)
├─ quality_assessment.py (100 lines)
├─ context_manager.py (100 lines)
└─ state_tracker.py (100 lines)
```
### Day 5: LLM Providers (1,254 lines)
**File**: `llm_providers.py`
**Estimated Reduction**: 350-500 lines (28-40%)
**Analysis Needed**:
- Provider-specific implementations
- API client wrappers
- Response formatting
- Error handling per provider
- Configuration management
**Extraction Opportunities**:
- `llm/providers/openai_provider.py` - OpenAI implementation
- `llm/providers/gemini_provider.py` - Google Gemini
- `llm/providers/ollama_provider.py` - Ollama local
- `llm/base_provider.py` - Base provider class
- `llm/response_formatter.py` - Response formatting
**Expected Structure**:
```
llm_providers.py (754-904 lines after)
└─ llm/
├─ base_provider.py (150 lines)
├─ response_formatter.py (100 lines)
└─ providers/
├─ openai_provider.py (150 lines)
├─ gemini_provider.py (150 lines)
└─ ollama_provider.py (150 lines)
```
## Secondary Targets (Days 6-8)
### Day 6: Advanced Search (1,023 lines)
**Estimated Reduction**: 250-350 lines (24-34%)
**Extraction Opportunities**:
- Faceted search logic
- Full-text indexing
- Aggregation functions
- Query builders
### Day 7: Server Core (983 lines)
**Estimated Reduction**: 200-300 lines (20-31%)
**Extraction Opportunities**:
- MCP server setup
- Tool registration logic
- Middleware configuration
- Request handling
### Day 8: Natural Scheduler (964 lines)
**Estimated Reduction**: 200-300 lines (21-31%)
**Extraction Opportunities**:
- Time parsing logic
- Schedule management
- Reminder system
- Task queue
## Estimated Impact Summary
### Priority Targets (Days 1-5)
| File | Current | Estimated After | Reduction | % |
|------|---------|----------------|-----------|---|
| crackerjack_integration.py | 1,632 | 1,032-1,232 | 400-600 | 25-37% |
| quality_engine.py | 1,256 | 756-956 | 300-500 | 24-40% |
| serverless_mode.py | 1,285 | 835-985 | 300-450 | 23-35% |
| session_manager.py | 947 | 547-697 | 250-400 | 26-42% |
| llm_providers.py | 1,254 | 754-904 | 350-500 | 28-40% |
| **TOTAL** | **6,374** | **3,924-4,774** | **1,600-2,450** | **25-38%** |
### Secondary Targets (Days 6-8)
| File | Current | Estimated After | Reduction | % |
|------|---------|----------------|-----------|---|
| advanced_search.py | 1,023 | 673-773 | 250-350 | 24-34% |
| server_core.py | 983 | 683-783 | 200-300 | 20-31% |
| natural_scheduler.py | 964 | 664-764 | 200-300 | 21-31% |
| **TOTAL** | **2,970** | **2,020-2,320** | **650-950** | **22-32%** |
### Phase 4 Total Potential
- **Files targeted**: 8 large files
- **Current lines**: 9,344 lines
- **Estimated reduction**: 2,250-3,400 lines (24-36%)
- **New modules created**: ~25-30 specialized modules
- **Duration**: 8 working days
## Implementation Principles
### 1. Module Extraction Pattern
```python
# Before: All in one file
class BigClass:
def method1(self): ...
def method2(self): ...
def method3(self): ...
# ... 50 more methods
# After: Separated by concern
# component1.py
class Component1:
def method1(self): ...
# component2.py
class Component2:
def method2(self): ...
# main_file.py
from .component1 import Component1
from .component2 import Component2
class BigClass:
def __init__(self):
self.comp1 = Component1()
self.comp2 = Component2()
```
### 2. Backend/Provider Pattern
```python
# Before: All providers in one file
class LLMProviders:
def openai_call(self): ...
def gemini_call(self): ...
def ollama_call(self): ...
# After: Each provider separate
# providers/base.py
class BaseProvider: ...
# providers/openai_provider.py
class OpenAIProvider(BaseProvider): ...
# llm_providers.py
from .providers.openai_provider import OpenAIProvider
```
### 3. Utility Extraction Pattern
```python
# Before: Utility functions mixed with business logic
class Manager:
def parse_output(self, text): ... # 50 lines
def calculate_score(self, data): ... # 40 lines
def main_logic(self): ... # Uses above
# After: Utilities extracted
# utils/parsers.py
def parse_output(text): ...
# utils/scoring.py
def calculate_score(data): ...
# manager.py
from utils.parsers import parse_output
from utils.scoring import calculate_score
class Manager:
def main_logic(self): ...
```
## Testing Strategy
### For Each Refactoring
1. **Before**: Read file, count lines, test imports
1. **Extract**: Create new modules with extracted code
1. **Integrate**: Update original file to use extracted modules
1. **Test**: Verify imports work, no functionality changes
1. **Commit**: Individual commit per file refactored
### Validation Checklist
- ✅ All imports successful
- ✅ Original file significantly smaller
- ✅ New modules are focused and cohesive
- ✅ No duplicate code between modules
- ✅ Clear module boundaries
- ✅ 100% functional compatibility
## Success Criteria
### Phase 4 Goals
- [ ] Reduce 8 large files by 2,250-3,400 lines (24-36%)
- [ ] Create 25-30 focused, reusable modules
- [ ] Improve codebase organization and maintainability
- [ ] Maintain 100% functional compatibility
- [ ] All tests passing
- [ ] Zero breaking changes
### Combined Progress (Phases 1-4)
- **Phase 1-2**: 34 lines (test infrastructure)
- **Phase 3**: 758 lines (tool file duplication)
- **Phase 4**: 2,250-3,400 lines (large file modularization)
- **Total**: 3,042-4,192 lines eliminated
- **Progress to goal**: 30-42% of 10,000 line target
## Risk Mitigation
### Potential Issues
1. **Import cycles** - Careful module organization needed
1. **Dependency injection** - May need to refactor DI setup
1. **Test updates** - Tests may reference old locations
1. **Documentation** - Need to update module references
### Mitigation Strategies
1. Use clear module hierarchies to avoid cycles
1. Leverage existing ACB DI patterns
1. Use import-based testing to catch issues early
1. Update CLAUDE.md as we go
## Next Steps
1. **Start with Day 1** - Analyze crackerjack_integration.py
1. **Create extraction plan** - Identify specific extraction boundaries
1. **Extract modules** - Create new focused modules
1. **Refactor main file** - Use extracted modules
1. **Test and commit** - Validate and commit changes
1. **Repeat** - Move to next file
Let's begin with **crackerjack_integration.py** (1,632 lines)! 🚀
______________________________________________________________________
*Previous phases: REFACTORING_PHASE3_COMPLETE.md*
