Hostaway MCP Server

# Skill Evolution System - Complete Guide ## Overview The Skill Evolution System is a meta-framework that prevents skills from drifting away from project reality by automatically detecting failures, analyzing root causes, and updating skills with learnings. **Core Principle**: *Skills must evolve with the project or they become technical debt.* --- ## System Components ### 1. Skill Evolution Manager (Meta-Skill) **Location**: `~/.claude/skills/skill-evolution-manager.md` **Purpose**: Provides the framework for detecting failures, analyzing them, and updating skills. **Key Features**: - Failure detection patterns - Root cause analysis templates - Update generation workflows - Effectiveness tracking - Drift prevention strategies ### 2. Skill Usage Tracker **Location**: `.automation/scripts/track-skill-usage.sh` **Purpose**: Records skill usage and outcomes for effectiveness analysis. **Commands**: ```bash # Record skill was used ./track-skill-usage.sh use deploy-validator # Record successful outcome ./track-skill-usage.sh success deploy-validator # Record failure with reason ./track-skill-usage.sh failure deploy-validator "lint-check-failed" # Show skill statistics ./track-skill-usage.sh stats deploy-validator # Show all skills statistics ./track-skill-usage.sh all-stats ``` ### 3. Enhanced Pre-PR Validation **Location**: `.automation/scripts/pre-pr-validate.sh` **Purpose**: Applies lessons from PR #8 - formats code, runs integration tests, catches issues before CI. **Usage**: ```bash # Apply fixes and validate (recommended) .automation/scripts/pre-pr-validate.sh # Check-only mode (no automatic fixes) .automation/scripts/pre-pr-validate.sh --check-only ``` ### 4. Skill Changelog **Location**: `~/.claude/skills/history/changelog.md` **Purpose**: Tracks all skill updates with context, rationale, and expected impact. ### 5. Metrics Tracking **Location**: `.automation/logs/skill-metrics.json` **Purpose**: Stores effectiveness data for all skills. **Structure**: ```json { "skills": { "skill-name": { "total_uses": 0, "successful_uses": 0, "failures": 0, "effectiveness_rate": 0.0, "last_used": "ISO-8601", "last_updated": "ISO-8601", "failure_history": [] } } } ``` --- ## Workflow ### When CI/CD Fails ``` 1. DETECT ─────────> 2. ANALYZE ─────────> 3. UPDATE │ │ │ ├─ CI logs ├─ Root cause ├─ Generate update ├─ Skill used ├─ Skill gap ├─ Validate prevents failure └─ Failure type └─ Impact └─ Apply to skill file │ v 4. TRACK │ ├─ Record metrics ├─ Update changelog └─ Monitor effectiveness ``` ### Practical Example (PR #8) **Failure**: PR #8 failed lint/format and integration tests **Applied Workflow**: 1. **DETECT**: CI logs showed formatting issues and 500 errors 2. **ANALYZE**: Skill said "check" but should have said "apply" 3. **UPDATE**: Changed guidance from passive to active commands 4. **TRACK**: Recorded in metrics, expected 100% effectiveness **Result**: Updated deploy-validator skill now prevents both failure types --- ## How to Use the System ### For New Skills When creating a new skill: 1. **Add Effectiveness Tracking** ```markdown ## Skill Effectiveness **Last Updated**: [Date] **Effectiveness Rate**: [From metrics] **Recent Updates**: - [Date]: [What changed and why] ``` 2. **Initialize Metrics** ```bash .automation/scripts/track-skill-usage.sh use new-skill ``` ### When Using a Skill **Before**: ```bash # Record you're using the skill .automation/scripts/track-skill-usage.sh use deploy-validator ``` **After (Success)**: ```bash # Record successful outcome .automation/scripts/track-skill-usage.sh success deploy-validator ``` **After (Failure)**: ```bash # Record failure with reason .automation/scripts/track-skill-usage.sh failure deploy-validator "integration-tests-500" # Trigger skill evolution process # (Use skill-evolution-manager to analyze and update) ``` ### When Updating a Skill 1. **Document the Update** ```markdown # In skill file **UPDATED** (2025-10-31): [What changed] **Previous**: ~~[Old guidance]~~ **Current**: [New guidance] **Reason**: [Why it changed] ``` 2. **Update Changelog** ```bash cat >> ~/.claude/skills/history/changelog.md <<EOF ## $(date +%Y-%m-%d) - [Skill Name] Update **Triggered By**: [PR/Issue] **Changes**: [List of changes] **Impact**: [Expected improvement] EOF ``` 3. **Update Metrics** ```bash # Increment update counter in skill-metrics.json # (Automated by track-skill-usage.sh) ``` ### Weekly Review ```bash # Check all skills effectiveness .automation/scripts/track-skill-usage.sh all-stats # Review skills with low effectiveness (<80%) # Identify patterns in failure_history # Plan improvements for struggling skills ``` --- ## Success Metrics ### Skill-Level Metrics - **Effectiveness Rate**: (Successful Uses) / (Total Uses) - Target: >95% - **Time to Update**: Time between failure and skill update - Target: <24 hours - **Recurrence Rate**: Same failure after update - Target: 0% ### System-Level Metrics - **Total Skills**: Number of active skills - **Average Effectiveness**: Mean effectiveness across all skills - **Update Velocity**: Updates per week - **Failure Prevention**: Failures prevented by skill updates ### Sample Dashboard ``` ┌─────────────────────────────────────────────────┐ │ Skill Evolution System Status │ ├─────────────────────────────────────────────────┤ │ Total Skills: 5 │ │ Average Effectiveness: 94.2% │ │ Total Uses (30 days): 127 │ │ Failures Prevented: 23 │ │ Last Update: 2 days ago │ │ │ │ Top Performers: │ │ deploy-validator 100.0% (15 uses) │ │ skill-creator 97.5% (40 uses) │ │ │ │ Needs Attention: │ │ legacy-skill 72.0% (25 uses) ⚠️ │ │ │ └─────────────────────────────────────────────────┘ ``` --- ## Anti-Patterns to Avoid ### ❌ Don't: Update Without Analysis **Wrong**: ```markdown Updated skill because something failed ``` **Right**: ```markdown **UPDATED** (2025-10-31): Prevent lint failures **Root Cause**: Skill said "check" but CI requires "apply" **Evidence**: PR #8 failed with "3 files would be reformatted" **Fix**: Changed guidance to apply formatting first **Validation**: Tested - would have prevented PR #8 failure ``` ### ❌ Don't: Ignore Patterns **Wrong**: Update skill once, ignore if same failure repeats **Right**: Track patterns, escalate if >2 recurrences, deep review skill design ### ❌ Don't: Vague Guidance **Wrong**: ```markdown - [ ] Check formatting ``` **Right**: ```markdown - [ ] Run `ruff format src/ tests/` to APPLY formatting - [ ] Verify with `ruff format --check src/ tests/` ``` ### ❌ Don't: No Effectiveness Tracking **Wrong**: Update skill and forget about it **Right**: Track usage, monitor effectiveness, validate updates work --- ## Advanced: Predictive Updates ### Detect Drift Before Failure **Monitor**: ```bash # Watch for project changes that may affect skills git log --oneline --since="7 days ago" -- \ .github/workflows/ \ pyproject.toml \ .pre-commit-config.yaml \ README.md \ CLAUDE.md ``` **Alert**: - New workflow: Review deploy-validator skill - New tool: Update skill-creator skill - New convention: Update all skills **Auto-Suggest**: - Generate draft skill updates - Submit for human review - Apply proactively ### Continuous Improvement Cycle ``` Week 1: Skills created and used Week 2: Failures detected, skills updated Week 3: Validate updates prevent recurrence Week 4: Review effectiveness, identify patterns Week 5: Proactive updates based on project changes [Repeat] ``` --- ## File Structure ``` Project Root ├── .automation/ │ ├── docs/ │ │ ├── PR-8-LESSONS-LEARNED.md # Failure analysis │ │ ├── SKILL-EVOLUTION-EXAMPLE-PR8.md # Practical example │ │ └── SKILL-EVOLUTION-SYSTEM-README.md # This file │ ├── logs/ │ │ └── skill-metrics.json # Effectiveness data │ └── scripts/ │ ├── track-skill-usage.sh # Metrics tracker │ └── pre-pr-validate.sh # Enhanced validation ~/.claude/skills/ ├── deploy-validator.md # Updated skill ├── skill-evolution-manager.md # Meta-skill └── history/ └── changelog.md # Skill updates log ``` --- ## ROI Analysis ### Investment - Create skill evolution system: ~3 hours - Apply to first failure (PR #8): ~1 hour - Weekly maintenance: ~30 minutes - **Total (Month 1)**: ~6 hours ### Return - Time saved per PR: ~15 minutes - PRs per month: ~20 - CI failures prevented: ~10 - Developer frustration: Significantly reduced **Savings** (Month 1): - Time: 15 min × 20 PRs = 300 minutes (5 hours) - CI resources: 10 failed runs saved - **ROI**: Positive after 1 month **Long-term** (Year 1): - Time saved: 5 hours × 12 months = 60 hours - Skills continuously improving - Zero drift from project reality - Team productivity significantly increased --- ## Quick Reference ### Common Commands ```bash # Track skill usage ./track-skill-usage.sh use <skill> ./track-skill-usage.sh success <skill> ./track-skill-usage.sh failure <skill> "<reason>" # View stats ./track-skill-usage.sh stats <skill> ./track-skill-usage.sh all-stats # Run enhanced validation .automation/scripts/pre-pr-validate.sh # Check skill changelog cat ~/.claude/skills/history/changelog.md # View metrics cat .automation/logs/skill-metrics.json | jq ``` ### Update Checklist When updating a skill: - [ ] Document what changed and why - [ ] Reference triggering PR/issue - [ ] Validate update prevents failure - [ ] Update changelog - [ ] Update effectiveness metrics - [ ] Add examples - [ ] Test with real scenario --- ## Next Steps 1. **Apply to all skills**: Add effectiveness tracking to existing skills 2. **Automate tracking**: Hook into CI to auto-record outcomes 3. **Build dashboard**: Visualize skill effectiveness over time 4. **Expand detection**: Add more failure pattern detectors 5. **Cross-skill analysis**: Identify overlaps and gaps between skills --- ## Conclusion The Skill Evolution System ensures skills remain: - **Current**: Always aligned with project reality - **Effective**: >95% success rate - **Actionable**: Specific commands, not vague guidance - **Validated**: Updates proven to prevent failures - **Tracked**: Metrics show continuous improvement **Result**: Skills that improve over time instead of becoming stale, preventing project drift and maintaining core task focus. --- **For questions or improvements**: Update this system using itself! Apply the skill evolution manager to improve the skill evolution manager. 🔄