# Production Safety & Validation
## Why This Matters
Ansible and Chef automation affects **production systems** - infrastructure, applications, databases, and ongoing operations. A bad conversion can cause outages, data loss, or security vulnerabilities. SousChef provides tools to validate migrations thoroughly before production deployment.
This guide explains how to:
- Verify generated Ansible is functionally equivalent to Chef
- Test conversions in safe environments
- Build confidence before production rollout
- Plan rollback strategies
- Execute phased migrations with minimal risk
## The Safety Pyramid
```
/\
/ \ Production Deploy (100% confidence)
/────\
/ \ Production Pilot (95% confidence)
/────────\
/ \ Staging Validation (90% confidence)
/────────────\
/ \ Development Testing (80% confidence)
/────────────────\
/ \ Static Analysis (baseline)
/────────────────────\
```
Work your way up the pyramid. Never skip steps.
## Phase 1: Static Analysis (Baseline)
### Validate Conversion Accuracy
**Use SousChef's validation tools** to check the conversion quality:
```
Validate the conversion from cookbook-name
```
Your AI assistant will use the `validate_conversion` tool to:
- Compare Chef resources vs generated Ansible tasks
- Identify missing resources or incorrect mappings
- Flag complex Chef patterns that need manual review
- Highlight security-sensitive resources (users, SSH keys, credentials)
**Manual Review Checklist:**
- [ ] All Chef resources have corresponding Ansible tasks
- [ ] Guards (`only_if`, `not_if`) properly converted to `when` conditions
- [ ] Resource notifications mapped to handlers
- [ ] File/template permissions preserved
- [ ] Service management actions correct (start, enable, restart)
- [ ] Package version specifications maintained
- [ ] Complex bash/ruby scripts reviewed for shell escaping
### Compare Side-by-Side
**For critical resources**, manually compare Chef and Ansible:
```ruby
# Chef original
package 'nginx' do
version '1.18.0-6ubuntu14.4'
action :install
notifies :restart, 'service[nginx]'
end
```
```yaml
# Generated Ansible
- name: Install nginx
ansible.builtin.package:
name: nginx
state: present
version: 1.18.0-6ubuntu14.4
notify: restart nginx
handlers:
- name: restart nginx
ansible.builtin.service:
name: nginx
state: restarted
```
**Key validation points:**
- Package lock versions matched exactly (especially for applications)
- Notification triggers are preserved
- Service state expectations align
- File paths are correct for target platform
### Security Review
**Critical for production systems:**
- [ ] Secrets management strategy defined (Ansible Vault vs Chef data bags)
- [ ] File permissions match Chef exactly (especially for keys, configs)
- [ ] User/group creation preserves UIDs/GIDs where necessary
- [ ] SSH key deployments are secure
- [ ] Application credentials handled correctly
- [ ] No secrets leaked in plain text playbooks
**Use SousChef security scanning:**
```
Check the generated playbooks for security issues
```
## Phase 2: Development Testing (80% Confidence)
### Test in Disposable Environment
**Create identical infrastructure** for testing:
1. **Use Infrastructure-as-Code** (Terraform, CloudFormation, Vagrant):
```hcl
# Terraform example
resource "aws_instance" "test" {
ami = "ami-ubuntu-22.04" # Match production
instance_type = "t3.medium" # Match production
tags = {
Environment = "ansible-migration-test"
}
}
```
2. **Snapshot baseline state** (pre-Chef run):
- System packages and versions
- Running services
- File checksums (especially configs)
- Network ports
- Application functionality
3. **Run Chef cookbook** to establish baseline:
```bash
chef-client --local-mode --runlist 'recipe[cookbook-name]'
```
4. **Verify baseline functionality**:
- Applications respond correctly
- Services are running
- Configurations are correct
- Day-2 operations work (backups, monitoring, log shipping)
5. **Rebuild fresh environment** (destroy and recreate)
6. **Run Ansible playbook**:
```bash
ansible-playbook -i inventory playbook-name.yml -vv
```
7. **Compare states** (Ansible vs Chef):
```bash
# Package comparison
diff <(chef-state-packages) <(ansible-state-packages)
# File comparison
diff -r /etc/app/chef-configs /etc/app/ansible-configs
# Service comparison
systemctl status service-name # Same on both?
```
### Automated Testing with InSpec/Testlnfra
**Convert Chef InSpec tests to Ansible validation:**
```
Generate InSpec tests from the recipe at recipes/default.rb
```
**Example InSpec test:**
```ruby
describe package('nginx') do
it { should be_installed }
its('version') { should match /^1\.18/ }
end
describe service('nginx') do
it { should be_running }
it { should be_enabled }
end
describe file('/etc/nginx/nginx.conf') do
it { should exist }
its('mode') { should cmp '0644' }
its('owner') { should eq 'root' }
end
describe port(80) do
it { should be_listening }
end
```
**Run against both Chef and Ansible results:**
```bash
# Chef-managed system
inspec exec tests/nginx-test.rb -t ssh://chef-node
# Ansible-managed system
inspec exec tests/nginx-test.rb -t ssh://ansible-node
# Compare outputs
```
**Ansible assert tasks** (alternative to InSpec):
```yaml
- name: Verify nginx is installed
ansible.builtin.package_facts:
- name: Assert nginx version
ansible.builtin.assert:
that:
- "'nginx' in ansible_facts.packages"
- "ansible_facts.packages.nginx[0].version is match('^1.18')"
```
### Application-Level Testing
**For application cookbooks**, functional tests are critical:
**Web applications:**
```bash
# Test endpoints
curl -f http://test-server/health || exit 1
curl -f http://test-server/api/v1/status || exit 1
# Test authentication
curl -u testuser:password http://test-server/private || exit 1
# Test database connectivity
curl -f http://test-server/api/v1/db-check || exit 1
```
**Databases:**
```bash
# Test connectivity
psql -h test-db -U appuser -c "SELECT 1" || exit 1
# Test permissions
psql -h test-db -U appuser -c "CREATE TABLE test_table (id INT)" || exit 1
# Test backup procedures
pg_dump -h test-db -U appuser appdb > /tmp/test-backup.sql || exit 1
```
**Message queues:**
```bash
# Test RabbitMQ
rabbitmqctl status || exit 1
rabbitmqadmin list queues || exit 1
# Test Kafka
kafka-topics --list --bootstrap-server test-kafka:9092 || exit 1
```
### Day-2 Operations Testing
**Critical for production confidence:**
**Backup and restore:**
```bash
# Test backup creation
ansible-playbook playbooks/backup.yml
# Verify backup files exist
ls -lh /backup/appdb-$(date +%Y%m%d)* || exit 1
# Test restore procedure
ansible-playbook playbooks/restore.yml -e backup_file=/backup/appdb-20260217.sql
```
**Scaling operations:**
```bash
# Test scale-up playbook
ansible-playbook playbooks/scale-up.yml -e target_instances=5
# Test scale-down playbook
ansible-playbook playbooks/scale-down.yml -e target_instances=3
```
**Configuration updates:**
```bash
# Test config change deployment
ansible-playbook playbooks/update-config.yml -e new_workers=8
# Verify change applied without downtime
curl -f http://test-server/health || exit 1
```
**Monitoring and logging:**
```bash
# Test monitoring agent deployment
ansible-playbook playbooks/monitoring.yml
# Verify metrics collection
curl -f http://monitoring-server/api/v1/query?query=up || exit 1
# Test log shipping
tail -f /var/log/app/app.log # Verify logs appear in aggregator
```
## Phase 3: Staging Validation (90% Confidence)
### Production-Like Environment
**Staging should mirror production:**
- Same OS versions and patches
- Same network topology (subnets, firewalls)
- Same data volumes (even if sanitized/synthetic)
- Same integrations (APIs, databases, message queues)
- Same monitoring and logging
- Same backup procedures
### Full Integration Testing
**Run complete workflows** that exercise the system:
**E-commerce example:**
1. User registration
2. Product browsing
3. Add to cart
4. Checkout process
5. Payment processing (test mode)
6. Order fulfilment
7. Email notifications
**API service example:**
1. Authentication via OAuth
2. CRUD operations on all resources
3. Pagination and filtering
4. Rate limiting behaviour
5. Error handling (4xx, 5xx responses)
6. Batch operations
7. Webhook delivery
### Load Testing
**Verify performance under Ansible-managed infrastructure:**
```bash
# Load test with Apache Bench
ab -n 10000 -c 100 http://staging-server/
# Load test with k6
k6 run --vus 100 --duration 30s load-test.js
# Database load test
pgbench -h staging-db -U appuser -c 50 -j 4 -T 300 appdb
```
**Compare metrics:**
- Chef-managed: X req/sec, Y ms p95 latency
- Ansible-managed: Should match or exceed
### Security Validation
**Run security scans on Ansible-managed systems:**
```bash
# Vulnerability scanning
nmap -sV -A staging-server
# Configuration hardening check
ansible-playbook --check playbooks/hardening.yml
# Compliance scanning (CIS benchmarks)
inspec exec profiles/cis-ubuntu-22.04 -t ssh://staging-server
```
### Disaster Recovery Testing
**Test failure scenarios:**
**Service failure:**
```bash
# Kill service
sudo systemctl stop nginx
# Verify monitoring alerts
# Verify Ansible remediation (if automated)
# Manual remediation
ansible-playbook playbooks/recover-nginx.yml
```
**Database failure:**
```bash
# Simulate database corruption
# Run restore from backup
ansible-playbook playbooks/restore-database.yml
# Verify data integrity
# Verify application functionality
```
**Full system failure:**
```bash
# Destroy staging infrastructure
terraform destroy -auto-approve
# Rebuild from scratch with Ansible
terraform apply -auto-approve
ansible-playbook site.yml
# Verify complete system functionality
smoke-test.sh
```
## Phase 4: Production Pilot (95% Confidence)
### Canary Deployment
**Deploy to a subset of production infrastructure first:**
**Strategy 1: Geographic rollout**
```
Week 1: Deploy to APAC region (lowest traffic)
Week 2: Deploy to EMEA region
Week 3: Deploy to Americas region (highest traffic)
```
**Strategy 2: Service-based rollout**
```
Week 1: Deploy to internal web services
Week 2: Deploy to API gateway
Week 3: Deploy to customer-facing web servers
```
**Strategy 3: Blue-green deployment**
```
1. Build new "green" environment with Ansible
2. Route 10% traffic to green
3. Monitor for 48 hours
4. Increase to 50% traffic
5. Monitor for 1 week
6. Complete cutover to green
7. Decommission "blue" (Chef-managed)
```
### Monitoring During Pilot
**Critical metrics to watch:**
**System metrics:**
- CPU, memory, disk usage (should match Chef baseline)
- Network traffic patterns
- Service response times
- Error rates
**Application metrics:**
- Request rates and latency (p50, p95, p99)
- Database query performance
- Cache hit rates
- Background job queue lengths
- Application-specific KPIs
**Business metrics:**
- Transaction success rates
- User engagement metrics
- Revenue impact (for e-commerce)
- SLA compliance
**Set up alerts:**
```yaml
# Prometheus alert example
- alert: AnsibleMigrationPerformanceRegression
expr: http_request_duration_p95 > 200
for: 10m
labels:
severity: critical
annotations:
summary: "Performance regression detected in Ansible-managed infrastructure"
```
### Rollback Plan
**Always have a rollback strategy before piloting:**
**Quick rollback (< 5 minutes):**
```bash
# Switch traffic back to Chef-managed infrastructure
# Update load balancer / DNS
terraform apply -var "active_pool=chef-pool"
```
**Full rollback (< 30 minutes):**
```bash
# Revert infrastructure changes
terraform workspace select chef-managed
terraform apply
# Re-run Chef
chef-client --local-mode --runlist 'recipe[cookbook-name]'
# Verify functionality
smoke-test.sh
```
**Document rollback triggers:**
- Error rate > X% increase
- Latency > Y ms increase
- Failed health checks
- Database connection issues
- Customer complaints > Z threshold
## Phase 5: Production Deploy (100% Confidence)
### Final Cutover
**After successful pilot**, complete the migration:
1. **Scheduled maintenance window** (if downtime required)
2. **Final Ansible deployment** to remaining infrastructure
3. **Verification** of all systems
4. **Decommission Chef infrastructure** (after retention period)
### Post-Deployment Monitoring
**Monitor for 30 days post-cutover:**
- Same metrics as pilot phase
- Watch for delayed issues (memory leaks, log rotation, backup failures)
- Validate day-2 operations (patches, config updates, scaling events)
### Documentation Update
**Update operational runbooks:**
- Replace Chef commands with Ansible equivalents
- Update disaster recovery procedures
- Update onboarding docs for new ops team members
- Document new CI/CD pipeline integration
## Troubleshooting Common Issues
### "Ansible generates different file than Chef"
**Cause**: Template variables, file paths, or permissions differ.
**Solution**:
1. Compare files byte-by-byte: `diff -u chef-file ansible-file`
2. Check template variable mapping: `parse_template path/to/template.erb`
3. Verify file permissions: `ls -la chef-file ansible-file`
4. Check for Chef-specific Ruby code in templates
### "Service fails to start under Ansible"
**Cause**: Dependencies not satisfied, configs incorrect, or timing issues.
**Solution**:
1. Check service dependencies: `systemctl list-dependencies service-name`
2. Review systemd logs: `journalctl -u service-name -n 50`
3. Validate configuration syntax: `service-name-binary check-config`
4. Add pre-start validation tasks in Ansible
5. Check for race conditions (services starting before dependencies ready)
### "Application breaks after Ansible deployment"
**Cause**: Environment variables, application configs, or runtime dependencies differ.
**Solution**:
1. Compare environment variables: `printenv` on both systems
2. Check application logs for specific errors
3. Verify database connection strings
4. Validate API endpoints and URLs
5. Check for hard-coded paths that differ between environments
6. Review application-specific Chef resources that may need special handling
### "Performance degradation after migration"
**Cause**: Configuration tuning, resource limits, or caching differ.
**Solution**:
1. Compare system resource limits: `ulimit -a`
2. Check application tuning (worker processes, thread pools, connection pool sizes)
3. Verify caching configuration (Redis, Memcached)
4. Review kernel parameters: `sysctl -a`
5. Check for missing performance optimizations from Chef
6. Profile application under load to find bottlenecks
## Best Practices Summary
✅ **DO:**
- Test in disposable environments first
- Compare Chef and Ansible results side-by-side
- Use automated testing (InSpec, assert tasks)
- Test day-2 operations thoroughly
- Test application-level functionality, not just infrastructure
- Deploy in phases (dev → staging → pilot → production)
- Monitor extensively during pilot
- Have a documented rollback plan
- Validate backups and disaster recovery
❌ **DON'T:**
- Skip testing phases
- Deploy directly to production
- Ignore security validation
- Assume generated Ansible is perfect
- Rush the migration timeline
- Forget about application dependencies
- Ignore monitoring and logging
- Deploy without a rollback plan
- Forget day-2 operations
## Additional Resources
- [SousChef Validation Tools](../user-guide/mcp-tools.md#validation-testing)
- [Migration Guide: Advanced Workflows](advanced-workflows.md)
- [SECURITY.md](../../SECURITY.md) - Security features and best practices
- [Ansible Testing Strategies](https://docs.ansible.com/ansible/latest/reference_appendices/test_strategies.html)
- [InSpec Documentation](https://docs.chef.io/inspec/)
## Getting Help
If you encounter migration issues:
1. Review validation tool output for specific problems
2. Compare Chef and Ansible results in detail
3. Check application logs and system logs
4. Test in isolated environment
5. Open a [GitHub issue](https://github.com/kpeacocke/souschef/issues) with:
- Chef cookbook excerpt
- Generated Ansible playbook
- Observed vs expected behaviour
- Test environment details
For security concerns, see [SECURITY.md](../../SECURITY.md).
