Chroma MCP Server

# Validation System The validation system provides a framework for objectively measuring the quality and impact of code changes. This is particularly valuable for: 1. Ensuring changes successfully address issues 2. Qualifying learning promotions with evidence 3. Establishing a feedback loop for improvement 4. Building a repository of validated best practices ## Key Concepts The validation system is built around several key concepts: ### Evidence Types The system supports three main types of validation evidence: 1. **Test Transitions**: Tests that change from failing to passing, demonstrating a fix works 2. **Runtime Error Resolutions**: Runtime errors that are detected, fixed, and verified 3. **Code Quality Improvements**: Measurable improvements in code quality metrics ### Validation Score Each piece of evidence contributes to an overall validation score, which is used to determine if a learning meets the promotion threshold. The score is calculated based on: - Number and significance of test transitions - Complexity and impact of error resolutions - Percentage improvements in code quality metrics ### Promotion Threshold A configurable threshold (default: 0.7 or 70%) that evidence must meet to qualify for promotion. This ensures only well-validated learnings are promoted. ## Using the Validation System ### Setting Up 1. Ensure required collections exist: ```bash chroma-mcp-client setup-collections ``` 2. Configure any custom thresholds or settings as needed (defaults work well for most cases) ### Workflow Overview The typical validation workflow consists of these steps: 1. **Identify Issue**: Document a bug, error, or quality problem 2. **Initial Logging**: Log the error or failing test 3. **Implement Fix**: Make code changes to address the issue 4. **Verify Fix**: Run tests or tools to verify the fix works 5. **Log Evidence**: Record the improvements as validation evidence 6. **Validate Changes**: Calculate validation score from evidence 7. **Promote Learning**: Use validated evidence to promote learning ### Logging Validation Evidence #### Runtime Errors Use the `log-error` command to record runtime errors: ```bash # Log an error that was encountered ./scripts/log_error.sh \ --error-type "TypeError" \ --message "Cannot read property 'value' of undefined" \ --stacktrace "Error at line 42 in utils.js" \ --files "src/utils.js,src/components/form.js" # After fixing, update with resolution ./scripts/log_error.sh \ --error-type "TypeError" \ --message "Cannot read property 'value' of undefined" \ --files "src/utils.js,src/components/form.js" \ --resolution "Added null checks before accessing properties" \ --verified ``` #### Test Results Use the `log-test-results` command to record test results: ```bash # Basic test logging ./scripts/log_test_results.sh --xml test-results.xml # Compare before/after to identify transitions ./scripts/log_test_results.sh \ --xml after-fix.xml \ --before-xml before-fix.xml \ --commit-before abc12345 \ --commit-after def67890 ``` #### Code Quality Use the `log-quality-check` command to record code quality metrics: ```bash # Log current quality metrics ./scripts/log_quality_check.sh --after pylint-output.txt # Compare before/after to measure improvements ./scripts/log_quality_check.sh \ --tool pylint \ --metric error_count \ --before before-refactor.txt \ --after after-refactor.txt ``` ### Validating Evidence Use the `validate-evidence` command to calculate validation scores: ```bash # Validate from a single source ./scripts/validate_evidence.sh --file evidence.json # Combine evidence from multiple sources ./scripts/validate_evidence.sh \ --test-ids test-transition-123,test-transition-456 \ --runtime-ids error-789 \ --quality-ids quality-check-012 # Save validation results to a file ./scripts/validate_evidence.sh \ --test-ids test-transition-123 \ --quality-ids quality-check-012 \ --output validation-report.json ``` ### Promoting Validated Learnings Use the validation evidence when promoting learnings: ```bash # Promote learning with validation evidence chroma-mcp-client promote-learning \ --description "Always check for null/undefined before accessing object properties" \ --pattern "if (obj && obj.property)" \ --code-ref "src/utils.js:abc123:42" \ --tags "javascript,error-handling,null-safety" \ --confidence 0.95 \ --require-validation \ --validation-evidence-id evidence-123 # Promote learning with manual validation score chroma-mcp-client promote-learning \ --description "Use try/catch blocks around JSON.parse" \ --pattern "try { JSON.parse(str) } catch (e) { /* handle error */ }" \ --code-ref "src/data.js:def456:78" \ --tags "javascript,error-handling,json" \ --confidence 0.9 \ --require-validation \ --validation-score 0.85 ``` ## Evidence Scoring The validation system calculates scores based on these factors: ### Test Transitions - **Weight**: 0.5 (50% of total score) - **Factors**: Number of transitions, test importance, coverage of affected code ### Runtime Error Resolutions - **Weight**: 0.3 (30% of total score) - **Factors**: Error severity, verification status, impact scope ### Code Quality Improvements - **Weight**: 0.2 (20% of total score) - **Factors**: Percentage improvement, metric type, affected code size The final score is a weighted average of these components, normalized to a 0-1 range. ## Best Practices - Log evidence as close to the time of implementation as possible - Capture "before" data before making changes - Be specific about affected files and components - Verify resolutions before marking them as verified - Combine multiple types of evidence for stronger validation - Use consistent naming and tagging conventions ## Reference See the CLI command reference for detailed parameter information: - [log-error](docs/scripts/log-error.md) - [log-test-results](docs/scripts/log-test-results.md) - [log-quality-check](docs/scripts/log-quality-check.md) - [validate-evidence](docs/scripts/validate-evidence.md) - [promote-learning](docs/scripts/promote-learning.md)