Documentation Generator MCP Server

{ "document_info": { "title": "audit_codebase Implementation Plan", "tool_id": 9, "version": "1.0.0", "created_date": "2025-10-10", "status": "ready_for_implementation", "estimated_effort": "10-12 hours", "description": "Comprehensive implementation plan for Tool #9: audit_codebase - a standards compliance auditing tool that detects violations and generates audit reports", "depends_on": "Tool #8 (establish_standards) must be run first to generate standards documents" }, "executive_summary": { "purpose": "Audit codebases against established standards to find violations, inconsistencies, and non-compliant code patterns. Generates actionable audit reports with compliance scores.", "value_proposition": "Now that you have standards (Tool #8), this tool finds where your code doesn't follow them. It's like a code reviewer that checks for consistency violations.", "real_world_analogy": "Like a building inspector checking if construction follows blueprints - finds all the places where implementation doesn't match specifications", "use_case": "Run periodically (weekly/monthly) or before releases to ensure codebase consistency", "output": "Timestamped audit report in coderef/audits/ directory with violations, compliance score, and fix suggestions", "trilogy_position": "Tool #9 is the SECOND tool in the consistency trilogy:\n1. establish_standards (Tool #8) - Document what's standard ✅ COMPLETE\n2. audit_codebase (Tool #9) - Find violations of standards 🔄 IN PROGRESS\n3. check_consistency (Tool #10) - Quality gate for new code ⏭️ NEXT" }, "risk_assessment": { "overall_risk": "Medium", "complexity": "High", "scope": "Large - 7 files affected", "risk_factors": { "file_system": "Medium risk - Reads standards files and writes audit reports. Uses safe Path.resolve() for all paths. No user-controlled write locations (hardcoded coderef/audits/).", "dependencies": "Low risk - No external dependencies. Reuses existing patterns from StandardsGenerator (Tool #8). All dependencies already in project (pathlib, json, datetime, re).", "performance": "Medium risk - Scanning large codebases (500+ files) could take 3-5 minutes. Regex-based violation detection may be slow on large files. Mitigation: Implement file size limits (reuse MAX_FILE_SIZE from Tool #8), add progress logging, consider async file I/O for phase 2 optimization.", "security": "Low risk - Inherits SEC-001, SEC-004, SEC-007, SEC-008 from Tool #8. New risks: SEC-009 (standards dir traversal), SEC-010 (report path traversal), SEC-011 (large standards files DoS). All mitigated via validation and hardcoded paths.", "breaking_changes": "None - New tool, no existing functionality to break. Tool #10 will depend on this tool's output format (AuditResultDict)." } }, "current_state_analysis": { "affected_files": [ "NEW: generators/audit_generator.py - Complete AuditGenerator class with 13 methods (standards parsing, violation detection, report generation)", "MODIFIED: server.py:260 - Add audit_codebase tool schema after establish_standards", "MODIFIED: tool_handlers.py - Add handle_audit_codebase handler + register in TOOL_HANDLERS dict", "MODIFIED: constants.py - Add Paths.AUDITS_DIR, Files.AUDIT_REPORT, AuditSeverity enum, AuditScope enum", "MODIFIED: type_defs.py - Add 5 new TypedDicts: StandardsDataDict, AuditViolationDict, ComplianceScoreDict, ViolationStatsDict, AuditResultDict", "MODIFIED: validation.py - Add validate_severity_filter() and validate_audit_scope() functions", "MODIFIED: generators/__init__.py - Export AuditGenerator class" ], "dependencies": { "existing": [ "ErrorResponse factory (ARCH-001) - Consistent error handling", "Handler registry pattern (QUA-002) - Tool registration", "Structured logging (ARCH-003) - Operation tracking", "Input validation (REF-003) - Boundary validation", "Constants (REF-002) - No magic strings", "TypedDict pattern (QUA-001) - Complex return types", "StandardsGenerator patterns - File scanning logic to reuse" ], "new": [ "AuditGenerator class - Core audit engine (13 methods)", "Regex-based pattern matching - Violation detection", "Markdown report generation - Formatted audit output", "Compliance scoring algorithm - Weighted violation scoring", "Fix suggestion generation - Actionable remediation guidance" ] }, "architecture_context": "Operates at the generator layer (similar to StandardsGenerator). Consumes output from Tool #8 (standards documents), produces output consumed by Tool #10 (audit history). Fits into consistency trilogy workflow: establish → audit → enforce." }, "key_features": [ "Parse 4 standards documents (UI-STANDARDS.md, BEHAVIOR-STANDARDS.md, UX-PATTERNS.md, COMPONENT-INDEX.md) into structured StandardsDataDict", "Detect UI violations: non-standard button sizes/variants, undocumented colors, non-standard modal sizes", "Detect behavior violations: non-standard error messages, missing loading states, inconsistent error handling", "Detect UX violations: missing ARIA attributes, inconsistent navigation patterns, missing keyboard support", "Assign severity levels (critical/major/minor) based on impact to functionality, accessibility, and UX", "Generate actionable fix suggestions for each violation with code examples", "Calculate compliance score (0-100) using weighted violations (critical=10pts, major=5pts, minor=1pt)", "Generate comprehensive markdown audit report with executive summary, violations by severity, violations by file, fix suggestions, and statistics", "Support severity filtering (critical/major/minor/all) to focus on high-priority issues", "Support scope filtering (ui_patterns/behavior_patterns/ux_patterns/all) to audit specific areas", "Gracefully handle missing/incomplete standards documents with warnings and partial audits" ], "tool_specification": { "name": "audit_codebase", "description": "Audit codebase for standards violations using established standards documents. Scans all source files, compares against standards, and generates comprehensive compliance report with violations, severity levels, and fix suggestions.", "input_schema": { "type": "object", "properties": { "project_path": { "type": "string", "description": "Absolute path to project directory", "required": true }, "standards_dir": { "type": "string", "description": "Path to standards directory (relative to project root)", "required": false, "default": "coderef/standards" }, "severity_filter": { "type": "string", "enum": ["critical", "major", "minor", "all"], "description": "Filter violations by severity level", "required": false, "default": "all" }, "scope": { "type": "array", "items": { "enum": ["ui_patterns", "behavior_patterns", "ux_patterns", "all"] }, "description": "Which areas to audit", "required": false, "default": ["all"] }, "generate_fixes": { "type": "boolean", "description": "Include automated fix suggestions in report", "required": false, "default": true } }, "required": ["project_path"] }, "output_format": { "report_file": "coderef/audits/AUDIT-REPORT-{timestamp}.md", "structure": { "executive_summary": "Compliance score, total violations, scan metadata", "critical_violations": "List of critical issues requiring immediate fix", "major_violations": "List of major inconsistencies", "minor_violations": "List of minor style violations", "file_breakdown": "Violations grouped by file with line numbers", "fix_suggestions": "Actionable remediation steps for each violation", "statistics": "Charts and metrics (files scanned, patterns checked, compliance trend)" } }, "success_response": { "type": "AuditResultDict", "fields": { "report_path": "Path to generated audit report", "compliance_score": "0-100 percentage", "total_violations": "Integer count", "violations_by_severity": "Dict with critical/major/minor counts", "violations_by_file": "Dict mapping file paths to violation counts", "scan_duration": "Seconds taken for audit", "files_scanned": "Integer count" } } }, "implementation_phases": { "phase_1_infrastructure": { "title": "Core Infrastructure Setup", "duration": "2 hours", "tasks": [ { "id": "INFRA-001", "task": "Add tool schema to server.py", "location": "server.py:260 (after establish_standards tool)", "details": "Add Tool definition with name='audit_codebase', description, inputSchema", "effort": "15 minutes" }, { "id": "INFRA-002", "task": "Create handler in tool_handlers.py", "location": "tool_handlers.py (new function at end)", "details": "Create async def handle_audit_codebase(arguments: dict) -> list[TextContent]", "effort": "30 minutes" }, { "id": "INFRA-003", "task": "Register handler in TOOL_HANDLERS dict", "location": "tool_handlers.py TOOL_HANDLERS dict", "details": "Add 'audit_codebase': handle_audit_codebase to registry", "effort": "5 minutes" }, { "id": "INFRA-004", "task": "Update constants.py", "location": "constants.py", "details": "Add AUDITS_DIR = 'coderef/audits', Files.AUDIT_REPORT, AuditSeverity enum, AuditScope enum", "effort": "20 minutes", "new_constants": { "Paths.AUDITS_DIR": "'coderef/audits'", "Files.AUDIT_REPORT": "'AUDIT-REPORT-{timestamp}.md'", "AuditSeverity": "Enum with CRITICAL, MAJOR, MINOR, ALL", "AuditScope": "Enum with UI_PATTERNS, BEHAVIOR_PATTERNS, UX_PATTERNS, ALL" } }, { "id": "INFRA-005", "task": "Create generators/audit_generator.py", "location": "generators/ (new file)", "details": "Create AuditGenerator class with standards parser, violation detector, report generator", "effort": "1 hour (skeleton only, implementation in Phase 2-4)" }, { "id": "INFRA-006", "task": "Add TypedDict definitions to type_defs.py", "location": "type_defs.py", "details": "Create 5 new TypedDict definitions for audit system (QUA-001 compliance)", "effort": "25 minutes", "type_dicts": { "StandardsDataDict": "Parsed standards data structure", "AuditViolationDict": "Single violation with file, line, severity, message, fix", "AuditResultDict": "Complete audit results with compliance score and violations", "ComplianceScoreDict": "Compliance metrics by category", "ViolationStatsDict": "Statistics about violations found" } }, { "id": "INFRA-007", "task": "Add validation functions to validation.py", "location": "validation.py", "details": "Create validate_severity_filter() and validate_audit_scope() for input validation (REF-003 compliance)", "effort": "15 minutes", "functions": { "validate_severity_filter": "Validates severity is one of: critical, major, minor, all", "validate_audit_scope": "Validates scope list contains only valid area names" } } ] }, "phase_2_standards_parser": { "title": "Standards Document Parser", "duration": "3 hours", "description": "Parse markdown standards documents (UI-STANDARDS.md, BEHAVIOR-STANDARDS.md, UX-PATTERNS.md, COMPONENT-INDEX.md) and build in-memory standards model", "tasks": [ { "id": "PARSE-001", "task": "Implement parse_standards_documents()", "method_signature": "def parse_standards_documents(self, standards_dir: Path) -> StandardsDataDict", "description": "Read all 4 standards documents and parse into structured data", "details": [ "Read UI-STANDARDS.md and extract button sizes, variants, modal configs, colors", "Read BEHAVIOR-STANDARDS.md and extract error patterns, loading states", "Read UX-PATTERNS.md and extract navigation patterns, accessibility rules", "Read COMPONENT-INDEX.md and extract component inventory", "Handle missing standards files gracefully (log warning, continue with available standards)" ], "effort": "1 hour" }, { "id": "PARSE-002", "task": "Implement parse_ui_standards()", "method_signature": "def parse_ui_standards(self, content: str) -> dict", "description": "Parse UI-STANDARDS.md to extract UI patterns", "parsing_strategy": { "buttons": "Extract sizes: regex for 'Discovered Sizes: (.+?)\\n', split by comma", "modals": "Extract sizes: regex for 'Discovered Sizes: (.+?)\\n'", "colors": "Extract hex codes: regex for '- `(#[0-9a-fA-F]{6})`'", "graceful_degradation": "If section says '*No X patterns discovered*', return empty list for that category" }, "effort": "45 minutes" }, { "id": "PARSE-003", "task": "Implement parse_behavior_standards()", "method_signature": "def parse_behavior_standards(self, content: str) -> dict", "description": "Parse BEHAVIOR-STANDARDS.md to extract behavior patterns", "parsing_strategy": { "error_messages": "Extract messages: regex for '- (.+?)\\n' under 'Discovered Error Messages'", "loading_patterns": "Detect if 'Loading state patterns detected' present", "toast_patterns": "Extract toast configurations if documented" }, "effort": "30 minutes" }, { "id": "PARSE-004", "task": "Implement parse_ux_standards()", "method_signature": "def parse_ux_standards(self, content: str) -> dict", "description": "Parse UX-PATTERNS.md to extract UX patterns", "parsing_strategy": { "navigation": "Check if 'Routing library detected' present", "accessibility": "Check if 'ARIA attributes detected' present", "permissions": "Extract permission patterns if documented" }, "effort": "30 minutes" }, { "id": "PARSE-005", "task": "Build StandardsDataDict structure", "description": "Combine parsed data into comprehensive standards model", "structure": { "ui_patterns": { "buttons": {"allowed_sizes": [], "allowed_variants": []}, "modals": {"allowed_sizes": []}, "colors": {"allowed_hex_codes": []} }, "behavior_patterns": { "error_messages": {"expected_patterns": []}, "loading_states": {"required": true/false} }, "ux_patterns": { "navigation": {"routing_detected": true/false}, "accessibility": {"aria_required": true/false} }, "components": {"known_components": []} }, "effort": "15 minutes" } ] }, "phase_3_violation_detection": { "title": "Violation Detection Engine", "duration": "3-4 hours", "description": "Scan codebase and compare against parsed standards to detect violations", "tasks": [ { "id": "DETECT-001", "task": "Implement scan_for_violations()", "method_signature": "def scan_for_violations(self, standards: StandardsDataDict) -> List[AuditViolationDict]", "description": "Main violation detection orchestrator", "workflow": [ "1. Scan all source files (reuse logic from StandardsGenerator)", "2. For each file, run violation checks", "3. Collect all violations with file/line locations", "4. Assign severity levels based on impact", "5. Generate fix suggestions for each violation" ], "effort": "30 minutes (orchestration only)" }, { "id": "DETECT-002", "task": "Implement detect_ui_violations()", "method_signature": "def detect_ui_violations(self, file_content: str, file_path: Path, standards: dict) -> List[AuditViolationDict]", "description": "Detect UI pattern violations (buttons, modals, colors)", "violation_checks": [ { "check": "Non-standard button sizes", "regex": "<Button.*?size=['\\\"](.+?)['\\\"]", "validation": "Check if extracted size is in standards['ui_patterns']['buttons']['allowed_sizes']", "violation": "If not in allowed list, create violation with severity=MAJOR", "example": "size='tiny' when standards only allow 'sm', 'md', 'lg'" }, { "check": "Non-standard button variants", "regex": "<Button.*?variant=['\\\"](.+?)['\\\"]", "validation": "Check if extracted variant is in standards['ui_patterns']['buttons']['allowed_variants']", "violation": "If not in allowed list, create violation with severity=MAJOR" }, { "check": "Undocumented colors", "regex": "#[0-9a-fA-F]{6}|#[0-9a-fA-F]{3}", "validation": "Check if extracted hex code is in standards['ui_patterns']['colors']['allowed_hex_codes']", "violation": "If not in allowed list, create violation with severity=MINOR", "example": "#FF0000 when standards only document #3B82F6, #EF4444, etc." }, { "check": "Non-standard modal sizes", "regex": "<Modal.*?size=['\\\"](.+?)['\\\"]", "validation": "Check if extracted size is in standards['ui_patterns']['modals']['allowed_sizes']", "violation": "If not in allowed list, create violation with severity=MAJOR" } ], "effort": "1 hour" }, { "id": "DETECT-003", "task": "Implement detect_behavior_violations()", "method_signature": "def detect_behavior_violations(self, file_content: str, file_path: Path, standards: dict) -> List[AuditViolationDict]", "description": "Detect behavior pattern violations (errors, loading)", "violation_checks": [ { "check": "Non-standard error messages", "regex": "throw new Error\\(['\\\"](.+?)['\\\"]\\)|toast\\.error\\(['\\\"](.+?)['\\\"]\\)", "validation": "Check if message follows patterns in standards['behavior_patterns']['error_messages']", "violation": "If message doesn't match expected pattern, create violation with severity=MINOR", "example": "Inconsistent capitalization, punctuation, or tone" }, { "check": "Missing loading states", "regex": "fetch\\(|axios\\.|api\\.", "validation": "Check if file contains isLoading/loading pattern when API calls present", "violation": "If API call without loading state, create violation with severity=MAJOR", "rationale": "UX requirement: users should see loading indicators during async operations" }, { "check": "Inconsistent error handling", "regex": "try\\s*{[\\s\\S]*?}\\s*catch", "validation": "Check if catch block follows error handling standards", "violation": "If error handling inconsistent, create violation with severity=MINOR" } ], "effort": "1 hour" }, { "id": "DETECT-004", "task": "Implement detect_ux_violations()", "method_signature": "def detect_ux_violations(self, file_content: str, file_path: Path, standards: dict) -> List[AuditViolationDict]", "description": "Detect UX pattern violations (navigation, accessibility)", "violation_checks": [ { "check": "Missing ARIA attributes", "regex": "<button[^>]*>", "validation": "If standards require ARIA, check if buttons have aria-label or aria-describedby", "violation": "If button missing ARIA when required, create violation with severity=CRITICAL", "rationale": "Accessibility requirement for screen readers" }, { "check": "Inconsistent navigation patterns", "regex": "useNavigate|useRouter|navigate\\(|router\\.", "validation": "Check if navigation pattern matches standard", "violation": "If using different routing approach, create violation with severity=MINOR" }, { "check": "Missing keyboard support", "regex": "onClick=", "validation": "Check if element also has onKeyPress/onKeyDown for keyboard accessibility", "violation": "If onClick without keyboard handler, create violation with severity=MAJOR" } ], "effort": "45 minutes" }, { "id": "DETECT-005", "task": "Implement assign_severity()", "method_signature": "def assign_severity(self, violation_type: str, context: dict) -> str", "description": "Assign severity level to violations based on impact", "severity_rules": { "CRITICAL": [ "Missing ARIA attributes (accessibility blocker)", "Security vulnerabilities", "Broken user flows" ], "MAJOR": [ "Non-standard button sizes/variants (user confusion)", "Missing loading states (poor UX)", "Inconsistent error handling" ], "MINOR": [ "Undocumented colors (style inconsistency)", "Non-standard error messages (copy inconsistency)", "Missing inline comments" ] }, "effort": "30 minutes" }, { "id": "DETECT-006", "task": "Implement generate_fix_suggestion()", "method_signature": "def generate_fix_suggestion(self, violation: AuditViolationDict) -> str", "description": "Generate actionable fix suggestion for each violation", "examples": [ { "violation": "Button size='tiny' not in standards", "suggestion": "Change size='tiny' to size='sm' (smallest standard size)" }, { "violation": "Color #FF0000 not documented", "suggestion": "Replace #FF0000 with standard error color #EF4444, or add #FF0000 to color palette" }, { "violation": "API call without loading state", "suggestion": "Add const [isLoading, setLoading] = useState(false) and show <Spinner /> during fetch" } ], "effort": "30 minutes" } ] }, "phase_4_report_generation": { "title": "Audit Report Generator", "duration": "2 hours", "description": "Generate comprehensive audit report with compliance score, violations, and fix suggestions", "tasks": [ { "id": "REPORT-001", "task": "Implement calculate_compliance_score()", "method_signature": "def calculate_compliance_score(self, violations: List[AuditViolationDict], total_patterns: int) -> ComplianceScoreDict", "description": "Calculate compliance percentage based on violations found", "formula": "compliance_score = 100 - (weighted_violations / total_checkable_patterns * 100)", "weighting": { "critical": "10 points per violation", "major": "5 points per violation", "minor": "1 point per violation" }, "output": { "overall_score": "0-100", "ui_compliance": "0-100", "behavior_compliance": "0-100", "ux_compliance": "0-100" }, "effort": "30 minutes" }, { "id": "REPORT-002", "task": "Implement generate_audit_report()", "method_signature": "def generate_audit_report(self, violations: List[AuditViolationDict], compliance: ComplianceScoreDict) -> str", "description": "Generate markdown audit report", "report_structure": [ "# Audit Report - {timestamp}", "## Executive Summary (score, total violations, scan metadata)", "## Compliance Score (overall + by category with visual progress bars)", "## Critical Violations (must fix immediately)", "## Major Violations (should fix soon)", "## Minor Violations (style improvements)", "## Violations by File (grouped by file path)", "## Fix Suggestions (actionable remediation steps)", "## Statistics (charts, trends, metrics)", "## Appendix (scan configuration, standards used)" ], "effort": "1 hour" }, { "id": "REPORT-003", "task": "Implement save_audit_report()", "method_signature": "def save_audit_report(self, report_content: str, audits_dir: Path) -> AuditResultDict", "description": "Save audit report to timestamped file", "filename_format": "AUDIT-REPORT-{YYYY-MM-DD-HHmmss}.md", "example": "AUDIT-REPORT-2025-10-10-153045.md", "output": "AuditResultDict with report_path, compliance_score, violation counts, scan metadata", "effort": "30 minutes" } ] } }, "code_structure": { "audit_generator_class": { "file": "generators/audit_generator.py", "class": "AuditGenerator", "methods": [ { "name": "__init__", "signature": "def __init__(self, project_path: Path, standards_dir: Path)", "description": "Initialize with project path and standards directory" }, { "name": "parse_standards_documents", "signature": "def parse_standards_documents(self, standards_dir: Path) -> StandardsDataDict", "description": "Parse all 4 standards documents into structured data model" }, { "name": "parse_ui_standards", "signature": "def parse_ui_standards(self, content: str) -> dict", "description": "Parse UI-STANDARDS.md and extract UI patterns" }, { "name": "parse_behavior_standards", "signature": "def parse_behavior_standards(self, content: str) -> dict", "description": "Parse BEHAVIOR-STANDARDS.md and extract behavior patterns" }, { "name": "parse_ux_standards", "signature": "def parse_ux_standards(self, content: str) -> dict", "description": "Parse UX-PATTERNS.md and extract UX patterns" }, { "name": "scan_for_violations", "signature": "def scan_for_violations(self, standards: StandardsDataDict) -> List[AuditViolationDict]", "description": "Scan codebase and detect all violations" }, { "name": "detect_ui_violations", "signature": "def detect_ui_violations(self, file_content: str, file_path: Path, standards: dict) -> List[AuditViolationDict]", "description": "Detect UI pattern violations (buttons, colors, modals)" }, { "name": "detect_behavior_violations", "signature": "def detect_behavior_violations(self, file_content: str, file_path: Path, standards: dict) -> List[AuditViolationDict]", "description": "Detect behavior pattern violations (errors, loading)" }, { "name": "detect_ux_violations", "signature": "def detect_ux_violations(self, file_content: str, file_path: Path, standards: dict) -> List[AuditViolationDict]", "description": "Detect UX pattern violations (accessibility, navigation)" }, { "name": "assign_severity", "signature": "def assign_severity(self, violation_type: str, context: dict) -> str", "description": "Assign severity level (critical/major/minor) based on impact" }, { "name": "generate_fix_suggestion", "signature": "def generate_fix_suggestion(self, violation: AuditViolationDict) -> str", "description": "Generate actionable fix suggestion for violation" }, { "name": "calculate_compliance_score", "signature": "def calculate_compliance_score(self, violations: List[AuditViolationDict], total_patterns: int) -> ComplianceScoreDict", "description": "Calculate compliance percentage (0-100)" }, { "name": "generate_audit_report", "signature": "def generate_audit_report(self, violations: List[AuditViolationDict], compliance: ComplianceScoreDict) -> str", "description": "Generate markdown audit report" }, { "name": "save_audit_report", "signature": "def save_audit_report(self, report_content: str, audits_dir: Path) -> AuditResultDict", "description": "Save audit report to timestamped file" } ] }, "handler_implementation": { "file": "tool_handlers.py", "function": "handle_audit_codebase", "pattern": "Standard handler pattern with validation, logging, error handling", "workflow": [ "1. Log tool invocation (ARCH-003)", "2. Validate inputs: project_path, standards_dir, severity_filter, scope (REF-003)", "3. Check if standards documents exist (FileNotFoundError if missing)", "4. Initialize AuditGenerator", "5. Parse standards documents", "6. Scan for violations", "7. Calculate compliance score", "8. Generate and save audit report", "9. Return AuditResultDict with report path and summary", "10. Handle all error types: ValueError, FileNotFoundError, PermissionError, OSError, Exception" ], "imports_required": [ "from mcp.types import TextContent", "from pathlib import Path", "import json", "from datetime import datetime", "from constants import Paths, Files, AuditSeverity, AuditScope", "from validation import validate_project_path_input, validate_severity_filter, validate_audit_scope", "from error_responses import ErrorResponse", "from logger_config import logger, log_tool_call, log_error, log_security_event", "from generators import AuditGenerator", "from type_defs import AuditResultDict, AuditViolationDict, StandardsDataDict" ] } }, "type_defs_additions": { "file": "type_defs.py", "new_type_dicts": [ { "name": "StandardsDataDict", "description": "Parsed standards data from markdown documents", "fields": { "ui_patterns": "dict with buttons, modals, colors", "behavior_patterns": "dict with error_messages, loading_states", "ux_patterns": "dict with navigation, accessibility", "components": "list of known components", "source_files": "list of standards files parsed", "parse_errors": "list of warnings during parsing" } }, { "name": "AuditViolationDict", "description": "Single violation detected during audit", "fields": { "id": "str - unique violation ID (e.g., 'V-001')", "type": "str - violation type (e.g., 'non_standard_button_size')", "severity": "str - 'critical', 'major', or 'minor'", "category": "str - 'ui_patterns', 'behavior_patterns', 'ux_patterns'", "file_path": "str - relative path to file", "line_number": "int - line where violation occurs", "column": "int - column position (optional)", "message": "str - human-readable violation description", "actual_value": "str - what was found in code", "expected_value": "str - what standards specify", "fix_suggestion": "str - actionable fix instruction", "code_snippet": "str - 3-5 lines of context around violation" } }, { "name": "ComplianceScoreDict", "description": "Compliance metrics by category", "fields": { "overall_score": "int - 0-100", "ui_compliance": "int - 0-100", "behavior_compliance": "int - 0-100", "ux_compliance": "int - 0-100", "grade": "str - A/B/C/D/F based on score", "passing": "bool - true if score >= 80" } }, { "name": "ViolationStatsDict", "description": "Statistics about violations found", "fields": { "total_violations": "int", "critical_count": "int", "major_count": "int", "minor_count": "int", "violations_by_file": "dict mapping file paths to counts", "violations_by_type": "dict mapping violation types to counts", "most_violated_file": "str - file with most violations", "most_common_violation": "str - most frequent violation type" } }, { "name": "AuditResultDict", "description": "Complete audit results", "fields": { "report_path": "str - path to generated report", "compliance_score": "int - 0-100", "compliance_details": "ComplianceScoreDict", "violation_stats": "ViolationStatsDict", "violations": "List[AuditViolationDict]", "scan_metadata": "dict with timestamp, duration, files_scanned", "success": "bool" } } ] }, "constants_additions": { "file": "constants.py", "new_constants": { "Paths": { "AUDITS_DIR": "'coderef/audits' # Audit reports storage" }, "Files": { "AUDIT_REPORT_TEMPLATE": "'AUDIT-REPORT-{timestamp}.md' # Timestamped audit reports" }, "AuditSeverity": { "description": "Enum for violation severity levels", "values": { "CRITICAL": "'critical' # Must fix immediately - breaks functionality/accessibility", "MAJOR": "'major' # Should fix soon - causes inconsistency or poor UX", "MINOR": "'minor' # Style improvement - cosmetic inconsistency", "ALL": "'all' # Filter for all severity levels" }, "class_definition": "class AuditSeverity(str, Enum):\n CRITICAL = 'critical'\n MAJOR = 'major'\n MINOR = 'minor'\n ALL = 'all'\n \n @classmethod\n def values(cls) -> list:\n return [cls.CRITICAL, cls.MAJOR, cls.MINOR, cls.ALL]" }, "AuditScope": { "description": "Enum for audit scope options", "values": { "UI_PATTERNS": "'ui_patterns' # Audit UI component patterns only", "BEHAVIOR_PATTERNS": "'behavior_patterns' # Audit behavior patterns only", "UX_PATTERNS": "'ux_patterns' # Audit UX patterns only", "ALL": "'all' # Audit all pattern categories" }, "class_definition": "class AuditScope(str, Enum):\n UI_PATTERNS = 'ui_patterns'\n BEHAVIOR_PATTERNS = 'behavior_patterns'\n UX_PATTERNS = 'ux_patterns'\n ALL = 'all'\n \n @classmethod\n def values(cls) -> list:\n return [cls.UI_PATTERNS, cls.BEHAVIOR_PATTERNS, cls.UX_PATTERNS, cls.ALL]" } } }, "validation_additions": { "file": "validation.py", "new_functions": [ { "name": "validate_severity_filter", "signature": "def validate_severity_filter(severity: str) -> str", "description": "Validate severity_filter parameter", "logic": "Check if severity is in AuditSeverity.values()", "raises": "ValueError if invalid severity", "example": "validate_severity_filter('critical') # OK\nvalidate_severity_filter('invalid') # ValueError" }, { "name": "validate_audit_scope", "signature": "def validate_audit_scope(scope: list) -> list", "description": "Validate audit scope parameter", "logic": "Check if all items in scope list are in AuditScope.values()", "raises": "ValueError if any invalid scope value", "example": "validate_audit_scope(['ui_patterns', 'all']) # OK\nvalidate_audit_scope(['invalid']) # ValueError" } ] }, "security_considerations": { "threats_and_mitigations": [ { "id": "SEC-009", "threat": "Standards Directory Traversal", "risk_level": "Medium", "description": "User could provide standards_dir='../../../etc' to read sensitive files", "mitigation": "Validate standards_dir is relative path and resolve within project_path only", "implementation": "standards_path = (project_path / standards_dir).resolve()\nif not standards_path.is_relative_to(project_path):\n raise ValueError('standards_dir must be within project directory')" }, { "id": "SEC-010", "threat": "Audit Report Path Traversal", "risk_level": "Low", "description": "Attacker could try to write audit report outside project directory", "mitigation": "Always write audit reports to hardcoded coderef/audits/ within project_path", "implementation": "audits_dir = project_path / Paths.AUDITS_DIR # No user input allowed" }, { "id": "SEC-011", "threat": "Large Standards Files DoS", "risk_level": "Low", "description": "Maliciously large standards files could cause memory exhaustion", "mitigation": "Reuse MAX_FILE_SIZE check from StandardsGenerator (10 MB limit)", "implementation": "if standards_file.stat().st_size > MAX_FILE_SIZE:\n logger.warning('Skipping large standards file')\n continue" } ], "security_checklist": [ "✅ SEC-001: Path traversal protection (project_path validation)", "✅ SEC-004: Excluded directory validation (reuse from Tool #8)", "✅ SEC-007: File size limits (reuse MAX_FILE_SIZE)", "✅ SEC-008: Symlink validation (reuse from Tool #8)", "✅ SEC-009: Standards directory validation (NEW - relative path only)", "✅ SEC-010: Audit report path hardcoded (NEW - no user control)", "✅ SEC-011: Standards file size limits (NEW - prevent DoS)" ] }, "testing_strategy": { "unit_tests": [ "test_parse_ui_standards - Verify parsing of UI-STANDARDS.md", "test_parse_behavior_standards - Verify parsing of BEHAVIOR-STANDARDS.md", "test_detect_ui_violations - Verify button/color/modal violation detection", "test_detect_behavior_violations - Verify error/loading violation detection", "test_calculate_compliance_score - Verify scoring formula", "test_generate_fix_suggestion - Verify fix suggestions are actionable" ], "integration_tests": [ { "test": "test_audit_with_violations", "setup": "Create sample React project with intentional violations", "expected": "Violations detected, compliance score < 100, fix suggestions generated" }, { "test": "test_audit_perfect_compliance", "setup": "Create project where all patterns match standards", "expected": "No violations, compliance score = 100" }, { "test": "test_audit_missing_standards", "setup": "Run audit when standards documents don't exist", "expected": "FileNotFoundError with helpful message to run establish_standards first" }, { "test": "test_audit_empty_standards", "setup": "Run audit when standards documents contain 'No patterns discovered'", "expected": "Report notes no standards available, skip violation checks gracefully" } ], "edge_cases": [ "Empty standards documents (all say 'No patterns discovered')", "Missing standards files (some or all of 4 files missing)", "Large codebase with thousands of violations", "File with violations on every line", "Standards with unusual patterns (edge case regex matches)" ] }, "success_criteria": { "functional_requirements": [ "Tool appears in MCP tool list with ✓ Connected status", "Successfully parses all 4 standards documents without errors", "Detects at least 5 violation types (button, color, error, loading, accessibility)", "Generates audit report with compliance score 0-100", "Report includes violations grouped by severity", "Each violation has file path, line number, and fix suggestion", "Handles missing standards gracefully with clear error message", "Completes audit in < 5 minutes for typical project (< 500 files)" ], "quality_requirements": [ "100% architecture compliance (ARCH-001, QUA-001, QUA-002, REF-002, REF-003)", "All 5 error types handled: ValueError, FileNotFoundError, PermissionError, OSError, Exception", "100% logging coverage (tool invocation, errors, security events, completion)", "All functions have complete type hints", "Documentation updated: README.md, API.md, ARCHITECTURE.md, CLAUDE.md" ], "performance_requirements": [ "Audit completes in < 5 minutes for 500-file project", "Memory usage < 500 MB for typical project", "Standards parsing < 10 seconds for all 4 documents", "Violation detection rate > 20 files/sec" ] }, "report_example": { "filename": "AUDIT-REPORT-2025-10-10-153045.md", "content_preview": "# Audit Report\n\n**Generated**: 2025-10-10 15:30:45\n**Project**: my-react-app\n**Compliance Score**: 78/100 (C)\n\n---\n\n## Executive Summary\n\n✅ **Passing** (score >= 80: No)\n- Total Violations: 47\n- Critical: 2\n- Major: 18\n- Minor: 27\n- Files Scanned: 143\n- Scan Duration: 3.2 minutes\n\n---\n\n## Compliance Score\n\n**Overall**: 78/100 ⬛⬛⬛⬛⬛⬛⬛⬛⬜⬜ (C)\n**UI Patterns**: 85/100 ⬛⬛⬛⬛⬛⬛⬛⬛⬜⬜ (B)\n**Behavior Patterns**: 72/100 ⬛⬛⬛⬛⬛⬛⬛⬜⬜⬜ (C)\n**UX Patterns**: 65/100 ⬛⬛⬛⬛⬛⬛⬜⬜⬜⬜ (D)\n\n---\n\n## Critical Violations (2)\n\n### V-001: Missing ARIA label on button\n- **File**: src/components/UserProfile.tsx:45\n- **Severity**: CRITICAL\n- **Message**: Button missing aria-label for screen readers\n- **Actual**: `<button onClick={handleDelete}>×</button>`\n- **Expected**: Button with aria-label or aria-describedby\n- **Fix**: Add aria-label=\"Close\" to button element\n\n---\n\n## Major Violations (18)\n\n### V-002: Non-standard button size\n- **File**: src/components/Dashboard.tsx:78\n- **Severity**: MAJOR\n- **Message**: Button size 'tiny' not in standards (allowed: sm, md, lg)\n- **Actual**: `<Button size=\"tiny\">`\n- **Expected**: One of: sm, md, lg\n- **Fix**: Change size=\"tiny\" to size=\"sm\"\n\n..." }, "integration_with_trilogy": { "tool_8_dependency": { "description": "audit_codebase REQUIRES establish_standards to run first", "validation": "Check if coderef/standards/ directory exists and contains all 4 standards documents", "error_if_missing": "FileNotFoundError: Standards documents not found. Please run establish_standards first to generate baseline standards.", "workflow": "1. User runs establish_standards (Tool #8) → generates standards documents\n2. User runs audit_codebase (Tool #9) → finds violations\n3. User fixes violations\n4. User runs audit_codebase again → compliance score improves\n5. Repeat until compliance >= 80%" }, "tool_10_integration": { "description": "audit_codebase provides historical compliance tracking for check_consistency", "future_enhancement": "Tool #10 (check_consistency) can compare new code against historical audit reports to detect regression", "example": "If last audit had compliance score of 85%, and new code would drop it to 75%, Tool #10 blocks the change" } }, "changelog_entry": { "tool": "add_changelog_entry", "parameters": { "project_path": "C:/Users/willh/.mcp-servers/docs-mcp", "version": "1.3.0", "change_type": "feature", "severity": "major", "title": "Add audit_codebase tool for standards compliance auditing", "description": "Implemented Tool #9 (audit_codebase) - audits codebases against established standards to detect violations. Parses standards documents (UI-STANDARDS.md, BEHAVIOR-STANDARDS.md, UX-PATTERNS.md, COMPONENT-INDEX.md) and scans codebase for inconsistencies. Generates comprehensive audit reports with compliance scores (0-100), violations grouped by severity (critical/major/minor), file locations, and actionable fix suggestions. Creates timestamped reports in coderef/audits/ directory. Second tool in consistency trilogy (after Tool #8: establish_standards, before Tool #10: check_consistency).", "files": [ "server.py", "tool_handlers.py", "constants.py", "validation.py", "type_defs.py", "generators/audit_generator.py", "generators/__init__.py" ], "reason": "Enable teams to measure and improve codebase consistency by detecting violations of established standards. Bridge between standards documentation (Tool #8) and quality gates (Tool #10). Provides visibility into compliance and identifies areas needing remediation.", "impact": "Users can now audit their codebase against established standards. Generated reports show exactly what's inconsistent, where violations are, and how to fix them. Compliance scores track improvement over time. Enables data-driven consistency improvements." } }, "future_enhancements": { "v1_1_improvements": [ "Automated fix application (--auto-fix flag to apply suggestions)", "Trend analysis (compare current audit to previous audits)", "HTML audit report viewer (interactive dashboard)", "CI/CD integration (fail build if compliance < threshold)", "Custom violation rules (user-defined checks)", "Violation exemptions (ignore specific violations with justification)" ] }, "timeline_estimate": { "phase_1": "2 hours - Infrastructure setup", "phase_2": "3 hours - Standards parser", "phase_3": "4 hours - Violation detection engine", "phase_4": "2 hours - Report generation", "testing": "1.5 hours - Integration testing", "documentation": "1 hour - Update docs", "total": "10-12 hours for complete implementation" }, "implementation_checklist": { "description": "Complete implementation checklist with all task IDs for progress tracking", "pre_implementation": [ "☐ Review this plan for completeness", "☐ Verify Tool #8 (establish_standards) is fully implemented and tested", "☐ Get user approval on audit approach and violation detection rules" ], "phase_1_infrastructure": [ "☐ INFRA-001: Add audit_codebase tool schema to server.py:260", "☐ INFRA-002: Create handle_audit_codebase handler in tool_handlers.py", "☐ INFRA-003: Register handler in TOOL_HANDLERS dict", "☐ INFRA-004: Add constants to constants.py (AUDITS_DIR, AUDIT_REPORT, enums)", "☐ INFRA-005: Create generators/audit_generator.py skeleton with 13 method stubs", "☐ INFRA-006: Add 5 TypedDicts to type_defs.py (StandardsDataDict, AuditViolationDict, ComplianceScoreDict, ViolationStatsDict, AuditResultDict)", "☐ INFRA-007: Add validation functions to validation.py (validate_severity_filter, validate_audit_scope)" ], "phase_2_standards_parser": [ "☐ PARSE-001: Implement parse_standards_documents() - orchestrator for all 4 documents", "☐ PARSE-002: Implement parse_ui_standards() - extract button sizes/variants, modal sizes, colors", "☐ PARSE-003: Implement parse_behavior_standards() - extract error patterns, loading states", "☐ PARSE-004: Implement parse_ux_standards() - extract navigation/accessibility patterns", "☐ PARSE-005: Build StandardsDataDict structure with all parsed data" ], "phase_3_violation_detection": [ "☐ DETECT-001: Implement scan_for_violations() - main orchestrator", "☐ DETECT-002: Implement detect_ui_violations() - check buttons, colors, modals", "☐ DETECT-003: Implement detect_behavior_violations() - check errors, loading states", "☐ DETECT-004: Implement detect_ux_violations() - check ARIA, navigation, keyboard support", "☐ DETECT-005: Implement assign_severity() - categorize violations as critical/major/minor", "☐ DETECT-006: Implement generate_fix_suggestion() - create actionable remediation steps" ], "phase_4_report_generation": [ "☐ REPORT-001: Implement calculate_compliance_score() - weighted scoring algorithm", "☐ REPORT-002: Implement generate_audit_report() - format markdown with all sections", "☐ REPORT-003: Implement save_audit_report() - write timestamped file, return AuditResultDict" ], "phase_5_testing": [ "☐ TEST-001: Unit tests for standards parsers (parse_ui_standards, parse_behavior_standards, parse_ux_standards)", "☐ TEST-002: Unit tests for violation detectors (detect_ui_violations, detect_behavior_violations, detect_ux_violations)", "☐ TEST-003: Integration test - audit React project with intentional violations, verify detection", "☐ TEST-004: Integration test - audit project with perfect compliance, verify score=100", "☐ TEST-005: Edge case test - missing standards documents, verify graceful handling", "☐ TEST-006: Edge case test - empty standards (no patterns discovered), verify partial audit", "☐ TEST-007: Manual validation - run on sample project, verify report quality and fix suggestions" ], "phase_6_documentation": [ "☐ DOC-001: Update README.md - Add audit_codebase to Available Tools section", "☐ DOC-002: Update API.md - Document audit_codebase endpoint with parameters, examples, error handling", "☐ DOC-003: Update ARCHITECTURE.md - Add AuditGenerator to Module Architecture section", "☐ DOC-004: Update CLAUDE.md - Add complete AI usage guidance for audit_codebase to Tool Catalog" ], "finalization": [ "☐ Add changelog entry via add_changelog_entry (version 1.3.0, change_type=feature)", "☐ Update tooling-plan.json - Mark Tool #9 status as 'implemented'", "☐ Run final integration test on docs-mcp itself (should show high compliance)", "☐ Commit changes with message: 'feat: implement audit_codebase tool (Tool #9)'", "☐ Begin planning Tool #10 (check_consistency) - final tool in consistency trilogy" ] }, "next_steps": [ "1. Begin Phase 1: Infrastructure setup", "2. Implement AuditGenerator class skeleton", "3. Build standards parser (Phase 2)", "4. Implement violation detection (Phase 3)", "5. Build report generator (Phase 4)", "6. Test on sample React project with intentional violations", "7. Test on docs-mcp project (should show high compliance since no UI)", "8. Update documentation", "9. Create changelog entry via add_changelog_entry", "10. Commit and push changes", "11. Plan Tool #10: check_consistency" ] }