Parquet MCP Server

# Audit Log Implementation Summary ## Changes Implemented ### 1. New Files Created #### `data/schemas/audit_log_schema.json` - Schema definition for audit log entries - Tracks operation type, affected fields, old/new values, timestamps #### `mcp-servers/parquet/AUDIT_LOG_GUIDE.md` - Complete documentation for audit log system - Usage examples, configuration, troubleshooting ### 2. Modified Files #### `mcp-servers/parquet/parquet_mcp_server.py` **New Constants:** - `LOGS_DIR`: Directory for audit log (`data/logs/`) - `AUDIT_LOG_PATH`: Path to audit log parquet file - `FULL_SNAPSHOT_ENABLED`: Environment variable for optional full snapshots (default: false) - `SNAPSHOT_FREQUENCY`: How often to create full snapshots (default: weekly) **New Functions:** - `should_create_full_snapshot()`: Determines if full snapshot needed based on configuration - `create_audit_entry()`: Creates audit log entry for data modifications - Stores operation type, record ID, affected fields, old/new values - Appends to `data/logs/audit_log.parquet` - Returns audit entry with unique `audit_id` **Modified Functions:** - `add_record()`: Now creates audit log entry instead of always creating full snapshot - Only creates full snapshot if `should_create_full_snapshot()` returns true - Returns `audit_id` in response - `update_records()`: Creates audit log entries for each updated record - Captures old values before modification - Stores complete change history - Returns list of `audit_ids` - `delete_records()`: Creates audit log entries for each deleted record - Stores complete record data before deletion - Enables rollback by restoring from audit log - Returns list of `audit_ids` **New Tools:** - `read_audit_log`: View audit log with filters (data_type, operation, record_id, limit) - `rollback_operation`: Rollback specific operation using audit_id - Add → Delete record - Update → Restore old values - Delete → Restore record ## Storage Efficiency Comparison ### Before (Full Snapshots) ``` Operation: Add 1 record to 10 MB parquet file Storage: 10 MB snapshot Result: 10 MB for 1 record change ``` ### After (Audit Log) ``` Operation: Add 1 record to 10 MB parquet file Storage: ~1 KB audit entry Result: 99.99% reduction ``` ### Hybrid (Recommended for Production) ``` Daily operations: Audit log entries (~1 KB each) Weekly snapshot: 10 MB full snapshot Result: 99%+ reduction vs snapshot-per-operation ``` ## Configuration ### Default (Development) ```bash # No environment variables needed # FULL_SNAPSHOT_ENABLED=false (default) # SNAPSHOT_FREQUENCY=weekly (default) ``` Result: Audit log only, no periodic snapshots ### Production ```bash export MCP_FULL_SNAPSHOTS=true export MCP_SNAPSHOT_FREQUENCY=weekly ``` Result: Weekly full snapshots + audit log for every operation ## Testing Required After MCP server restart, test: 1. **Add Operation** ```python mcp_parquet_add_record( data_type="beliefs", record={"belief_id": "test-123", ...} ) ``` - Verify audit log created at `data/logs/audit_log.parquet` - Verify `audit_id` returned in response - Verify no snapshot created (unless configured) 2. **Read Audit Log** ```python mcp_parquet_read_audit_log(limit=10) ``` - Verify audit entries returned with timestamps, operations, old/new values 3. **Update Operation** ```python mcp_parquet_update_records( data_type="beliefs", filters={"belief_id": "test-123"}, updates={"notes": "Updated"} ) ``` - Verify audit entry captures old and new values - Verify `audit_ids` returned in response 4. **Rollback** ```python mcp_parquet_rollback_operation(audit_id="...") ``` - Verify operation reversed correctly - Verify rollback creates new audit entry 5. **Delete Operation** ```python mcp_parquet_delete_records( data_type="beliefs", filters={"belief_id": "test-123"} ) ``` - Verify audit entry stores complete record data - Verify record can be restored via rollback ## Restart Instructions The MCP server must be restarted for changes to take effect: ### Option 1: Restart Cursor Close and reopen Cursor to restart all MCP servers ### Option 2: Restart MCP Server (if supported) Check Cursor settings for MCP server restart option ### Option 3: Verify Current State ```bash # Check if audit log exists (won't exist until server restart) ls -lh data/logs/audit_log.parquet # After restart and first write operation, file should exist ``` ## Migration Notes - Existing snapshots in `data/snapshots/` are preserved - No data migration required - System automatically uses audit log for new operations - Old snapshots remain available for recovery ## Rollback Plan (If Issues) To revert to old snapshot-only approach: 1. Restore previous version of `parquet_mcp_server.py` 2. Remove audit log functionality 3. Restart MCP server Backup of original implementation is in git history. ## Status ✅ Implementation complete ✅ Documentation created ✅ No linting errors ⏳ Awaiting MCP server restart ⏳ Testing pending (post-restart) ## Next Steps 1. Restart MCP server (Cursor restart) 2. Run test suite (see Testing Required section) 3. Verify audit log creation 4. Verify rollback functionality 5. Monitor storage efficiency gains 6. Configure environment variables for production (if needed)