basic-memory

# .gitignore Integration **Status**: New Feature **PR**: #314 **Impact**: Improved security and reduced noise ## What's New v0.15.0 integrates `.gitignore` support into the sync process. Files matching patterns in `.gitignore` are automatically skipped during synchronization, preventing sensitive files and build artifacts from being indexed. ## How It Works ### Ignore Pattern Sources Basic Memory combines patterns from two sources: 1. **Global user patterns**: `~/.basic-memory/.bmignore` - User's personal ignore patterns - Applied to all projects - Useful for global exclusions (OS files, editor configs) 2. **Project-specific patterns**: `{project}/.gitignore` - Project's standard gitignore file - Applied to that project only - Follows standard gitignore syntax ### Automatic .gitignore Respect When syncing, Basic Memory: 1. Loads patterns from `~/.basic-memory/.bmignore` (if exists) 2. Loads patterns from `.gitignore` in project root (if exists) 3. Combines both pattern sets 4. Skips files matching any pattern 5. Does not index ignored files ### Pattern Matching Uses standard gitignore syntax: ```gitignore # Comments are ignored *.log # Ignore all .log files build/ # Ignore build directory node_modules/ # Ignore node_modules .env # Ignore .env files !important.log # Exception: don't ignore this file ``` ## Benefits ### 1. Security **Prevents indexing sensitive files:** ```gitignore # Sensitive files automatically skipped .env .env.* secrets.json credentials/ *.key *.pem cloud-auth.json ``` **Result:** Secrets never indexed or synced ### 2. Performance **Skips unnecessary files:** ```gitignore # Build artifacts and caches node_modules/ __pycache__/ .pytest_cache/ dist/ build/ *.pyc ``` **Result:** Faster sync, smaller database ### 3. Reduced Noise **Ignores OS and editor files:** ```gitignore # macOS .DS_Store .AppleDouble # Linux *~ .directory # Windows Thumbs.db desktop.ini # Editors .vscode/ .idea/ *.swp ``` **Result:** Cleaner knowledge base ## Setup ### Default Behavior If no `.gitignore` exists, Basic Memory uses built-in patterns: ```gitignore # Default patterns .git .DS_Store node_modules __pycache__ .pytest_cache .env ``` ### Global .bmignore (Optional) Create global ignore patterns for all projects: ```bash # Create global ignore file cat > ~/.basic-memory/.bmignore <<'EOF' # OS files (apply to all projects) .DS_Store .AppleDouble Thumbs.db desktop.ini *~ # Editor files (apply to all projects) .vscode/ .idea/ *.swp *.swo # Always ignore these .env .env.* *.secret EOF ``` **Use cases:** - Personal preferences (editor configs) - OS-specific files - Global security rules ### Project-Specific .gitignore Create `.gitignore` in project root for project-specific patterns: ```bash # Create .gitignore cat > ~/basic-memory/.gitignore <<'EOF' # Project-specific secrets credentials.json *.key # Project build artifacts dist/ build/ *.pyc __pycache__/ node_modules/ # Project-specific temp files *.tmp *.cache EOF ``` **Use cases:** - Build artifacts - Dependencies (node_modules, venv) - Project-specific secrets ### Sync with .gitignore and .bmignore ```bash # Sync respects both .bmignore and .gitignore bm sync # Ignored files are skipped # → ".DS_Store skipped (global .bmignore)" # → ".env skipped (gitignored)" # → "node_modules/ skipped (gitignored)" ``` **Pattern precedence:** 1. Global `.bmignore` patterns checked first 2. Project `.gitignore` patterns checked second 3. If either matches, file is skipped ## Use Cases ### Git Repository as Knowledge Base Perfect synergy when using git for version control: ```bash # Project structure ~/my-knowledge/ ├── .git/ # ← git repo ├── .gitignore # ← shared ignore rules ├── notes/ │ ├── public.md # ← synced │ └── private.md # ← synced ├── .env # ← ignored by git AND sync └── build/ # ← ignored by git AND sync ``` **Benefits:** - Same ignore rules for git and sync - Consistent behavior - No sensitive files in either system ### Sensitive Information ```gitignore # .gitignore *.key *.pem credentials.json secrets/ .env* ``` **Result:** ```bash $ bm sync Syncing... → Skipped: api-key.pem (gitignored) → Skipped: .env (gitignored) → Skipped: secrets/passwords.txt (gitignored) ✓ Synced 15 files (3 skipped) ``` ### Development Environment ```gitignore # Project-specific node_modules/ venv/ .venv/ __pycache__/ *.pyc .pytest_cache/ .coverage .tox/ dist/ build/ *.egg-info/ ``` **Result:** Clean knowledge base without dev noise ## Pattern Examples ### Common Patterns **Secrets:** ```gitignore .env .env.* *.key *.pem *secret* *password* credentials.json auth.json ``` **Build Artifacts:** ```gitignore dist/ build/ *.o *.pyc *.class *.jar node_modules/ __pycache__/ ``` **OS Files:** ```gitignore .DS_Store .AppleDouble .LSOverride Thumbs.db desktop.ini *~ ``` **Editors:** ```gitignore .vscode/ .idea/ *.swp *.swo *~ .project .settings/ ``` ### Advanced Patterns **Exceptions (!):** ```gitignore # Ignore all logs *.log # EXCEPT this one !important.log ``` **Directory-specific:** ```gitignore # Ignore only in root /.env # Ignore everywhere **/.env ``` **Wildcards:** ```gitignore # Multiple extensions *.{log,tmp,cache} # Specific patterns test_*.py *_backup.* ``` ## Integration with Cloud Sync ### .bmignore Files Overview Basic Memory uses `.bmignore` in two contexts: 1. **Global user patterns**: `~/.basic-memory/.bmignore` - Used for **local sync** - Standard gitignore syntax - Applied to all projects 2. **Cloud bisync filters**: `.bmignore.rclone` - Used for **cloud sync** - rclone filter format - Auto-generated from .gitignore patterns ### Automatic Pattern Conversion Cloud bisync converts .gitignore to rclone filter format: ```bash # Source: .gitignore (standard gitignore syntax) node_modules/ *.log .env # Generated: .bmignore.rclone (rclone filter format) - node_modules/** - *.log - .env ``` **Automatic conversion:** Basic Memory handles conversion during cloud sync ### Sync Workflow 1. **Local sync** (respects .bmignore + .gitignore) ```bash bm sync # → Loads ~/.basic-memory/.bmignore (global) # → Loads {project}/.gitignore (project-specific) # → Skips files matching either ``` 2. **Cloud bisync** (respects .bmignore.rclone) ```bash bm cloud bisync # → Generates .bmignore.rclone from .gitignore # → Uses rclone filters for cloud sync # → Skips same files as local sync ``` **Result:** Consistent ignore behavior across local and cloud sync ## Verification ### Check What's Ignored ```bash # Dry-run sync to see what's skipped bm sync --dry-run # Output shows: # → Syncing: notes/ideas.md # → Skipped: .env (gitignored) # → Skipped: node_modules/package.json (gitignored) ``` ### List Ignore Patterns ```bash # View .gitignore cat .gitignore # View effective patterns bm sync --show-patterns ``` ### Test Pattern Matching ```bash # Check if file matches pattern git check-ignore -v path/to/file # Example: git check-ignore -v .env # → .gitignore:5:.env .env ``` ## Migration ### From v0.14.x **Before v0.15.0:** - .gitignore patterns not respected - All files synced, including ignored ones - Manual exclude rules needed **v0.15.0+:** - .gitignore automatically respected - Ignored files skipped - No manual configuration needed **Action:** Just add/update .gitignore - next sync uses it ### Cleaning Up Already-Indexed Files If ignored files were previously synced: ```bash # Option 1: Re-sync (re-indexes from scratch) bm sync --force-resync # Option 2: Delete and re-sync specific project bm project remove old-project bm project add clean-project ~/basic-memory bm sync --project clean-project ``` ## Troubleshooting ### File Not Being Ignored **Problem:** File still synced despite being in .gitignore **Check:** 1. Is .gitignore in project root? ```bash ls -la ~/basic-memory/.gitignore ``` 2. Is pattern correct? ```bash # Test pattern git check-ignore -v path/to/file ``` 3. Is file already indexed? ```bash # Force resync bm sync --force-resync ``` ### Pattern Not Matching **Problem:** Pattern doesn't match expected files **Common issues:** ```gitignore # ✗ Wrong: Won't match subdirectories node_modules # ✓ Correct: Matches recursively node_modules/ **/node_modules/ # ✗ Wrong: Only matches in root /.env # ✓ Correct: Matches everywhere .env **/.env ``` ### .gitignore Not Found **Problem:** No .gitignore file exists **Solution:** ```bash # Create default .gitignore cat > ~/basic-memory/.gitignore <<'EOF' .git .DS_Store .env node_modules/ __pycache__/ EOF # Re-sync bm sync ``` ## Best Practices ### 1. Use Global .bmignore for Personal Preferences Set global patterns once, apply to all projects: ```bash # Create global ignore file cat > ~/.basic-memory/.bmignore <<'EOF' # Personal editor/OS preferences .DS_Store .vscode/ .idea/ *.swp # Never sync these anywhere .env .env.* EOF ``` ### 2. Use .gitignore for Project-Specific Patterns Even if not using git, create .gitignore for project-specific sync: ```bash # Create project .gitignore cat > .gitignore <<'EOF' # Project build artifacts dist/ node_modules/ __pycache__/ # Project secrets credentials.json *.key EOF ``` ### 3. Ignore Secrets First Start with security (both global and project-specific): ```bash # Global: ~/.basic-memory/.bmignore .env* *.key *.pem # Project: .gitignore credentials.json secrets/ api-keys.txt ``` ### 4. Ignore Build Artifacts Reduce noise in project .gitignore: ```gitignore # Build outputs dist/ build/ node_modules/ __pycache__/ *.pyc ``` ### 5. Use Standard Templates Start with community templates for .gitignore: - [GitHub .gitignore templates](https://github.com/github/gitignore) - Language-specific ignores (Python, Node, etc.) - Framework-specific ignores ### 6. Test Your Patterns ```bash # Verify pattern works git check-ignore -v file.log # Test sync bm sync --dry-run ``` ## See Also - `cloud-bisync.md` - Cloud sync and .bmignore.rclone conversion - `env-file-removal.md` - Why .env files should be ignored - gitignore documentation: https://git-scm.com/docs/gitignore - GitHub gitignore templates: https://github.com/github/gitignore ## Summary Basic Memory provides flexible ignore patterns through: - **Global**: `~/.basic-memory/.bmignore` - personal preferences across all projects - **Project**: `.gitignore` - project-specific patterns - **Cloud**: `.bmignore.rclone` - auto-generated for cloud sync Use global .bmignore for OS/editor files, project .gitignore for build artifacts and secrets.