Jinni: Bring Your Project Into Context

# Jinni Project - Detailed Design This document outlines the detailed design for the components of the Jinni project, expanding on the high-level plan in `PLAN.md`. ## 1. Introduction This document details the internal design of the `jinni` MCP Server, `jinni` CLI, Core Logic Module, and Configuration System. ## 2. Core Logic Module Design *(Based on `dev-reference/prototype.py` and `PLAN.md`)* * **File Discovery:** * Utilizes `os.walk(topdown=True)` for recursive traversal starting from one or more target paths provided by the user. * Symbolic links encountered during traversal will be **skipped**. * **Filtering (Dynamic Rule Application):** * Determines the `rule_root` for each target: * Targets within the project root (or CWD if no project root) use the project root/CWD as their rule root * External targets (outside project root/CWD) use themselves as rule root * Determines the `output_rel_root` (explicit project root or common ancestor) for final output path relativity. * For each directory visited during `os.walk`: * If overrides are active (CLI `--overrides` or MCP `rules`): Uses a pre-compiled `PathSpec` based exclusively on the provided override rules. Path matching is relative to `walk_target_path`. * If no overrides: Finds relevant `.contextfiles` from `rule_root` down to the current directory, loads rules, combines with defaults, and compiles a `PathSpec` specific to that directory's context. Path matching is relative to `walk_target_path`. * Applies the active `PathSpec` to filter files and prune subdirectories. * **Explicit Target Inclusion:** Explicitly provided file targets bypass rule checks (but not binary/size checks). * **Rule Root Isolation:** Each target has its own rule root, ensuring external targets have self-contained rule sets. * Handles `list_only` flag. * **File Reading:** * Attempts multiple encodings (UTF-8, Latin-1, CP1252). * Handles file read errors gracefully (permissions, non-existent files). * **Output Formatting:** * Concatenates file content. * Prepends a path header (e.g., ````path=src/app.py`) for each file unless `list_only` is true. Content is enclosed in triple backticks. * When not `list_only`, file entries are separated by a blank line. * **Large Context Handling:** * Implement a configurable total content size limit (default: 100MB, configurable via `JINNI_MAX_SIZE_MB` env var). Accumulate file sizes during traversal. If the limit is exceeded, **abort** processing and return/print an explanatory error message (e.g., "Error: Total content size exceeds limit of 100MB"). ## 3. Configuration System (`.contextfiles` & Overrides) Design * **Core Principle:** Files and directories are filtered based on `.gitignore`-style rules. Files are included by default, but filtered by built-in defaults and custom rules. The effective rules are determined dynamically during traversal. * **Implementation Library:** `pathspec` (using `'gitwildmatch'` syntax). * **File Format:** * Plain text file named `.contextfiles`. * One pattern per line. * Encoding: UTF-8. * **Rule Syntax (`gitwildmatch`):** * Follows standard `.gitignore` syntax rules. See `pathspec` documentation and `.gitignore` documentation for full details. * Lines starting with `#` are comments. * Empty lines or lines with only whitespace are ignored. * Patterns specify files/directories to match. By default, matching an item includes it, unless the pattern starts with `!`. * Patterns starting with `!` denote an **exclusion** rule (these files/dirs should be skipped, even if matched by an inclusion pattern). * Patterns are matched against the path relative to the **target directory** being processed (or the project root if no specific target is given). * A leading `/` anchors the pattern to the directory containing the `.contextfiles`. * A trailing `/` indicates the pattern applies only to directories. * `**` matches zero or more directories (recursive wildcard). * Example `.contextfiles` (Inclusion-focused): ``` # Include all Python files in the src directory and subdirectories src/**/*.py # Include all Markdown files anywhere *.md # Exclude temporary files anywhere, even if they are .py or .md !*.tmp # Include the specific config file at the root of this context /config.yaml # Exclude the entire build directory relative to this file build/ ``` * **Rule Loading & Compilation (`config_system.py`):** * Provides helper functions: * `load_rules_from_file(path)`: Reads lines from a `.contextfile` or override file. * `compile_spec_from_rules(rules_list)`: Compiles a list of rule strings into a `pathspec.PathSpec` object. * Defines `DEFAULT_RULES` (common excludes like `.git/`, `node_modules/`, etc.). * **Rule Application Logic (`context_walker.py`):** 1. **Determine Rule Root:** The `rule_root` is passed from `core_logic.py` based on target location (project root for internal targets, self for external). 2. **Override Check:** Determines if override rules are provided (CLI `--overrides <file>` or MCP `rules` argument). 3. **Dynamic Spec Generation (During `os.walk`):** For each directory visited: * Finds all `.contextfiles` and `.gitignore` files starting from `rule_root` down to the current directory (no rules above `rule_root` are considered). * Loads rules from these files. * Combines rules in order: * `DEFAULT_RULES` (common excludes like `.git/`, `node_modules/`, etc.) * `.gitignore` rules (converted to Jinni-style patterns) * `.contextfiles` rules (respecting order: root rules first, current dir rules last) * **If overrides are provided:** Override rules are added as high-priority rules at the end * Compiles a new `PathSpec` object specific to this directory's context. 4. **Filtering Decision:** * The active `PathSpec` for the current directory is used to match files and subdirectories. * The path used for matching is calculated **relative to `walk_target_path`**. * Standard `pathspec` matching applies (last matching pattern wins, `!` negates). If no user-defined pattern matches, the item is included unless it matches a built-in default exclusion. If no pattern matches at all, it's included. 5. **Explicit Target Inclusion:** Explicitly provided file targets bypass rule checks (but not binary/size checks). 6. **Directory Pruning:** During `os.walk` (`topdown=True`), subdirectories are checked against the active `PathSpec` (using the path relative to `walk_target_path`). If a subdirectory doesn't match (is excluded) and wasn't an explicit target, it's removed from `dirnames` to prevent traversal. 7. **Output Path Relativity:** Final output paths (in headers or list mode) are always calculated relative to the original `output_rel_root` determined in `core_logic.py`. ## 4. `jinni` MCP Server Design * **Server Name:** `jinni` (Update from `codedump` in `prototype.py`). * **Transport:** Stdio. * **Library:** `mcp.server.fastmcp`. * **Tool: `read_context`** * **Description:** Generates a concatenated view of relevant code files from a specified directory, applying filtering rules from defaults, `.contextfiles`, and optional inline rules. * **Input Schema:** * `project_root` (string, required): The absolute path to the project root directory. Rule discovery and output paths are relative to this root. * `targets` (JSON array of strings, **required**): Specifies the file(s)/director(y/ies) within `project_root` to process. Must be a JSON array of string paths (e.g., `["path/to/file1", "path/to/dir2"]`). Paths can be absolute or relative to CWD. If an empty list `[]` is provided, the entire `project_root` is processed. All target paths must resolve to locations inside `project_root`. * `rules` (JSON array of strings, **required**): A list of inline filtering rules (using `.gitignore`-style syntax, e.g., `["src/**/*.py", "!*.tmp"]`). Provide an empty list `[]` if no specific rules are needed (uses built-in defaults). If non-empty, these rules are used exclusively, ignoring built-in defaults and `.contextfiles`. * `list_only` (boolean, optional, default: false): Only list file paths. * `size_limit_mb` (integer, optional): Override context size limit. * `debug_explain` (boolean, optional, default: false): Enable debug logging on server stderr. * **Output:** String containing concatenated content or file list. * **Error Handling:** Returns standard MCP errors for invalid input or internal failures. * **Capabilities:** Reports `tools` capability. ## 5. `jinni` CLI Design * **Command:** `jinni` * **Arguments:** * `paths` (optional, positional, zero or more): Target directory or file paths to process. Defaults to `['.']` if none provided. * `--root <DIR>` / `-r <DIR>` (optional): Specify the project root directory. This primarily affects the base for calculating relative output paths. If omitted, the root is inferred from the common ancestor of the `<PATH...>` arguments. Rule discovery and matching are relative to the target path(s). * `--output <file>` / `-o <file>` (optional): Write output to a file instead of stdout. * `--list-only` / `-l` (optional): Only list file paths found. * `--overrides <file>` (optional): Specify an overrides file. If provided, rules from this file are used exclusively, ignoring built-in defaults and all `.contextfiles`. * `--size-limit-mb <int>` / `-s <int>` (optional): Override context size limit. * `--debug-explain` (optional): Enable detailed debug logging to stderr and `jinni_debug.log`. * `--no-copy` (optional): Prevent automatically copying output content to the clipboard when printing to stdout (default is to copy). * **Output:** Prints concatenated content or file list to stdout or specified output file. * **Error Handling:** Prints user-friendly error messages to stderr. ## 6. Data Flow / Interaction * Both the `jinni` MCP Server (`read_context` tool) and the `jinni` CLI utilize the `core_logic.read_context` function. * The `core_logic.read_context` function handles target validation, determines the `output_rel_root` for output paths, and determines the initial `walk_target_path`(s). * It calls `context_walker.walk_and_process` for each target directory. * `context_walker.walk_and_process` manages the directory traversal (`os.walk`), determines if overrides are active, discovers `.contextfiles` relative to the `walk_target_path`, compiles the `PathSpec`, and applies it to filter files/directories using paths relative to `walk_target_path`. It calls `file_processor.process_file` for included files.