TokenScope
Note: This package is published as tokenscope but the repository name remains directory-explorer-mcp for historical reasons.
A Model Context Protocol (MCP) server for token-aware directory exploration and analysis, designed for Large Language Models (LLMs).
Overview
TokenScope provides intelligent directory structure analysis and token-aware file content exploration. It helps LLMs like Claude understand codebases and directory structures by:
- Scanning directory structures with token-efficient summaries
- Extracting and analyzing file contents with token awareness
- Finding important files for codebase understanding
- Generating reports with relevant information
Features
- Token-Aware Directory Scanning
- Explores directories recursively with configurable depth
- Provides intelligent summaries for large directories
- Respects .gitignore files and custom ignore patterns
- File Content Analysis
- Smart extraction of file contents that respects token limits
- Special handling for JSON and other structured files
- File selection prioritization based on importance
- Token Usage Statistics
- Estimates tokens required to process directories
- Breaks down token usage by file extension
- Identifies token-heavy files
- Comprehensive Reporting
- Generates markdown reports with directory structure
- Includes token usage statistics
- Shows samples of important files
Installation
Prerequisites
- Python 3.10 or higher
- uv (recommended for easy dependency management)
1. Main Installation (PyPI)
This is the recommended method for most users who just want to use TokenScope:
# Install from PyPI using uv (recommended)
uv pip install tokenscope
Running TokenScope
The --base-path argument is mandatory for security reasons. It restricts all file operations to the specified directory.
# Run using the installed package
uv run --with tokenscope tokenscope --base-path /path/to/allowed/directory
Configuring in Claude Desktop
- Locate Claude Desktop's configuration file (typically in
~/.config/claude/config.json) - Add TokenScope to the
mcpServers section:"mcpServers": {
"TokenScope": {
"command": "uv",
"args": [
"run",
"--with",
"tokenscope",
"tokenscope",
"--base-path",
"/your/secure/base/path"
]
}
}
- Replace
/your/secure/base/path with the directory you want to restrict operations to - Save the configuration file and restart Claude Desktop
2. Development Installation (from GitHub)
For contributors or users who want to modify the code:
# Clone the repository
git clone https://github.com/cdgaete/directory-explorer-mcp.git
cd directory-explorer-mcp
# Install development dependencies with uv
uv pip install -e ".[dev]"
Running in Development Mode
# Run the server directly with uv
uv run --with fastmcp --with tiktoken server.py --base-path /path/to/allowed/directory
Configuring Development Version in Claude Desktop
- Locate Claude Desktop's configuration file
- Add TokenScope to the
mcpServers section with development paths:"mcpServers": {
"TokenScope (Dev)": {
"command": "uv",
"args": [
"run",
"--with",
"fastmcp",
"--with",
"tiktoken",
"/path/to/your/directory-explorer-mcp/server.py",
"--base-path",
"/your/secure/base/path"
]
}
}
- Replace
/path/to/your/directory-explorer-mcp/server.py with the actual path to the server.py file - Replace
/your/secure/base/path with your secure directory
Security Features
The --base-path argument is mandatory for security reasons:
- All file operations are validated to ensure they're within the specified directory
- Attempts to access or modify files outside the base path will be rejected
- The base path is set once when starting the server and cannot be changed without restart
Example Prompts
Here are some examples of how to use TokenScope with Claude:
Please scan my project directory at /path/to/project and tell me about its structure, focusing on the most important files.
Analyze the token usage in my project directory at /path/to/project and tell me how many tokens would be needed to process the entire codebase with an LLM.
Generate a comprehensive directory report about my project at /path/to/project, including structure, token statistics, and samples of the most important files.
The server provides the following MCP tools:
scan_directory_structure
Scans a directory and returns its structure in a token-efficient way.
Extracts the content of a specific file, respecting token limits and format.
search_files_by_pattern
Searches for files matching specified patterns within a directory structure.
analyze_token_usage
Analyzes token usage for a directory or file to estimate LLM processing requirements.
copy_file_to_destination
Copy a file from source path to destination path.
generate_directory_report
Generates a comprehensive markdown report about a directory with token statistics.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Acknowledgments
- Built with FastMCP
- This same concept was implemented originally in repoai
- Inspired by the need to efficiently analyze codebases with LLMs