Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Token Optimization System find the acceptance criteria we set for the chunking experiment in my notes"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Token Optimization System is a small MCP server for Claude Desktop. It indexes folders you choose on your own computer. When Claude needs something from those files, it asks the server, and the server returns the few sections that answer the question instead of whole documents. Claude reads less, so a conversation uses less of its context window and of your plan's usage.
Everything runs on your machine. The index is one SQLite file in the install folder. There is no account, no telemetry and no network connection.
You ask Claude: "What acceptance criteria did we set for the chunking experiment?"
Claude calls optimizer_context with the question. The server returns two sections, about 1,500 tokens, each under a short file label. Without the server, Claude would open the records one by one: about 80,000 tokens for the same answer. That is one measured case on this project's own notes, not a benchmark.
Related MCP server: MCP-RAG
Tool | What it does |
| The main tool. Returns the most relevant sections for a question, at most |
| Lists the best-matching sections with their file and heading. |
| Returns one indexed section, chosen by file and heading. |
| Shows what is indexed and how many tokens were delivered compared with the full files. |
Supported files: Markdown, Word (.docx), PDF, PowerPoint (.pptx), Excel (.xlsx), plain text, CSV, JSON, YAML, HTML, CSS, Python, JavaScript and TypeScript. Markdown and Word files are split by heading, PDFs by page, presentations by slide and spreadsheets by sheet.
Windows 10 or 11
Python 3.10 or newer, from python.org
Each step shows the command to run and what you should see. If a step does not end as shown, stop there and read Troubleshooting.
1. Download this repository. Either click Code > Download ZIP, then right-click the file and choose Extract All. Or, with Git installed:
git clone https://github.com/HEX-1000/token-optimization-system.git2. Open PowerShell in the downloaded folder.
Open the folder in File Explorer (the one that contains README.md), click the address bar, type powershell and press Enter. Check that you are in the right place:
dir README.mdYou should see README.md listed.
3. Install.
powershell -ExecutionPolicy Bypass -File scripts\install.ps1This copies the program to %USERPROFILE%\Programs\TokenOptimizationSystem, creates a private Python environment there and installs its packages. It ends with Installed. Next steps:.
4. Index a folder. Replace the path with a folder of your documents:
powershell -ExecutionPolicy Bypass -File scripts\index.ps1 -Folder "C:\Users\you\Documents\Notes"It lists the files it reads and ends with a summary in which Failed files: is 0. Repeat this step for every folder you want Claude to search.
5. Quit Claude Desktop completely. Right-click its icon in the system tray and choose Quit. Closing the window is not enough.
6. Connect Claude Desktop.
powershell -ExecutionPolicy Bypass -File scripts\connect-claude-desktop.ps1It makes a backup of Claude Desktop's config file, adds one entry named token-optimization-system, and ends with Done.
7. Start Claude Desktop and test. Under Settings > Developer, token-optimization-system should show as running. In a new chat, ask: "Run optimizer_stats." The answer starts with Token Optimization System v0.25.2 and shows your file count.
Tip: Claude uses the tools when it thinks they help. To make that the default, add this to your project instructions in Claude: "Before reading local files, call optimizer_context with the question. Read whole files only if the returned sections are not enough."
Run step 4 again after files change. Only new or changed files are read again. Add -Rebuild to index a folder from scratch.
Quit Claude Desktop and run powershell -ExecutionPolicy Bypass -File scripts\connect-claude-desktop.ps1 -Undo.
Delete %USERPROFILE%\Programs\TokenOptimizationSystem. This also deletes the index.
Message or symptom | What to do |
| Run the commands exactly as shown, including |
| Install Python from python.org with Add python.exe to PATH ticked, open a new PowerShell window and repeat step 3. |
| Quit it from the tray icon, not just the window. |
| Run step 6 again with |
| In Claude Desktop open Settings > Developer > Edit Config once, quit Claude Desktop, and repeat step 6. |
The server shows as failed in Claude Desktop | Python from the Microsoft Store sometimes fails when another app starts it. Install Python from python.org and repeat step 3 with |
Search is keyword based (SQLite FTS5 with BM25). It finds sections that share words with the question, not paraphrases. There are no embeddings yet.
Queries are reduced to the letters a to z and digits, so words with other letters (ö, é, ß) do not match. English documents work best.
Text files other than Markdown (.txt, .json, .py and so on) are stored as one piece, which can be large. Folders of Markdown, Word, PDF, PowerPoint or Excel files work best.
Tables and speaker notes in Word and PowerPoint files are not extracted yet.
The scripts are for Windows. The server is plain Python and may run elsewhere, but that is untested.
Retrieval quality was measured on a small benchmark: 54 sections and 89 questions. Results on your files will differ.
Indexing reads only the folders you pass to index.ps1.
The index is stored in %USERPROFILE%\Programs\TokenOptimizationSystem\data\optimizer.db.
When Claude asks a question, the server reads only its own index. It does not open files on disk and makes no network connections.
The sections the server returns become part of your conversation with Claude, like anything else you share with Claude. Index only what you would share.
Index. Files are split into sections by structure and stored in a SQLite full-text index.
Retrieve. The question is matched against the index in several passes (all words, any word, word prefixes).
Rank. Candidates are scored on how much of the question they cover, heading matches, phrase matches and BM25 relevance.
Select. The best section is returned only if its score clears a fixed threshold; otherwise the server says nothing matched. Further sections are added only when they add new information.
Return. Selected sections go back to Claude in a compact format.
Details and the evaluation method: docs/how-it-works.md.
This is free, open-source software, and it stays usable without contributing anything back. Every installation works alone, with its own index.
The long-term plan is a privacy-safe way to learn from many installations. Users who opt in would share experiment results, such as a method, its settings and aggregate scores, never their documents, queries or file names. Improvements found on one machine could then be tested and released for everyone. None of this exists yet. The current stage is a reliable local release; a release process, an experiment format and opt-in contribution come next, in that order.
Current: 0.25.2, retrieval V0.25.0 with compact output. See CHANGELOG.md.
Development and experiments happen in a separate repository. Changes reach this one after they are measured on the evaluation suite. Bug reports and questions are welcome as issues.
This server cannot be deployed
Token-efficient search for coding agents over public and private documentation.
Ingest, manage, and retrieve documents for RAG-powered AI applications
Provide your AI coding tools with token-efficient access to up-to-date technical documentation for…
Read and search folders on your own computer with explicit consent. Files are never uploaded.
