SmartDrive 🧠☁️

Semantic search for your entire OneDrive, powered by RAG architecture with Pinecone vector search and Azure Blob Storage.

SmartDrive is an MCP (Model Context Protocol) server that brings intelligent semantic search to your Microsoft OneDrive documents. Ask Claude to find "tax forms" and it'll surface your 1099s, W-2s, and related docs—even if those exact words aren't in the filename. Built with a true RAG architecture: hybrid vector search (semantic + keyword) in Pinecone, full document storage in Azure Blob.

🔥 Features

Core Capabilities

• RAG Architecture: True retrieval-augmented generation with vectors in Pinecone, full text in Azure Blob

• Hybrid Search: Combines semantic (dense vectors) + keyword (sparse BM25) for maximum accuracy

• Semantic Search: Natural language queries - "tax forms" finds W-2s, 1099s, etc.

• Flexible Embeddings: Choose local (free, optional), Voyage AI (recommended), Pinecone inference, or OpenAI-compatible APIs

• ONE Vector Per File: No chunking = 12.5x faster indexing, simpler search, better results

• 100K Char Embeddings: Full small docs embedded, intelligent sampling (80% beginning + 20% end) for large files

• Auto-Refreshing Index: Background delta sync (Graph /delta) — boot + every 15 min, with stale-token fallback

• Interactive Folder Selection: Choose which folders to index, skip what you don't need

• Smart Caching: Remembers authentication, folder choices, and delta state between runs

• MCP Integration: Two tools via Streamable HTTP — search_onedrive and read_document (any MCP client: Open WebUI, Claude, Inspector)

Document Support

• Documents: PDF (with OCR for scanned docs!), DOCX, DOC

• Presentations: PPTX (legacy .ppt not supported - convert to .pptx)

• Spreadsheets: XLSX, XLSM, CSV

• Data: JSON, TXT, Markdown (MD)

• Images: PNG, JPG, TIFF, BMP, GIF (with OCR)

• Archives: ZIP files (list contents or extract and index)

• Graceful Fallbacks: Corrupted/malformed files indexed with metadata only

OCR & Document Intelligence

• Local OCR: EasyOCR for scanned PDFs and images (free, no external software!)

• Cloud OCR: Azure Computer Vision for 10-20x faster processing (optional)

• Azure Document Intelligence: Premium AI for forms, tables, invoices, receipts with handwriting support

• Flexible Modes: Never, selective (smart detection), or always use Document Intelligence

• No Setup Required: Local OCR works out of the box

• Smart Detection: Automatically detects scanned PDFs and applies OCR

Related MCP server: FAQ RAG MCP Server

📦 Installation

Prerequisites

• Python 3.10+

• Microsoft 365 account with OneDrive

• Azure account (for Blob Storage - free tier available)

• Pinecone account (free tier available with hybrid search support)

• Any MCP client supporting Streamable HTTP (Open WebUI, Claude, MCP Inspector)

Quick Setup

1. Clone the repo

   git clone https://github.com/1818TusculumSt/smartdrive-mcp.git
   cd smartdrive-mcp

2. Install dependencies

   pip install -r requirements.txt

   The default requirements do not install local sentence-transformer embeddings. If using EMBEDDING_PROVIDER=local, install them separately. For CPU-only machines, install the CPU PyTorch wheel first to avoid CUDA packages:

   pip install --index-url https://download.pytorch.org/whl/cpu torch
   pip install -r requirements-local.txt

   Alternatively, use EMBEDDING_PROVIDER=voyage, api, or pinecone and skip requirements-local.txt.

Streamable HTTP

The server is Streamable HTTP-only (no stdio). Default is 127.0.0.1:8000, override with --host/--port:

python smartdrive_server.py
# or
python smartdrive_server.py --host 127.0.0.1 --port 8083

The MCP endpoint is http://127.0.0.1:8000/mcp (adjust port if overridden); the health check is http://127.0.0.1:8000/healthz. The server binds 0.0.0.0 inside Docker so the host mapping works. Clients that support Streamable HTTP can use:

{
  "type": "http",
  "url": "http://127.0.0.1:8000/mcp"
}

Only run one SmartDrive server instance at a time, since each instance owns its background delta sync loop. Keep it running (terminal, tmux, systemd) — clients connect over HTTP, they don't spawn it.

3. Create Azure App Registration

   • Go to Azure Portal → App Registrations → New registration

   • Name: SmartDrive MCP

   • Supported accounts: Personal Microsoft accounts only

   • Redirect URI: Leave blank

   • After creation, go to API permissions → Add:

     • Files.Read.All

     • User.Read

   • Go to Authentication → Enable Allow public client flows

   • Copy Application (client) ID and Directory (tenant) ID

4. Create Pinecone Index

   Option A: Manual Creation (Recommended for most users)

   • Go to Pinecone → Create Index

   • Name: smartdrive

   • Dimensions: Choose based on your embedding provider:

     • 384 for local (all-MiniLM-L6-v2)

     • 1024 for Pinecone inference (llama-text-embed-v2)

     • 2048 for Voyage AI (voyage-3-large, recommended)

   • Metric: cosine

   • Cloud: AWS (free tier available)

   • Region: Choose closest to you (e.g., us-east-1)

   • Important: Check "Enable Hybrid Search" for best results (combines semantic + keyword search)

   • Copy your API Key and Index Host after creation

   Option B: Automated Creation (Advanced users)

   # Configure your .env with Pinecone credentials first
   python create_hybrid_index.py

   • Automatically creates a hybrid search index optimized for Voyage AI

   • Uses 2048 dimensions and dotproduct metric

   • Deletes and recreates existing index (use with caution!)

   • Useful for emergency recovery or scripted deployments

5. Create Azure Blob Storage Container

   • Go to Azure Portal → Storage Accounts → Create new (or use existing)

   • Choose Standard performance tier (general purpose v2)

   • After creation, go to Access keys → Copy Connection string

   • Create a container named documents (or use your own name)

6. Configure .env

   Copy .env.example to .env and fill in your values:

   # Pinecone (required)
   PINECONE_API_KEY=your_pinecone_api_key
   PINECONE_INDEX_NAME=smartdrive
   PINECONE_HOST=smartdrive-xxxxx.svc.aped-xxxx-xxxx.pinecone.io
   
   # Microsoft (required)
   MICROSOFT_CLIENT_ID=your_azure_client_id
   MICROSOFT_TENANT_ID=consumers
   
   # Azure Blob Storage (required for RAG)
   AZURE_STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=...
   AZURE_STORAGE_CONTAINER_NAME=documents
   
   # Embedding provider (optional, default: local)
   EMBEDDING_PROVIDER=local
   EMBEDDING_MODEL=all-MiniLM-L6-v2
   
   # For Voyage AI (recommended - 32K token context, 2048 dims, $0.10/1M tokens):
   # EMBEDDING_PROVIDER=voyage
   # VOYAGE_API_KEY=your_voyage_api_key
   # VOYAGE_MODEL=voyage-3-large
   
   # Azure Computer Vision OCR (optional - 10-20x faster than local)
   # AZURE_VISION_KEY=your_azure_vision_key
   # AZURE_VISION_ENDPOINT=https://your-region.api.cognitive.microsoft.com/

7. Index your OneDrive

   python onedrive_crawler.py

   You'll see an interactive menu:

   ============================================================
   📋 SmartDrive Crawler - Main Menu
   ============================================================
   1. Run crawler (use cached folder choices)
   2. Reset folder choices and start fresh
   3. View/edit cached folder choices
   4. Exit

   • First time: Choose option 1

   • Authenticate with your Microsoft account (device code flow)

   • Choose ZIP handling (list contents or extract - default is list)

   • Set file limit (or press Enter for no limit to index everything)

   • Answer Yes/No for each folder as the crawler discovers them

   • Use "always yes" or "skip always" to remember your choices!

8. Start the Streamable HTTP server

   python smartdrive_server.py
   # MCP endpoint: http://127.0.0.1:8000/mcp
   # Health check: http://127.0.0.1:8000/healthz

   The server must be running before clients connect. Keep it running in a terminal, tmux, or a systemd service. Env vars come from the server's shell/systemd/Docker environment. Base requirements no longer include sentence-transformers — for EMBEDDING_PROVIDER=local also run pip install -r requirements-local.txt (CPU-only: install torch CPU wheel first).

   Add the server to any MCP client that supports Streamable HTTP:

   {
     "mcpServers": {
       "smartdrive": {
         "type": "http",
         "url": "http://127.0.0.1:8000/mcp"
       }
     }
   }

   Note: The MCP server only needs Pinecone and Azure Blob Storage credentials. The crawler/delta sync need additional credentials (Microsoft Graph API, OCR services, embedding API keys).

9. Verify the server

   curl http://127.0.0.1:8000/healthz
   # Docker (host port may differ if you remapped 127.0.0.1:8083:8000):
   docker exec openwebui curl -s http://host.docker.internal:8000/healthz  # needs extra_hosts on the client container
   # or via shared network:
   # docker exec openwebui curl -s http://smartdrive-mcp:8000/healthz

🐳 Docker Setup (Recommended)

Why Docker?

• ✅ Zero system pollution - isolated environment

• ✅ No dependency conflicts - all Python packages contained

• ✅ Reproducible - works identically everywhere

• ✅ Easy cleanup - remove container, done

• ✅ Persistent cache - OAuth tokens and folder choices survive restarts

Docker Quick Start

1. Clone and configure

   git clone https://github.com/1818TusculumSt/smartdrive-mcp.git
   cd smartdrive-mcp
   cp .env.example .env
   # Edit .env with your credentials

2. Build and run

   docker-compose up -d

3. Index your OneDrive (first time)

   docker-compose run --rm smartdrive-mcp python onedrive_crawler.py

4. Subsequent runs (use cached folder choices)

   docker-compose run --rm smartdrive-mcp

Docker Commands

# Build the image
docker-compose build

# Run crawler interactively (uses host .env + cache mounts)
docker-compose run --rm smartdrive-mcp python onedrive_crawler.py
# Manual delta sync
docker-compose run --rm smartdrive-mcp python delta_sync.py sync

# View logs
docker-compose logs -f

# Stop container
docker-compose down

# Rebuild after code changes
docker-compose build --no-cache

# Clean up everything (keeps .env and cache files)
docker-compose down --rmi all

Cache Persistence

Docker mounts these from your host (create empty files first to avoid Docker creating directories):

touch ~/.smartdrive_token_cache.json ~/.smartdrive_folder_skip_cache.json ~/.smartdrive_delta_store.json

• ~/.smartdrive_token_cache.json - OAuth tokens (survives restarts)

• ~/.smartdrive_folder_skip_cache.json - Folder choices (remembers skip/process decisions)

• ~/.smartdrive_delta_store.json - Delta sync state (deltaLink + item map; avoids full resync)

• ~/.EasyOCR/ - OCR models (avoids re-downloading 100MB)

Using with MCP clients (Open WebUI, Claude, Inspector)

Start the container (it runs the Streamable HTTP server on 0.0.0.0:8000 inside, mapped to 127.0.0.1:8000 on the host):

docker-compose up -d
curl http://127.0.0.1:8000/healthz   # verify (use your host-mapped port if you changed 8000 → 8083)

If your client runs in Docker on a separate network (e.g., Open WebUI), keep them separate and use host.docker.internal — add to the client's compose:

extra_hosts:
  - "host.docker.internal:host-gateway"

Then point the client at the host-mapped URL:

{
  "mcpServers": {
    "smartdrive": {
      "type": "http",
      "url": "http://host.docker.internal:8000/mcp"
    }
  }
}

If they share smartdrive-net (docker network connect smartdrive-net openwebui), use the in-network name instead:

{
  "mcpServers": {
    "smartdrive": {
      "type": "http",
      "url": "http://smartdrive-mcp:8000/mcp"
    }
  }
}

Common pitfall: inside the client container 127.0.0.1 is itself. host.docker.internal or the shared service name is required when the client is containerized.

🚀 Usage

MCP tools

SmartDrive provides two MCP tools to any connected client:

1. search_onedrive - Hybrid semantic + keyword search

• Searches Pinecone with dense (semantic) + sparse (BM25/keyword) vectors

• Returns top-k results with file paths, dates, scores, and content previews

• Automatically fetches full text from Azure Blob for matched documents

• Smart truncation keeps responses under 900KB (shows first 2K chars per doc)

2. read_document - Retrieve full document text

• Fetches complete document content from Azure Blob Storage by doc_id

• Use this when you need the full text of a search result

• Returns entire document (no truncation)

Simply ask your assistant natural language questions:

• "Search my OneDrive for resume"

• "Find tax documents from 2024"

• "Show me project proposals"

• "Where are my meeting notes about the Q4 budget?"

• "Read the full content of document doc_abc123" (after getting doc_id from search)

The client will automatically use search_onedrive to find relevant documents and read_document to retrieve full content.

Interactive Crawler Menu

The crawler has a full menu system for managing your indexing:

Option 1: Run Crawler

• Choose ZIP handling (list or extract contents)

• Set file limit (or no limit for full index)

• Interactive folder selection

• Beautiful progress tracking with OCR status

Option 2: Reset Folder Choices

• Clear all cached folder preferences

• Start fresh with folder selection

Option 3: View/Edit Cached Folder Choices

• See all your saved folder decisions

• Toggle folders between SKIP and PROCESS

• Delete specific cached choices

Processing Summary

After crawling, you'll get a detailed summary:

============================================================
📊 Processing Summary:
============================================================
✅ Successfully extracted: 847 files

❌ Failed extractions (3):
   • corrupted_report.xlsx (.xlsx)
   • malformed_doc.pdf (.pdf)

⚠️ Unsupported file types (5):
   • .mp4: 2 file(s)
   • .zip: 3 file(s)
============================================================

🏗️ Architecture

SmartDrive uses a true RAG (Retrieval Augmented Generation) architecture that separates vector embeddings from document storage for optimal performance and unlimited document size support.

┌─────────────────┐
│  MCP Client     │  (Open WebUI / Claude / Inspector)
└────────┬────────┘
         │ Streamable HTTP
         ▼
┌─────────────────────┐       ┌──────────────────┐
│ smartdrive_server.py│◄──────┤  Pinecone Index  │
│  (Streamable HTTP)  │       │ (Hybrid Vectors) │
└────────┬────────────┘       └──────────────────┘
         │                             │
         │  background delta sync      ├─ Dense vectors (semantic)
         │  (Graph /delta)             ├─ Sparse vectors (BM25/keyword)
         │  every 15 min               ├─ Minimal metadata
         │                             └─ doc_id references
         │
         └──► ┌──────────────────┐
              │  Azure Blob      │
              │  (Full Texts)    │
              └──────────────────┘
                      ├─ Complete documents
                      ├─ Unlimited size
                      └─ Fast retrieval (~50ms)

How It Works

1. Indexing (onedrive_crawler.py + delta_sync.py)

Manual onedrive_crawler.py and background delta_sync.py share the same extract→embed→upsert pipeline (indexing_core.py). For each file:

1. Authenticate via Microsoft Graph API (device code flow, cached)

2. Crawl OneDrive recursively with interactive folder selection

3. Extract text from documents:

   • PDFs: PyMuPDF (fitz) extracts text directly

   • Scanned PDFs/Images: OCR with Azure Document Intelligence → Azure Computer Vision → EasyOCR fallback chain

   • Office docs: python-docx (DOCX), python-pptx (PPTX), openpyxl (XLSX)

   • Text files: Direct read (TXT, JSON, MD, CSV)

   • Archives: List or extract ZIP contents

4. Generate embeddings:

   • Dense vector: Configurable provider (local/Voyage AI/Pinecone/OpenAI-compatible API)

   • Sparse vector: BM25 encoder for keyword matching (auto-truncates to 2048 terms)

   • Up to 100K chars embedded (smart sampling: 80% beginning + 20% end for large files)

5. Store in two places:

   • Azure Blob Storage: Full document text → returns doc_id (SHA256 hash of file path)

   • Pinecone: Dense + sparse vectors + minimal metadata + doc_id reference

6. Incremental/manual sync: Crawler checks Pinecone metadata; delta sync uses item_map + modified/size to skip unchanged, deletes via ContainerNotFound-safe path

7. Auto-refresh: delta_sync.py via Graph root/delta — full enumeration on first run/410, incremental thereafter; respects folder skip cache; runs inside the server process

8. Cleanup: Removes stale vectors from Pinecone + orphaned blobs from Azure (including moves/renames via path-derived IDs)

2. Searching (smartdrive_server.py)

When any MCP client searches your OneDrive:

1. Query embedding: Convert natural language query to dense + sparse vectors

2. Hybrid search: Query Pinecone with both vectors for semantic + keyword matching

3. Retrieve matches: Get top-k results with doc_id and metadata

4. Fetch full text: Retrieve complete documents from Azure Blob using doc_id

5. Smart truncation: Preview first 2K chars per result, keep total response <900KB

6. Return to Claude: Formatted results with file paths, dates, scores, and content

Components

Core Files:

• smartdrive_server.py - Streamable HTTP MCP server exposing search_onedrive and read_document; owns the background delta sync loop

• delta_sync.py - Graph delta sync (/me/drive/root/delta, client-side /Documents filter, 410 fallback, item_map for id-only deletes)

• indexing_core.py - Shared side-effect-free extract→embed→upsert pipeline (used by both crawler and delta sync)

• onedrive_crawler.py - Interactive manual CLI (imports from indexing_core.py, never imported by the server)

• embeddings.py - Embedding provider abstraction (local/Voyage/Pinecone/OpenAI-compatible APIs)

• document_storage.py - Azure Blob Storage interface for full document text

• document_intelligence.py - Azure Document Intelligence integration for advanced form/table extraction

• config.py - Configuration management with pydantic-settings

Dependencies:

• Pinecone: Vector database for hybrid search (dense + sparse vectors)

• Azure Blob Storage: Document storage (full text, unlimited size)

• Microsoft Graph API: OneDrive file access (device code flow auth; delta sync is silent-only)

• PyMuPDF (fitz): PDF text extraction

• python-docx, python-pptx, openpyxl: Office document parsing

• EasyOCR: Local OCR fallback (CPU-based, ~10-30 sec/page)

• Azure Computer Vision (optional): Cloud OCR (10-20x faster, ~1-3 sec/page)

• Azure Document Intelligence (optional): Advanced form/table extraction

• sentence-transformers (optional, requirements-local.txt): Local embedding model (default: all-MiniLM-L6-v2)

• pinecone-text: BM25 encoder for sparse vectors (keyword matching)

Key Architecture Decisions

Why RAG (vectors separate from full text)?

• ✅ No metadata limits: Pinecone has 40KB metadata cap, Azure Blob has unlimited storage

• ✅ ONE vector per file: No chunking = 12.5x faster indexing, simpler search

• ✅ Full context retrieval: Search finds relevant docs, then retrieves complete text

• ✅ Cost-efficient: ~$0.02/GB/month Azure storage vs expensive vector metadata

Why hybrid search (dense + sparse)?

• ✅ Dense vectors: Semantic understanding ("tax forms" matches "W-2", "1099")

• ✅ Sparse vectors: Exact keyword matching (filename search, acronyms)

• ✅ Better accuracy: Combines semantic similarity with keyword precision

Why 100K char embeddings?

• ✅ Full document understanding: Entire small docs embedded, smart sampling for large ones

• ✅ No chunking overhead: 1 vector vs 10+ per file

• ✅ Faster search: Fewer vectors to query

• ✅ More context: Voyage AI supports 32K tokens (128K chars), we use 100K for efficiency

Why incremental sync?

• ✅ Speed: Skips unchanged files (~100x faster for re-indexing)

• ✅ Cost savings: No re-embedding unchanged documents

• ✅ Metadata comparison: Checks modified date + file size in Pinecone before extraction

🛠️ Configuration

Embedding Providers

SmartDrive supports four embedding providers:

Local (Free, Private)

EMBEDDING_PROVIDER=local
EMBEDDING_MODEL=all-MiniLM-L6-v2

• ✅ Runs on your machine (sentence-transformers)

• ✅ No API calls or costs

• ✅ Complete privacy

• 📊 384 dimensions, ~512 token context

Voyage AI (Recommended for Large Documents) 🚀

EMBEDDING_PROVIDER=voyage
VOYAGE_API_KEY=your_voyage_api_key
VOYAGE_MODEL=voyage-3-large

• ✅ 32,000 token context (128K chars) - embed entire 50+ page PDFs!

• ✅ 2048 dimensions for maximum quality

• ✅ Fast cloud API, optimized for long documents

• 💰 $0.10 per 1M tokens (~$0.10-0.50 for 600 typical files)

• 🎯 Best for: Academic papers, books, reports, large documents

Pinecone Inference

EMBEDDING_PROVIDER=pinecone
EMBEDDING_MODEL=llama-text-embed-v2

• Hosted embedding models via Pinecone

• 1024 dimensions (high quality)

• Requires Pinecone API key

• Access to specialized models

Custom API

EMBEDDING_PROVIDER=api
EMBEDDING_API_URL=https://your-api.com/embeddings
EMBEDDING_API_KEY=your_api_key
EMBEDDING_MODEL=your-model-name

• OpenAI-compatible API format

• Use any embedding service (OpenAI, Cohere, etc.)

• Self-hosted options supported

Incremental Sync

SmartDrive intelligently skips unchanged files to save time and API costs:

• ✅ Pre-extraction check: Checks Pinecone before downloading/extracting files

• ✅ Metadata comparison: Compares file modified date and size

• ✅ Skip unchanged: Files that haven't changed are skipped entirely

• ✅ Update only modified: Only re-indexes files that changed

• ⚡ ~100x faster for re-indexing mostly unchanged folders

New Folder Detection: When running with cached folder choices, you can optionally check for new folders:

• Press Enter = Skip check (fast, uses cache only)

• Type 'check' = Discover new folders and prompt for each one

OCR Configuration

SmartDrive supports two OCR methods:

Local OCR (EasyOCR - Default)

• Free and works out of the box

• Downloads models automatically on first use (~100MB)

• Speed: 10-30 seconds per page

• No external dependencies

Cloud OCR (Azure Computer Vision - Optional)

Add to your .env:

AZURE_VISION_KEY=your_azure_vision_key
AZURE_VISION_ENDPOINT=https://your-region.api.cognitive.microsoft.com/

Benefits:

• 10-20x faster: 1-3 seconds per page vs 10-30 seconds

• More accurate OCR results

• No CPU/GPU load on your machine

• Free tier: 5,000 pages/month

• Paid tier: $1.50 per 1,000 pages (~$0.50-$2 for typical use)

Setup:

1. Go to Azure Portal

2. Create "Computer Vision" resource

3. Choose "Free F0" tier (5,000 pages/month) or "Standard S1"

4. Copy your API key and endpoint to .env

OCR Strict Mode (Optional)

Force Azure OCR only (no EasyOCR fallback):

OCR_STRICT_MODE=true

When enabled:

• ✅ Only uses Azure OCR (10-20x faster)

• ❌ Files fail if Azure OCR fails (no slow EasyOCR fallback)

• 💡 Use this for speed when you have Azure credits

SmartDrive will automatically use Azure OCR if credentials are provided, otherwise falls back to local EasyOCR (unless strict mode is enabled).

Azure Document Intelligence (Advanced)

Azure Document Intelligence (formerly Form Recognizer) is a premium AI service that provides advanced extraction capabilities beyond basic OCR. It's specifically designed for structured documents like forms, invoices, receipts, and tax documents.

What It Does:

• Intelligent form extraction: Automatically identifies and extracts key-value pairs from forms

• Table extraction: Preserves table structure with rows, columns, and cell relationships

• Handwriting recognition: Accurately recognizes handwritten text

• Layout analysis: Understands document structure (headers, sections, paragraphs)

• Pre-built models: Optimized for invoices, receipts, tax forms, ID documents

Three Operating Modes:

1. never (default): Document Intelligence is disabled

   • Uses standard Azure OCR → EasyOCR fallback chain

   • Fastest and most cost-effective for simple documents

2. selective (smart detection): Automatically enabled for specific document types

   • Activates when filenames contain keywords: tax, invoice, receipt, form, w2, 1099, w-2, 1040

   • Perfect balance of cost and capability

   • Recommended for mixed document libraries

3. always: Uses Document Intelligence for ALL documents

   • Maximum extraction quality for every file

   • Higher cost - only use if you need advanced extraction for all documents

Pricing & Limits:

Free Tier (F0):

• Cost: Free

• Limitations: Only processes first 2 pages of multi-page documents

• Monthly limit: 500 pages per month

• Speed: 1 transaction per second (TPS)

• Best for: Testing, small document sets, or documents that are 1-2 pages

Standard Tier (S0):

• Cost: $1.50 per 1,000 pages

• Full document processing: All pages extracted, no page limits

• No monthly limits: Pay-per-use

• Speed: 15 TPS

• Typical cost: ~$0.75 for 500 pages (vs free tier's first-2-pages limitation)

• Recommended for: Production use, multi-page documents

Setup Instructions:

1. Go to Azure Portal

2. Create a "Document Intelligence" resource (or search "Form Recognizer")

3. Choose tier:

   • Free F0: Testing or 1-2 page documents only

   • Standard S0: Production use with full document extraction

4. After creation, go to "Keys and Endpoint"

5. Copy KEY 1 and Endpoint URL

6. Add to your .env:

AZURE_FORM_RECOGNIZER_KEY=your_key_here
AZURE_FORM_RECOGNIZER_ENDPOINT=https://your-region.cognitiveservices.azure.com/

# Choose your mode:
USE_DOCUMENT_INTELLIGENCE=selective  # never, selective, or always

Fallback Chain:

SmartDrive uses a sophisticated fallback system:

1. Azure Document Intelligence (if enabled and conditions met)

2. Azure Computer Vision OCR (if credentials provided)

3. EasyOCR (local, always available)

Performance:

• Processing time: 5-15 seconds per document (varies with page count and complexity)

• Timeout: 2-minute safety timeout prevents hanging on problematic files

• Progress indicator: Real-time page-by-page progress for multi-page documents

• Reliability: Automatic fallback if service is unavailable or times out

Best Use Cases:

• Tax documents (W-2, 1099, 1040 forms)

• Invoices and receipts with complex layouts

• Business forms with structured fields

• Contracts with tables and signatures

• Handwritten notes and forms

• Documents requiring precise table extraction

Tips:

• Start with selective mode to balance cost and quality

• Use always mode only if you need advanced extraction for every document

• Free tier (F0) is fine for testing, but upgrade to S0 for production multi-page documents

• Monitor your usage in Azure Portal to stay within budget

Indexing Customization

File Limits:

• Test with 50-100 files first

• Then press Enter for no limit to index everything

Folder Selection:

• Interactive prompts for every folder

• Use "always" options to cache your choices

• Edit choices anytime via the menu (Option 3)

ZIP File Handling:

• Default: List contents (fast, searchable by filename and file list)

• Extract: Full text extraction from files inside ZIPs (slower, comprehensive)

📊 Supported File Formats

Note: Legacy PowerPoint (.ppt) files are not supported. Convert to .pptx for full-text extraction.

🎯 Best Practices

For Large OneDrive Libraries (10GB+)

1. Test First: Start with 100-file limit

2. Choose Folders Wisely: Skip temp folders, downloads, etc.

3. ZIP Strategy: Use "list" mode for most ZIPs (faster)

4. Run Overnight: Full indexing of large libraries can take hours

5. Monitor Progress: OCR shows page-by-page progress

For Best Search Results

1. Descriptive Queries: "Find project proposals from Q4" works better than "proposals"

2. Use Context: Include timeframes, topics, or people names

3. Iterative Search: Refine based on initial results

Maintaining Your Index

1. Automatic: Background delta sync keeps the index fresh (boot + every SMARTDRIVE_SYNC_INTERVAL seconds, default 15 min; 0 = boot only)

2. Manual: Re-run python onedrive_crawler.py for interactive control, or python delta_sync.py sync for a one-shot delta sync

3. Cached Choices: Folder preferences (~/.smartdrive_folder_skip_cache.json) and delta state (~/.smartdrive_delta_store.json) persist between runs

🐛 Troubleshooting

Common Issues

"OCR failed" warnings

• This is expected for some scanned PDFs

• Text extraction falls back to whatever is available

• Most documents work fine

Excel parsing errors

• Some complex XLSX files may fail

• CSV is more reliable for data files

Authentication timeout

• Tokens are cached - just re-run if expired

• Delete ~/.smartdrive_token_cache.json to force re-auth

Slow processing

• OCR takes 3-10 seconds per page

• Normal for scanned documents

• Progress indicators show it's working

Need Help?

Open a GitHub issue with:

• Error message (if any)

• File type causing issues

• Steps to reproduce

🗺️ Roadmap

Completed ✅

Core Features:

• ✅ Recursive folder crawling with interactive selection

• ✅ Interactive folder selection with caching

• ✅ New folder detection (optional pre-crawl check)

• ✅ Incremental sync (pre-extraction Pinecone check)

• ✅ Token caching for Microsoft authentication

• ✅ Progress indicators and comprehensive error reporting

• ✅ Graceful fallbacks for corrupted files

File Format Support:

• ✅ Documents: PDF, DOCX, DOC

• ✅ Spreadsheets: XLSX, XLSM, CSV

• ✅ Data: JSON, TXT, Markdown (.md)

• ✅ Images: PNG, JPG, TIFF, BMP, GIF (with OCR)

• ✅ Archives: ZIP (list + extract modes)

OCR & Document Intelligence:

• ✅ Local OCR (EasyOCR) with automatic model download

• ✅ Cloud OCR (Azure Computer Vision) for 10-20x speedup

• ✅ Azure Document Intelligence with three modes (never/selective/always)

• ✅ Scanned PDF OCR with page-by-page progress

• ✅ Image OCR via Document Intelligence (all formats)

• ✅ Smart timeout handling (2-minute safety)

• ✅ OCR strict mode (Azure-only, no fallback)

RAG Architecture:

• ✅ True RAG implementation: Vectors in Pinecone, full text in Azure Blob Storage

• ✅ ONE vector per file (no chunking, 12.5x faster uploads)

• ✅ 100K char embeddings (entire small docs, intelligent sampling for large)

• ✅ 2048-dimension Voyage AI embeddings for maximum quality (configurable: 384/1024/2048)

• ✅ Hybrid search: Dense (semantic) + sparse (BM25/keyword) vectors

• ✅ Rich metadata: File type categorization, size, dates, coverage indicator

• ✅ Azure Blob Storage: Unlimited document size storage (~$0.02/GB/month)

• ✅ Smart cleanup: Removes stale docs from both Pinecone and Azure

• ✅ Duplicate prevention: Azure checks existence before upload

• ✅ Sparse vector handling: Auto-truncates to 2048 terms (Pinecone limit)

• ✅ Two MCP tools: search_onedrive (hybrid search) + read_document (full text retrieval)

• ✅ Smart result truncation: Keeps responses under 900KB to prevent MCP 1MB limit issues

Embedding Providers:

• ✅ Local embeddings (sentence-transformers, free)

• ✅ Voyage AI (32K token context, 2048 dims, optimized for long docs)

• ✅ Pinecone inference (llama-text-embed-v2, 1024 dims)

• ✅ Custom API (OpenAI-compatible endpoints)

Recently Completed ✅ — Incremental Sync Daemon

• Microsoft Graph Delta API (/me/drive/root/delta) — incremental via deltaLink, full enumeration on 410/404; no ?token= bootstrap on personal accounts

• Background inside the one server — asyncio.to_thread so search never blocks; silent-only auth; log-never-raise

• item_map for id-only deletes — Graph deletes are id-only; path-derived doc_id/vector_id resolved via persistent map; moves/renames delete old + re-index new

• Folder-aware — respects ~/.smartdrive_folder_skip_cache.json (process / list-only / skip)

• Streamable HTTP-only transport — Uvicorn + Starlette + StreamableHTTPSessionManager (stateless=True, json_response=True); host port remapping documented for host.docker.internal

Other Features

• Support for SharePoint/Teams files

• Configurable crawl depth

• Custom metadata extraction

• Multi-language OCR

🤝 Contributing

Built for the community, by the community. PRs welcome!

Areas we'd love help with:

• Performance optimizations

• Documentation improvements

• Unit tests

• Additional file formats (e.g., RTF, ODT)

📄 License

MIT License - do whatever you want with this, just keep it free and accessible.

🙏 Acknowledgments

• Built with MCP by Anthropic

• Embeddings via sentence-transformers

• Vector storage by Pinecone

• Microsoft Graph API for OneDrive access

• OCR powered by EasyOCR

• PDF processing via PyMuPDF

💬 Support

Questions? Issues? Open a GitHub issue or reach out.

Built with 🔥 by @1818TusculumSt

💰 Cost Breakdown

Free Tier Setup (Recommended for Testing):

• ✅ Embeddings: Local (sentence-transformers) - $0/month

• ✅ Pinecone: Free tier - 100K vectors, hybrid search enabled - $0/month

• ✅ Azure Blob Storage: Free tier - 5GB, 20K read ops/month - $0/month

• ✅ OCR: Local EasyOCR - $0/month (slower but free)

• Total: $0/month for small-to-medium OneDrive libraries (<1000 files)

Production Setup (Recommended for Large Libraries):

• 💰 Embeddings: Voyage AI - ~$0.10-0.50 for 600 typical files (one-time indexing cost)

• 💰 Pinecone: Serverless - ~$0.03/month per 100K vectors (pay-as-you-go)

• 💰 Azure Blob Storage: $0.02/GB/month ($0.02/month for 500 docs @ 50KB avg)

• 💰 OCR (optional): Azure Computer Vision - Free tier: 5K pages/month, Paid: $1.50/1000 pages

• Total: ~$0.50-2.00/month for typical use (1000-5000 files)

Tips to Minimize Costs:

• Use local embeddings (free) instead of Voyage AI if you don't need 32K token context

• Azure Blob free tier covers most personal use cases (5GB = ~100K documents)

• Pinecone free tier covers up to 100K vectors (plenty for personal OneDrive)

• Local EasyOCR is free but slow - use Azure OCR only if you have lots of scanned docs