TDZ C64 Knowledge

MCP server for managing and searching Commodore 64 documentation. Ingest PDFs, text, Markdown, HTML, Excel, and web pages into a searchable knowledge base accessible via Claude Code or other MCP clients.

🚀 Quick Start

# 1. Install
python -m venv .venv
.venv\Scripts\activate
pip install -e .

# 2. Configure Claude Code
claude mcp add tdz-c64-knowledge -- .venv\Scripts\python.exe server.py

# 3. Add documents
.venv\Scripts\python.exe cli.py add-folder "C:\c64docs" --tags reference --recursive

# 4. Search via Claude Code
# Ask: "Search the C64 docs for VIC-II sprite registers"

See QUICKSTART.md for detailed setup.

Related MCP server: ElasticMind-MCP

Features

Search & Retrieval

• FTS5 full-text search - 480x faster queries (50ms vs 24s)

• Semantic search - Find by meaning, not keywords (e.g., "movable objects" → "sprites")

• RAG question answering - Answer questions by synthesizing docs with citations

• Fuzzy search - Typo tolerance ("VIC2" → "VIC-II", "asembly" → "assembly")

• Progressive refinement - Search within results to narrow down

• Hybrid search - Combines keyword + semantic with configurable weighting

• Similarity search - Discover related documentation automatically

• Query preprocessing - NLTK stemming and stopword removal

• Smart tagging - AI-powered tag suggestions by category

• Table/code search - Search extracted tables and code blocks

Document Management

• Multi-format - PDF, TXT, MD, HTML, Excel, web scraping

• Duplicate detection - Content-based deduplication

• Chunked retrieval - Get specific sections without loading entire docs

• Metadata extraction - Author, subject, page numbers

• Persistent index - Documents stay indexed between sessions

AI-Powered Features

• Entity extraction - Extract hardware, memory addresses, instructions, concepts (5000x faster with C64 regex patterns)

• Relationship mapping - Co-occurrence analysis with distance-based strength scoring

• Document comparison - Side-by-side analysis with similarity scores

• Natural language query translation - Parse queries into structured search parameters

• Anomaly detection - ML-based baseline learning for URL-sourced content (3400+ docs/second)

• Temporal analysis - Event detection, timeline construction, historical context (5 event types, 8 date formats)

• Advanced visualizations - 3D knowledge graphs, hierarchical bundling, Sankey flow diagrams

Wiki Export (NEW in v2.23.15)

• Static HTML wiki - Export entire knowledge base to browsable website

• Document similarity map - 2D visualization using UMAP/t-SNE dimensionality reduction

• Interactive timeline - Horizontal scrollable timeline with zoom levels and event filters

• Knowledge graph - D3.js force-directed graph (178 entities, 20 relationships)

• Enhanced UI - Explanation boxes, prominent ASK AI button, file type detection

• Clickable clusters - Browse k-means clusters with linked documents

• No server required - Pure client-side JavaScript, works offline

• Full-text search - Fuse.js powered search across all content

• See WIKI_EXPORT_GUIDE.md for usage

REST API (Optional)

• 18 endpoints - Full CRUD, search, analytics, export

• OpenAPI/Swagger docs - Interactive API at /api/docs

• API authentication - Secure via X-API-Key header

• See docs/REST_API.md for details

Performance

• Scalability - Tested to 5,000+ documents

• Concurrent throughput - 5,712 queries/sec (10 workers)

• Lazy loading - 100k+ document support

• Search caching - 50-100x speedup for repeated queries

Installation (Windows)

Prerequisites

• Python 3.10+ - https://python.org (check "Add Python to PATH")

• uv (recommended) or pip: pip install uv

Setup

cd C:\Users\YourName\mcp-servers\tdz-c64-knowledge

# Using uv (faster)
uv venv
.venv\Scripts\activate
uv pip install mcp pypdf rank-bm25 nltk

# Or using pip
python -m venv .venv
.venv\Scripts\activate
pip install mcp pypdf rank-bm25 nltk

# Test
python server.py  # Press Ctrl+C to stop

Configuration

Claude Code

claude mcp add tdz-c64-knowledge -- C:\path\.venv\Scripts\python.exe C:\path\server.py

Or add to .claude/settings.json:

{
  "mcpServers": {
    "tdz-c64-knowledge": {
      "command": "C:\\path\\.venv\\Scripts\\python.exe",
      "args": ["C:\\path\\server.py"],
      "env": {
        "TDZ_DATA_DIR": "C:\\c64-knowledge-data"
      }
    }
  }
}

Claude Desktop

Add to %APPDATA%\Claude\claude_desktop_config.json:

{
  "mcpServers": {
    "tdz-c64-knowledge": {
      "command": "C:\\path\\.venv\\Scripts\\python.exe",
      "args": ["C:\\path\\server.py"],
      "env": {
        "TDZ_DATA_DIR": "C:\\c64-knowledge-data"
      }
    }
  }
}

Environment Variables

Search Features

FTS5 Full-Text Search (Recommended)

Enable with USE_FTS5=1 for maximum performance:

• 480x faster than BM25

• Native SQLite BM25 ranking

• Porter stemming tokenizer

Semantic Search

Enable with USE_SEMANTIC_SEARCH=1:

• Meaning-based search (e.g., "movable objects" finds "sprites")

• FAISS vector similarity with sentence-transformers

• ~7-16ms per query after embeddings built

• Pre-build embeddings: pip install sentence-transformers faiss-cpu

Phrase Search

Use double quotes for exact phrases:

search_docs(query='"VIC-II chip" registers')

Fuzzy Search

Handles typos automatically with USE_FUZZY_SEARCH=1:

• "VIC-I" → "VIC-II" (83% similarity)

• "grafics" → "graphics" (88% similarity)

• Configurable threshold (default: 80%)

OCR for Scanned PDFs

Automatic with USE_OCR=1:

• Detects scanned PDFs (< 100 chars extracted)

• Uses Tesseract OCR

• Install: pip install pytesseract pdf2image Pillow + Tesseract binary

• ~1-2 seconds per page

Temporal Analysis & Visualizations

Extract events, construct timelines, and visualize knowledge graphs.

Event Detection

Automatically detect significant events in documents:

• 5 Event Types - Product releases, company milestones, technical innovations, cultural events, version updates

• 8 Date Formats - Full dates, month-year, year ranges, decades, parenthetical dates

• Confidence Scoring - Pattern matching with proximity-based confidence (0.0-1.0)

• Entity Association - Automatically link entities to events

# Extract events from a document
result = kb.extract_document_events('doc_id', min_confidence=0.7)
# Returns: event_count, filtered_count, stored_count, events list

Timeline Construction

Build chronological timelines with flexible querying:

• Automatic Timeline Building - Chronologically sorted by date (YYYYMMDD integer sort)

• Category Organization - Group by decade-type combinations (e.g., "1980s-release")

• Importance Levels - 1-5 scale based on confidence

• Date Range Filtering - Query events by year range, type, importance

# Build timeline from events
timeline_result = kb.build_timeline(min_confidence=0.5)

# Query timeline
timeline = kb.get_timeline(start_year=1980, end_year=1989, min_importance=3)

# Get historical context
context = kb.get_historical_context(year=1982, context_years=2)

Interactive Visualizations

Generate interactive HTML visualizations with Plotly and NetworkX:

Timeline Visualizations:

• Interactive Timeline - Horizontal timeline with zoom/pan, color-coded by event type

• Event Network - Spring layout showing event relationships

• Trend Charts - Multi-subplot dashboard (bar chart, stacked area, cumulative line)

Advanced Graph Visualizations:

• 3D Knowledge Graph - Interactive 3D entity-relationship graph with rotation controls

• Hierarchical Bundling - Circular layout with curved edges bundled through center

• Sankey Diagrams - Topic flow over time (decade or year grouping)

# Generate visualizations
kb.visualize_timeline(start_year=1980, end_year=1990, output_path="timeline.html")
kb.visualize_knowledge_graph_3d(max_entities=50, output_path="graph_3d.html")
kb.visualize_hierarchical_bundling(max_entities=30, output_path="bundling.html")
kb.visualize_topic_flow_sankey(time_period='decade', output_path="flow.html")

MCP Tools for Timeline

4 timeline-specific MCP tools:

• extract_document_events - Extract and store events from documents

• get_timeline - Query chronological timeline with filters

• search_events_by_date - Search events by date range and type

• get_historical_context - Get events around a specific year

See PHASE3_TEMPORAL_ANALYSIS.md for complete documentation.

Tools

92 MCP tools organized by category. Key tools listed below.

Search Tools

search_docs - Full-text search

search_docs(query="SID register", max_results=5, tags=["sid"])

semantic_search - Meaning-based search

semantic_search(query="How do sprites work?", max_results=5)

hybrid_search - Combined keyword + semantic

hybrid_search(query="SID chip", semantic_weight=0.7, max_results=10)

answer_question - RAG-based Q&A with citations

answer_question(
  question="How do I program sprites on the VIC-II?",
  max_sources=5,
  search_mode="auto"
)

search_figures - Search text OCR'd out of document figures

search_figures(query="raster interrupt", max_results=10)

Finds content that exists only inside images - memory-map diagrams, register tables, pinout drawings - which plain document search cannot reach. Requires a figure-OCR pass first (see below).

Figure OCR Tools

Ingest-time OCR only runs on PDFs detected as entirely scanned. A normal text PDF gets its text layer indexed while its embedded figures are never read - and in C64 documentation those figures are often where the reference data actually lives. These tools run a background batch pass over PDFs already in the knowledge base, extracting each embedded image and OCRing it into searchable text stored separately from the document's own text.

batch_ocr_figures - Queue the whole knowledge base

batch_ocr_figures(limit=50, reprocess=false)

Returns immediately; work proceeds on the background worker.

ocr_document_figures - Queue one document

ocr_document_figures(doc_id="89d0943d6009")

figure_ocr_status - Coverage and pending-job report

figure_ocr_status()

get_document_figures - List one document's figures with OCR text

get_document_figures(doc_id="89d0943d6009", with_text_only=true)

Requires PyMuPDF, Tesseract, and USE_OCR=1. Note this path does not need Poppler - it rasterizes with PyMuPDF rather than pdf2image, unlike the scanned-page OCR fallback. figure_ocr_status reports exactly what is missing if the feature is unavailable. If Tesseract is installed but not on PATH, point TESSERACT_PATH at the executable or its directory. Tuning: TDZ_FIGURE_MIN_WIDTH, TDZ_FIGURE_MIN_HEIGHT (ignore images too small to be figures) and TDZ_FIGURE_MIN_CHARS (discard OCR noise).

By default only embedded images are read. Typeset manuals usually draw their schematics, memory maps and timing diagrams as vector paths, which embed no image at all - set TDZ_FIGURE_RASTERIZE_PAGES=1 to also render each page's drawing regions (at TDZ_FIGURE_RASTER_DPI, default 200) and OCR those. Only the clustered drawing regions are rendered, never the whole page: a full-page render would OCR the body text a second time on top of what the PDF's own text layer already contributed. Clusters covering more than TDZ_FIGURE_RASTER_MAX_AREA (default 0.9) of the page are treated as borders and skipped. Each row records its source (embedded or vector).

Rasterizing is much more expensive per page, so pair it with TDZ_FIGURE_OCR_WORKERS (default 1): OCR blocks in a tesseract subprocess rather than in Python, so a pool scales roughly with core count.

fuzzy_search - Typo-tolerant search

fuzzy_search(query="VIC2 asembly", similarity_threshold=80)

search_within_results - Progressive refinement

# Broad search, then refine
results = search_docs(query="VIC-II", max_results=50)
refined = search_within_results(results, "sprite collision", max_results=5)

find_similar - Find related documents

find_similar(doc_id="abc123", max_results=5)

Document Management

add_document - Add a file

add_document(
  filepath="C:/docs/c64_ref.pdf",
  title="C64 Programmer's Reference",
  tags=["reference", "memory-map"]
)

add_documents_bulk - Bulk import

add_documents_bulk(
  directory="C:/c64docs",
  pattern="**/*.{pdf,txt}",
  tags=["reference"],
  recursive=true
)

list_docs - List all documents

get_chunk - Get specific chunk

get_chunk(doc_id="abc123", chunk_id=5)

remove_document - Remove a document

remove_documents_bulk - Bulk remove by IDs or tags

remove_documents_bulk(tags=["outdated"])

check_updates - Check for file changes

check_updates(auto_update=false)

URL Scraping

scrape_url - Scrape documentation website

scrape_url(
  url="https://www.c64-wiki.com/wiki/VIC",
  tags=["wiki"],
  depth=2,
  threads=5
)

rescrape_document - Re-scrape for updates

rescrape_document(doc_id="abc123", force=false)

check_url_updates - Check all scraped docs

check_url_updates(auto_rescrape=false, check_structure=true)

AI & Analytics

extract_entities - Extract named entities

extract_entities(doc_id="abc123", confidence_threshold=0.6)

search_entities - Search across entities

search_entities(query="VIC-II", entity_types=["hardware"])

get_entity_analytics - Comprehensive entity statistics

extract_entity_relationships - Extract co-occurrences

extract_entity_relationships(doc_id="abc123", min_strength=0.3)

search_entity_pair - Find docs with entity pair

search_entity_pair(entity1="VIC-II", entity2="sprite")

compare_documents - Side-by-side comparison

compare_documents(doc_id_1="abc", doc_id_2="def", comparison_type="full")

suggest_tags - AI-powered tag suggestions

suggest_tags(doc_id="abc123", confidence_threshold=0.6)

get_tags_by_category - Browse tags by category

translate_query - Parse natural language queries

translate_query(query="find sprites on VIC-II chip")

Export Tools

export_entities - Export to CSV/JSON

export_entities(format="csv", output_path="entities.csv", min_confidence=0.7)

export_relationships - Export relationships

export_relationships(format="json", output_path="rels.json", min_strength=0.5)

System

kb_stats - Knowledge base statistics

health_check - System diagnostics

Data Storage

SQLite database with 22 tables:

• documents - Document metadata

• chunks - Chunked content (1500 words, 200 overlap)

• document_tables - Extracted PDF tables

• document_code_blocks - Detected code blocks

• document_entities - Extracted entities

• entity_relationships - Co-occurrence tracking

• Plus: summaries, extraction_jobs, monitoring_history, etc.

Benefits:

• Lazy loading (metadata at startup, chunks on-demand)

• ACID transactions

• Scalable to 100k+ documents

• FTS5 full-text indexes

Default location: ~/.tdz-c64-knowledge or TDZ_DATA_DIR

Usage Examples

Ask Claude Code:

• "Search the C64 docs for SID voice registers"

• "What does the memory map say about $D400?"

• "Find information about sprite multiplexing"

• "Add C:/docs/mapping_the_c64.pdf with tags memory-map, reference"

• "How do I program raster interrupts on the VIC-II?" (uses RAG)

Suggested Tags

Organize docs with consistent tags:

• reference, memory-map, basic, assembly

• sid, vic-ii, cia, kernal

• hardware, disk, graphics, sound

Troubleshooting

"pypdf not installed" - Run: pip install pypdf rank-bm25

"mcp module not found" - Run: pip install mcp

Server not responding - Use Python from virtual environment, not system Python

PDF extraction issues - Use OCR or add plain text version

BM25 issues - Check logs in TDZ_DATA_DIR/server.log, try USE_BM25=0

Development

Testing

pip install -e ".[dev]"

# Run all tests
pytest test_card_updates.py test_pdf_viewer.py -v

# With coverage
pytest test_card_updates.py -v --cov=server --cov-report=term

Test Coverage:

• test_card_updates.py - Card upsert/supersede semantics and chunk cleanup on remove

• test_pdf_viewer.py - PDF viewer support (requires PyMuPDF)

A larger suite (35 files) lives under archive/tests/ but is not currently wired into CI. CI itself only runs test_card_updates.py; test_pdf_viewer.py is included above for local runs but requires PyMuPDF (pip install -e ".[dev]" covers it).

CI/CD

GitHub Actions workflow tests on Python 3.10/3.11/3.12 across Windows/Linux/macOS with Ruff code quality checks.

Documentation

Core Documentation

• README.md (this file) - Installation, features, tools, usage

• QUICKSTART.md - Fast setup guide (5 minutes)

• ARCHITECTURE.md - Technical deep dive, database schema, algorithms

• CONTEXT.md - Project status, quick stats, version history

• CLAUDE.md - Quick reference for Claude Code integration

• CHANGELOG.md - Complete version history

Feature Documentation

Browse docs/ for detailed guides on specific features:

API & Integration:

• REST API - FastAPI REST server (18 endpoints)

AI-Powered Features:

• Entity Extraction - Extract hardware, memory addresses, instructions

• Anomaly Detection - ML-based URL content monitoring

• Summarization - AI-powered document summarization

Data Sources:

• Web Scraping - Scrape documentation websites

• Web Monitoring - Track URL-sourced content changes

Setup & Deployment:

• Deployment Guide - Production deployment

• Docker Setup - Docker configuration

• Environment Setup - Environment variables

• Poppler Setup - Poppler installation for PDFs

User Interfaces:

• GUI Guide - Streamlit web interface

Development:

• Testing Guide - Test suite and CI/CD

• Examples - Usage examples and performance analysis

• Monitoring Setup - Scheduled monitoring configuration

• Roadmap - Future improvements and features

Version History

v2.23.0 - RAG Question Answering & Advanced Search (Phase 2 Complete)

• RAG-based answer_question with citations

• Fuzzy search with rapidfuzz

• Progressive search refinement

• Smart tagging system

v2.22.0 - Search Improvements (Phase 1 Complete)

• Enhanced entity analytics

• C64-specific regex patterns (5000x faster)

• Performance optimizations

v2.21.0 - Anomaly Detection

• ML-based baseline learning

• 1500x performance improvement

v2.18.0 - REST API & Background Processing

• FastAPI REST server (18 endpoints)

• Background entity extraction

v2.15.0+ - Entity Intelligence

• Entity extraction, relationships, analytics

See CONTEXT.md for complete version history.

License

MIT License - Use freely for your retro computing projects!