# TDZ C64 Knowledge

[![Version](https://img.shields.io/badge/version-2.24.0-brightgreen.svg)](https://github.com/MichaelTroelsen/tdz-c64-knowledge)
[![CI/CD Pipeline](https://github.com/MichaelTroelsen/tdz-c64-knowledge/actions/workflows/ci.yml/badge.svg)](https://github.com/MichaelTroelsen/tdz-c64-knowledge/actions/workflows/ci.yml)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)

MCP server for managing and searching Commodore 64 documentation. Ingest PDFs, text, Markdown, HTML, Excel, and web pages into a searchable knowledge base accessible via Claude Code or other MCP clients.

## 🚀 Quick Start

```cmd
# 1. Install
python -m venv .venv
.venv\Scripts\activate
pip install -e .

# 2. Configure Claude Code
claude mcp add tdz-c64-knowledge -- .venv\Scripts\python.exe server.py

# 3. Add documents
.venv\Scripts\python.exe cli.py add-folder "C:\c64docs" --tags reference --recursive

# 4. Search via Claude Code
# Ask: "Search the C64 docs for VIC-II sprite registers"
```

See [QUICKSTART.md](docs/QUICKSTART.md) for detailed setup.

## Features

### Search & Retrieval
- **FTS5 full-text search** - 480x faster queries (50ms vs 24s)
- **Semantic search** - Find by meaning, not keywords (e.g., "movable objects" → "sprites")
- **RAG question answering** - Answer questions by synthesizing docs with citations
- **Fuzzy search** - Typo tolerance ("VIC2" → "VIC-II", "asembly" → "assembly")
- **Progressive refinement** - Search within results to narrow down
- **Hybrid search** - Combines keyword + semantic with configurable weighting
- **Similarity search** - Discover related documentation automatically
- **Query preprocessing** - NLTK stemming and stopword removal
- **Smart tagging** - AI-powered tag suggestions by category
- **Table/code search** - Search extracted tables and code blocks

### Document Management
- **Multi-format** - PDF, TXT, MD, HTML, Excel, web scraping, and (optional
  `markitdown` extra) DOCX, PPTX, EPUB, CSV, JSON, XML - install with
  `pip install -e ".[markitdown]"`; disable at runtime with `TDZ_MARKITDOWN=0`
  even when installed. PDF and ZIP (HVSC SID archives) never route through
  markitdown - see `docs/ARCHITECTURE.md`'s "Extending File Type Support"
  for why.
- **Duplicate detection** - Content-based deduplication
- **Chunked retrieval** - Get specific sections without loading entire docs
- **Metadata extraction** - Author, subject, page numbers
- **Persistent index** - Documents stay indexed between sessions

### AI-Powered Features
- **Entity extraction** - Extract hardware, memory addresses, instructions, concepts (5000x faster with C64 regex patterns)
- **Relationship mapping** - Co-occurrence analysis with distance-based strength scoring
- **Document comparison** - Side-by-side analysis with similarity scores
- **Natural language query translation** - Parse queries into structured search parameters
- **Anomaly detection** - ML-based baseline learning for URL-sourced content (3400+ docs/second)
- **Temporal analysis** - Event detection, timeline construction, historical context (5 event types, 8 date formats)
- **Advanced visualizations** - 3D knowledge graphs, hierarchical bundling, Sankey flow diagrams

### Wiki Export (NEW in v2.23.15)
- **Static HTML wiki** - Export entire knowledge base to browsable website
- **Document similarity map** - 2D visualization using UMAP/t-SNE dimensionality reduction
- **Interactive timeline** - Horizontal scrollable timeline with zoom levels and event filters
- **Knowledge graph** - D3.js force-directed graph (178 entities, 20 relationships)
- **Enhanced UI** - Explanation boxes, prominent ASK AI button, file type detection
- **Clickable clusters** - Browse k-means clusters with linked documents
- **No server required** - Pure client-side JavaScript, works offline
- **Full-text search** - Fuse.js powered search across all content
- See [WIKI_EXPORT_GUIDE.md](docs/WIKI_EXPORT_GUIDE.md) for usage

### REST API (Optional)
- **18 endpoints** - Full CRUD, search, analytics, export
- **OpenAPI/Swagger docs** - Interactive API at `/api/docs`
- **API authentication** - Secure via X-API-Key header
- See [docs/REST_API.md](docs/REST_API.md) for details

### Performance
- **Scalability** - Tested to 5,000+ documents
- **Concurrent throughput** - 5,712 queries/sec (10 workers)
- **Lazy loading** - 100k+ document support
- **Search caching** - 50-100x speedup for repeated queries

## Installation (Windows)

### Prerequisites
- **Python 3.10+** - https://python.org (check "Add Python to PATH")
- **uv** (recommended) or pip: `pip install uv`

### Setup

```cmd
cd C:\Users\YourName\mcp-servers\tdz-c64-knowledge

# Using uv (faster)
uv venv
.venv\Scripts\activate
uv pip install mcp pypdf rank-bm25 nltk

# Or using pip
python -m venv .venv
.venv\Scripts\activate
pip install mcp pypdf rank-bm25 nltk

# Test
python server.py  # Press Ctrl+C to stop
```

## Configuration

### Claude Code

```cmd
claude mcp add tdz-c64-knowledge -- C:\path\.venv\Scripts\python.exe C:\path\server.py
```

Or add to `.claude/settings.json`:
```json
{
  "mcpServers": {
    "tdz-c64-knowledge": {
      "command": "C:\\path\\.venv\\Scripts\\python.exe",
      "args": ["C:\\path\\server.py"],
      "env": {
        "TDZ_DATA_DIR": "C:\\c64-knowledge-data"
      }
    }
  }
}
```

### Claude Desktop

Add to `%APPDATA%\Claude\claude_desktop_config.json`:
```json
{
  "mcpServers": {
    "tdz-c64-knowledge": {
      "command": "C:\\path\\.venv\\Scripts\\python.exe",
      "args": ["C:\\path\\server.py"],
      "env": {
        "TDZ_DATA_DIR": "C:\\c64-knowledge-data"
      }
    }
  }
}
```

## Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `TDZ_DATA_DIR` | Database directory | `~/.tdz-c64-knowledge` |
| `TDZ_MCP_TRANSPORT` | `stdio` (client spawns the server) or `http` (one hosted instance serves several machines) | `stdio` |
| `TDZ_MCP_HOST` | Bind address for the HTTP transport | `127.0.0.1` |
| `TDZ_MCP_PORT` | Port for the HTTP transport | `8765` |
| `TDZ_API_KEYS` | Comma-separated API keys; shared with the REST API. Required to bind the HTTP transport off loopback | *(unset)* |
| `TDZ_MCP_ALLOW_INSECURE` | **Disables a refusal.** `1` permits the MCP HTTP transport to bind off loopback with no `TDZ_API_KEYS` configured, i.e. no authentication at all | `0` |
| `TDZ_MCP_ALLOWED_HOSTS` | `Host` values accepted by DNS-rebinding protection; `*` disables it | *(unset)* |
| `TDZ_MARKITDOWN` | Set to `0` to disable the markitdown fallback extraction (.docx, .pptx, .epub, .csv, .json, .xml) even when the `markitdown` extra is installed | `1` |
| `TDZ_ALLOW_CWD` | **Widens the path allowlist.** `1` adds the process's current working directory to the paths `add_document`/`scrape_url` may use, on top of `ALLOWED_DOCS_DIRS` | `0` |
| `TDZ_REST_ALLOW_INSECURE` | **Disables a refusal.** `1` lets the REST API (`rest_server.py`) bind with no `TDZ_API_KEYS` configured, i.e. no authentication at all | `0` |
| `TDZ_RESPECT_ROBOTS` | Honour `robots.txt` when scraping. `0` overrides it (self-hosted mirrors only) | `1` |
| `TDZ_USER_AGENT` | User-Agent string sent on outbound scrapes | `tdz-c64-knowledge/<version> (+https://github.com/Thordanielz/tdz-c64-knowledge)` |
| `TDZ_DEEPSID_BASE_URL` | Base URL for DeepSID ingestion (`add_deepsid_document`/`add_deepsid_folder`) | `https://deepsid.chordian.net` |
| `TDZ_DB_BUSY_TIMEOUT_MS` | SQLite busy-timeout, in ms, for WAL-mode concurrent server processes | `30000` |
| `TDZ_TOOL_TIMEOUT_S` | Max seconds an ordinary MCP tool call may run before it is aborted; `0` disables the bound (unbounded, not zero seconds) | `600` |
| `TDZ_LONG_TOOL_TIMEOUT_S` | Same bound, applied instead of `TDZ_TOOL_TIMEOUT_S` to tools flagged as long-running; `0` disables it | `0` (unbounded) |
| `TDZ_TOOL_LOCK_WAIT_S` | Max seconds a tool call waits to acquire the shared dispatch lock before giving up; `0` disables the bound | `600` |
| `TDZ_PROGRESS_INTERVAL_S` | Seconds between progress notifications on a long-running tool call | `15` |
| `TDZ_EMBEDDINGS_LOAD_TIMEOUT_S` | Max seconds to wait for the sentence-transformers model to load on first semantic-search use before continuing in the background | `20` |
| `TDZ_NLTK_IMPORT_TIMEOUT_S` | Max seconds to wait for the lazy NLTK import/download before giving up | `5` |
| `TDZ_STDERR_LOG_LEVEL` | Console log level; `INFO` restores pre-quiet-mode verbosity | `WARNING` |
| `TDZ_RECOVER_EXTRACTION_JOBS` | Recover interrupted extraction jobs found stale at startup. `0` disables recovery | `1` |
| `TDZ_EXTRACTION_STALE_MINUTES` | Minutes since last update before an in-progress extraction job is considered stale and eligible for recovery | `60` |
| `TDZ_EXTRACTION_RECOVER_LIMIT` | Max number of stale extraction jobs recovered per startup pass | `100` |
| `USE_FTS5` | Enable FTS5 search (recommended) | `0` |
| `USE_SEMANTIC_SEARCH` | Enable semantic search | `0` |
| `SEMANTIC_MODEL` | Sentence-transformers model | `all-MiniLM-L6-v2` |
| `USE_BM25` | Enable BM25 fallback | `1` |
| `USE_QUERY_PREPROCESSING` | Enable NLTK preprocessing | `1` |
| `USE_FUZZY_SEARCH` | Enable fuzzy search | `1` |
| `FUZZY_THRESHOLD` | Fuzzy similarity (0-100) | `80` |
| `USE_OCR` | Enable OCR for scanned PDFs | `1` |
| `TESSERACT_PATH` | Tesseract exe or its directory, when not on PATH | *(unset)* |
| `SEARCH_CACHE_SIZE` | Max cached results | `100` |
| `SEARCH_CACHE_TTL` | Cache TTL (seconds) | `300` |
| `ALLOWED_DOCS_DIRS` | Document directory whitelist | None |
| `EMBEDDING_WINDOW_WORDS` | Words per encoder window when building embeddings (0 = whole chunk, no windowing) | `200` |
| `EMBEDDING_WINDOW_OVERLAP` | Word overlap between consecutive embedding windows | `40` |
| `RAG_CONTEXT_TOKENS` | Token budget for `answer_question`'s LLM context | `8000` |
| `HYBRID_FUSION` | `rrf` (rank fusion) or `weighted` (legacy normalized-score blend) | `rrf` |
| `RRF_K` | Rank-fusion damping constant for `hybrid_search` | `60` |
| `USE_RERANKER` | Rerank hybrid/semantic results with a cross-encoder | `0` |
| `RERANK_MODEL` | Cross-encoder model for reranking | `cross-encoder/ms-marco-MiniLM-L-6-v2` |
| `RERANK_CANDIDATES` | First-stage candidates fed to the reranker | `30` |
| `USE_ANSWER_VERIFICATION` | Check `answer_question`'s cited claims against their source before trusting them | `1` |
| `VERIFY_PASSAGE_CHARS` | Max characters per cited source shown to the grounding check | `800` |
| `USE_NLI_VERIFICATION` | Use a local NLI cross-encoder for claim verification instead of a second LLM call | `0` |
| `NLI_MODEL` | Cross-encoder model for NLI-based claim verification | `cross-encoder/nli-deberta-v3-base` |
| `NLI_MAX_LENGTH` | Max input tokens for the NLI model | `512` |

## Search Features

### FTS5 Full-Text Search (Recommended)

Enable with `USE_FTS5=1` for maximum performance:
- **480x faster** than BM25
- Native SQLite BM25 ranking
- Porter stemming tokenizer

### Semantic Search

Enable with `USE_SEMANTIC_SEARCH=1`:
- Meaning-based search (e.g., "movable objects" finds "sprites")
- FAISS vector similarity with sentence-transformers
- ~7-16ms per query after embeddings built
- Pre-build embeddings: `pip install sentence-transformers faiss-cpu`

### Phrase Search

Use double quotes for exact phrases:
```
search_docs(query='"VIC-II chip" registers')
```

### Fuzzy Search

Handles typos automatically with `USE_FUZZY_SEARCH=1`:
- "VIC-I" → "VIC-II" (83% similarity)
- "grafics" → "graphics" (88% similarity)
- Configurable threshold (default: 80%)

### OCR for Scanned PDFs

Automatic with `USE_OCR=1`:
- Detects scanned PDFs (< 100 chars extracted)
- Uses Tesseract OCR
- Install: `pip install pytesseract pdf2image Pillow` + Tesseract binary
- ~1-2 seconds per page

## Temporal Analysis & Visualizations

Extract events, construct timelines, and visualize knowledge graphs.

### Event Detection

Automatically detect significant events in documents:
- **5 Event Types** - Product releases, company milestones, technical innovations, cultural events, version updates
- **8 Date Formats** - Full dates, month-year, year ranges, decades, parenthetical dates
- **Confidence Scoring** - Pattern matching with proximity-based confidence (0.0-1.0)
- **Entity Association** - Automatically link entities to events

```python
# Extract events from a document
result = kb.extract_document_events('doc_id', min_confidence=0.7)
# Returns: event_count, filtered_count, stored_count, events list
```

### Timeline Construction

Build chronological timelines with flexible querying:
- **Automatic Timeline Building** - Chronologically sorted by date (YYYYMMDD integer sort)
- **Category Organization** - Group by decade-type combinations (e.g., "1980s-release")
- **Importance Levels** - 1-5 scale based on confidence
- **Date Range Filtering** - Query events by year range, type, importance

```python
# Build timeline from events
timeline_result = kb.build_timeline(min_confidence=0.5)

# Query timeline
timeline = kb.get_timeline(start_year=1980, end_year=1989, min_importance=3)

# Get historical context
context = kb.get_historical_context(year=1982, context_years=2)
```

### Interactive Visualizations

Generate interactive HTML visualizations with Plotly and NetworkX:

**Timeline Visualizations:**
- **Interactive Timeline** - Horizontal timeline with zoom/pan, color-coded by event type
- **Event Network** - Spring layout showing event relationships
- **Trend Charts** - Multi-subplot dashboard (bar chart, stacked area, cumulative line)

**Advanced Graph Visualizations:**
- **3D Knowledge Graph** - Interactive 3D entity-relationship graph with rotation controls
- **Hierarchical Bundling** - Circular layout with curved edges bundled through center
- **Sankey Diagrams** - Topic flow over time (decade or year grouping)

```python
# Generate visualizations
kb.visualize_timeline(start_year=1980, end_year=1990, output_path="timeline.html")
kb.visualize_knowledge_graph_3d(max_entities=50, output_path="graph_3d.html")
kb.visualize_hierarchical_bundling(max_entities=30, output_path="bundling.html")
kb.visualize_topic_flow_sankey(time_period='decade', output_path="flow.html")
```

### MCP Tools for Timeline

4 timeline-specific MCP tools:
- `extract_document_events` - Extract and store events from documents
- `get_timeline` - Query chronological timeline with filters
- `search_events_by_date` - Search events by date range and type
- `get_historical_context` - Get events around a specific year

See [PHASE3_TEMPORAL_ANALYSIS.md](docs/PHASE3_TEMPORAL_ANALYSIS.md) for complete documentation.

## Tools

95 MCP tools organized by category. Key tools listed below.

### Search Tools

**search_docs** - Full-text search
```
search_docs(query="SID register", max_results=5, tags=["sid"])
```

**semantic_search** - Meaning-based search
```
semantic_search(query="How do sprites work?", max_results=5)
```

**hybrid_search** - Combined keyword + semantic
```
hybrid_search(query="SID chip", semantic_weight=0.7, max_results=10)
```

**answer_question** - RAG-based Q&A with citations
```
answer_question(
  question="How do I program sprites on the VIC-II?",
  max_sources=5,
  search_mode="auto"
)
```

**search_figures** - Search text OCR'd out of document figures
```
search_figures(query="raster interrupt", max_results=10)
```
Finds content that exists only inside images - memory-map diagrams, register
tables, pinout drawings - which plain document search cannot reach. Requires a
figure-OCR pass first (see below).

### Figure OCR Tools

Ingest-time OCR only runs on PDFs detected as *entirely* scanned. A normal text
PDF gets its text layer indexed while its embedded figures are never read - and
in C64 documentation those figures are often where the reference data actually
lives. These tools run a background batch pass over PDFs already in the
knowledge base, extracting each embedded image and OCRing it into searchable
text stored separately from the document's own text.

**batch_ocr_figures** - Queue the whole knowledge base
```
batch_ocr_figures(limit=50, reprocess=false)
```
Returns immediately; work proceeds on the background worker.

**ocr_document_figures** - Queue one document
```
ocr_document_figures(doc_id="89d0943d6009")
```

**figure_ocr_status** - Coverage and pending-job report
```
figure_ocr_status()
```

**get_document_figures** - List one document's figures with OCR text
```
get_document_figures(doc_id="89d0943d6009", with_text_only=true)
```

Requires `PyMuPDF`, Tesseract, and `USE_OCR=1`. Note this path does **not**
need Poppler - it rasterizes with PyMuPDF rather than pdf2image, unlike the
scanned-page OCR fallback. `figure_ocr_status` reports exactly what is missing
if the feature is unavailable. If Tesseract is installed but not on `PATH`,
point `TESSERACT_PATH` at the executable or its directory. Tuning:
`TDZ_FIGURE_MIN_WIDTH`, `TDZ_FIGURE_MIN_HEIGHT` (ignore images too small to be
figures) and `TDZ_FIGURE_MIN_CHARS` (discard OCR noise).

By default only *embedded* images are read. Typeset manuals usually draw their
schematics, memory maps and timing diagrams as vector paths, which embed no
image at all - set `TDZ_FIGURE_RASTERIZE_PAGES=1` to also render each page's
drawing regions (at `TDZ_FIGURE_RASTER_DPI`, default 200) and OCR those. Only
the clustered drawing regions are rendered, never the whole page: a full-page
render would OCR the body text a second time on top of what the PDF's own text
layer already contributed. Clusters covering more than
`TDZ_FIGURE_RASTER_MAX_AREA` (default 0.9) of the page are treated as borders
and skipped. Each row records its `source` (`embedded` or `vector`).

Rasterizing is much more expensive per page, so pair it with
`TDZ_FIGURE_OCR_WORKERS` (default 1): OCR blocks in a `tesseract` subprocess
rather than in Python, so a pool scales roughly with core count.

**fuzzy_search** - Typo-tolerant search
```
fuzzy_search(query="VIC2 asembly", similarity_threshold=80)
```

**search_within_results** - Progressive refinement
```
# Broad search, then refine
results = search_docs(query="VIC-II", max_results=50)
refined = search_within_results(results, "sprite collision", max_results=5)
```

**find_similar** - Find related documents
```
find_similar(doc_id="abc123", max_results=5)
```

### Document Management

**add_document** - Add a file
```
add_document(
  filepath="C:/docs/c64_ref.pdf",
  title="C64 Programmer's Reference",
  tags=["reference", "memory-map"]
)
```

**add_documents_bulk** - Bulk import
```
add_documents_bulk(
  directory="C:/c64docs",
  pattern="**/*.{pdf,txt}",
  tags=["reference"],
  recursive=true
)
```

**list_docs** - List all documents

**get_chunk** - Get specific chunk
```
get_chunk(doc_id="abc123", chunk_id=5)
```

**remove_document** - Remove a document

**remove_documents_bulk** - Bulk remove by IDs or tags
```
remove_documents_bulk(tags=["outdated"])
```

**check_updates** - Check for file changes
```
check_updates(auto_update=false)
```

### URL Scraping

**scrape_url** - Scrape documentation website
```
scrape_url(
  url="https://www.c64-wiki.com/wiki/VIC",
  tags=["wiki"],
  depth=2,
  threads=5
)
```

**rescrape_document** - Re-scrape for updates
```
rescrape_document(doc_id="abc123", force=false)
```

**check_url_updates** - Check all scraped docs
```
check_url_updates(auto_rescrape=false, check_structure=true)
```

### AI & Analytics

**extract_entities** - Extract named entities
```
extract_entities(doc_id="abc123", confidence_threshold=0.6)
```

**search_entities** - Search across entities
```
search_entities(query="VIC-II", entity_types=["hardware"])
```

**get_entity_analytics** - Comprehensive entity statistics

**extract_entity_relationships** - Extract co-occurrences
```
extract_entity_relationships(doc_id="abc123", min_strength=0.3)
```

**search_entity_pair** - Find docs with entity pair
```
search_entity_pair(entity1="VIC-II", entity2="sprite")
```

**compare_documents** - Side-by-side comparison
```
compare_documents(doc_id_1="abc", doc_id_2="def", comparison_type="full")
```

**suggest_tags** - AI-powered tag suggestions
```
suggest_tags(doc_id="abc123", confidence_threshold=0.6)
```

**get_tags_by_category** - Browse tags by category

**translate_query** - Parse natural language queries
```
translate_query(query="find sprites on VIC-II chip")
```

### Export Tools

**export_entities** - Export to CSV/JSON
```
export_entities(format="csv", output_path="entities.csv", min_confidence=0.7)
```

**export_relationships** - Export relationships
```
export_relationships(format="json", output_path="rels.json", min_strength=0.5)
```

### System

**kb_stats** - Knowledge base statistics

**health_check** - System diagnostics

## Data Storage

SQLite database with 22 tables:
- **documents** - Document metadata
- **chunks** - Chunked content (1500 words, 200 overlap)
- **document_tables** - Extracted PDF tables
- **document_code_blocks** - Detected code blocks
- **document_entities** - Extracted entities
- **entity_relationships** - Co-occurrence tracking
- Plus: summaries, extraction_jobs, monitoring_history, etc.

**Benefits:**
- Lazy loading (metadata at startup, chunks on-demand)
- ACID transactions
- Scalable to 100k+ documents
- FTS5 full-text indexes

Default location: `~/.tdz-c64-knowledge` or `TDZ_DATA_DIR`

## Usage Examples

Ask Claude Code:
- "Search the C64 docs for SID voice registers"
- "What does the memory map say about $D400?"
- "Find information about sprite multiplexing"
- "Add C:/docs/mapping_the_c64.pdf with tags memory-map, reference"
- "How do I program raster interrupts on the VIC-II?" (uses RAG)

## Suggested Tags

Organize docs with consistent tags:
- `reference`, `memory-map`, `basic`, `assembly`
- `sid`, `vic-ii`, `cia`, `kernal`
- `hardware`, `disk`, `graphics`, `sound`

## Troubleshooting

**"pypdf not installed"** - Run: `pip install pypdf rank-bm25`

**"mcp module not found"** - Run: `pip install mcp`

**Server not responding** - Use Python from virtual environment, not system Python

**PDF extraction issues** - Use OCR or add plain text version

**BM25 issues** - Check logs in `TDZ_DATA_DIR/server.log`, try `USE_BM25=0`

## Development

### Testing

```cmd
pip install -e ".[dev]"

# Run all tests
pytest test_card_updates.py test_pdf_viewer.py -v

# With coverage
pytest test_card_updates.py -v --cov=server --cov-report=term
```

**Test Coverage:**
- `test_card_updates.py` - Card upsert/supersede semantics and chunk cleanup on remove
- `test_pdf_viewer.py` - PDF viewer support (requires PyMuPDF)

A larger suite (35 files) lives under `archive/tests/` but is not currently wired into CI. CI
itself only runs `test_card_updates.py`; `test_pdf_viewer.py` is included above for local runs
but requires PyMuPDF (`pip install -e ".[dev]"` covers it).

### CI/CD

GitHub Actions workflow tests on Python 3.10/3.11/3.12 across Windows/Linux/macOS with Ruff code quality checks.

## Documentation

### Core Documentation

- **[README.md](README.md)** (this file) - Installation, features, tools, usage
- **[QUICKSTART.md](docs/QUICKSTART.md)** - Fast setup guide (5 minutes)
- **[ARCHITECTURE.md](docs/ARCHITECTURE.md)** - Technical deep dive, database schema, algorithms
- **[CONTEXT.md](CONTEXT.md)** - Project status, quick stats, version history
- **[CLAUDE.md](CLAUDE.md)** - Quick reference for Claude Code integration
- **[CHANGELOG.md](docs/CHANGELOG.md)** - Complete version history

### Feature Documentation

Browse [docs/](docs/) for detailed guides on specific features:

**API & Integration:**
- [REST API](docs/REST_API.md) - FastAPI REST server (18 endpoints)

**AI-Powered Features:**
- [Entity Extraction](docs/ENTITY_EXTRACTION.md) - Extract hardware, memory addresses, instructions
- [Anomaly Detection](docs/ANOMALY_DETECTION.md) - ML-based URL content monitoring
- [Summarization](docs/SUMMARIZATION.md) - AI-powered document summarization

**Data Sources:**
- [Web Scraping](docs/WEB_SCRAPING.md) - Scrape documentation websites
- [Web Monitoring](docs/WEB_MONITORING.md) - Track URL-sourced content changes

**Setup & Deployment:**
- [Deployment Guide](docs/DEPLOYMENT.md) - Production deployment
- [Docker Setup](docs/DOCKER.md) - Docker configuration
- [Environment Setup](docs/ENVIRONMENT_SETUP.md) - Environment variables
- [Poppler Setup](docs/POPPLER_SETUP.md) - Poppler installation for PDFs

**User Interfaces:**
- [GUI Guide](docs/GUI.md) - Streamlit web interface

**Development:**
- [Testing Guide](docs/TESTING.md) - Test suite and CI/CD
- [Examples](docs/EXAMPLES.md) - Usage examples and performance analysis
- [Monitoring Setup](docs/MONITORING.md) - Scheduled monitoring configuration
- [Roadmap](docs/ROADMAP.md) - Future improvements and features

## Version History

**v2.23.0** - RAG Question Answering & Advanced Search (Phase 2 Complete)
- RAG-based answer_question with citations
- Fuzzy search with rapidfuzz
- Progressive search refinement
- Smart tagging system

**v2.22.0** - Search Improvements (Phase 1 Complete)
- Enhanced entity analytics
- C64-specific regex patterns (5000x faster)
- Performance optimizations

**v2.21.0** - Anomaly Detection
- ML-based baseline learning
- 1500x performance improvement

**v2.18.0** - REST API & Background Processing
- FastAPI REST server (18 endpoints)
- Background entity extraction

**v2.15.0+** - Entity Intelligence
- Entity extraction, relationships, analytics

See CONTEXT.md for complete version history.

## License

MIT License - Use freely for your retro computing projects!