# Phase 3: Temporal Analysis & Advanced Visualizations
**Status:** ✅ COMPLETE
**Version:** 2.23.14+
**Completion Date:** 2026-01-03
## Overview
Phase 3 adds comprehensive temporal analysis capabilities to the TDZ C64 Knowledge Base, enabling event detection, timeline construction, and advanced graph visualizations. This phase transforms the knowledge base from a static document repository into a dynamic temporal knowledge graph.
## Features
### 3.1 Database Schema Enhancements
**Tables Added:**
1. **`events`** - Stores detected temporal events
- `event_id` (TEXT, PRIMARY KEY) - Unique event identifier
- `event_type` (TEXT) - Event category (release, milestone, innovation, cultural, update)
- `title` (TEXT) - Event title/summary
- `description` (TEXT) - Detailed event description
- `date_extracted` (TEXT) - Original date text from document
- `date_normalized` (TEXT) - ISO 8601 normalized date
- `year`, `month`, `day` (INTEGER) - Parsed date components
- `confidence` (REAL) - Detection confidence (0.0-1.0)
- `entities` (TEXT) - JSON array of related entities
- `metadata` (TEXT) - JSON object with additional data
- `created_date` (TEXT) - Record creation timestamp
2. **`document_events`** - Maps events to documents
- `mapping_id` (TEXT, PRIMARY KEY)
- `doc_id` (TEXT, FOREIGN KEY) - References documents table
- `event_id` (TEXT, FOREIGN KEY) - References events table
- `context` (TEXT) - Surrounding text context
- `position` (INTEGER) - Character position in document
- `created_date` (TEXT)
3. **`timeline_entries`** - Chronologically sorted timeline
- `entry_id` (TEXT, PRIMARY KEY)
- `event_id` (TEXT, FOREIGN KEY) - References events table
- `display_date` (TEXT) - Formatted date for display
- `sort_order` (INTEGER) - Chronological sort key (YYYYMMDD)
- `category` (TEXT) - Timeline category (e.g., "1980s-release")
- `importance` (INTEGER) - Importance level (1-5)
- `created_date` (TEXT)
**Indexes Added:**
- `idx_events_type` - Event type filtering
- `idx_events_year` - Year-based queries
- `idx_events_date` - Date range searches
- `idx_document_events_doc` - Document-to-events lookup
- `idx_document_events_event` - Event-to-documents lookup
- `idx_timeline_entries_event` - Event references
- `idx_timeline_entries_sort` - Chronological sorting
- `idx_timeline_entries_category` - Category filtering
### 3.2 Date/Time Extraction
**Method:** `extract_dates_from_text(text: str) -> list[dict]`
Extracts temporal references using regex patterns:
**Supported Formats:**
- Full dates: "January 15, 1982", "15 Jan 1982"
- Month-year: "August 1982", "Aug 1982"
- Year only: "1982"
- Year ranges: "1982-1994", "1982 to 1994"
- Decades: "1980s", "the 80s", "early 80s", "mid-1980s", "late 80s"
- Parenthetical: "(1982)", "[1982]"
**Output Structure:**
```python
{
'text': 'August 1982', # Original text
'type': 'month_year', # Format type
'year': 1982, # Extracted year
'month': 8, # Extracted month (or None)
'day': None, # Extracted day (or None)
'start_pos': 45, # Character position
'end_pos': 57 # End position
}
```
**Method:** `normalize_date(date_dict: dict) -> tuple[str, int, int, int]`
Normalizes dates to ISO 8601 format:
- Full date: `(1982-08-15, 1982, 8, 15)`
- Month-year: `(1982-08, 1982, 8, 0)`
- Year only: `(1982, 1982, 0, 0)`
### 3.3 Event Detection
**Method:** `detect_events_in_text(text: str, doc_id: Optional[str] = None) -> list[dict]`
Pattern-based event detection using regex matching:
**Event Types:**
1. **release** - Product releases, launches
- Triggers: "released", "launched", "introduced", "unveiled", "announced", "shipped", "available"
- Confidence: 0.85
2. **milestone** - Company milestones
- Triggers: "founded", "established", "acquired", "merged", "created", "formed", "incorporated"
- Confidence: 0.80
3. **innovation** - Technical innovations
- Triggers: "first", "invented", "developed", "created", "designed", "pioneered", "innovated", "breakthrough"
- Confidence: 0.75
4. **cultural** - Cultural events
- Triggers: "competition", "contest", "demo", "demonstration", "conference", "convention", "meeting", "expo", "exhibition", "show"
- Confidence: 0.70
5. **update** - Version updates
- Triggers: "updated", "upgraded", "revised", "version", "release", "edition"
- Confidence: 0.65
**Confidence Scoring:**
- Base confidence from event type
- +0.1 if date within 50 characters
- -0.2 if no date found
- Extracts entities (capitalized words) from context
**Method:** `extract_document_events(doc_id: str, min_confidence: float = 0.5) -> dict`
Processes entire document to extract and store events:
1. Combines all chunks into full text
2. Detects events using pattern matching
3. Filters by confidence threshold
4. Stores events to database with document mapping
5. Returns summary statistics
### 3.4 Timeline Construction
**Method:** `build_timeline(min_confidence: float = 0.5, categorize: bool = True) -> dict`
Builds chronological timeline from events:
1. Queries events above confidence threshold
2. Creates timeline entries with sort order (YYYYMMDD as integer)
3. Assigns categories (decade-type combinations)
4. Calculates importance (1-5 scale based on confidence)
5. Returns statistics (date range, categories, year distribution)
**Method:** `get_timeline(...) -> list[dict]`
Flexible timeline querying with filters:
- `start_year`, `end_year` - Date range filtering
- `category` - Category filtering
- `min_importance` - Importance threshold (1-5)
- `limit` - Result count limit
Returns enriched entries with event details.
**Method:** `search_events_by_date(...) -> list[dict]`
Search events by date range and type:
- Year range filtering
- Event type filtering
- Confidence threshold
- Returns event details with entities
**Method:** `get_historical_context(year: int, context_years: int = 2) -> dict`
Retrieves events around a target year for temporal context:
- Returns events from target year ± context_years
- Groups by year
- Provides year range and statistics
### 3.5 MCP Tools for Timeline
**4 New MCP Tools Added:**
1. **extract_document_events**
- Extracts temporal events from a document
- Parameters: `doc_id`, `min_confidence` (default 0.5)
- Returns: Event count, filtered count, stored count, event list
- Auto-detects events and stores to database
2. **get_timeline**
- Retrieves chronological timeline with filtering
- Parameters: `start_year`, `end_year`, `category`, `min_importance`, `limit`
- Returns: Timeline entries with event details
- Auto-builds timeline if not exists
- Markdown formatting with star ratings (⭐) for importance
3. **search_events_by_date**
- Searches events within date range
- Parameters: `start_year`, `end_year`, `event_type`, `min_confidence`
- Returns: Filtered event list
- Grouped by year in output
4. **get_historical_context**
- Gets temporal context for a specific year
- Parameters: `year`, `context_years` (default 2)
- Returns: Events from target year ± context_years
- Emoji markers (🎯 for target year, 📅 for context years)
**Total MCP Tools:** 62 (58 previous + 4 timeline)
### 3.6 Timeline Visualizations
**Method:** `visualize_timeline(...) -> str`
Interactive horizontal timeline using Plotly:
- **Parameters:**
- `start_year`, `end_year` - Date range filtering
- `output_path` - HTML output file path
- **Features:**
- Color-coded by event type
- Marker size based on importance (importance × 5)
- Hover tooltips with details
- Zoomable and pannable
- Date range support (handles partial dates)
- **Output:** HTML file with interactive Plotly chart
**Method:** `visualize_event_network(...) -> str`
Network graph of event relationships:
- **Parameters:**
- `start_year`, `end_year` - Date filtering
- `output_path` - HTML output file path
- **Features:**
- NetworkX spring layout algorithm
- Nodes = events (sized by connection count)
- Edges weighted by:
- Shared entities (×2)
- Same year (+1)
- Same type (+0.5)
- Interactive Plotly visualization
- Hover details for nodes and edges
- **Output:** HTML file with network graph
**Method:** `visualize_event_trends(...) -> str`
Multi-chart trend analysis:
- **Parameters:** `output_path` - HTML output file path
- **Features:**
- Three subplots:
1. Events per year (bar chart)
2. Events by type over time (stacked area chart)
3. Cumulative events (line chart)
- Database aggregation by year and type
- Color-coded by event type
- Interactive zoom and pan
- **Output:** HTML file with 3-subplot dashboard
### 3.7 Advanced Graph Visualizations
**Method:** `visualize_knowledge_graph_3d(...) -> str`
3D interactive knowledge graph:
- **Parameters:**
- `max_entities` - Top N entities to include (default 50)
- `min_confidence` - Entity confidence threshold (default 0.7)
- `output_path` - HTML output file path
- **Features:**
- NetworkX 3D spring layout (dim=3)
- Entities as nodes (frequency-based sizing)
- Relationships as edges
- Color-coded by entity type:
- PERSON: #FF6B6B (red)
- ORG: #4ECDC4 (teal)
- PRODUCT: #95E1D3 (mint)
- TECH: #FFA07A (orange)
- LOCATION: #9B59B6 (purple)
- Plotly Scatter3d with rotation controls
- Hover information per node
- **Output:** HTML file with 3D interactive graph
**Method:** `visualize_hierarchical_bundling(...) -> str`
Circular hierarchical edge bundling:
- **Parameters:**
- `max_entities` - Maximum entities (default 30)
- `output_path` - HTML output file path
- **Features:**
- Circular layout with entities grouped by type
- Curved Bezier edges through center (bundling effect)
- Edge opacity and width based on relationship strength
- Node size based on frequency
- Type-based color coding
- Beautiful hierarchical structure
- **Output:** HTML file with circular bundled graph
**Method:** `visualize_topic_flow_sankey(...) -> str`
Sankey diagram for topic flow over time:
- **Parameters:**
- `time_period` - Grouping ('year' or 'decade')
- `output_path` - HTML output file path
- **Features:**
- Tracks entities flowing between time periods
- Top 10 entities per period
- Flow strength = min(count in period1, count in period2)
- Color-coded by time period
- Shows topic evolution and persistence
- Plotly Sankey diagram
- **Output:** HTML file with Sankey flow diagram
- **Requires:** Events with entities across multiple time periods
## Database Statistics
**Current Knowledge Base (as of 2026-01-03):**
- Documents: 215
- Events: 3
- Timeline entries: 3
- Entities: 3,576 (across 10 types)
- product: 384 unique
- hardware: 202 unique
- person: 177 unique
- instruction: 108 unique
- memory_address: 97 unique
- Entity relationships: 128 (avg strength: 0.136)
## Usage Examples
### Event Detection and Timeline
```python
from server import KnowledgeBase
kb = KnowledgeBase()
# Extract events from a document
result = kb.extract_document_events('doc_id_here', min_confidence=0.7)
print(f"Found {result['stored_count']} events")
# Build timeline
timeline_result = kb.build_timeline(min_confidence=0.5)
print(f"Timeline: {timeline_result['timeline_entries']} entries")
print(f"Date range: {timeline_result['date_range']}")
# Query timeline
timeline = kb.get_timeline(start_year=1980, end_year=1989, min_importance=3)
for entry in timeline:
print(f"[{entry['display_date']}] {entry['title']}")
```
### Visualizations
```python
# Create timeline visualization
kb.visualize_timeline(
start_year=1980,
end_year=1990,
output_path="c64_timeline.html"
)
# Create 3D knowledge graph
kb.visualize_knowledge_graph_3d(
max_entities=50,
min_confidence=0.7,
output_path="knowledge_graph_3d.html"
)
# Create hierarchical bundling
kb.visualize_hierarchical_bundling(
max_entities=30,
output_path="hierarchical_bundling.html"
)
# Create topic flow Sankey
kb.visualize_topic_flow_sankey(
time_period='decade',
output_path="topic_flow.html"
)
```
### MCP Tools (via Claude Desktop)
```
Extract events from a document:
> Use extract_document_events tool with doc_id "abc123"
Get timeline for 1980s:
> Use get_timeline tool with start_year 1980, end_year 1989
Search for release events:
> Use search_events_by_date with event_type "release"
Get historical context for 1982:
> Use get_historical_context with year 1982
```
## Testing
**Test Files:**
- `test_phase3_schema.py` - Database schema validation
- `test_date_extraction.py` - Date parsing tests (8 test cases)
- `test_event_detection.py` - Event pattern matching (6 test cases)
- `test_timeline_construction.py` - Timeline building (6 test categories)
- `test_timeline_visualizations.py` - Plotly timeline tests (6 tests)
- `test_advanced_visualizations.py` - 3D/Sankey tests (7 tests)
**Run All Tests:**
```bash
.venv/Scripts/python.exe test_phase3_schema.py
.venv/Scripts/python.exe test_date_extraction.py
.venv/Scripts/python.exe test_event_detection.py
.venv/Scripts/python.exe test_timeline_construction.py
.venv/Scripts/python.exe test_timeline_visualizations.py
.venv/Scripts/python.exe test_advanced_visualizations.py
```
**All Tests:** ✅ PASSING
## Dependencies
**Required:**
- `plotly` >= 6.5.0 - Interactive visualizations
- `networkx` >= 3.6.0 - Graph algorithms
- `numpy` - Numerical operations
**Already Installed:**
All dependencies are included in the base installation.
## Performance
**Event Detection:**
- ~100-500 events/sec depending on text density
- Confidence scoring: O(n) where n = text length
**Timeline Construction:**
- ~1000 events/sec for timeline building
- Database queries: O(log n) with indexes
**Visualizations:**
- 3D graph: ~1-2 seconds for 50 entities
- Hierarchical bundling: ~2-3 seconds for 30 entities
- Sankey: ~1 second for decade grouping
- HTML files: ~4-5 MB per visualization
## Limitations
1. **Event Detection:**
- Pattern-based (not ML-based)
- English language only
- Requires explicit temporal markers
- No entity disambiguation
2. **Sankey Diagrams:**
- Requires events with entities across multiple time periods
- Current data has limited temporal spread (1976-1983)
- Needs more event extraction for richer flow visualization
3. **Visualization File Size:**
- HTML files are large (4-5 MB) due to embedded Plotly.js
- Contains full data in HTML for offline viewing
## Future Enhancements
1. **ML-Based Event Detection:**
- Use transformer models for better accuracy
- Entity linking to knowledge bases
- Temporal relation extraction
2. **Enhanced Timeline Features:**
- Multi-scale timeline (zoom from decades to days)
- Timeline comparison (multiple knowledge bases)
- Event clustering and summarization
3. **Advanced Visualizations:**
- Temporal heatmaps
- Event flow animations
- Graph evolution over time
- VR/AR knowledge graph exploration
4. **Performance Optimization:**
- Incremental timeline updates
- Cached visualization generation
- Progressive rendering for large graphs
## Migration Notes
**Upgrading from v2.23.14 to v2.24.0+:**
Phase 3 tables are created automatically on first run. No migration script needed - the `_create_tables()` method in KnowledgeBase handles all schema creation.
**Existing Databases:**
- Automatically upgraded on initialization
- No data loss
- Indexes created automatically
- Foreign key constraints enforced
## Contributors
- **Phase 3 Implementation:** Claude Sonnet 4.5 (Anthropic)
- **Architecture Design:** Claude Sonnet 4.5
- **Testing & Validation:** Claude Sonnet 4.5
- **Documentation:** Claude Sonnet 4.5
## References
1. NetworkX Documentation: https://networkx.org/
2. Plotly Python Graphing Library: https://plotly.com/python/
3. ISO 8601 Date Format: https://en.wikipedia.org/wiki/ISO_8601
4. Hierarchical Edge Bundling: Holten, D. (2006)
5. Sankey Diagrams: Schmidt, M. (2008)
---
**Phase 3 Status:** ✅ COMPLETE
**Last Updated:** 2026-01-03
**Version:** 2.23.14+
