Fabric Data Engineering MCP Server

# Changelog ## [2.3.0] - 2026-02-04 ### 🎉 100% Fabric Portal Parity - Real-Time Hub & Data Wrangler This release completes feature parity with the Microsoft Fabric Web Portal by adding Real-Time Hub (Eventstreams) and Data Wrangler with interactive visual designers. ### Added #### 17. Real-Time Hub (NEW) **Visual Eventstream Designer (Webview Panel)** - Canvas layout showing Sources → Transformations → Destinations - Real-time metrics dashboard (events/sec, latency, backlog) - Add/remove sources, destinations, and transformations - Start/Stop eventstream controls - Per-component status indicators **Data Sources (12 types)** - Azure: Event Hub, IoT Hub, Service Bus - Streaming: Apache Kafka, Amazon Kinesis, Google Pub/Sub - Custom: Custom App (SDK), Webhook, MQTT - Database: Change Data Capture (CDC) - Sample Data for testing **Destinations (8 types)** - Fabric: KQL Database, Lakehouse, Reflex (Data Activator) - Azure: Event Hub, Data Lake Gen2 - Custom: Custom Endpoint, Derived Stream **Transformations** - Filter: Filter events by condition - Manage Fields: Add, remove, rename fields - Aggregate: Calculate aggregations over time windows - Group By: Group events by key - Union: Combine multiple streams - Expand: Expand arrays into rows - Join: Join with reference data **Metrics Tracked** - Input/Output events per second - Input/Output bytes per second - Processing latency (ms) - Backlog event count - Errors in last 24 hours **Commands (11)** - `fabric.realtimehub.createEventstream` - Create new eventstream - `fabric.realtimehub.openDesigner` - Open visual designer - `fabric.realtimehub.startEventstream` - Start streaming - `fabric.realtimehub.stopEventstream` - Stop streaming - `fabric.realtimehub.deleteEventstream` - Delete eventstream - `fabric.realtimehub.addSource` - Add data source - `fabric.realtimehub.addDestination` - Add destination - `fabric.realtimehub.addTransform` - Add transformation - `fabric.realtimehub.viewMetrics` - View metrics dashboard - `fabric.realtimehub.viewStreamData` - Live data preview - `fabric.realtimehub.refresh` - Refresh tree view #### 18. Data Wrangler (NEW) **Visual Data Wrangler (Webview Panel)** - Interactive data grid with scrollable preview - Column headers with data type and quality indicators - Step history sidebar with undo capability - Column statistics panel (unique, nulls, distribution) - Toolbar with transform and export options **Transformations (35+ types)** *Selection Operations* - SelectColumns, DropColumns, ReorderColumns *Filtering Operations* - FilterRows, DropDuplicates, DropNulls, SampleRows *Type/Value Transformations* - RenameColumn, ChangeType, FillMissing, Replace - SplitColumn, MergeColumns, Extract, Normalize *Text Operations* - Trim, Uppercase, Lowercase, PadString, RegexReplace *DateTime Operations* - ExtractDate, ExtractTime, DateDiff, FormatDate *Math Operations* - Round, Ceil, Floor, Abs, Log, Exp, Normalize *Aggregation* - GroupBy, Pivot, Unpivot, Aggregate *Custom* - CustomFormula, PythonCode **Data Quality Features** - Automatic issue detection - Issue types: Missing, Outlier, Invalid, Inconsistent, Duplicate - Severity levels: Low, Medium, High - Suggested fixes for each issue - Quality score per column (0-100%) **Code Export Formats** - Python (Pandas) - Standard data science - Python (PySpark) - Spark notebooks - Python (Polars) - High-performance alternative **Commands (10)** - `fabric.datawrangler.newSession` - Create wrangler session - `fabric.datawrangler.openWrangler` - Open visual wrangler - `fabric.datawrangler.transform` - Apply transformation - `fabric.datawrangler.undoStep` - Undo transformation step - `fabric.datawrangler.exportCode` - Export as Python code - `fabric.datawrangler.viewProfile` - View column profile - `fabric.datawrangler.fixIssue` - Fix data quality issue - `fabric.datawrangler.saveSession` - Save session - `fabric.datawrangler.deleteSession` - Delete session - `fabric.datawrangler.refresh` - Refresh tree view ### Technical Improvements - Added 18 tree view providers (2 new) - Added 3 visual webview designers (2 new) - Added 21 new commands - 50 passing tests (8 new) - 20 sidebar views total - ~150 total commands - **100% Fabric Portal feature parity achieved** --- ## [2.2.0] - 2026-02-04 ### 🔀 Dataflows Gen2 with Visual Designer Release This release adds full Dataflows Gen2 support with a visual Power Query-style designer, bringing the extension to 99% feature parity with Microsoft Fabric Portal. ### Added #### 16. Dataflows Gen2 (NEW) **Visual Designer (Webview Panel)** - **Canvas View** - Drag-and-drop query nodes with visual connections - **Query List Sidebar** - Browse and select queries - **Transform Palette** - 35+ transformation types via dropdown menus - **Data Preview** - Preview data at any transformation step - **M Code View** - View and copy generated Power Query M code - **Properties Panel** - Edit query properties, sources, and destinations **Data Sources (22 types)** - Fabric: Lakehouse, Warehouse - Azure: SQL Database, Synapse, Data Lake Gen2, Blob Storage - Files: CSV/Text, Excel, JSON, Parquet - Online: Web, OData, SharePoint List, SharePoint Folder - Databases: SQL Server, PostgreSQL, MySQL, Oracle, Snowflake, Databricks, BigQuery, Dataverse **Transformations (35+ types)** - Column Operations: Select, Remove, Rename, Change Type, Duplicate - Row Operations: Filter, Sort, Remove Duplicates, Remove Blanks, Remove Errors - Combine: Group By, Merge Queries (all join types), Append Queries - Reshape: Pivot, Unpivot, Transpose, Reverse Rows - Add Columns: Custom Column, Conditional Column, Index Column - Text: Split Column, Merge Columns, Replace Values, Trim, Clean, Extract - Table: Promote Headers, Demote Headers, Fill Down, Fill Up **Output Destinations** - Lakehouse Tables (Replace, Append, Merge) - Warehouse Tables - KQL Database - Azure SQL - Dataverse **Tree View Features** - Browse dataflows with state indicators (Active, Paused, Error, Refreshing) - View queries by type (Source, Transform, Output) - Browse refresh history with status and duration - View and edit refresh schedules **Commands (11)** - `fabric.dataflow.create` - Create new dataflow - `fabric.dataflow.openDesigner` - Open visual designer - `fabric.dataflow.run` - Run dataflow refresh - `fabric.dataflow.cancelRefresh` - Cancel running refresh - `fabric.dataflow.editSchedule` - Edit refresh schedule - `fabric.dataflow.delete` - Delete dataflow - `fabric.dataflow.duplicate` - Duplicate dataflow - `fabric.dataflow.viewRefreshDetails` - View refresh details - `fabric.dataflow.exportJson` - Export definition to JSON - `fabric.dataflow.importJson` - Import from JSON - `fabric.dataflow.refresh` - Refresh tree view ### Technical Improvements - Added 16 tree view providers (1 new) - Added visual webview designer panel - Added DataflowService with full API operations - Added M code generation for all transformations - Added 11 new commands - 42 passing tests (6 new) - 18 sidebar views total - ~129 total commands --- ## [2.1.0] - 2026-02-04 ### 🤖 ML Models & Data Activator Release This release adds ML Models (MLflow) and Data Activator (Reflex) support, bringing the extension to 98% feature parity with Microsoft Fabric Portal. ### Added #### 14. ML Models (NEW) - **Model Registry** - Browse and manage MLflow registered models - **Model Versions** - Track model versions with staging workflows - **Experiments** - View ML experiments and runs - **Run Metrics** - Explore run metrics, parameters, and artifacts - **Stage Transitions** - Move models between None/Staging/Production/Archived - **Model Deployment** - Deploy models to endpoints - **Artifact Download** - Download model artifacts - **Run Comparison** - Compare metrics across runs - **Notebook Integration** - View run source notebooks - **Model Flavors** - Support for sklearn, pytorch, tensorflow, onnx, spark, etc. #### 15. Data Activator (NEW) - **Reflex Browser** - Browse Data Activator reflexes - **Object Management** - Create and manage data objects - **Property Definitions** - Define properties with aggregations - **Trigger Creation** - Create triggers with conditions and actions - **Condition Types** - Threshold, change, anomaly, pattern, absence - **Action Types** - Email, Teams, Power Automate, custom webhooks - **Trigger Testing** - Test triggers before activation - **Trigger History** - View trigger execution history - **Activate/Pause** - Control trigger state - **Multi-source Support** - Eventstream, Lakehouse, KQL, Power BI sources ### Technical Improvements - Added 15 tree view providers (2 new) - Added 24 new commands - 36 passing tests - 17 sidebar views total - ~118 total commands --- ## [2.0.0] - 2026-02-04 ### 🚀 Major Release: Full Fabric Portal Feature Parity This release adds 10 major features to achieve near-complete feature parity with the Microsoft Fabric Web Portal. ### Added #### 1. Lakehouse Explorer (NEW) - **Browse OneLake** - Navigate files and folders in OneLake - **Delta Tables** - View and preview Delta table data - **File Operations** - Upload, download, delete files and folders - **Shortcuts** - Create and manage OneLake shortcuts to external data - **Table Maintenance** - Optimize and vacuum Delta tables - **Data Preview** - Preview table data with column selection #### 2. Pipeline Management (NEW) - **Pipeline Browser** - View all Data Factory pipelines - **Run Pipelines** - Trigger pipeline executions from VS Code - **Run Monitoring** - Track pipeline runs with live status updates - **Run History** - View historical runs with duration and status - **Activity Logs** - Access detailed activity-level logs - **Cancel Runs** - Cancel running pipelines #### 3. Notebook Support (NEW) - **Notebook Browser** - View all Spark notebooks - **Create Notebooks** - Create new notebooks (Python/Scala/R/SQL) - **Run Notebooks** - Execute notebooks with output capture - **Session Management** - Start, stop, and monitor Spark sessions - **Delete Notebooks** - Remove notebooks from workspace #### 4. Monitoring Hub (NEW) - **Active Refreshes** - Real-time view of in-progress refreshes - **Recent Refreshes** - Historical refresh status and timing - **Running Queries** - Monitor active SQL queries - **Activity Timeline** - Chronological view of all workspace events - **Cancel Operations** - Cancel running refreshes and queries #### 5. Deployment Pipelines (CI/CD) (NEW) - **Pipeline Browser** - View deployment pipeline stages - **Stage Management** - Assign workspaces to stages - **Deploy Content** - Deploy between Dev/Test/Prod stages - **Compare Stages** - Visual diff between environments - **Deployment History** - Track deployment operations - **Deployment Notes** - Add deployment descriptions #### 6. Scheduling Manager (NEW) - **Schedule Browser** - View all refresh schedules - **Create Schedules** - Set up new refresh schedules - **Edit Schedules** - Modify times and frequency - **Enable/Disable** - Toggle schedules on/off - **Manual Refresh** - Trigger immediate refresh - **Refresh History** - View refresh results and errors #### 7. Lineage Viewer (NEW) - **Upstream Sources** - View data sources feeding an item - **Downstream Dependents** - See what depends on an item - **Impact Analysis** - Assess change impact - **Navigate Graph** - Click through lineage relationships - **Full Lineage Graph** - Visual graph representation #### 8. SQL Analytics (Warehouse) (NEW) - **Warehouse Browser** - Browse all SQL analytics endpoints - **Schema Explorer** - Tables, views, stored procedures - **Column Details** - Data types, keys, constraints - **Generate SELECT** - Auto-generate SELECT statements - **View Definitions** - View object DDL - **Connection Strings** - Copy connection details - **Pause/Resume** - Control warehouse compute #### 9. Git Integration Enhancement (NEW) - **Connect to Git** - Link workspace to Azure DevOps or GitHub - **Pending Changes** - View uncommitted changes - **Commit Changes** - Commit and push to repository - **Sync Operations** - Pull, push, and sync - **Branch Management** - Switch and create branches - **Conflict Resolution** - Resolve merge conflicts - **Undo Changes** - Revert uncommitted modifications #### 10. Capacity Admin (NEW) - **Capacity Browser** - View all Fabric capacities - **Real-time Metrics** - CPU, memory, CU usage - **Pause/Resume** - Control capacity state - **Scale Capacity** - Change SKU (F2-F512) - **Workload Management** - Enable/disable workloads - **Alert Management** - View and acknowledge alerts - **Metrics Dashboard** - Visual capacity dashboard ### New Views Added | View ID | Name | Description | |---------|------|-------------| | `fabricLakehouse` | Lakehouse Explorer | Browse OneLake files and tables | | `fabricPipelines` | Pipelines | Data Factory pipeline management | | `fabricNotebooks` | Notebooks | Spark notebook management | | `fabricWarehouse` | SQL Analytics | Warehouse schema browser | | `fabricMonitoring` | Monitoring Hub | Real-time operation monitoring | | `fabricDeployment` | Deployment Pipelines | CI/CD pipeline management | | `fabricScheduling` | Schedules | Refresh schedule management | | `fabricLineage` | Lineage | Data lineage visualization | | `fabricGit` | Git Integration | Source control management | | `fabricCapacity` | Capacity Admin | Capacity administration | ### New Commands (80+ added) Key commands by feature: **Lakehouse**: `fabric.lakehouse.refresh`, `upload`, `download`, `previewTable`, `createShortcut`, `optimizeTable` **Pipelines**: `fabric.pipeline.run`, `cancel`, `viewHistory`, `viewLogs` **Notebooks**: `fabric.notebook.create`, `open`, `run`, `delete`, `manageSession` **Warehouse**: `fabric.warehouse.newQuery`, `previewTable`, `generateSelect`, `pause`, `resume` **Monitoring**: `fabric.monitoring.cancelRefresh`, `viewDetails`, `cancelQuery` **Deployment**: `fabric.deployment.deploy`, `compare`, `assignWorkspace` **Scheduling**: `fabric.scheduling.create`, `edit`, `toggle`, `runNow` **Lineage**: `fabric.lineage.showGraph`, `impactAnalysis`, `navigate` **Git**: `fabric.git.commit`, `sync`, `pull`, `push`, `switchBranch`, `createBranch`, `resolveConflict` **Capacity**: `fabric.capacity.pause`, `resume`, `scale`, `toggleWorkload`, `openDashboard` ### Architecture Updates ``` src/ ├── extension.ts # Entry point with all 10 features ├── commands/ │ ├── index.ts # Core commands │ └── feature-commands.ts # All feature commands (NEW) ├── core/api/ │ └── fabric-api-service.ts # Unified API service (NEW) ├── lakehouse/ │ └── lakehouse-service.ts # Lakehouse operations ├── pipelines/ │ └── pipeline-service.ts # Pipeline operations ├── notebooks/ │ └── notebook-service.ts # Notebook operations ├── warehouse/ │ └── warehouse-service.ts # SQL Analytics operations ├── monitoring/ │ └── monitoring-service.ts # Monitoring operations ├── deployment/ │ └── deployment-service.ts # Deployment pipeline operations ├── scheduling/ │ └── scheduling-service.ts # Schedule operations ├── lineage/ │ └── lineage-service.ts # Lineage operations ├── git-integration/ │ └── git-service.ts # Git operations ├── capacity/ │ └── capacity-service.ts # Capacity operations └── views/ ├── lakehouse-tree-provider.ts # Lakehouse tree (NEW) ├── pipeline-tree-provider.ts # Pipeline tree (NEW) ├── notebook-tree-provider.ts # Notebook tree (NEW) ├── warehouse-tree-provider.ts # Warehouse tree (NEW) ├── monitoring-tree-provider.ts # Monitoring tree (NEW) ├── deployment-tree-provider.ts # Deployment tree (NEW) ├── scheduling-tree-provider.ts # Scheduling tree (NEW) ├── lineage-tree-provider.ts # Lineage tree (NEW) ├── git-tree-provider.ts # Git tree (NEW) └── capacity-tree-provider.ts # Capacity tree (NEW) ``` --- ## [1.0.0] - 2026-02-04 ### Added #### Core Features - **Workspace Explorer** - Browse all Fabric workspaces with hierarchical tree view - **Authentication** - Azure CLI, Service Principal, and Interactive browser login - **Status Bar** - Connection status and workspace selector #### SQL Features - **SQL Editor** - Full-featured query editor with syntax highlighting - **Query Execution** - Execute T-SQL against Lakehouses and Warehouses - **Results Grid** - Sortable columns, export to CSV/JSON - **Query History** - Access previous queries - **Query Snippets** - Save and reuse common queries #### Notebooks & Pipelines - **Notebook Viewer** - View notebook cells (code + markdown) - **Notebook Execution** - Run notebooks with progress tracking - **Pipeline Viewer** - Visual DAG of pipeline activities - **Pipeline Execution** - Trigger and monitor pipeline runs - **Run History** - Track all executions in Recent Runs panel #### Semantic Models (NEW) - **Model Explorer** - View tables, columns, and measures - **Refresh Management** - Trigger refreshes, view history - **Take Over** - Take ownership of models - **Parameter Updates** - Modify model parameters #### Power BI Reports (NEW) - **Report Viewer** - View report info and pages - **Open in Service** - Quick link to Power BI Service - **Clone Reports** - Duplicate reports - **Rebind Reports** - Change data source #### Capacity Monitoring (NEW) - **Capacity Dashboard** - View all capacities - **Status Monitoring** - See active/inactive status - **Usage Metrics** - CPU and memory visualization #### Git Integration (NEW) - **Connection Status** - View Git connection state - **Commit Changes** - Push workspace to Git - **Pull Updates** - Sync from Git repository - **Connect/Disconnect** - Manage Git connections #### OneLake File Browser (NEW) - **Directory Navigation** - Browse lakehouse files - **File Preview** - View text-based files - **File Management** - Delete files #### Other - **Favorites** - Bookmark frequently used items - **Context Menus** - Right-click actions on all items - **Keyboard Shortcuts** - Ctrl+Alt+N for new query, etc. - **Settings UI** - Full configuration options ### Architecture ``` src/ ├── extension.ts # Entry point ├── commands/ │ └── index.ts # All command registrations ├── core/ │ ├── api/ │ │ ├── fabric-client.ts # REST API (workspaces, notebooks, pipelines) │ │ ├── fabric-client-extended.ts # Extended API (semantic models, reports, git) │ │ └── sql-client.ts # SQL execution engine │ ├── auth/ │ │ └── auth-provider.ts # Azure authentication │ └── utils/ │ ├── config-manager.ts # VS Code settings │ └── logger.ts # Logging └── views/ ├── workspace-tree.ts # Sidebar explorer ├── recent-runs-tree.ts # Run history ├── favorites-tree.ts # Bookmarks ├── status-bar.ts # Status bar ├── sql-editor-provider.ts # SQL query webview ├── notebook-viewer.ts # Notebook display ├── pipeline-viewer.ts # Pipeline DAG ├── semantic-model-manager.ts # Model management ├── report-viewer.ts # Power BI reports ├── capacity-monitor.ts # Capacity dashboard ├── git-integration.ts # Git panel ├── onelake-browser.ts # File browser └── query-snippets.ts # Snippet manager ``` ### Commands Added | Command | Description | |---------|-------------| | `fabric.signIn` | Sign in to Microsoft Fabric | | `fabric.signOut` | Sign out | | `fabric.refresh` | Refresh explorer | | `fabric.newSqlQuery` | Open SQL editor | | `fabric.runNotebook` | Run selected notebook | | `fabric.runPipeline` | Run selected pipeline | | `fabric.viewNotebookContent` | View notebook cells | | `fabric.viewPipelineHistory` | View pipeline runs | | `fabric.connectLakehouse` | Connect to lakehouse SQL | | `fabric.openSemanticModel` | Open model manager | | `fabric.refreshSemanticModel` | Trigger model refresh | | `fabric.openReport` | View Power BI report | | `fabric.openCapacityMonitor` | Open capacity dashboard | | `fabric.openGitIntegration` | Manage Git connection | | `fabric.browseOneLake` | Open file browser | | `fabric.openQuerySnippets` | View saved snippets | | `fabric.saveQuerySnippet` | Save current query | ### Settings | Setting | Default | Description | |---------|---------|-------------| | `fabricTools.authentication.tenantId` | - | Azure AD tenant | | `fabricTools.authentication.method` | azureCli | Auth method | | `fabricTools.security.readOnlyMode` | false | Prevent modifications | | `fabricTools.sqlEditor.defaultRowLimit` | 1000 | Max rows returned | | `fabricTools.sqlEditor.queryTimeout` | 30 | Query timeout (sec) | --- ## Roadmap ### v1.1.0 (Planned) - [ ] IntelliSense for SQL editor - [ ] Drag-and-drop file upload to OneLake - [ ] Real-time capacity metrics via Azure Monitor - [ ] Deployment pipeline management - [ ] KQL query support for Eventhouse ### v1.2.0 (Planned) - [ ] Notebook diff viewer - [ ] Pipeline visualization improvements - [ ] Multi-workspace comparison - [ ] Custom dashboard builder