# ADF Cost Intelligence MCP

![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)
![MCP](https://img.shields.io/badge/MCP-stdio-green)
![Azure ADF](https://img.shields.io/badge/Azure-Data%20Factory-0078D4)
![License MIT](https://img.shields.io/badge/license-MIT-brightgreen)
![CI](https://github.com/harinarayn/adf-cost-intelligence-mcp/actions/workflows/ci.yml/badge.svg)

**Ask your AI assistant which ADF pipeline is bleeding your Azure budget, and exactly why.**

An enterprise FinOps intelligence system for Azure Data Factory, exposed via the Model Context Protocol (MCP). Converts raw ADF consumption metrics (DIU-hours, vCore-hours, activity runs) into dollar estimates, detects waste, identifies cost spikes, and recommends specific fixes with quantified savings. Pricing is fetched live from the Azure Retail Prices API, not hardcoded constants.

Works with **Claude Desktop, Claude Code, GitHub Copilot, Cursor, Windsurf, or any MCP-compatible client.**

---

## What You Can Ask

```
Which ADF pipeline cost the most this month?

Why did my ADF bill spike last week?

Are any of my pipelines running wastefully?

Give me your top recommendations to cut my ADF spend.

Break down the cost of my IngestSalesData pipeline.
```

**Example response:**

> Your top 3 most expensive pipelines this month:
> 1. **IngestSalesData** - $142.30 (driver: DIU hours from Copy Activity)
> 2. **TransformCustomerData** - $87.50 (driver: Data Flow vCore hours)
> 3. **LoadDailyReports** - $34.20 - running hourly with zero rows processed
>
> Recommendation: Switch LoadDailyReports to an event-based trigger. Estimated saving: **$34.20/month ($410/year)**.

---

## The 5 Tools

| Tool | What it does |
|---|---|
| `get_top_costly_pipelines` | Ranks all pipelines by estimated monthly cost with trend vs prior period |
| `get_pipeline_cost_breakdown` | Itemised cost by activity type for a specific pipeline |
| `get_wasteful_pipelines` | Flags zero-row runs, inactive pipelines, and debug runs in production |
| `get_cost_spike_analysis` | Compares current vs prior period, explains sudden cost increases |
| `get_optimization_recommendations` | 7-rule engine with specific fixes and estimated annual savings |

---

## Get Started in 5 Minutes

**New user? Follow this path:**

```
Step 1 - Clone and install         (~1 min)
Step 2 - Try with mock data        (~1 min)   no Azure needed
Step 3 - Connect your Azure ADF    (~10 min)  one-time setup
Step 4 - Add to your MCP client    (~2 min)
```

Jump to [Quick Start](#quick-start) below for the full steps.

**Already have Python and an ADF instance?** The shortest path to real data:

```bash
git clone https://github.com/harinarayn/adf-cost-intelligence-mcp
cd adf-cost-intelligence-mcp
pip install -r requirements.txt

# Create a service principal (one-time)
az ad sp create-for-rbac --name adf-cost-mcp-sp --role Reader \
  --scopes /subscriptions/{YOUR_SUB_ID}

# Grant it ADF read access (one-time)
az role assignment create --assignee {APP_ID} \
  --role "Data Factory Contributor" \
  --scope /subscriptions/{SUB}/resourceGroups/{RG}/providers/Microsoft.DataFactory/factories/{ADF}

# Configure credentials
cp .env.example .env
# Edit .env with your values

# Add to your MCP client config (see Connect Your MCP Client below)
# Then ask: "Which ADF pipeline cost the most this month?"
```

---

## Security and Privacy

**This server runs entirely in your own environment. No data ever leaves your infrastructure.**

- Runs locally or in your own cloud, not our servers
- Only reads ADF run metadata (pipeline names, durations, status, DIU counts) - never your pipeline data or business content
- Service principal needs read-only access (see setup below)
- No telemetry, no callbacks, no third-party dependencies except the public [Azure Retail Prices API](https://prices.azure.com/api/retail/prices) (unauthenticated, read-only pricing data)
- Fully open source - every line of code is auditable

See [SECURITY.md](SECURITY.md) for the full security posture.

---

## Prerequisites

| Requirement | How to Check |
|---|---|
| Python 3.11+ | `python --version` |
| Azure subscription | With at least one ADF instance |
| ADF run history | At least 7 days of pipeline runs for meaningful results |
| MCP-compatible client | Claude Desktop, Copilot, Cursor, Windsurf, or any MCP client |

---

## Quick Start

### Step 1 - Clone and install

```bash
git clone https://github.com/harinarayn/adf-cost-intelligence-mcp
cd adf-cost-intelligence-mcp
pip install -r requirements.txt
cp .env.example .env
```

### Step 2 - Try it immediately with mock data (no Azure needed)

```bash
USE_MOCK_DATA=true python server.py
```

Open your MCP client and ask: **"Which ADF pipeline cost the most this month?"**

You'll see responses using realistic synthetic data covering 10 mock pipelines, including wasteful runs, cost spikes, and inactive pipelines.

### Step 3 - Connect your real Azure ADF (~10 minutes, one-time)

**3a. Login and find your subscription ID**
```bash
az login
az account show --query id -o tsv
```

**3b. Create a service principal with minimal permissions**
```bash
az ad sp create-for-rbac \
  --name adf-cost-mcp-sp \
  --role Reader \
  --scopes /subscriptions/{YOUR_SUBSCRIPTION_ID}
# Save the output - you need: appId, password, tenant
```

**3c. Grant read access to your ADF instance**
```bash
az role assignment create \
  --assignee {APP_ID_FROM_ABOVE} \
  --role "Data Factory Contributor" \
  --scope /subscriptions/{SUB_ID}/resourceGroups/{RG}/providers/Microsoft.DataFactory/factories/{ADF_NAME}
```

> **Minimum permissions:** The service principal only needs to read pipeline runs and activity runs. `Data Factory Contributor` is the closest built-in role. For tighter security, create a custom role with `Microsoft.DataFactory/factories/read`, `Microsoft.DataFactory/factories/pipelineruns/read`, and `Microsoft.DataFactory/factories/activityruns/read`.

**3d. Enable per-pipeline billing in ADF (important)**
1. Open [ADF Studio](https://adf.azure.com)
2. Go to **Manage** > **Factory Settings**
3. Set **Billing by pipeline** to **ON** and save

> Without this, cost data is at factory level only, not per-pipeline.

### Step 4 - Configure .env

```env
AZURE_TENANT_ID=paste-from-step-3b
AZURE_CLIENT_ID=paste-appId-from-step-3b
AZURE_CLIENT_SECRET=paste-password-from-step-3b
AZURE_SUBSCRIPTION_ID=paste-from-step-3a
AZURE_RESOURCE_GROUP=your-resource-group-name
AZURE_DATA_FACTORY_NAME=your-adf-factory-name
USE_MOCK_DATA=false
```

---

## Connect Your MCP Client

### Claude Desktop

**Mac/Linux:** `~/.claude/claude_desktop_config.json`
**Windows:** `%APPDATA%\Claude\claude_desktop_config.json`

```json
{
  "mcpServers": {
    "adf-cost-intelligence": {
      "command": "python",
      "args": ["/absolute/path/to/adf-cost-intelligence-mcp/server.py"],
      "env": {
        "AZURE_TENANT_ID": "your-tenant-id",
        "AZURE_CLIENT_ID": "your-app-id",
        "AZURE_CLIENT_SECRET": "your-secret",
        "AZURE_SUBSCRIPTION_ID": "your-subscription-id",
        "AZURE_RESOURCE_GROUP": "your-resource-group",
        "AZURE_DATA_FACTORY_NAME": "your-adf-name",
        "USE_MOCK_DATA": "false"
      }
    }
  }
}
```

Restart Claude Desktop. The tools appear automatically.

### Claude Code (CLI)

```bash
claude mcp add adf-cost-intelligence python /path/to/server.py
```

Or add a `.mcp.json` to your project root (safe to commit, uses env var references):

```json
{
  "servers": {
    "adf-cost-intelligence": {
      "type": "stdio",
      "command": "python",
      "args": ["server.py"],
      "env": {
        "AZURE_TENANT_ID": "${AZURE_TENANT_ID}",
        "AZURE_CLIENT_ID": "${AZURE_CLIENT_ID}",
        "AZURE_CLIENT_SECRET": "${AZURE_CLIENT_SECRET}",
        "AZURE_SUBSCRIPTION_ID": "${AZURE_SUBSCRIPTION_ID}",
        "AZURE_RESOURCE_GROUP": "${AZURE_RESOURCE_GROUP}",
        "AZURE_DATA_FACTORY_NAME": "${AZURE_DATA_FACTORY_NAME}",
        "USE_MOCK_DATA": "false"
      }
    }
  }
}
```

### GitHub Copilot (VS Code)

Add to `.vscode/mcp.json` in your workspace:

```json
{
  "servers": {
    "adf-cost-intelligence": {
      "type": "stdio",
      "command": "python",
      "args": ["/absolute/path/to/server.py"],
      "env": {
        "AZURE_TENANT_ID": "${env:AZURE_TENANT_ID}",
        "AZURE_CLIENT_ID": "${env:AZURE_CLIENT_ID}",
        "AZURE_CLIENT_SECRET": "${env:AZURE_CLIENT_SECRET}",
        "AZURE_SUBSCRIPTION_ID": "${env:AZURE_SUBSCRIPTION_ID}",
        "AZURE_RESOURCE_GROUP": "${env:AZURE_RESOURCE_GROUP}",
        "AZURE_DATA_FACTORY_NAME": "${env:AZURE_DATA_FACTORY_NAME}",
        "USE_MOCK_DATA": "false"
      }
    }
  }
}
```

### Cursor / Windsurf

Both support MCP via `~/.cursor/mcp.json` or `~/.windsurf/mcp.json` using the same format as the Claude Desktop config above.

### MCP Inspector (for testing and demos)

```bash
npx @modelcontextprotocol/inspector python server.py
```

Opens a browser UI to call each tool individually and inspect raw JSON responses. Useful for testing your setup or running demos.

---

## Testing with MCP Inspector

MCP Inspector lets you call each tool directly in a browser UI and inspect the raw JSON responses, with no AI client needed. Great for testing your setup, debugging, or demos.

![MCP Inspector showing all 5 tools connected and returning real Azure data](docs/images/mcp-inspector.png)

### Launch (Windows)

```cmd
scripts\inspect.cmd
```

This loads credentials from your `.env` file and opens the inspector automatically.

### Launch (Mac/Linux)

```bash
source .env && npx @modelcontextprotocol/inspector python server.py
```

### What you will see

1. Browser opens at `http://localhost:6274`
2. Green **Connected** indicator - server is live
3. **Tools** tab lists all 5 tools with their descriptions
4. Click any tool, enter arguments, hit **Run Tool**, and see the exact JSON your AI client receives

### Example - test the cost ranking tool

Click `get_top_costly_pipelines`, enter:
```json
{ "days": 30, "top_n": 8 }
```
Hit **Run Tool**. You will see your real ADF pipelines ranked by estimated monthly cost with waste percentages and trend data.

---

## Running Tests

```bash
# All tests - no Azure credentials required (uses mock data)
USE_MOCK_DATA=true python -m pytest tests/ -v

# Specific modules
python -m pytest tests/test_cost_calculator.py -v
python -m pytest tests/test_pricing_client.py -v
python -m pytest tests/test_recommendations.py -v
python -m pytest tests/test_tools.py -v
```

---

## How Costs Are Calculated

Costs are estimated from ADF activity run metadata using the same billing model as Azure:

| Activity Type | Billing Model |
|---|---|
| Copy (Azure IR) | DIU-hours x $0.25/DIU-hour (4-min minimum per run) |
| Copy (Self-hosted IR) | DIU-hours x $0.10/DIU-hour |
| Mapping Data Flow | vCore-hours x rate by compute type (1-min minimum) |
| Pipeline / Lookup / ForEach | Activity execution hours x $0.005/hour |
| External (Databricks, etc.) | Activity execution hours x $0.00025/hour |
| Inactive pipeline | $0.80/month flat fee |

When available, **actual billed DIU-hours are read directly from `billingReference` in ADF activity output**, not estimated from duration. Live pricing is fetched from the [Azure Retail Prices API](https://prices.azure.com/api/retail/prices) at startup and cached for 24 hours.

---

## Optimization Rules

The recommendations engine applies 7 rules across your pipeline history:

| # | Condition | Fix | Typical Saving |
|---|---|---|---|
| 1 | Schedule trigger + >20% zero-row runs | Switch to event-based trigger | 20-80% of wasteful run cost |
| 2 | Copy DIU count > 4, data < 100 MB | Reduce maxDataIntegrationUnits | 40% of DIU cost |
| 3 | ForEach sequential=true, items > 10 | Enable parallelism (isSequential=false) | 30% of pipeline duration |
| 4 | DataFlow without IR cluster TTL | Set Time-to-Live on Integration Runtime | 15% of DataFlow cost |
| 5 | Pipeline with zero runs in 30 days | Decommission (save $0.80/month each) | $9.60/pipeline/year |
| 6 | Daily full table load, large dataset | Implement watermark incremental load | 70% of data movement cost |
| 7 | Debug runs in production factory | Enforce dev/test factory separation | ~8% of monthly pipeline cost |

---

## How This Differs from azure-mcp and Azure Advisor

| | azure-mcp (official) | Azure Advisor | This Tool |
|---|---|---|---|
| ADF pipeline run data | No | No | Yes - every run, 30-day history |
| Cost per pipeline | No | No | Yes - estimated to 4 decimal places |
| Waste detection | No | No | Yes - zero-row runs, inactive pipelines |
| Cost spike analysis | No | No | Yes - current vs prior period |
| Specific $ savings estimates | No | Generic | Yes - per recommendation |
| Works without Log Analytics | N/A | N/A | Yes - reads ADF APIs directly |

azure-mcp gives your AI assistant a hammer. This gives it a scalpel.

---

## Cost Disclaimer

> Cost estimates are based on Azure PAYG pricing. Actual costs vary by Azure agreement type (Enterprise Agreement, CSP, PAYG), reserved capacity, region, and currency. Always cross-check significant decisions with Azure Cost Management for exact billing figures.

---

## V2 Roadmap

- **Remediation with code**: Generate ADF JSON pipeline/trigger configs as fixes, not just advice
- **Multi-factory support**: Query across multiple ADF instances in one conversation
- **Trend analysis**: Week-over-week cost trending with anomaly detection
- **Databricks integration**: Include Databricks job cost as part of ADF external activity cost
- **Budget alerts**: Set thresholds and get proactive warnings
- **Remote MCP (enterprise)**: Deploy as remote MCP server on Azure Container Apps with Managed Identity, no local credentials, full audit trail

---

## Contributing

Contributions welcome. Key areas:

- Additional recommendation rules (open an issue to propose)
- New Azure regions in the pricing parser
- Synapse Analytics support
- Integration tests with real ADF instances

```bash
# Dev setup
pip install -r requirements.txt
USE_MOCK_DATA=true python -m pytest tests/ -v
```

**License:** MIT