# Marketing Data Hub

**Your marketing data, on your machine, free.** An open-source Windsor.ai
alternative: pulls Google Analytics 4, Search Console (plus Google Ads, Meta
Ads, YouTube) into one local database — queryable via a REST API, scheduled
CSV exports, and by AI assistants like Claude (MCP). No hosted service, no
subscription, your tokens never leave your computer.

## Quick start (no config editing needed)

```bash
pip install marketing-data-hub
mkdir my-marketing-hub && cd my-marketing-hub
hub setup
```

(Developers: `git clone https://github.com/rallabandibhargava-dev/marketing-data-hub`
and `pip install -e ".[dev]"` instead.)

`hub setup` opens a page in your browser where you:

1. **Connect Google** — sign in, done (multiple Google accounts supported)
2. **Tick the GA4 properties / Search Console sites** you want
3. Optionally paste Google Ads / Meta Ads tokens
4. **Run the first sync** and watch it load
5. **Copy the Claude snippet** to ask questions in plain English

One prerequisite: a Google OAuth client file at `secrets/google_client.json`
(one-time, ~5 minutes — see [SETUP.md](SETUP.md) step 2; teams share one file).

**Never used a terminal? → [GUIDE.md](GUIDE.md)** is a complete, plain-English
walkthrough from installing Python through asking your first question — written
for non-technical teammates, and made to be shared.

Then ask Claude things like *"How did organic traffic do in June vs May?"* or
*"Top non-branded search queries this month?"* — or automate a daily 6am sync
(SETUP.md, step 8).

## Reports (analysis shapes)

Each source syncs several named *reports* — different dimensional shapes of the
same data, stored side by side and never mixed (mixing granularities would
double-count):

| Source | Report | Answers |
|---|---|---|
| ga4 | `core` | daily campaign totals (sessions, users, conversions, revenue) |
| ga4 | `channels` | traffic mix: organic vs paid vs direct, engagement, pageviews |
| ga4 | `landing_pages` | entry-page performance per channel |
| ga4 | `pages` | page behaviour: views, engagement time, events per path |
| ga4 | `audience` | device × country segmentation |
| ga4 | `visitors` | new vs returning (cohort-lite) |
| gsc | `core` | exact daily search totals per site |
| gsc | `queries` | per-query performance (branded split = string-match) |
| gsc | `pages` | per-URL search performance |
| gsc | `devices` / `countries` | mobile/desktop and geo splits |
| ga4 | `events` | per-event counts by name (brand-specific: form_submit, call_click...) |

GA4 breakdown reports exclude GA4's unattributable `(other)` bucket, so they
sum to slightly **under** the topline (on very large properties, well under
for high-cardinality dims like landing pages) — use `core` for exact totals,
breakdowns for composition/ranking. Same idea as GSC query anonymisation.

Pass `report=<name>` to the API/MCP `query_metrics`; default is `core`.
MCP `query_metrics` also supports `compare=` (prev_period / prev_day / prev_week /
prev_month / prev_year — returns value, previous, and %-change per metric for any
date range) and `filters=` (exact match on any dimension incl. report extras,
e.g. `{"event": "form_submit"}` or `{"device": "MOBILE"}`).
Rates are computed, not stored: engagement rate = engaged_sessions/sessions,
ctr = clicks/impressions, avg engagement time = engagement_seconds/pageviews.
GSC breakdown reports undercount totals slightly (Google anonymises rare
queries) — use `core` for toplines. True user-level cohorts need the GA4
BigQuery export; `visitors` + the live tools cover cohort-lite analysis.

For anything the synced reports don't cover, the MCP tools `query_ga4_live`
and `query_gsc_live` pass arbitrary dimension/metric combinations straight to
the APIs on demand.

## Setup

**New here / installing on another machine? Follow [SETUP.md](SETUP.md)** — a
step-by-step guide including the Google Cloud OAuth setup. Quick version:

1. `python -m pip install -e ".[dev]"`
2. Copy `config.yaml.example` → `config.yaml`; fill in your GA4 `property_id`
   and Search Console `site_url`. Have multiple GA4 properties or Search Console
   sites under the same Google login? Use `property_ids: [...]` / `site_urls: [...]`
   instead — all of them sync, and every row is tagged with its own `account_id`
   so they stay distinguishable downstream.
3. Copy `.env.example` → `.env`; set a random `HUB_API_KEY`.
4. Google Cloud Console → create a project → enable **Google Analytics Data API**,
   **Google Analytics Admin API**, **Search Console API**, **YouTube Analytics
   API** → create an **OAuth client (Desktop app)** → download JSON to
   `secrets/google_client.json`. (See [SETUP.md](SETUP.md) for the OAuth
   consent-screen steps and the 7-day token-expiry gotcha.)
5. `hub doctor` — first run opens a browser to authorize; then all checks go green.
6. `hub accounts --add` — pick which GA4 properties / GSC sites to sync from
   everything your Google login can see.

## Daily use

| Command | What it does |
|---|---|
| `hub sync all` | sync every configured source (rolling 30-day window) |
| `hub backfill ga4 --from 2024-01-01` | load history in 90-day chunks |
| `hub status` | row counts + last sync per source |
| `hub serve` | query API on 127.0.0.1:8000 + cron scheduler |
| `hub export all` | write configured CSVs to exports/ |
| `hub mcp` | MCP server (stdio) for Claude |

## Query API

```
GET /connectors/all/data?fields=date,source,clicks,spend&date_preset=last_30d
X-API-Key: <HUB_API_KEY>
```
`format=csv` for CSV, `report=<name>` for a breakdown report. `/connectors`
lists sources; `/connectors/{source}/reports` lists report shapes;
`/connectors/{source}/fields?report=<name>` lists fields.

## Claude MCP

`claude mcp add marketing-hub -- python -m hub.cli mcp --config <absolute-path>/config.yaml`
Then ask Claude: "How did my campaigns do last week?"

Note: use an absolute path for --config; the MCP process may be launched from a
different working directory.

`trigger_sync` starts the sync in the background and returns immediately
(output goes to `logs/mcp_sync.log`); poll `sync_status` to see when it
finishes. While a sync holds the write lock, query tools return a readable
"database is busy" error instead of hanging.

## Activating the ad connectors

- **Google Ads:** apply for a developer token (API Center), then uncomment
  `google_ads` in config.yaml and fill options.
- **Meta Ads:** create a Meta app, generate a long-lived token with `ads_read`,
  uncomment `meta_ads` and fill options.

## Known limitations

- DuckDB allows one writer: run `hub mcp` OR `hub serve`, not both at once
  (trigger_sync from MCP spawns the CLI, which needs the write lock free).
  While any sync runs, MCP query tools report "database is busy" until it
  finishes (~3 min for `sync all`).
- Extras fields (e.g. position, ctr, views) are returned as strings by the query
  API — cast numerically as needed.