shopping-mcp
by leodinh
README.md
# Shopping MCP
**Shopping with Agent** lets a shopper ask their AI assistant to search and compare products across connected stores. Merchants connect a Shopify store; it is searched live, through Shopify's API, nothing copied and exposed to assistants through an MCP server. "Shopping MCP" is the protocol, config, and repo name.
- `/`: shopper-facing home page with an example conversation and the `shopping-mcp.json` download.
- `/docs`: how to add Shopping MCP to an assistant.
- `/products`: the results of a search an assistant ran, linked from its answer (`browseUrl`). Not a storefront: there is no search box, and without a search it explains where results come from. The shopper can filter by store, max price (with its currency) and stock, switch between grid and list, and page with Previous/Next. The URL keeps the search, filters, page and view.
- `/products/[id]`: one product, live from its store (`pageUrl`): images, price, options, description, and **Purchase**, which opens the product on the store's own site. Checkout happens there.
- `/profile`: a signed-in user's account and connected stores: status (**Connected** or **Disconnected**, from the connection alone), **Disconnect** (search stops reaching the store, after an inline confirmation), and **Connect store**. One user can own several stores. Protected: signed out, it redirects to the home page.
- `/login`: Continue with Google, or an email magic link; shared by the dashboard and MCP clients' OAuth.
This is a development app: auth and data isolation are not ready for public deployment. Product context lives in `PRODUCT.md`; domain terms (Merchant, MerchantConnection, Store connection, …) in `CONTEXT.md`.
More docs in [`docs/`](docs):
- [Authentication](docs/authentication.md): sign-in, sessions, how assistants sign in (CIMD, DCR, tokens), magic-link email, Google, and local vs tunnel profiles.
- [Development tunnel](docs/development-tunnel.md): exposing local servers over HTTPS through Cloudflare for external assistants and provider callbacks.
## Local setup
Prerequisites: Node.js 22+, pnpm, PostgreSQL 17+, and a Shopify app for connecting stores (see [Shopify app setup](#shopify-app-setup)). Optional: Docker, for Redis (the search cache).
1. Create an empty `shopping-mcp` database.
2. Install and configure:
```sh
pnpm install
cp .env.example .env
```
3. Fill in `.env` (see [Configuration](#configuration)). At minimum: `DATABASE_URL`, `BETTER_AUTH_SECRET`, `CREDENTIALS_KEY`, and the `SHOPIFY_*` values.
4. Migrate and start:
```sh
pnpm run db:migrate
docker compose up -d redis # optional: the search cache (REDIS_URL)
pnpm dev
```
`pnpm dev` starts two processes: the Next.js UI at http://127.0.0.1:3000 and the Nest API and MCP server at http://127.0.0.1:3001. Start them separately with `pnpm dev:web` and `pnpm dev:api`.
Then open http://localhost:3000/login and sign in. Without `RESEND_API_KEY` the magic link is printed in the API log; with it, it's emailed. Then connect a Shopify store from `/profile`. Details in [docs/authentication.md](docs/authentication.md).
## Configuration
All variables live in the root `.env`.
| Variable | Required | Purpose |
| ----------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DATABASE_URL` | yes | PostgreSQL URL including the user, e.g. `postgresql://USER@localhost:5432/shopping-mcp`. Add `:PASSWORD` only if the server needs one. |
| `SESSION_SECRET` | legacy | Only decrypts Shopify credentials stored before `CREDENTIALS_KEY` existed (re-encrypted the next time they're used). Not used for sign-in. |
| `CREDENTIALS_KEY` | yes | Encrypts stored Shopify tokens. **Must not change after use** (see below). |
| `SHOPIFY_API_KEY` | yes | Your Shopify app's client ID. |
| `SHOPIFY_API_SECRET` | yes | Your Shopify app's client secret. Verifies OAuth callbacks (HMAC) and exchanges tokens. |
| `SHOPIFY_SCOPES` | no | Requested scopes; default `read_products`. |
| `API_PORT` | no | API port; default `3001`. |
| `BETTER_AUTH_SECRET` | yes | Signs Better Auth sessions and OAuth state (dashboard and MCP sign-in). Generate separately. |
| `RESEND_API_KEY` | no | Resend API key (send-only) for magic-link emails. Unset: links are printed to the API console (local development). |
| `EMAIL_FROM` | no | Sender, on a domain verified in Resend. Default `onboarding@resend.dev` only delivers to your own Resend account email. |
| `GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET` | no | Enable "Continue with Google". Register `<API_ORIGIN>/api/auth/callback/google` as an authorized redirect URI. Signing in with Google and by email link for the same address is the same user. |
| `WEB_ORIGIN` | no | Web UI origin (CORS, trusted origin, login/consent pages, and the `browseUrl` / `pageUrl` links in tool results); default `http://localhost:3000`. |
| `API_ORIGIN` | no | Public API origin: Better Auth issuer (`<origin>/api/auth`) and MCP resource (`<origin>/api/mcp`); default `http://localhost:3001`. |
| `COOKIE_DOMAIN` | no | Parent domain for cross-subdomain session cookies (e.g. `leodev.online` for `app.` / `api.` siblings); unset on localhost. |
| `NEXT_PUBLIC_API_ORIGIN` | no | API origin the web UI calls; default `http://localhost:3001`. |
| `REDIS_URL` | no | Redis for the search cache, e.g. `redis://localhost:6379` (`docker compose up -d redis`). Unset or down: every search asks the stores live. |
| `SEARCH_CACHE_FRESH_S` / `SEARCH_CACHE_MAX_AGE_S` | no | A cached store page is fresh for 300s; it may serve as a stale fallback until 1800s after it was fetched, never later. |
| `SEARCH_STORE_TIMEOUT_MS` | no | Deadline for one store's page, including waiting for another pod's fetch; default 5000. |
| `SEARCH_STORE_CONCURRENCY` / `SEARCH_POD_CONCURRENCY` | no | Calls this pod makes at once to one store (default 4) and to all stores (default 32). Per pod, not cluster-wide. |
Generate each secret separately, e.g. `openssl rand -hex 32`. Changing `CREDENTIALS_KEY` makes stored Shopify tokens unreadable, and those stores must reconnect. Credentials encrypted before `CREDENTIALS_KEY` existed (with the old `SESSION_SECRET`-derived key) still decrypt and are re-encrypted on their next sync.
### Shopify app setup
One Shopify app serves every merchant; merchants never configure anything. In your app's settings (Dev Dashboard at dev.shopify.com, or the Partner Dashboard for older apps):
1. Copy the client ID and secret into `SHOPIFY_API_KEY` and `SHOPIFY_API_SECRET`.
2. Add `<API_ORIGIN>/api/connections/shopify/callback` to the allowed redirect URLs, character for character (`127.0.0.1` ≠ `localhost`, `http` ≠ `https`, port and trailing slash matter): `http://localhost:3001/api/connections/shopify/callback` locally, `https://shopping-api-dev.leodev.online/api/connections/shopify/callback` on the tunnel. The callback always follows `API_ORIGIN`, because the browser must return to the host that set its `oauth_binding` cookie. On the Dev Dashboard, settings live in a version: create and **release** it.
3. Add one redirect URL per environment you run (local, staging, production), or use a separate app per environment.
4. Only stores the app's distribution allows can connect: custom distribution limits installs to specific stores; public distribution allows any store.
5. Point the mandatory compliance webhooks (`customers/data_request`, `customers/redact`, `shop/redact`) at `https://<api-host>/api/webhooks/shopify`. They can only be set in app settings, not through the API, and public distribution requires them.
The `app/uninstalled` webhook needs no setup: after each connect, the API subscribes the shop to the same endpoint, whose URL comes from `API_ORIGIN`. Shopify only delivers to HTTPS, so with a plain `http://` `API_ORIGIN` (local development) the subscription is skipped with a warning.
If the dashboard rejects an `http://` URL, use the HTTPS tunnel ([docs/development-tunnel.md](docs/development-tunnel.md)): run with `API_ORIGIN` set to the tunnel and register its callback.
## Commands
| Command | Purpose |
| ------------------------------ | -------------------------------------------------------- |
| `pnpm dev` | Web :3000 and API :3001 |
| `pnpm run db:migrate` | Apply outstanding SQL migrations transactionally |
| `pnpm run db:setup` | Same as `db:migrate` |
| `pnpm test` | Workspace tests and architecture boundaries; no database |
| `pnpm run test:integration` | Real PostgreSQL tests; migrate first |
| `pnpm typecheck` | TypeScript validation |
| `pnpm lint` / `lint:fix` | ESLint across all workspaces |
| `pnpm format` / `format:check` | Prettier |
| `pnpm build` | Typecheck every workspace, then build the Next.js app |
ESLint checks code quality; Prettier handles formatting (`eslint-config-prettier` disables conflicting rules). Generated output and environment files are excluded from formatting. Never run local tooling against a production database.
## MCP server
Endpoint: `http://127.0.0.1:3001/api/mcp`. `GET /mcp.json` downloads a `shopping-mcp.json` config pointing at it; add that to an assistant as a custom MCP server. Tools call catalog functions; they never talk to Shopify themselves.
| Tool | Backend | Purpose |
| ------------------ | ------------------- | --------------------------------------------------------------------------------- |
| `search_products` | `searchProducts()` | Search every connected store live by keywords, price, merchant, stock, country |
| `get_product` | `getProduct()` | One product, live from its store, by the ID search returned |
| `compare_products` | `compareProducts()` | Consistent, live details for 2–5 products so an assistant can explain differences |
| `get_checkout` | `getCheckout()` | Look up a merchant checkout URL; currently returns `supported: false` |
| `get_account` | `getAccount()` | The signed-in User and their stores; requires sign-in |
Tool inputs use the catalog's own schemas from `packages/contracts`, so a tool never advertises input the catalog would reject. `search_products` also returns `browseUrl`, the web page showing the same search (query and filters, never the cursor), and every product from `search_products`, `get_product` and `compare_products` carries `pageUrl`, its product page. Both are built from `WEB_ORIGIN`. `search_products` takes real types: `maxPrice` is a number in major units (`19.99` = 19.99 USD) and requires `currency`; `inStock` is a boolean; `country` (ISO 3166) prices products for the buyer's market. A product ID is `shopify:<merchantId>:<productId>`.
## HTTP API
```sh
curl 'http://127.0.0.1:3001/api/products?q=black+backpack¤cy=USD&maxPrice=100'
curl 'http://127.0.0.1:3001/api/products?inStock=true&limit=5'
curl 'http://127.0.0.1:3001/api/products?inStock=true&limit=5&cursor=<nextCursor>'
curl 'http://127.0.0.1:3001/api/products/shopify%3A<merchantId>%3A<productId>'
curl 'http://127.0.0.1:3001/api/merchants'
```
`GET /api/products` accepts `q` (up to 200 characters), optional `merchantId` (UUID), `country` (ISO 3166, e.g. `CA`), `currency` (ISO 4217; only products priced in it are returned), `maxPrice` (non-negative number in major units; requires `currency`), `inStock` (`true`/`false`), `limit` (1–100, default 24), and `cursor`. It returns `{ products, nextCursor, hasMore, partial, unavailable, stores }`; for the next page, send the same parameters plus `cursor=<nextCursor>`. There is no total: it can't be computed cheaply across live stores. Each product has its ID, merchant, name, description, `price { amountMinor, currency }` (its cheapest variant), `available`, `brand`, `gtin`, `sku`, images, product URL and up to 20 variants. Invalid filters or a bad cursor return 400; 503 means no store could be reached. The controller only converts query strings into the same typed search the MCP tool uses.
`GET /api/products/:id` returns `{ product }`, read live from its store (optional `country` for market prices). A malformed ID returns 400, a product that no longer exists 404, and a store that didn't answer 503. The product page uses it; it is never cached.
Prices are in each currency's own minor units (cents for USD, none for JPY, three digits for KWD). The API filters by currency; it never converts between currencies.
## Architecture
```text
apps/web (Next :3000) ──credentialed fetch──> apps/api (Nest :3001) ──> packages/commerce ──> PostgreSQL
│ (users, stores,
│ connections)
└──> Shopify Admin API
(live search, per store)
```
This is a **pnpm workspace**, not a distributed commerce backend. Business operations live in `packages/commerce`. Import its feature entry points (`@shopping-mcp/commerce/catalog`, `/merchants`, `/auth`, `/store-connection`, and `/connectors/shopify`); there is no root barrel export.
- `packages/database`: Drizzle schema, pool, migrations, and a test-only isolated-schema helper (`@shopping-mcp/database/testing`).
- `packages/contracts`: browser-safe schemas and JSON types shared by the API, MCP tools, and web client.
- `packages/commerce`: catalog search (the `ProductProvider` contract and live federated search), merchants, accounts, Store connection (Shopify OAuth), and the Shopify provider.
- `packages/config`: server-only environment helpers.
- `apps/api`: Nest HTTP controllers, cookies, OAuth responses, and MCP tools.
- `apps/web`: shopper, docs, results and product pages, and profile UI only. Results and product pages read the HTTP API; they never call Shopify.
### Data model
`Merchant 1 — 1 MerchantConnection`. Table shapes are declared in `packages/database/src/schema.ts`. `SellerStatus` and `CatalogProduct` are read models, not table rows. Products are never stored: Shopify is the source of truth.
A MerchantConnection stores the connector type, non-secret configuration, the shop domain and Shopify shop GID, encrypted credentials with their expiry times, when the shop was verified, the enabled flag, and why it was last disabled. A Merchant is bound to one shop and never switches shops; a shop belongs to at most one Merchant.
### Store connection
`@shopping-mcp/commerce/store-connection` owns the Shopify OAuth lifecycle behind two operations:
1. **Begin** (signed-in User only): validate the `*.myshopify.com` domain, reject a shop owned by another User (`shop_taken`), and store a single-use OAuth attempt for the User and shop, tied to the browser by an `oauth_binding` cookie (10 minutes). Nothing is created for the shop yet.
2. **Complete**: verify the callback HMAC, consume the attempt, exchange the code for tokens and **verify** the new token with one Admin GraphQL `shop { id name }` call, both outside any DB transaction. Then in one transaction create the Merchant for the User, or **claim** it if it predates Users, set its name to the shop's name, and store encrypted credentials with the shop GID and `verified_at`. Nothing is imported: the store is searchable at once. The browser returns to `/profile`; Shopify never signs anyone in.
Both return typed outcomes (`{ ok: false, reason }`); the API maps each reason to an HTTP status. Connecting while signed out is a 401. A failed token exchange uses up the attempt; the User starts again. **Disconnect** (dashboard) and the `app/uninstalled` webhook share one path: disable the connection, drop its credentials, hide its products.
### Search
Search is **live and federated**: `LiveProductSearchService` (behind the `ProductSearchService` interface) asks every enabled store at once, through one `ProductProvider` per store.
1. **Fan-out.** Only the stores a page can reach are asked: round-robin takes one product per store per turn, so a page of `limit` products needs at most the next `limit` stores in rotation. Those are queried concurrently (at most 8 at a time), each with a timeout (`SEARCH_STORE_TIMEOUT_MS`, default 5s), and asked for `max(1, ceil(limit / stores asked))` products. Stores further along stay unvisited and are reached on later pages; one that fails or comes back empty hands its turn to the next. A new search visits stores in `orderStores()` order (the connection order for now); a resumed one keeps the cursor's order.
2. **Merge.** Results are interleaved round-robin, so every merchant gets balanced exposure. Stores that still have results are asked again until the page is full or every store is exhausted. The rotation carries over between pages.
3. **Paging.** `nextCursor` is stateless, opaque and HMAC-signed (key derived from `BETTER_AUTH_SECRET`). It holds each store's Shopify position and whether it's exhausted, the store order, whose turn is next, a fingerprint of the query and filters, a version, and a 30-minute expiry. A store's position advances only past products actually returned, so fetched but unreturned products come back on the next page. Each request re-checks the signature, expiry, parameters and which stores are still connected. Nothing is stored server-side.
4. **Failures.** A store that errors or times out with no usable data is listed in `unavailable`, and only then is the page `partial`. It keeps its position and is retried on the next page. An empty store is a success, not a failure. The search fails (503) only if every store tried fails; with no connected stores it returns an empty result.
5. **Cache** (when `REDIS_URL` is set). Each store's page is cached in Redis, keyed by the store's connection generation (a reconnect starts empty) and everything that changes the answer (query, filters, `country`, `currency`, position, batch size). Only public product data is cached.
- **Fresh** (under 5 minutes): served without asking the store.
- **Otherwise** one pod fetches it: identical requests in a pod share one call, and across pods a Redis lock per page (unique token, TTL, atomic owner-checked release) lets only the holder ask the store. Others wait within their deadline for its page; they never fetch without the lock. At the deadline they use an eligible stale page or report the store unavailable.
- **Writes are guarded**: a holder whose lock expired, or whose data is older than what's stored, can't overwrite it.
- **Stale fallback**: when a store fails, a cached page up to 30 minutes old (since it was fetched) is served instead; never older, and only for stores connected right now.
- **Freshness is visible**: `stores[]` gives each store's `outcome` (`cache_hit`, `cache_miss`, `stale_fallback`, `provider_error`, `provider_timeout`) and `fetchedAt`, and every product has `fetchedAt`.
- **Redis is optional**: commands time out fast and never queue; on any Redis failure the store is fetched live (without writing), within the per-pod limits. The API connects at startup (so the first search can use the cache) and closes the connection on shutdown.
- Lookups (`get_product`, `compare_products`, `get_checkout`) are never cached. `get_checkout` validates the current offer live and refuses when the product is gone, sold out, or its store doesn't answer.
6. **Shopify.** Only products that are active and published to the online store are searchable. Search words are quoted, so text can never become a Shopify filter. `maxPrice` goes to Shopify's `price` filter, or with `country` (market prices) is checked after fetching. `inStock` (any variant for sale) and `currency` (the product's returned currency) are checked after fetching. Tokens are refreshed when used, under a per-store Postgres advisory lock, because Shopify refresh tokens work only once.
### Shopify webhooks
`POST /api/webhooks/shopify` verifies `X-Shopify-Hmac-Sha256` against the raw request body, then acts on `X-Shopify-Topic`:
- `app/uninstalled`: disable the connection and delete its credentials, so search stops reaching the store at once. Reconnecting re-enables it.
- `shop/redact` (48h after uninstall): delete the Merchant and everything held for the shop.
- `customers/data_request`, `customers/redact`: acknowledged; no customer data is stored.
Bad signatures get 401; processing errors get 503 so Shopify retries.
### Boundaries
Do not expose this development app to the public internet without auth. Provider responses are untrusted data: every product is normalized and validated before it leaves the provider, and malformed products are skipped. Shopify credentials are encrypted with `CREDENTIALS_KEY` (AES-256-GCM), never stored in connection config.
`apps/web` contains Next.js UI. `packages/commerce` and `packages/database` contain backend implementation. Timestamps in responses are ISO strings. Boundary tests check package subpaths, relative imports, and web aliases: shared packages cannot import apps, commerce cannot import HTTP/MCP frameworks, and web and contracts cannot import backend packages (Next.js's own APIs, such as `next/headers`, are allowed in web).
## Tests
Tests live in their owning workspace's `tests/` directory:
- `apps/api/tests`: HTTP basics (signed-out dashboard, invalid search parameters and product IDs) and the Store connection failure → status table. `integration/`: Shopify connect end to end over HTTP (signed in), MCP OAuth token checks, and JSON-RPC calls to `/api/mcp` plus REST search.
- `apps/web/tests`: API URL helpers, profile response validation, and the results and product pages: their URLs, the per-tab page cache (cached pages, cursor reuse, expiry, restarts), price formatting per currency, purchase-link checks, and response parsing.
- `packages/commerce/tests`: money (per-currency minor units), the Shopify provider (query building, normalization, filters), the page cache (outcomes, stale-age boundaries, lock waiting and ownership, cache failures, per-pod limits; against an in-memory cache), checkout offers, and live federated search (merge, refill, paging, cursors, failures, timeouts). `integration/`: Store connection, Shopify token refresh (including concurrent refresh), and webhooks against PostgreSQL.
- `packages/contracts/tests`: search and product-ID schemas, and results-page and product-page URLs.
- `packages/config/tests`: environment helpers.
- Root `tests/`: repository architecture boundaries only.
Redis integration tests (`packages/commerce/tests/integration/redis-cache.test.ts`) run against a real Redis at `REDIS_URL`: pods racing for one page, slow lock holders, waiter deadlines, lock ownership and expiry, guarded writes, TTLs, and an unreachable Redis. Locally they're skipped without `REDIS_URL`; with `CI` set they're required.
CI (`.github/workflows/ci.yml`) runs on pushes to `main` and on pull requests, with Postgres 17 and Redis 7 service containers: frozen install, migrate, typecheck, lint, `test`, `test:integration`, and the web build.
Run one workspace with `pnpm --filter @shopping-mcp/api test` or `pnpm --filter @shopping-mcp/commerce test`. Root `pnpm test` runs all database-free suites.
For integration tests, point `DATABASE_URL` at a disposable PostgreSQL database, run `pnpm db:migrate`, then `pnpm test:integration`. Workspace integration scripts load the root `.env`; an explicitly set `DATABASE_URL` takes precedence. Most integration tests run in a throwaway schema from `createTestDatabase()`; the MCP HTTP and dashboard tests use the migrated database and create uniquely named records. Shopify is never called: Store connection tests inject a fake token exchange and shop check, token-refresh tests stub `fetch`, search tests use fake providers, and the MCP HTTP test answers every `*.myshopify.com` request itself.
The API runs TypeScript through `tsx`; they do not emit build artifacts. `pnpm build` checks all workspaces and builds the web app.
## Troubleshooting
- **`redirect_uri is not whitelisted`** from Shopify: add `<API_ORIGIN>/api/connections/shopify/callback`, exactly, to the Shopify app's allowed redirect URLs and release the version ([Shopify app setup](#shopify-app-setup)).
- **"Shopify access expired. Reconnect your store."** on `/profile`: the refresh token expired (Shopify tokens are refreshed only when used, so 90 days without a search reaching the store) or the app was uninstalled. The store shows **Disconnected**. Connect the same shop again.
- **"Could not send the sign-in link"**: the API log shows Resend's reason. With the default test sender, Resend only delivers to your own Resend account email; verify a domain in Resend and set `EMAIL_FROM` to an address on it.
- **`CREDENTIALS_KEY is required`**: set it in `.env`; see [Configuration](#configuration).
- **Catalog setup screen / 503**: check `.env`, database health, and `pnpm run db:migrate`.
- **No products**: only products that are active and published to the online store are searchable. Check the store is **Connected** on `/profile`. If a result lists it in `unavailable`, it didn't answer in time.
- **"Shopify app uninstalled. Reconnect your store."** on `/profile`: the app was uninstalled from the shop. Connect it again.
- **Database connection refused**: make sure `DATABASE_URL` matches your PostgreSQL host and port.
- **Migration ledger errors on an old database**: databases built by the old SQL runner must be recreated before `pnpm run db:migrate`.
This server cannot be deployed
Maintenance
ActivityNo data
ResponsivenessOutdated data