# Pantry Pal: an Alexa+ MCP server for zero-waste cooking

[![CI](https://github.com/karthik-258-code/pantry-pal-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/karthik-258-code/pantry-pal-mcp/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) ![MCP 2025-11-25](https://img.shields.io/badge/MCP-2025--11--25-0f766e) ![Streamable HTTP](https://img.shields.io/badge/transport-Streamable%20HTTP-0f766e)

> "Alexa, what can I cook tonight?"
> "You could make a spinach and chicken rice bowl in 25 minutes. It uses the spinach that expires tomorrow."

Pantry Pal is a **self-hosted MCP server** (Streamable HTTP, MCP spec `2025-11-25`) that gives Alexa+, or any MCP client, a memory of your kitchen. It tracks what you have and when it expires, uses **Amazon Bedrock** to suggest recipes that use the soonest-expiring food first, and puts missing ingredients straight on your shopping list.

**Hackathon entry:** Primary track **Alexa+ (MCP server + MCP App)** · Mini challenges **AWS Builder** (Bedrock, DynamoDB, Lambda) and **Open Source** ([mcp-bedrock-bridge](https://github.com/karthik-258-code/mcp-bedrock-bridge))

**Live endpoint (AWS Lambda, Sydney):** `https://5c2h3n7uby2kadki2e37pnftiy0hmwkm.lambda-url.ap-southeast-2.on.aws/mcp`, Streamable HTTP, bearer key per household. Health: [`/health`](https://5c2h3n7uby2kadki2e37pnftiy0hmwkm.lambda-url.ap-southeast-2.on.aws/health). Judges: the key is in the Devpost submission notes.

## Architecture

```
  Alexa+ / VS Code Copilot Chat / Claude / ChatGPT / MCP Inspector / Pantry Pal web app
        │  MCP 2025-11-25 over Streamable HTTP (JSON-RPC), bearer key or /h/<key>/mcp
        ▼
  AWS Lambda (ap-southeast-2) + Function URL          local: node src/server.ts
  ├─ Express app (src/app.ts)  ── same code ──►  http://127.0.0.1:3000/mcp
  ├─ 17 MCP tools + ui:// MCP App resource (src/mcp.ts, apps/)
  ├─ household context per request (src/context.ts)
  ├─ Amazon Bedrock, Converse API, Nova Lite (us-east-1): recipes, plans, vision, tool use
  └─ Amazon DynamoDB (pantry-pal table): one item per household      local: data/*.json
```

The same Express app runs locally and on Lambda. Each request resolves a household from its key, and every tool reads and writes that household's document. Tools answer with spoken text **and** `structuredContent`, so voice hosts and screen hosts (MCP Apps) share one server.

## How it works

```
 Alexa+ / MCP Inspector / demo agent
            │  JSON-RPC over Streamable HTTP (POST /mcp)
            ▼
 ┌───────────────────────────┐      ┌──────────────────────────────┐
 │ Pantry Pal MCP server     │─────▶│ Amazon Bedrock (Converse API) │
 │ Express + MCP TS SDK      │      │ recipe generation             │
 │ Origin check + Bearer auth│      └──────────────────────────────┘
 └────────────┬──────────────┘
              ▼
      data/pantry.json (pantry + shopping list)
```

- `src/server.ts`: Express app exposing `POST /mcp` with the SDK's `StreamableHTTPServerTransport` in stateless mode. Validates `Origin` and an optional bearer token.
- `src/mcp.ts`: the 17 MCP tools with voice-friendly text plus `structuredContent`, and the `ui://` MCP App resource.
- `apps/`: the MCP Apps View (pantry, recipes, plan, order receipt) that hosts render inline.
- `src/bedrock.ts`: Amazon Bedrock Runtime `Converse` call that returns structured recipes.
- `src/store.ts`: JSON-file storage, sorted by expiry.
- `src/demo-agent.ts`: a **simulated Alexa+** terminal assistant. A Bedrock model plans and calls the MCP tools through the MCP client SDK.
- `src/seed.ts`: demo data.

## MCP tools

| Tool | Example utterance |
|---|---|
| `scan_groceries` (Bedrock vision) | "Scan fridge.jpg" / "Here's my grocery receipt" |
| `update_household_profile` | "My son is allergic to peanuts, we're a family of 4" |
| `get_household_profile` | "What do you remember about us?" |
| `rate_recipe` | "The kids loved the frittata" / "That was too spicy" |
| `plan_meals` (Bedrock) | "Plan dinners for the next 5 days" |
| `get_meal_plan` | "What's for dinner tomorrow?" |
| `prepare_grocery_order` | "Order it" (prices the list and asks for confirmation) |
| `confirm_grocery_order` | "Yes, place it" (simulated checkout, no real payment) |
| `receive_grocery_order` | "My groceries arrived" |
| `add_pantry_item` | "I bought 6 eggs that expire in 10 days" |
| `use_pantry_item` | "I used 200 grams of chicken" |
| `list_pantry` | "What's about to go bad?" |
| `suggest_recipes` (Bedrock) | "What can I make for a quick vegetarian dinner?" |
| `add_missing_ingredients` | "Add what I'm missing for the frittata" |
| `add_to_shopping_list` | "Add bread and butter to my list" |
| `get_shopping_list` | "What's on my shopping list?" |
| `mark_shopping_done` | "I bought everything" |

## Quick start

Requirements: Node.js 20+, an AWS account with Amazon Bedrock access.

```bash
npm install
cp .env.example .env          # set AWS_REGION / BEDROCK_MODEL_ID if needed
aws configure                 # or export AWS_PROFILE=...
npm run seed                  # load demo pantry
npm run dev                   # MCP server on http://127.0.0.1:3000/mcp
```

Check it is up with `curl http://127.0.0.1:3000/health`. The response includes the MCP protocol version that the SDK negotiates.

### Interactive UI in any MCP Apps host (VS Code, Claude, ChatGPT, ...)

Pantry Pal is also an **MCP App**: seven tools carry `_meta.ui.resourceUri`, pointing at the `ui://pantry-pal/app.html` resource (`text/html;profile=mcp-app`). Compliant hosts render it inline in the chat, and the View calls other tools through the host: tap **Used it** on a pantry row, **Add missing to list** on a recipe, or **Place order** on the receipt (`confirm_grocery_order` only runs on that click).

```bash
npm run build:apps        # bundles apps/ into apps/dist/mcp-app.html (one-time, and after editing apps/)
npm run dev               # MCP server
```

- **VS Code (local):** this repo ships `.vscode/mcp.json`. Open the folder, open Chat, choose **Agent** mode, make sure the `pantry-pal` server is started (Command Palette: *MCP: List Servers*), then ask *"what's in my pantry?"*. The pantry View renders inline.
- **VS Code (hosted):** Command Palette: *MCP: Add Server...* > HTTP > `https://<id>.lambda-url.<region>.on.aws/h/<key>/mcp` > save to **User** settings (keeps the key out of the repo). VS Code's prompted-header config (`${input:...}` in `headers`) failed silently for us, so the capability URL is the reliable route.
- **Reference host:** `git clone https://github.com/modelcontextprotocol/ext-apps && cd ext-apps && npm install && cd examples/basic-host && SERVERS='["http://127.0.0.1:3000/mcp"]' npm start`, then open http://localhost:8080, pick `list_pantry` and click **Call Tool**.

The View source is in [`apps/`](apps/) (vanilla TypeScript, `@modelcontextprotocol/ext-apps`, bundled by Vite into a single HTML file). Tools return `structuredContent` with a `view` field that the single View switches on.

### Try it with MCP Inspector

```bash
npm run inspector
```

Choose **Transport: Streamable HTTP**, URL `http://127.0.0.1:3000/mcp`, click **Connect**, then **Tools > List Tools** and run `suggest_recipes`.

### Try the simulated Alexa+ assistant

With the server running, in a second terminal:

```bash
npm run agent
```

```
You: what's about to expire?
  [MCP] list_pantry {"expiring_within_days":3}
Alexa: Your spinach expires tomorrow, and the chicken thighs and tomatoes within 3 days.
You: what can I cook with those tonight?
  [MCP] suggest_recipes {"meal":"dinner"}
Alexa: ...
```

### Try the Echo Show-style web app (simulated Alexa+)

With the MCP server running, in a second terminal:

```bash
npm run web
```

Open http://127.0.0.1:3001 in **Chrome or Edge**. Tap the mic and talk, or type. Use the camera button to scan a fridge or receipt photo. Replies are spoken with **Amazon Polly**, and recipes, meal plans and orders show as cards. The right-hand panel shows every MCP tool call live.

### Scan a fridge photo or receipt

Put a JPEG or PNG under 3.75 MB in the `photos/` folder, for example `photos/fridge.jpg`, then tell the agent `scan fridge.jpg`. Bedrock vision identifies the food, estimates expiry dates and adds everything to the pantry. For receipts, say `scan receipt.jpg, it's a receipt`.

### Household memory

Allergies, diet, household size, dislikes and recipe ratings are saved in `data/pantry.json` and survive restarts. Every recipe request uses them, and any generated recipe that mentions an allergen is filtered out as a safety net.

### Plan the week and check out

`plan_meals` asks Bedrock for a multi-day plan that cooks the soonest-expiring food first and reuses bought ingredients across days, then adds what's missing to the shopping list. `prepare_grocery_order` prices the list at a simulated store (`src/catalog.ts`) and returns a quote. `confirm_grocery_order` places it only after the user says yes, and quotes expire after 30 minutes. `receive_grocery_order` stocks the pantry on delivery. **No real purchases are made.**

### Households and hosted storage

Every request runs in a **household context** (`src/context.ts`). Locally there's one household (`default`) stored in `data/pantry.json`. For a shared server, set `HOUSEHOLD_KEYS` so each bearer key maps to a household, and `STORE=dynamodb` so data lives in **Amazon DynamoDB** (one item per household, `pk = household#<id>`). Pantry, shopping list, memory, plans, orders and recipe follow-ups are all isolated per household; nothing else in the code changes.

### Deploy to AWS Lambda (public HTTPS endpoint)

The server is stateless, so it runs as a single Lambda function behind a Function URL (`src/lambda.ts`, via `serverless-http`). On Lambda, POST responses use JSON instead of SSE (`enableJsonResponse`), credentials come from the function's IAM role, and data lives in DynamoDB.

1. **DynamoDB:** table `pantry-pal`, partition key `pk` (String).
2. **Lambda:** create a function (Node.js 22, x86_64). Handler `dist/lambda.handler`, timeout 60 s, memory 512 MB. Environment: `STORE=dynamodb`, `DYNAMODB_TABLE=pantry-pal`, `BEDROCK_REGION=us-east-1`, `BEDROCK_MODEL_ID=us.amazon.nova-lite-v1:0`, `HOUSEHOLD_KEYS=<key>:<household>,...`. Add a **Function URL** (auth type NONE; the app checks bearer keys itself).
3. **Execution role policy:** `bedrock:InvokeModel` on the model/inference profile and `dynamodb:GetItem`/`dynamodb:PutItem` on the table.
4. **GitHub Actions:** set repository **variables** `LAMBDA_FUNCTION_NAME` and `AWS_REGION`, and **secrets** `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` for a user allowed to `lambda:UpdateFunctionCode`. `.github/workflows/deploy-lambda.yml` then deploys every push to `main`.
5. Check `https://<id>.lambda-url.<region>.on.aws/health`, then connect any MCP host to `https://<id>.lambda-url.<region>.on.aws/mcp` with `Authorization: Bearer <key>`. Hosts that can't send headers (ChatGPT connectors, quick VS Code "Add Server") can use the capability URL `https://<id>.lambda-url.<region>.on.aws/h/<key>/mcp` instead; treat that URL as a secret.

### Expose it from your laptop instead

The server binds to `127.0.0.1` by default. To reach it from a hosted client without deploying, put it behind an HTTPS tunnel and **set `MCP_AUTH_TOKEN`** or `HOUSEHOLD_KEYS`. Clients must then send `Authorization: Bearer <token>`.

## AWS integration (AWS Builder)

| Service | Where | What for |
|---|---|---|
| **Amazon Bedrock Runtime, Converse API** | `src/bedrock.ts` `generateRecipes()` | Turns the expiry-sorted pantry plus constraints (meal, servings, diet, time) into structured JSON recipes with used and missing ingredients |
| **Amazon Bedrock Runtime, Converse tool use** | `src/agent.ts` | The simulated Alexa+ assistant: MCP tool schemas are mapped to Bedrock `toolConfig`, and the model decides which MCP tools to call, in a loop of up to 8 steps |
| **Amazon Bedrock Runtime, Converse with images** | `src/bedrock.ts` `scanGroceries()` | Nova vision reads fridge, shelf and receipt photos and estimates expiry dates |
| **Amazon Bedrock Runtime, Converse** | `src/bedrock.ts` `generateMealPlan()` | Multi-day, waste-aware meal plans |
| **Amazon DynamoDB** | `src/persistence.ts` | Per-household persistence for the hosted, multi-tenant server (`STORE=dynamodb`) |
| **AWS Lambda + Function URL** | `src/lambda.ts`, `.github/workflows/deploy-lambda.yml` | Hosts the MCP server (Streamable HTTP, JSON responses) with IAM-role credentials, deployed from GitHub Actions |
| **Amazon Polly (neural TTS)**, optional | `src/web.ts` `/api/tts` | Alexa-style spoken replies in the web app (`POLLY_VOICE`, default Joanna). If Polly is unavailable (for example, blocked by an organization SCP), the app falls back to the most natural browser voice. |

The default model is `us.amazon.nova-lite-v1:0` (Amazon Nova Lite through a cross-region inference profile). You can switch to any Converse-compatible model with `BEDROCK_MODEL_ID`. The IAM identity needs `bedrock:InvokeModel` on that model or inference profile, and `polly:SynthesizeSpeech` for the web app's voice. Without Polly access, the web app falls back to the browser's built-in voice.

## Configuration

| Variable | Default | Description |
|---|---|---|
| `AWS_REGION` | `us-east-1` | Bedrock region |
| `BEDROCK_MODEL_ID` | `us.amazon.nova-lite-v1:0` | Converse-compatible model ID |
| `HOST` / `PORT` | `127.0.0.1` / `3000` | Bind address |
| `MCP_AUTH_TOKEN` | (unset) | Require `Authorization: Bearer <token>` |
| `ALLOWED_ORIGINS` | (unset) | Comma-separated browser origins allowed on `/mcp` |
| `STORE` | `file` | `file` (local JSON) or `dynamodb` (hosted, multi-household) |
| `DATA_FILE` | `data/pantry.json` | Storage location in `file` mode |
| `DYNAMODB_TABLE` / `DYNAMODB_REGION` | `pantry-pal` / `AWS_REGION` | Table (String partition key `pk`) used in `dynamodb` mode |
| `HOUSEHOLD_KEYS` | (unset) | `key:household,key:household`. Each bearer key gets its own pantry, memory and orders |
| `MCP_URL` | `http://127.0.0.1:3000/mcp` | Server URL used by the demo agent |

## Open-source spin-off: mcp-bedrock-bridge

[`bridge/`](bridge/), published at [karthik-258-code/mcp-bedrock-bridge](https://github.com/karthik-258-code/mcp-bedrock-bridge), is a standalone, MIT-licensed library that lets any MCP server's tools be used with the Amazon Bedrock Converse API. It covers tool schema mapping, result conversion and a tested agent loop. It's published as its own repository for the Open Source mini challenge.

## License

[MIT](LICENSE)