# GlassCloud

Cloud server for Rokid Bridge and GlassBridge — handles device linking, MCP tool relay, image storage with gallery, and server-side LLM scene description.

## What It Does

GlassCloud is the backend for Rokid smart glasses apps. It provides:

1. **Device Linking** — QR code or token-based pairing between the web console and the Android app
2. **MCP Tool Relay** — Proxies Gmail and Calendar tool calls from the glasses to Google APIs via WebSocket
3. **Image Gallery** — Receives scene images uploaded from the glasses, stores them with metadata, and displays them in a web gallery
4. **Server-Side LLM** — When an image arrives without a description, automatically calls Gemini or ChatGPT to generate one, then returns it to the app for display on the glasses
5. **Public Gallery Links** — Per-user public share tokens so galleries can be viewed without logging in

```
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Rokid Glasses  │────▶│   GlassCloud    │────▶│  Google APIs    │
│  + Android App  │ WS  │  (This Server)  │     │  Gmail/Calendar │
└─────────────────┘     └─────────────────┘     └─────────────────┘
                               │  │
                    REST API   │  │  Server-side LLM
                               ▼  ▼
                        ┌─────────────────┐
                        │  Web Console    │
                        │  Gallery · LLM  │
                        │  Settings · QR  │
                        └─────────────────┘
```

---

## Quick Start

```bash
# Install dependencies
npm install

# Copy and fill in environment variables
cp .env.example .env
# Edit .env — see Configuration section below

# Development (auto-reload)
npm run dev

# Production build + run
npm run build
npm start
```

Open **http://localhost:3000/console** to access the web console.

---

## Configuration

All environment variables are validated at startup with Zod. The server will print the missing fields and exit if required variables are absent.

### Generating Secrets

```bash
node -e "console.log(require('crypto').randomBytes(32).toString('base64'))"
```

Run this twice — once for `JWT_SECRET`, once for `ENCRYPTION_KEY`.

### Required Variables

| Variable | Description |
|---|---|
| `JWT_SECRET` | Signs device JWTs. Min 32 chars. Must be random and secret. |
| `ENCRYPTION_KEY` | Encrypts OAuth tokens and LLM API keys at rest (AES-256-GCM). Min 32 chars. |
| `GOOGLE_CLIENT_ID` | Google OAuth 2.0 client ID |
| `GOOGLE_CLIENT_SECRET` | Google OAuth 2.0 client secret |
| `GOOGLE_CALLBACK_URL` | OAuth redirect URI — must match what's registered in Google Cloud Console (e.g. `https://yourserver.com/auth/google/callback`) |

### Optional Variables

| Variable | Default | Description |
|---|---|---|
| `PORT` | `3000` | HTTP server port |
| `NODE_ENV` | `development` | `development` \| `production` \| `test` |
| `LOG_LEVEL` | `info` | `trace` \| `debug` \| `info` \| `warn` \| `error` \| `fatal` |
| `DATABASE_PATH` | `./data/glasscloud.db` | SQLite database file path |
| `CORS_ORIGINS` | `http://localhost:3000` | Comma-separated list of allowed CORS origins |
| `RATE_LIMIT_WINDOW_MS` | `60000` | Rate limit window in milliseconds |
| `RATE_LIMIT_MAX_REQUESTS` | `60` | Max requests per window per IP (applies to `/api/*`) |
| `WS_PING_INTERVAL_MS` | `30000` | WebSocket keepalive ping interval |
| `WS_PONG_TIMEOUT_MS` | `10000` | Time to wait for pong before closing connection |

### Example `.env`

```bash
PORT=3000
NODE_ENV=production
LOG_LEVEL=info
DATABASE_PATH=./data/glasscloud.db

JWT_SECRET=<generate with command above>
ENCRYPTION_KEY=<generate with command above>

CORS_ORIGINS=https://yourserver.com

GOOGLE_CLIENT_ID=123456789.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=GOCSPX-xxxxx
GOOGLE_CALLBACK_URL=https://yourserver.com/auth/google/callback
```

### Google OAuth Setup

1. Go to [Google Cloud Console → APIs & Credentials](https://console.cloud.google.com/apis/credentials)
2. Create an **OAuth 2.0 Client ID** (Web application)
3. Add your callback URL to **Authorized redirect URIs**: `https://yourserver.com/auth/google/callback`
4. Enable the **Gmail API** and **Google Calendar API** in the project
5. Add test users under **OAuth consent screen → Test users** (required while app is in Testing mode)

> **Note on Google Scopes:** This app requests `gmail.readonly` and `calendar.events`. For public deployment with more than 100 users, Google requires a CASA security assessment. For private or testing use, keep the app in Testing mode.

---

## LLM Settings (Server-Side Scene Description)

GlassCloud can automatically generate scene descriptions for images uploaded without one (e.g. when the Rokid app is in "Bypass LLM" mode). The generated description is:
- Saved to the database and shown in the gallery
- Returned in the upload API response so the app can display it on the glasses immediately

### Configuration

1. Sign in at `/console` and click **LLM Settings →** in the header
2. Select a provider (**Gemini** or **ChatGPT**) and paste your API key
3. Optionally add custom prompts and select one as active

The default prompt is: *"Describe the scene in this image concisely."*

### API Keys

| Provider | Where to get it |
|---|---|
| **Gemini** | [Google AI Studio → API keys](https://aistudio.google.com/app/apikey) |
| **ChatGPT** | [OpenAI Platform → API keys](https://platform.openai.com/api-keys) |

API keys are encrypted at rest using `ENCRYPTION_KEY` (AES-256-GCM). If no API key is configured, server-side LLM is silently skipped and images with empty descriptions are stored as-is.

---

## Web Console Pages

| URL | Description |
|---|---|
| `/console` | Dashboard: stats, device linking, MCP service management |
| `/console/link` | Generate QR code / link token for pairing a device |
| `/console/gallery` | Image gallery (table view with thumbnails, details, delete) |
| `/console/gallery/photo/:id` | Full-size image with metadata |
| `/console/settings/llm` | LLM provider, API key, and prompt management |
| `/public/gallery/:token` | Public gallery (no login required) |
| `/public/gallery/:token/photo/:id` | Public photo details |

---

## REST API Endpoints

### Device & Auth

| Endpoint | Method | Auth | Description |
|---|---|---|---|
| `/health` | GET | None | Health check with connection stats |
| `/auth/google` | GET | None | Initiate Google OAuth flow |
| `/auth/google/callback` | GET | None | OAuth callback |
| `/api/link/exchange` | POST | None | Exchange link token for device JWT |
| `/api/devices` | GET | Device JWT | List linked devices |
| `/api/devices/:id` | DELETE | Session | Unlink a device |

### Images

| Endpoint | Method | Auth | Description |
|---|---|---|---|
| `/api/images/upload` | POST | Device JWT | Upload image + metadata (multipart/form-data) |
| `/api/images/:id` | GET | Device JWT | Get image metadata by ID |

#### Upload fields

| Field | Type | Required | Description |
|---|---|---|---|
| `image` | file | Yes | JPEG, PNG, or WebP — max 20 MB |
| `scene_description` | string | Yes | Scene description (may be empty string) |
| `device_model` | string | Yes | Device model name |
| `device_instance_id` | string | Yes | Unique device identifier |
| `captured_at` | string | No | ISO 8601 timestamp when image was captured |
| `image_width` | number | No | Image width in pixels |
| `image_height` | number | No | Image height in pixels |
| `llm_provider` | string | No | LLM used on-device (e.g. `Gemini`, `ChatGPT`, `None`) |

If `scene_description` is empty and a server-side LLM is configured, the server will call the LLM and include `llmDescription` in the response:

```json
{
  "imageId": "uuid",
  "originalFilename": "uuid.jpg",
  "thumbnailFilename": "uuid_thumb.jpg",
  "receivedAt": 1714000000000,
  "capturedAt": 1714000000000,
  "llmDescription": "A person walking through a crowded market..."
}
```

### MCP

| Endpoint | Method | Auth | Description |
|---|---|---|---|
| `/api/mcp/services` | GET | Device JWT | List available MCP services and tools |

---

## WebSocket Protocol

Connect at: `ws://localhost:3000/ws?deviceId=YOUR_DEVICE_ID`

### Client → Server

```json
// Execute a tool
{ "type": "tool_execute", "requestId": "uuid", "serverId": "gmail",
  "toolName": "gmail.get_unread", "arguments": { "maxResults": 10 } }

// Link device to user account
{ "type": "link_device", "requestId": "uuid",
  "linkToken": "from-qr-code", "deviceId": "..." }

// Get user account info
{ "type": "get_user_account", "requestId": "uuid", "deviceId": "..." }
```

### Server → Client

```json
// Immediate progress (for voice feedback while tool runs)
{ "type": "tool_progress", "requestId": "uuid",
  "status": "executing", "message": "Checking your emails..." }

// Tool result
{ "type": "tool_result", "requestId": "uuid",
  "result": { "success": true, "content": "You have 3 unread emails..." } }

// Error
{ "type": "error", "requestId": "uuid", "error": "Token expired" }
```

---

## Available MCP Tools

### Gmail

| Tool | Description |
|---|---|
| `gmail.get_unread` | Get unread email count and summaries |
| `gmail.search` | Search emails by query |
| `gmail.get_message` | Get full email content by ID |

### Calendar

| Tool | Description |
|---|---|
| `calendar.get_today` | Get today's events |
| `calendar.get_events` | Get events for N days |
| `calendar.create_event` | Create a new event |

---

## Database Schema

The database is SQLite with WAL mode. Schema is created on startup; new columns are added via idempotent migrations.

```sql
-- Users (from Google OAuth)
users (
  id, google_id, email, display_name, profile_picture_url,
  public_gallery_token,          -- UUID for public gallery share link
  llm_provider,                  -- 'Gemini' or 'ChatGPT' (default: 'Gemini')
  openai_api_key_encrypted,      -- AES-256-GCM encrypted
  gemini_api_key_encrypted,      -- AES-256-GCM encrypted
  active_prompt_id,              -- FK to llm_prompts.id
  created_at, updated_at
)

-- Linked devices
devices (id, user_id, device_name, device_model, last_seen_at, last_heartbeat_at, linked_at, created_at)

-- QR code link tokens (single-use, 5 min expiry)
link_tokens (id, user_id, expires_at, used_at, used_by_device_id, created_at)

-- Encrypted Google OAuth tokens
oauth_tokens (id, user_id, provider, access_token_encrypted, refresh_token_encrypted, ...)

-- Scene images from glasses
glass_images (
  id, device_model, device_instance_id,
  scene_description,             -- May be empty; server-side LLM fills it if blank
  original_filename, thumbnail_filename,
  captured_at, received_at, created_at,
  image_width, image_height,     -- Pixels (nullable)
  user_id,                       -- FK to users.id (nullable for legacy rows)
  llm_provider                   -- Which LLM generated the description
)

-- User-defined LLM prompts
llm_prompts (id, user_id, name, text, created_at)

-- Third-party MCP servers registered via console
third_party_mcp_services (id, name, description, owner_user_id, endpoint_url, auth_type, auth_config_encrypted, tools_json, ...)
```

---

## Directory Structure

```
src/
├── index.ts                    # Entry point — starts HTTP + WebSocket servers
├── config/
│   ├── env.ts                  # Zod environment validation (exits on invalid config)
│   └── mcp-services.ts         # Built-in MCP service definitions
├── server/
│   ├── express.ts              # Express app — middleware, route mounting, error handling
│   ├── sessions.ts             # In-memory session store for console auth
│   └── websocket.ts            # WebSocket server with zombie connection cleanup
├── routes/
│   ├── auth.ts                 # Google OAuth flow
│   ├── console.ts              # Web console dashboard
│   ├── devices.ts              # Device management API
│   ├── gallery.ts              # Image gallery (authenticated + public)
│   ├── health.ts               # Health check
│   ├── images.ts               # Image upload + retrieval API
│   ├── link.ts                 # QR code token generation
│   ├── mcp.ts                  # MCP services API
│   └── settings.ts             # LLM settings page
├── services/
│   ├── auth.service.ts         # OAuth + token refresh mutex
│   ├── device.service.ts       # Device CRUD
│   ├── image.service.ts        # Image save/list/delete + thumbnail generation
│   ├── link.service.ts         # QR code token handling
│   ├── llm.service.ts          # LLM settings CRUD + callLlm() (Gemini / ChatGPT)
│   ├── mcp-proxy.service.ts    # MCP tool execution with input coercion
│   └── mcp-registry.service.ts # Third-party MCP server registry
├── websocket/
│   ├── handler.ts              # Message routing with progress feedback
│   ├── protocol.ts             # Message type definitions
│   └── connection.ts           # Connection tracking
├── db/
│   ├── index.ts                # SQLite connection + WAL mode
│   └── schema.ts               # CREATE TABLE + ALTER TABLE migrations
├── utils/
│   ├── crypto.ts               # AES-256-GCM encrypt/decrypt + token generation
│   ├── logger.ts               # Pino structured logging
│   └── cache.ts                # LRU cache for MCP tool results
└── types/
    ├── api.ts                  # REST API types
    ├── mcp.ts                  # MCP types
    └── websocket.ts            # WebSocket message types
```

Data files are written to `./data/`:
- `./data/glasscloud.db` — SQLite database
- `./data/images/` — Original uploaded images
- `./data/thumbnails/` — 400×400 JPEG thumbnails (generated by Sharp)

---

## Security

- **OAuth tokens** encrypted at rest with AES-256-GCM using `ENCRYPTION_KEY`
- **LLM API keys** encrypted at rest with the same key
- **Device JWTs** signed with `JWT_SECRET` (HS256); verified on every API request
- **Link tokens** stored as SHA-256 hash only; single-use; 5-minute expiry
- **Rate limiting** on all `/api/*` routes (configurable via env)
- **Helmet** security headers on all responses
- **Public gallery tokens** are random UUIDs; regenerating one instantly revokes the previous link

---

## License

MIT