# Enhanced AI Chat Plan
**Status**: ⏸️ PAUSED (Phase 1 complete, Phases 2-4 deferred)
Add dynamic model registry, provider detection, and embedded desktop chat to the MCP server template.
**Project**: `/home/jez/Documents/mcp-server-template-cloudflare`
**Created**: 2026-01-01
**Updated**: 2026-01-02
---
## Goals
1. **Dynamic Model Registry** - Fetch current models from OpenRouter instead of hardcoding
2. **Smart Filtering** - Only show tool-capable, recent models from configured providers
3. **Provider Detection** - Know which providers have BYOK keys configured
4. **Embedded Desktop Chat** - Side panel chat for desktop, bubble for mobile
5. **Extended Providers** - Support more AI Gateway providers (DeepSeek, Mistral, etc.)
---
## Current State
| Component | Status |
|-----------|--------|
| AI Chat | ✅ Working via floating bubble |
| Providers | ✅ Dynamic from OpenRouter API |
| Models | ✅ Fetched from OpenRouter, filtered for tool support |
| Provider detection | None - shows all providers |
| Desktop UX | Same as mobile (bubble only) |
| Conversation Memory | ✅ D1-backed persistent storage |
| Internal Agent | ✅ Optional `ask_agent` tool with Workers AI gatekeeper |
---
## Phase 1: Dynamic Model Registry ✅ COMPLETE
**Goal**: Fetch current models from OpenRouter API with smart filtering.
**Implemented**: 2026-01-01 - Models now fetched dynamically from OpenRouter API with 24h KV cache.
### New Endpoint
`GET /api/admin/models`
**Query params**:
- `provider` (optional) - Filter to specific provider
- `refresh` (optional) - Force cache refresh
**Response**:
```json
{
"models": [
{
"id": "anthropic/claude-3-5-sonnet-20241022",
"name": "Claude 3.5 Sonnet",
"provider": "anthropic",
"contextLength": 200000,
"created": "2024-10-22",
"pricing": { "prompt": 0.003, "completion": 0.015 }
}
],
"providers": ["openai", "anthropic", "google", "groq", "deepseek", "mistralai"],
"cached": true,
"cachedAt": "2026-01-01T00:00:00Z"
}
```
### Filtering Logic
```typescript
// Filter criteria
const isToolCapable = model.supported_parameters?.includes('tools');
const isRecent = model.created > (Date.now() / 1000) - (180 * 24 * 60 * 60);
const isGatewayProvider = GATEWAY_PROVIDERS.includes(provider);
// Only include if ALL criteria met
return isToolCapable && isRecent && isGatewayProvider;
```
### AI Gateway Provider Mapping
OpenRouter uses different provider names than AI Gateway:
| OpenRouter | AI Gateway | Notes |
|------------|------------|-------|
| `openai` | `openai` | Same |
| `anthropic` | `anthropic` | Same |
| `google` | `google-ai-studio` | Different! |
| `groq` | `groq` | Same |
| `mistralai` | `mistral` | Different! |
| `deepseek` | `deepseek` | Same |
| `cohere` | `cohere` | Same |
| `meta-llama` | N/A | Use via Groq/Together |
### Files to Create/Modify
| File | Action |
|------|--------|
| `src/lib/ai/openrouter.ts` | NEW: OpenRouter API client |
| `src/lib/ai/providers.ts` | UPDATE: Add provider mapping |
| `src/admin/routes.ts` | UPDATE: Add `/api/admin/models` endpoint |
| `src/admin/ui.ts` | UPDATE: Fetch models dynamically |
### Implementation
**`src/lib/ai/openrouter.ts`**:
```typescript
const OPENROUTER_API = 'https://openrouter.ai/api/v1/models';
const CACHE_KEY = 'openrouter:models';
const CACHE_TTL = 24 * 60 * 60; // 24 hours
// Gateway-supported providers
const GATEWAY_PROVIDERS = [
'openai', 'anthropic', 'google', 'groq',
'mistralai', 'deepseek', 'cohere'
];
export async function fetchModels(kv: KVNamespace, forceRefresh = false) {
// Check cache first
if (!forceRefresh) {
const cached = await kv.get(CACHE_KEY, 'json');
if (cached) return cached;
}
// Fetch from OpenRouter
const response = await fetch(OPENROUTER_API, {
headers: { 'User-Agent': 'MCP-Server-Template/1.0' }
});
const data = await response.json();
// Filter and transform
const cutoff = Date.now() / 1000 - (180 * 24 * 60 * 60);
const models = data.data
.filter(m => {
const provider = m.id.split('/')[0];
const hasTools = m.supported_parameters?.includes('tools');
const isRecent = m.created > cutoff;
return hasTools && isRecent && GATEWAY_PROVIDERS.includes(provider);
})
.map(m => ({
id: m.id,
name: m.name,
provider: m.id.split('/')[0],
contextLength: m.context_length,
created: new Date(m.created * 1000).toISOString().split('T')[0],
pricing: {
prompt: parseFloat(m.pricing.prompt),
completion: parseFloat(m.pricing.completion),
},
}));
// Cache result
await kv.put(CACHE_KEY, JSON.stringify({
models,
providers: [...new Set(models.map(m => m.provider))],
cachedAt: new Date().toISOString(),
}), { expirationTtl: CACHE_TTL });
return { models, providers: [...new Set(models.map(m => m.provider))] };
}
```
---
## Phase 2: Provider Detection
**Goal**: Only show providers that have BYOK keys configured.
### Options Evaluated
| Option | Pros | Cons |
|--------|------|------|
| **A: Env var config** | Simple, explicit | Manual maintenance |
| **B: Secrets Store API** | Automatic | Requires account API token |
| **C: Try-and-fail** | No config needed | Slow, wasteful |
### Recommended: Option A + B Hybrid
1. **Primary**: Use Cloudflare API to list secrets with `ai_gateway` scope
2. **Fallback**: Env var `ENABLED_PROVIDERS` if API token not configured
### Secrets Store API Integration
```typescript
// List secrets with ai_gateway scope
async function getConfiguredProviders(env: Env): Promise<string[]> {
// If no CF API token, fall back to env var
if (!env.CF_API_TOKEN) {
return (env.ENABLED_PROVIDERS || 'cloudflare').split(',');
}
// Query Secrets Store
const response = await fetch(
`https://api.cloudflare.com/client/v4/accounts/${env.CF_ACCOUNT_ID}/secrets_store/stores`,
{ headers: { 'Authorization': `Bearer ${env.CF_API_TOKEN}` } }
);
const data = await response.json();
// Parse secret names to extract provider names
// e.g., "ai_gateway_openai_key" -> "openai"
// ...
}
```
### New Env Vars
| Variable | Required | Description |
|----------|----------|-------------|
| `CF_API_TOKEN` | Optional | Cloudflare API token with Secrets Store read |
| `ENABLED_PROVIDERS` | Optional | Fallback: comma-separated provider list |
### Files to Modify
| File | Action |
|------|--------|
| `src/lib/ai/providers.ts` | Add `getConfiguredProviders()` |
| `src/admin/routes.ts` | Include enabled providers in `/api/admin/models` |
| `src/types.ts` | Add new env vars to `Env` interface |
---
## Phase 3: Embedded Desktop Chat
**Goal**: Side panel chat for desktop (>1024px), floating bubble for mobile.
### Layout Design
```
┌──────────────────────────────────────────────────────────────────┐
│ Header (sticky) │
├────────────────────────────────────┬─────────────────────────────┤
│ │ │
│ Main Content │ Chat Panel (400px) │
│ - Server Info │ ┌─────────────────────────┐│
│ - Tools List │ │ Provider: [Anthropic ▼] ││
│ - Resources │ │ Model: [claude-3.5... ▼]││
│ - Prompts │ ├─────────────────────────┤│
│ - Tokens │ │ ││
│ │ │ Chat Messages ││
│ │ │ ││
│ │ │ - User: test hello ││
│ │ │ - AI: [tool call...] ││
│ │ │ - AI: The hello tool...││
│ │ │ ││
│ │ ├─────────────────────────┤│
│ │ │ [Message input...] [→] ││
│ │ └─────────────────────────┘│
│ │ │
└────────────────────────────────────┴─────────────────────────────┘
Mobile (<1024px): Floating bubble in bottom-right, opens modal
```
### Responsive Breakpoints
```css
/* Desktop: side panel */
@media (min-width: 1024px) {
.admin-layout {
display: grid;
grid-template-columns: 1fr 400px;
gap: 1.5rem;
}
.chat-panel {
position: sticky;
top: 80px;
height: calc(100vh - 100px);
}
.chat-bubble { display: none; }
}
/* Mobile: floating bubble */
@media (max-width: 1023px) {
.chat-panel { display: none; }
.chat-bubble { display: flex; }
}
```
### Chat Panel Features
- **Header**: Provider dropdown, model dropdown, "New" button
- **Messages**: Scrollable, auto-scroll on new message
- **Tool calls**: Collapsible with syntax-highlighted JSON
- **Input**: Multiline textarea, send on Enter (Shift+Enter for newline)
- **Streaming**: Real-time SSE display
### Files to Modify
| File | Action |
|------|--------|
| `src/admin/ui.ts` | Major refactor for new layout |
### UI Component Structure
```typescript
// New component structure
function renderAdminPage() {
return `
<div class="admin-layout">
<main class="main-content">
${renderServerInfo()}
${renderTools()}
${renderResources()}
${renderPrompts()}
${renderTokens()}
</main>
<aside class="chat-panel">
${renderChatPanel()}
</aside>
</div>
<!-- Mobile only -->
<div class="chat-bubble">...</div>
<div class="chat-modal">...</div>
`;
}
```
---
## Phase 4: Extended Providers
**Goal**: Support more AI Gateway providers.
### New Providers to Add
| Provider | OpenRouter ID | AI Gateway ID | Tool Support |
|----------|---------------|---------------|--------------|
| DeepSeek | `deepseek` | `deepseek` | ✅ Yes |
| Mistral | `mistralai` | `mistral` | ✅ Yes |
| Cohere | `cohere` | `cohere` | ✅ Yes |
| xAI (Grok) | `x-ai` | `xai` | ✅ Yes |
### Provider ID Mapping
```typescript
// OpenRouter ID -> AI Gateway ID
const PROVIDER_MAP: Record<string, string> = {
'openai': 'openai',
'anthropic': 'anthropic',
'google': 'google-ai-studio',
'groq': 'groq',
'mistralai': 'mistral',
'deepseek': 'deepseek',
'cohere': 'cohere',
'x-ai': 'xai',
};
// Reverse for UI display
const PROVIDER_DISPLAY: Record<string, string> = {
'openai': 'OpenAI',
'anthropic': 'Anthropic',
'google-ai-studio': 'Google AI Studio',
'groq': 'Groq',
'mistral': 'Mistral AI',
'deepseek': 'DeepSeek',
'cohere': 'Cohere',
'xai': 'xAI (Grok)',
};
```
### Files to Modify
| File | Action |
|------|--------|
| `src/lib/ai/providers.ts` | Add mapping constants |
| `src/lib/ai/index.ts` | Use mapping in Compat endpoint calls |
---
## Implementation Order
| Phase | Effort | Status |
|-------|--------|--------|
| Phase 1: Model Registry | ~45 min | ✅ Complete |
| Phase 2: Provider Detection | ~30 min | ⏸️ Pending |
| Phase 3: Desktop Chat UI | ~60 min | ⏸️ Pending |
| Phase 4: Extended Providers | ~15 min | ⏸️ Pending |
**Recommended order**: 1 → 4 → 2 → 3
Phase 4 is quick and adds value immediately after Phase 1.
Phase 3 is the largest and can be done last.
---
## API Reference
### OpenRouter Models API
```
GET https://openrouter.ai/api/v1/models
```
No auth required. Returns all models with:
- `id`: `provider/model-name`
- `name`: Display name
- `created`: Unix timestamp
- `context_length`: Max tokens
- `supported_parameters`: Array including 'tools' if supported
- `pricing`: `{ prompt, completion }` per token
### Cloudflare Secrets Store API
```
GET https://api.cloudflare.com/client/v4/accounts/{account_id}/secrets_store/stores
Authorization: Bearer {api_token}
```
Requires API token with "Secrets Store Read" permission.
### AI Gateway Compat Endpoint
```
POST https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/compat/chat/completions
cf-aig-authorization: Bearer {gateway_token}
Content-Type: application/json
{
"model": "provider/model-name",
"messages": [...],
"tools": [...]
}
```
---
## Testing Checklist
### Phase 1
- [ ] `/api/admin/models` returns filtered models
- [ ] Cache works (subsequent calls fast)
- [ ] `?refresh=true` forces fresh fetch
- [ ] Graceful fallback if OpenRouter down
### Phase 2
- [ ] Providers filtered to configured only
- [ ] Env var fallback works
- [ ] Secrets Store API integration (if implemented)
### Phase 3
- [ ] Desktop shows side panel
- [ ] Mobile shows bubble
- [ ] Chat persists across tab navigation
- [ ] Tool calls render correctly
- [ ] Streaming works
### Phase 4
- [ ] DeepSeek models appear and work
- [ ] Mistral models appear and work
- [ ] Provider ID mapping correct
---
## Open Questions
1. **Cache invalidation**: Should admin be able to force refresh models?
2. **Model sorting**: By date? By provider? By popularity?
3. **Pricing display**: Show cost per 1K tokens in UI?
4. **Chat persistence**: Keep chat state on page refresh?
5. **Multi-session**: Support multiple chat sessions?
---
## References
- [OpenRouter API](https://openrouter.ai/api/v1/models)
- [AI Gateway Compat Endpoint](https://developers.cloudflare.com/ai-gateway/usage/chat-completion/)
- [Secrets Store API](https://developers.cloudflare.com/api/resources/secrets_store/)
- [AI Gateway BYOK](https://developers.cloudflare.com/ai-gateway/configuration/bring-your-own-keys/)
