> **Public reference fork** — sanitized copy of Elephant's internal investors-mcp for [X Engagement Reply Agent](https://github.com/prismteam-ai/x-engagement-reply-agent) candidates. Operational watchlists, scraped data, and internal profiles are excluded. See [docs/public-reference.md](docs/public-reference.md).

# Investors MCP Server

A Next.js-based MCP (Model Context Protocol) server that provides semantic search over investor content using Upstash Vector and Vercel Blob storage.

**Uses `mcp-handler`**

## Features

- **Semantic Search**: Query investor content using natural language via `queryInvestorContent` tool
- **Content Ingestion**: Add new paragraphs via `addInvestorParagraph` tool (authenticated)
- **Author Filtering**: Optionally filter results by investor/author name
- **Temporal Filtering**: Optionally filter by publication date (`dateFrom`/`dateTo`), original content creation datetime (`contentCreatedAtFrom`/`contentCreatedAtTo`), and indexed record creation datetime (`createdAtFrom`/`createdAtTo`)
- **RAG-Ready**: Returns full document content from Vercel Blob for retrieval-augmented generation
- **Streamable HTTP Transport**: Built for modern MCP clients

## Deploy to Vercel

[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https://github.com/YOUR_USERNAME/investors-mcp)

### Quick Deploy Steps

1. **Deploy the project** to Vercel via the button above or by connecting your repo

2. **Add Upstash Vector** from Vercel Marketplace:
   - Go to your project's **Storage** tab in Vercel dashboard
   - Click **Connect Store** → **Create New** → **Upstash Vector**
   - **Important**: Create an index with **3072 dimensions** (for `text-embedding-3-large`)
   - This automatically sets `UPSTASH_VECTOR_REST_URL` and `UPSTASH_VECTOR_REST_TOKEN`

3. **Add Vercel Blob** storage:
   - In the **Storage** tab, click **Connect Store** → **Create New** → **Blob**
   - This automatically sets `BLOB_READ_WRITE_TOKEN`

4. **Add AI Gateway API Key**:
   - Go to [Vercel AI Gateway](https://vercel.com/dashboard/ai/api-keys) and create an API key
   - In your project's **Settings** → **Environment Variables**, add:
     - `AI_GATEWAY_API_KEY` = your API key

5. **Redeploy** to pick up the new environment variables

## Environment Variables

| Variable | Source | Description |
|----------|--------|-------------|
| `AI_GATEWAY_API_KEY` | [Vercel AI Gateway](https://vercel.com/dashboard/ai/api-keys) | Required for embeddings |
| `UPSTASH_VECTOR_REST_URL` | Vercel Marketplace (auto) | Upstash Vector endpoint |
| `UPSTASH_VECTOR_REST_TOKEN` | Vercel Marketplace (auto) | Upstash Vector auth token |
| `BLOB_READ_WRITE_TOKEN` | Vercel Storage (auto) | Vercel Blob access token |
| `MCP_WRITE_TOKEN` | Self-generated | Optional. Bearer token for write operations (`addInvestorParagraph`) |
| `X_BEARER_TOKEN` | X Developer Portal | Optional. Enables authenticated X API article/full-text enrichment in ingestion scripts |
| `AUTOMATION_TRIGGER_TOKEN` | Self-generated | Bearer token for `/api/automation/monitor-x` trigger endpoint |
| `CRON_SECRET` | Self-generated | Optional. If set, Vercel Cron can authenticate to trigger endpoint |
| `ASANA_ACCESS_TOKEN` | Asana | Required to create tasks from monitored posts |
| `ASANA_PROJECT_GID` | Asana | Project where monitored-post tasks are created |
| `ASANA_ASSIGNEE_GID` | Asana | Optional default assignee for created tasks |
| `ASANA_ARTICLE_THRESHOLD_ASSIGNEE_GID` | Asana | Optional assignee when a post meets the article recommendation threshold |
| `ASANA_SECTION_GID` | Asana | Optional section to place newly created tasks into |
| `ASANA_WORKSPACE_GID` | Asana | Optional workspace override for task creation |
| `ASANA_SIMILARITY_SCORE_CUSTOM_FIELD_GID` | Asana | Optional shared number custom field GID for both parent tasks and recommendation subtasks |
| `ASANA_TASK_SIMILARITY_SCORE_CUSTOM_FIELD_GID` | Asana | Optional parent-task-specific number custom field GID; overrides the shared GID for parent tasks |
| `ASANA_SUBTASK_SIMILARITY_SCORE_CUSTOM_FIELD_GID` | Asana | Optional subtask-specific number custom field GID; overrides the shared GID for recommendation subtasks |
| `AUTOMATION_WATCH_AUTHORS` | Optional list | Optional comma-separated subset of watched authors |
| `RAG_AUTOREPLY_MODEL` | Optional | Legacy draft-model override (currently unused by Asana task creation) |
| `X_MONITOR_TWSTALKER_ORIGIN` | Optional | TwStalker base URL override (default `https://twstalker.com`) |

When the similarity-score custom field env vars are configured, the Asana create flow writes the raw vector similarity value into Asana:
- parent task: highest recommended-article raw similarity score
- recommendation subtask: that subtask's article raw similarity score

### Generating MCP_WRITE_TOKEN

To enable write operations, generate a secure random token:

```bash
# Linux/macOS
openssl rand -hex 32

# Or with Node.js
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```

Add the generated token as `MCP_WRITE_TOKEN` in your Vercel Environment Variables. Clients must include this token in the `Authorization: Bearer <token>` header when calling write tools.

### Index Requirements

Your Upstash Vector index **must be 3072 dimensions** to match `openai/text-embedding-3-large` embeddings.

## Available Tools

### `echo`
Simple echo tool for testing connectivity.

**Parameters:**
- `message` (string, required): Message to echo back

### `queryInvestorContent`
Query relevant investor content using semantic search.

**Parameters:**
- `query` (string, required): The search query to find relevant investor content
- `author` (string, optional): Author name to filter results (e.g., "Paul Graham", "Marc Andreessen")
- `company` (string, optional): Company name to filter results (e.g., "16z", "Y Combinator")
- `dateFrom` (string, optional): Publication date lower bound, inclusive (`YYYY-MM-DD`)
- `dateTo` (string, optional): Publication date upper bound, inclusive (`YYYY-MM-DD`)
- `contentCreatedAtFrom` (string, optional): Original content creation datetime lower bound, inclusive (ISO 8601 datetime with timezone)
- `contentCreatedAtTo` (string, optional): Original content creation datetime upper bound, inclusive (ISO 8601 datetime with timezone)
- `createdAtFrom` (string, optional): Record creation-time lower bound, inclusive (ISO 8601 datetime with timezone)
- `createdAtTo` (string, optional): Record creation-time upper bound, inclusive (ISO 8601 datetime with timezone)
- `topK` (number, optional): Number of results to return (1-20, default: 5)

**Response Structure:**
```json
{
  "query": "what makes a good startup idea?",
  "author": "Paul Graham",
  "authorNormalized": "paulgraham",
  "company": "Y Combinator",
  "topK": 5,
  "matchCount": 3,
  "matches": [
    {
      "id": "essay-123",
      "score": 0.89,
      "key": "investors/paulgraham/essay-123.json",
      "metadata": {
        "blobKey": "investors/paulgraham/essay-123.json",
        "authorNormalized": "paulgraham",
        "company": "Y Combinator",
        "date": "2012-11-01"
      },
      "blob": {
        "title": "How to Get Startup Ideas",
        "content": "...",
        "author": "Paul Graham",
        "company": "Y Combinator",
        "date": "2012-11-01"
      }
    }
  ]
}
```

### `addInvestorParagraph`
Add a new investor paragraph to the knowledge base. **Requires authentication** with `write:investors` scope.

**Authentication:**
Include `Authorization: Bearer <MCP_WRITE_TOKEN>` header in your MCP client request.

**Parameters:**
- `content` (string, required): The paragraph text content (10-10000 characters)
- `author` (string, required): The primary author name (e.g., "Paul Graham")
- `company` (string, optional): Company/organization name (e.g., "16z", "Y Combinator")
- `date` (string, optional): Publication date in ISO format (YYYY-MM-DD)
- `contentCreatedAt` (string, optional): Original content creation datetime in ISO 8601 format with timezone
- `title` (string, optional): Title or context for the paragraph
- `authors` (string[], optional): List of all authors if multiple
- `sourceUri` (string, optional): Source URL where the content was originally published

**Request Example:**
```json
{
  "content": "The best startup ideas are often ones that the founders themselves want...",
  "author": "Paul Graham",
  "company": "Y Combinator",
  "date": "2012-11-01",
  "contentCreatedAt": "2012-11-01T00:00:00.000Z",
  "title": "How to Get Startup Ideas",
  "sourceUri": "https://paulgraham.com/startupideas.html"
}
```

**Response Structure:**
```json
{
  "success": true,
  "vectorId": "investors/paulgraham/paragraph-abc123.json",
  "blobKey": "investors/paulgraham/paragraph-abc123.json",
  "blobUrl": "https://xyz.public.blob.vercel-storage.com/investors/paulgraham/paragraph-abc123.json",
  "authorNormalized": "paulgraham",
  "company": "Y Combinator",
  "date": "2012-11-01",
  "contentCreatedAt": "2012-11-01T00:00:00.000Z",
  "createdAt": "2024-01-15T10:30:00.000Z"
}
```

**Error Response (Unauthorized):**
```json
{
  "error": "Unauthorized",
  "message": "This tool requires authentication with write:investors scope. Please provide a valid Authorization header with Bearer token."
}
```

## Data Schema

### Upstash Vector Metadata

Each vector record should have metadata with:
- `blobKey` (string, required): Key/URL to retrieve full content from Vercel Blob
- `authorNormalized` (string, optional): Normalized author name for filtering (lowercase, no spaces)
- `company` (string, optional): Company/organization name for filtering
- `date` (string, optional): Publication date in ISO format (YYYY-MM-DD)
- `title` (string, optional): Title or context
- `sourceUri` (string, optional): Source URL
- `dateTs` (number, optional): Publication date as UTC epoch milliseconds (used for range filtering)
- `contentCreatedAt` (string, optional): Original content creation datetime in ISO 8601 format
- `contentCreatedAtTs` (number, optional): Original content creation datetime as UTC epoch milliseconds (used for range filtering)
- `createdAt` (string, optional): When the record was added to the system
- `createdAtTs` (number, optional): Record creation datetime as UTC epoch milliseconds (used for range filtering)
- `kind` (string, optional): Type of content (e.g., "paragraph")

### Vercel Blob Content

Store your content as JSON in Vercel Blob. The full JSON object is returned in the `blob` field of each match.

## Local Development

```bash
# Install dependencies
pnpm install

# Copy environment variables
cp .env.example .env.local
# Then fill in your values in .env.local

# Start development server
pnpm dev
```

### Sample MCP Client

```bash
# Test with Streamable HTTP transport
node scripts/test-streamable-http-client.mjs http://localhost:3000

# Test with SSE transport (requires Redis - see below)
node scripts/test-client.mjs http://localhost:3000
```

## Data Output Structure

Generated artifacts are stored outside the repository root:

- `data/scraping/`: scraper and enrichment outputs (for example `scraping-output-*.json`)
- `data/reports/`: ingestion and dedupe reports

## Ingesting X Article Text (Authenticated)

For `x.com/i/article/...` stubs and long-form post content, use authenticated X API enrichment:

```bash
export X_BEARER_TOKEN=<your-x-api-bearer-token>

# Option A: fetch + enrich in one step (TwStalker + X API + syndication fallback)
node scripts/fetch-twstalker-profile.mjs \
  --handle ssafavi \
  --author "Soofi Safavi" \
  --company elephant-xyz \
  --output data/scraping/scraping-output-soofi.json

# Option B: enrich an existing scraping-output file
node scripts/enrich-x-article-posts.mjs \
  --input data/scraping/scraping-output-soofi.json \
  --output data/scraping/scraping-output-soofi-enriched.json
```

Then upload to MCP:

```bash
MCP_WRITE_TOKEN=<token> node scripts/bulk-upload-paragraphs.mjs \
  --input data/scraping/scraping-output-soofi-enriched.json \
  --origin https://investors-mcp.vercel.app
```

## Reporting Ingestion Coverage

Use the reporting script to audit who is in the RAG and each author's date coverage:

```bash
# Table output
BLOB_READ_WRITE_TOKEN=<token> node scripts/report-rag-ingestion.mjs

# JSON output (with full publication date list per author)
BLOB_READ_WRITE_TOKEN=<token> node scripts/report-rag-ingestion.mjs \
  --json \
  --include-date-list \
  --output data/reports/ingestion-report.json
```

## Tracking Author Replies (e.g., Soofi on X)

Use this script to detect who an author replied to, based on reply-style posts
that begin with one or more `@handle` mentions. The JSON output also includes
daily post and reply counts for the author:

```bash
BLOB_READ_WRITE_TOKEN=<token> node scripts/report-author-replies.mjs \
  --author "Soofi Safavi" \
  --output data/reports/soofi-replies-report.json
```

Optional JSON output:

```bash
BLOB_READ_WRITE_TOKEN=<token> node scripts/report-author-replies.mjs \
  --author "Soofi Safavi" \
  --json
```

## Reporting X Polling Runs

Use this script to summarize `/api/automation/monitor-x` runs from Postgres-backed run records, including
when polling ran and how many posts were fetched/new:

```bash
POSTGRES_URL_NON_POOLING=<url> node scripts/report-x-monitor-runs.mjs \
  --output data/reports/x-monitor-runs-report.json
```

Optional JSON output:

```bash
 POSTGRES_URL_NON_POOLING=<url> node scripts/report-x-monitor-runs.mjs \
  --json
```

## Real-Time X Trigger Automation

`GET/POST /api/automation/monitor-x` runs the real-time polling pipeline:

1. polls watched X authors for new posts
2. ingests new posts into RAG
3. contextually compares each post against Soofi Safavi's RAG content
4. drafts a response
5. creates an Asana task with source post + draft + context matches

Watchlist source:

- `managed_authors` in Postgres (author/handle/company registry and polling source of truth)
- `data/reports/ingestion-report.json` (author/company universe used to fill company context)

### Trigger Parameters

- `dryRun=true|false` (default `false`): run pipeline without RAG writes or Asana task creation
- `backfill=true|false` (default `false`): process historical unseen posts on first run
- `ingestToRag=true|false` (default `true`): control RAG ingestion step
- `batchSize=<n>` (default `5`): number of authors processed per trigger
- `maxPostsPerAuthor=<n>` (default `20`): cap per-author post scan
- `topK=<n>` (default `6`): Soofi context matches used for drafting
- `author=<author-or-handle>`: run only one author

### Example (manual dry-run)

```bash
curl -H "Authorization: Bearer $AUTOMATION_TRIGGER_TOKEN" \
  "https://investors-mcp.vercel.app/api/automation/monitor-x?dryRun=true&batchSize=2"
```

## Deduplicating Records

Remove duplicate ingested records by `(author + sourceUri)` while keeping one canonical record:

```bash
# Preview only
BLOB_READ_WRITE_TOKEN=<token> \
UPSTASH_VECTOR_REST_URL=<url> \
UPSTASH_VECTOR_REST_TOKEN=<token> \
node scripts/dedupe-rag-records.mjs --report data/reports/dedupe-report.json

# Apply deletions in vector + blob
BLOB_READ_WRITE_TOKEN=<token> \
UPSTASH_VECTOR_REST_URL=<url> \
UPSTASH_VECTOR_REST_TOKEN=<token> \
node scripts/dedupe-rag-records.mjs --apply --report data/reports/dedupe-report.json
```

## Reporting UI

A simple reporting dashboard is available at:

- `https://investors-mcp.vercel.app/reporting` (production)
- `http://localhost:3000/reporting` (local)

The page reads `data/reports/ingestion-report.json` and can also read:
- `data/reports/soofi-replies-report.json`
- Postgres-backed `monitor_runs` records for the Polling Runs tab

It provides:

- per-author coverage table
- search and sorting
- CSV export
- Soofi replies reporting

## Dashboard Auth

The reporting routes (`/authors`, `/polling`, `/replies`, `/reporting`) are protected by Google sign-in.

Required environment variables:

- `NEXTAUTH_SECRET`
- `GOOGLE_CLIENT_ID`
- `GOOGLE_CLIENT_SECRET`
- `AUTH_ALLOWED_GOOGLE_DOMAIN`

Recommended value for `AUTH_ALLOWED_GOOGLE_DOMAIN`:

- `elephant-labs.xyz`

For local development, add Google OAuth callback URLs for:

- `http://localhost:3000/api/auth/callback/google`
- `https://investors-mcp.vercel.app/api/auth/callback/google`
- polling run reporting (run timestamp, duration, fetched/new posts, and task counts)

## Notes for Production on Vercel

- **Fluid Compute**: Enable [Fluid compute](https://vercel.com/docs/functions/fluid-compute) for efficient execution
- **Max Duration**: With Vercel Pro/Enterprise, you can increase `maxDuration` to 800 in `vercel.json`
- **SSE Transport**: Requires Redis. Add Upstash Redis from Vercel Marketplace and set `disableSse: false` in `app/mcp/route.ts`

## Architecture

```
┌─────────────┐     ┌──────────────────┐     ┌─────────────────┐
│ MCP Client  │────▶│ app/mcp/route.ts │────▶│ Vercel AI       │
│             │     │                  │     │ Gateway         │
└─────────────┘     └──────────────────┘     │ (embeddings)    │
                            │                └─────────────────┘
                            │
                            ▼
                    ┌──────────────────┐
                    │ Upstash Vector   │
                    │ (semantic search)│
                    └──────────────────┘
                            │
                            ▼
                    ┌──────────────────┐
                    │ Vercel Blob      │
                    │ (full content)   │
                    └──────────────────┘
```

## License

See [LICENSE](./LICENSE) file.