# The Agentic SEO Engine

**A headless, API-first MCP server that transforms unstructured creative portfolios into validated, machine-readable `llms.txt` manifests for the agentic web.**

Live deployment: `https://agentic-seo-engine-796932810904.us-central1.run.app/sse`

Built for the [Kaggle AI Agents: Intensive Vibe Coding Capstone](https://www.kaggle.com/) — Agents for Business track.

---

## The Problem

Traditional SEO is becoming obsolete in the agentic web. Autonomous agents — systems that crawl the web to procure talent, data, and services on behalf of enterprises — rely on clean, machine-readable specifications rather than visual layouts. Visual-heavy portfolios, experiential landing pages, and creative directories are fundamentally invisible to these programmatic crawlers, leaving human creators undiscoverable by the very systems increasingly responsible for sourcing them.

## The Solution

The Agentic SEO Engine bridges this gap. Given any portfolio URL, it:

1. **Ingests** the page deterministically, stripping layout noise and presentation styling
2. **Synthesizes** the creator's implicit disciplines, tone, and structure using Gemini inside a strict instruction harness
3. **Validates** the output against a rigid Pydantic V2 schema, guaranteeing standards-compliant, structured JSON every time

The result is a compiled `llms.txt` manifest — a foundational SEO artifact for the agentic internet, ensuring human creators remain discoverable by enterprise procurement agents.

## Why Agents

A static scraper can extract text, but it cannot reason about implicit creative intent — the difference between a portfolio's literal content and its actual thematic identity. This task is fundamentally interpretive, which is why an LLM-driven synthesis step, not a fixed extraction template, sits at the core of the pipeline.

## Architecture
[ Unstructured URL ] ➔ [ Deterministic Ingestion ] ➔ [ Gemini Inference ] ➔ [ Pydantic V2 Validation ] ➔ [ Verified llms.txt JSON ]

- **Ingestion layer:** `httpx` + `BeautifulSoup` strip scripts, styles, nav, and footer noise, leaving core structural text.
- **Inference layer:** The cleaned text is passed to `gemini-2.5-flash-lite` inside a disciplined system instruction that rejects conversational filler and assumption.
- **Validation layer:** Pydantic V2 models (`CreatorProfile`, `ProjectOrAsset`, `MetadataPair`) enforce strict typing on the model's output, with a recursive schema-cleaning step that strips permissive `additionalProperties` markers before the request is even sent.
- **Interface layer:** Exposed as a single tool, `generate_llms_manifest`, via a headless Model Context Protocol (MCP) server — no UI, pure machine-to-machine utility.

## Security

- No API keys or credentials are hardcoded anywhere in the codebase.
- The Gemini API key is injected at the infrastructure level as a Cloud Run environment variable, never bundled into the container image.
- All model inputs/outputs pass through strict Pydantic V2 schema validation, preventing malformed or injected data from reaching the final manifest.
- Service access is scoped via GCP IAM policy bindings controlling which identities can build, push, and invoke the deployed container.

## Tech Stack

- **MCP:** Official MCP Python SDK (`mcp.server.fastmcp.FastMCP`)
- **LLM:** Google Gemini (`google-genai` SDK, `gemini-2.5-flash-lite`)
- **Validation:** Pydantic V2
- **Scraping:** `httpx`, `BeautifulSoup4`
- **Deployment:** Google Cloud Run (Docker, `python:3.11-slim`)

## Setup & Local Development

### Prerequisites
- Python 3.11+
- A Google Gemini API key ([Google AI Studio](https://aistudio.google.com/))

### Installation

```bash
git clone https://github.com/<your-username>/agentic-seo-engine.git
cd agentic-seo-engine
pip install -r requirements.txt
```

### Configuration

Create a `.env` file in the project root (this file is git-ignored and never committed):
GEMINI_API_KEY=your_actual_key_here

### Running Locally

```bash
python server.py
```

With no `PORT` environment variable set, the server runs in `stdio` transport — ideal for local testing with the [MCP Inspector](https://github.com/modelcontextprotocol/inspector) or Claude Desktop.

### Deploying to Google Cloud Run

```bash
gcloud run deploy agentic-seo-engine \
  --source . \
  --region us-central1 \
  --allow-unauthenticated \
  --port 8080

gcloud run services update agentic-seo-engine \
  --region us-central1 \
  --update-env-vars GEMINI_API_KEY=your_actual_key_here
```

When `PORT` is present in the environment (as Cloud Run sets automatically), the server switches to `sse` transport and binds to `0.0.0.0` on the assigned port.

## Usage

Connect any MCP-compatible client to the server's SSE endpoint and invoke the `generate_llms_manifest` tool with a target portfolio URL:

```python
from mcp import ClientSession
from mcp.client.sse import sse_client

async with sse_client("https://agentic-seo-engine-796932810904.us-central1.run.app/sse") as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()
        result = await session.call_tool(
            "generate_llms_manifest",
            {"portfolio_url": "https://example-creator-portfolio.com"}
        )
        print(result)
```

### Example Output

```json
{
  "profile": {
    "name": "Vibe Code",
    "role_title": "AI App Builder",
    "summary": "Builds apps and websites with AI, focusing on rapid development and user-friendly interfaces...",
    "disciplines": ["AI Application Development", "Web Development", "Product Prototyping"]
  },
  "featured_works": [
    {
      "title": "Maison",
      "description": "Editorial home goods storefront.",
      "tags": ["ecommerce", "storefront", "template"]
    }
  ]
}
```

## Project Status

This project was built as a rapid capstone prototype. It is deployed live and functioning, and serves as the foundation for continued development in a subsequent, more extensive agentic build.

## Author

[Ralph Torres](https://www.kaggle.com/ralphtorresjr)