Skip to main content
Glama
Pulipuli Blog Knowledge MCP

Pulipuli Blog Knowledge MCP

Official
by pulipulichen
README.md
# Pulipuli Blog Knowledge MCP

A public, read-only knowledge base for articles from [Pulipuli Blog](https://blog.pulipuli.info/). Cloudflare Workers hosts the search website, JSON API, and MCP Streamable HTTP endpoint. Article metadata and full text are stored in Cloudflare D1, semantic vectors in Qdrant Cloud, and embeddings are generated by Workers AI.

## Features

- Read Blogger's lightweight summary JSON feed and backfill historical posts using a pagination cursor; generic Atom and RSS feeds are also supported.
- Fetch actual article pages and extract their full text instead of treating RSS excerpts as full content.
- Respect `robots.txt` and support retries, rate limiting, and retrying failed articles.
- Store titles, URLs, dates, tags, full text, and content hashes in D1; refresh the vector index when content changes.
- Support keyword, semantic, and hybrid search. Hybrid ranking uses Reciprocal Rank Fusion (RRF).
- Expose read-only `search_articles`, `list_recent_articles`, and `get_article` tools at `/mcp` for MCP clients supporting Streamable HTTP.
- List the latest indexed articles with one click on the search site, or call `list_recent_articles(n=5)` without embeddings.
- Run synchronization every two days on GitHub Actions; resume the initial backfill from D1 checkpoints.
- Parse article HTML, split text, and write vectors on the cloud runner. Each Worker embedding request handles one small text, avoiding Free Worker CPU limits for full articles.

## Tests and CI

GitHub Actions automatically runs CI in the cloud with Node.js 22. The following commands are optional for contributors; **end users do not need to run anything locally**:

```bash
npm install
npm run typecheck
npm test
```

The GitHub Actions [CI workflow](.github/workflows/ci.yml) performs type checking and offline unit tests on pushes to `main`, pull requests, and manual runs. See the [Testing Guide](docs/testing.md) for coverage and limitations.

## Quick Start (100% Cloud-Based, No Local Environment Required)

1. Create a D1 database named `pulipuli-blog-knowledge` in the Cloudflare Dashboard, then copy its Database ID.
2. Create a cluster in Qdrant Cloud and obtain its Cluster URL and API Key.
3. In GitHub Repository → Settings → Secrets and variables → Actions, configure:
   - **Variables:** `D1_DATABASE_ID` and `QDRANT_COLLECTION` (for example, `pulipuli_blog_chunks`).
   - **Secrets:** `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`, `INGEST_TOKEN`, `QDRANT_URL`, and `QDRANT_API_KEY`.
4. Open GitHub → Actions → **Deploy Cloudflare Worker** → **Run workflow**. The GitHub-hosted runner installs Node.js/Wrangler, runs tests, applies remote D1 migrations, and deploys the Worker automatically. No local setup is required.
5. After deployment, set `https://<worker-host>/internal/ingest` as the GitHub Secret `INGEST_URL`, then manually run **Refresh blog index** to start importing articles.
6. Once the initial deployment has been verified, optionally set the GitHub Variable `ENABLE_AUTO_DEPLOY=true` to deploy automatically after successful CI runs on `main`.

> The production `wrangler.toml` is generated from `wrangler.example.toml` only on the temporary GitHub Actions runner. **You do not need to install, clone, or deploy anything locally.**

For detailed instructions and Secret permissions, see the [Cloud-Only Deployment Guide](docs/deployment.md) and [Configuration Guide](docs/configuration.md).

## Automatic Worker and Qdrant Keep-Alive

The [Keep Alive - Blog Worker and Qdrant](.github/workflows/keep-alive.yml) GitHub Actions workflow performs **read-only checks once every three UTC days**. GitHub schedules it daily at **03:23 UTC** and a UTC epoch-day modulo-3 gate skips the other days, avoiding the month-boundary drift of `*/3` in a day-of-month cron field. You can also run it immediately from **Actions → Keep Alive - Blog Worker and Qdrant → Run workflow**.

- **Cloudflare Worker + D1:** `GET /health` must report `status: ok` and `database: ok`.
- **Qdrant:** `POST /collections/{QDRANT_COLLECTION}/points/scroll` reads at most one vector point, without creating, updating, or deleting anything. This checks Qdrant directly; the Worker health endpoint alone does not contact Qdrant.
- **Configuration:** Reuses the existing production-environment Secrets `INGEST_URL` (ending in `/internal/ingest`), `QDRANT_URL`, and `QDRANT_API_KEY`, plus the production Variable `QDRANT_COLLECTION`. There are no additional tokens to configure.
- **Failure reporting:** If an endpoint fails, the Action fails and the logs identify whether the Worker/D1 or Qdrant check failed. It does not automatically recreate a missing database.

**Important:** Qdrant Cloud free clusters can be suspended after a week of inactivity and deleted after four weeks without reactivation. This recurring check helps prevent idle suspension, but does **not** guarantee data retention. GitHub can also automatically disable scheduled workflows on public repositories after 60 days with no repository activity, and scheduled jobs can be delayed or skipped. Check that the workflow remains enabled, and retain an independent Qdrant backup or snapshot for important data.

## Endpoints

- `GET /`: Blog search page.
- `GET /api/search?q=...&limit=...&mode=hybrid|keyword|semantic`: Search articles.
- `GET /api/recent?n=5`: List newest indexed published posts (default 5; 1–50) with title, URL, date, tags, and excerpt.
- `GET /api/articles/:id`: Retrieve the fully indexed text of an article.
- `GET /health`: Worker and D1 health status.
- `POST /internal/ingest`: Bearer-token-protected batch synchronization endpoint; results include each article's title, URL, and outcome.
- `GET /internal/ingest/status`: Bearer-token-protected status endpoint showing the last reported in-progress articles and D1 lease expiry.
- `GET /internal/ingest/embedding`: Bearer-token-protected model/configuration check for the Actions runner.
- `POST /internal/ingest/embedding`: Bearer-token-protected single-text embedding proxy (maximum 3,000 characters).
- `/mcp`: Read-only MCP Streamable HTTP endpoint. Call `list_recent_articles` with no arguments for five posts, or `{"n":10}` for ten.

## Configuration and Secrets

- `wrangler.example.toml`, `config.example.json`, and `.dev.vars.example` are safe-to-publish examples.
- The production D1 UUID is stored as the GitHub Actions Variable `D1_DATABASE_ID`. The GitHub runner creates an ignored `wrangler.toml`.
- `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`, `INGEST_TOKEN`, `QDRANT_URL`, and `QDRANT_API_KEY` belong in GitHub Actions Secrets. The deployment workflow synchronizes the required Worker Secrets.
- `QDRANT_COLLECTION` and `ENABLE_AUTO_DEPLOY` belong in GitHub Actions Variables; `INGEST_URL` belongs in GitHub Actions Secrets.

## Documentation

- [Technical Specification](docs/SPEC.md)
- [Configuration Guide](docs/configuration.md)
- [Deployment and Initial Backfill](docs/deployment.md)
- [Testing Guide](docs/testing.md)

## Limitations

Free-tier quotas and limits for Cloudflare, Workers AI, D1, GitHub Actions, and Qdrant may change; verify current limits with each provider. Keyword search currently uses D1 substring matching, which is a practical starting point for Chinese-language queries against this blog. The indexing strategy can be revisited as the dataset grows.

Use **Refresh blog index** for production backfills. The older direct `POST /internal/ingest` path still runs a whole article inside the Worker and can exceed its CPU budget. Deploy the latest Worker before starting a new refresh run; existing articles and cursors do not need to be reset.