# MarketDocs India

MarketDocs India is an MIT-licensed, agent-first metadata index for NSE/BSE company documents. It indexes a rolling five years of concall transcripts, annual reports, and investor presentations, deduplicates companies by ISIN, and exposes provenance through server-rendered HTML, REST, and a stateless Streamable HTTP MCP server. This edition never stores or republishes source documents.

The Worker is deployed as a D1-only starter index. Its initial seed contains three verified company identities and two source-only records; it makes **no** claim of complete market coverage. Expanding to the requested ~5,000-company / ~70,000-document five-year index requires a licensed exchange dataset, issuer-approved feeds, or a user-supplied registry—none of those records are fabricated by this project.

## What is implemented

- Cloudflare Worker serving SSR HTML, REST v1, OpenAPI, MCP, `llms.txt`, `llms-full.txt`, robots, and partitionable sitemaps.
- D1 schema for companies, aliases, source health, documents, revisions, tombstones, and takedown audits.
- D1-only metadata records with official source URLs, provenance, quality, and rights state.
- SQLite Durable Object daily anonymous rate limiter with tighter content/MCP budgets.
- ISIN-first company identity, document classification, fiscal-period parsing, checksum deduplication, revisions, and rolling five-year retention.
- Issuer RSS, sitemap, and investor-relations adapters with conditional HTTP support. Exchange collection is hard-disabled; licensed NSE/BSE interfaces are explicit extension points.
- Public GitHub Actions boundary for future source discovery and validation; the live edition writes metadata only to D1.
- Deployment guard that refuses to run unless the supplied Cloudflare account ID exactly matches the separately supplied expected account ID.
- Provenance-first, no-JavaScript UI with explicit partial coverage, quality, rights, source, checksum, extraction, and freshness.

## Local development

Requirements: Node.js 22+ and Python 3.11+.

```sh
npm install
npm run db:migrate:local
npm run db:seed:local
npm run dev
```

Then open `http://localhost:8787`. The seed is metadata-only: it does not republish issuer documents.

Run all Worker/domain tests:

```sh
npm run check
```

Run ingestion tests without installing Docling:

```sh
python -m venv ingest/.venv
ingest/.venv/bin/python -m pip install -e './ingest[test]'
ingest/.venv/bin/python -m pytest ingest/tests
```

Docling conversion is installed in public CI with `./ingest[conversion,test]`. Source PDFs live only in a temporary directory during validation/conversion.

## Public surfaces

| Surface | Route |
|---|---|
| Search/latest | `/` |
| Company | `/companies/:slug` |
| Document provenance | `/companies/:slug/documents/:id` |
| Canonical Markdown | `/documents/:id.md` |
| Coverage/freshness | `/coverage` |
| Rights/takedowns | `/rights` |
| REST | `/api/v1/*` |
| OpenAPI/docs | `/openapi.json`, `/api/docs` |
| Remote MCP | `/mcp` |
| Agent guides | `/llms.txt`, `/llms-full.txt` |

MCP tools are `search_companies`, `find_documents`, `get_document`, and `get_coverage`. `get_document` returns bounded sections with continuation cursors; resource templates expose company and document provenance records.

## Source and rights policy

Do not scrape NSE or BSE websites. Their public sites are not production inputs. Add only reviewed issuer-controlled feeds/pages to `registry/sources.json`, or implement the `LicensedExchangeAdapter` boundary after obtaining suitable rights.

Full text can become public only when `rights_status` is `cleared` and quality is `provisional` or `verified`. Other records are source-only. Revocation requires immediate R2 deletion, a `source_only` transition, and a takedown audit entry.

The software is MIT-licensed. Source documents remain subject to their owners’ terms. See `/rights` in the app.

## Real-time ingestion

“Real-time” begins when an issuer publishes a document. Hourly Cloudflare Cron dispatches the public GitHub workflow; GitHub’s hourly schedule is the fallback. The workflow reads the reviewed issuer registry, discovers supported links, validates and hashes PDFs, publishes metadata for all rights states, and publishes Markdown only for cleared records. Configured live sources target p90 provisional availability within 60 minutes; `/coverage` reports actual freshness.

The workflow promotes provisional text immediately after the fast text-layer pass, then lets Docling replace it with verified output after layout/OCR checks. Protected GitHub `production` environments keep both publication steps behind the separate-account secrets. Scheduled runs leave those publication jobs skipped (not failed) unless the repository variable `PUBLISH_TO_CLOUDFLARE` is exactly `true`; enable it only after all three production secrets are configured.

## Separate Cloudflare account deployment

Verify the active Wrangler account before deployment. Create the D1 resource in the separate account, replace placeholder IDs/names in `wrangler.jsonc`, configure GitHub’s protected `production` environment, and use a narrowly scoped API token for automation.

```sh
export CLOUDFLARE_ACCOUNT_ID="separate-account-id"
export EXPECTED_CLOUDFLARE_ACCOUNT_ID="separate-account-id"
export CLOUDFLARE_API_TOKEN="scoped-token"
npm run deploy
```

The guard fails closed if any value is absent or the IDs differ. Initially retain the configurable `PUBLIC_ORIGIN` workers.dev URL. After buying a domain, update `PUBLIC_ORIGIN`; canonical links, OpenAPI, MCP examples, sitemaps, and agent files derive from it.

## Free-tier budgets

- Warn at 350 MB D1, below 500 MB per database.
- Store metadata and source URLs only; do not retain source files.
- Bound API/MCP pages and apply daily hashed-IP rate limits.
- Keep the public source registry and coverage denominator honest as issuer permissions expand.

Relevant primary documentation: [Workers limits](https://developers.cloudflare.com/workers/platform/limits/), [R2 pricing](https://developers.cloudflare.com/r2/pricing/), [D1 limits](https://developers.cloudflare.com/d1/platform/limits/), [Cloudflare remote MCP](https://developers.cloudflare.com/agents/model-context-protocol/guides/remote-mcp-server/), [GitHub Actions billing](https://docs.github.com/en/actions/concepts/billing-and-usage), [NSE terms](https://www.nseindia.com/static/nse-terms-of-use), and [Docling](https://github.com/docling-project/docling).