# Shayari: Devanagari-first shayari search MCP

A small, reproducible Python package for searching and reading Hindi/Devanagari shayari. The user-facing language is Hindi. Urdu is not required for search or reading, and this project does not transliterate or generate missing Hindi text.

## Status

The first milestone uses the approved Kaggle source:

- [Urdu Ghazals Rekhta](https://www.kaggle.com/datasets/shabanaftab/urdu-ghazals-rekhta)
- Kaggle version 1, downloaded without credentials
- 1,314 source stems, 30 poets
- 1,306 Hindi files, of which one is empty
- 1,305 source-valid Hindi-text records are active by explicit owner authorization. Nine source stems remain quarantined because they have no usable Hindi text.

The source metadata declares CC0, but the underlying Rekhta-derived rights are not independently verified. Read [`corpus/SOURCE.md`](corpus/SOURCE.md) before redistribution.

## Design decisions

- `text_devanagari` is the canonical field and is required for active records.
- No Urdu-to-Devanagari transliteration, translation, repair, or generation occurs.
- Invalid or incomplete records are written to quarantine, never silently indexed.
- `title_devanagari` and `poet_devanagari` remain optional because the inspected archive supplies only Roman source-path metadata.
- `title_source` and `poet_source` preserve that metadata explicitly; it is searchable but is not mislabeled as Hindi.
- Downloaded archives are ignored by Git. The generated active snapshot is versioned so a fresh checkout can search immediately; it contains no credentials.

## Requirements

- Python 3.9+
- `pytest` for tests
- Optional: the `mcp` Python package for the standard MCP server adapter

The core importer, validator, search service, and CLI use only the Python standard library.

## Reproduce the corpus

From the project root, place the downloaded archive at `.cache/urdu-ghazals-rekhta.zip`, then run:

```bash
PYTHONPATH=src python3 scripts/import_kaggle.py .cache/urdu-ghazals-rekhta.zip
```

The command writes these files under `corpus/generated/`:

- `records.jsonl`: active, owner-authorized source-valid records. Every record retains explicit provenance and authorization-status fields.
- `research.jsonl`: empty under the default owner-authorized policy; available when using `--activation-policy research_only` for comparison.
- `quarantine.jsonl`: rejected or incomplete source stems with reasons
- `audit.json`: counts, source version, checksum, script distribution, and activation policy

To audit an existing generated directory:

```bash
PYTHONPATH=src python3 scripts/audit_corpus.py corpus/generated
```

## CLI

Search the active, owner-authorized Hindi Devanagari corpus:

```bash
PYTHONPATH=src python3 -m shayari search 'दोस्ती' --limit 5
PYTHONPATH=src python3 -m shayari search 'मोहब्बत' --limit 5
```

The default importer policy is `owner_authorized`, which activates every source-valid Devanagari record while preserving the source's provenance and limitations. The activation is an owner decision, not a claim of independent legal or linguistic verification. To reproduce the earlier research-only layout explicitly:

```bash
PYTHONPATH=src python3 scripts/import_kaggle.py .cache/urdu-ghazals-rekhta.zip --activation-policy research_only
```

The current source has 1,305 active records after import. The remaining nine source stems stay quarantined because they have missing or empty Hindi text.

```bash
PYTHONPATH=src python3 -m shayari status
```

Set `SHAYARI_CORPUS` if the generated corpus lives outside `corpus/generated`:

```bash
SHAYARI_CORPUS=/path/to/generated shayari status
```

The explicitly named research-only MCP tool is available for inspection.

Search title and poet fields too, and inspect a result by ID:

```bash
PYTHONPATH=src python3 -m shayari show <record-id>
PYTHONPATH=src python3 -m shayari audit
```

All normal search results include the full `text_devanagari` field.

## MCP server

The package exposes a stdio server through either the `shayari-mcp` command or the module entry point. With the optional `mcp` dependency installed:

```bash
python3 -m pip install 'mcp>=1.0,<2.0'
PYTHONPATH=src python3 -m shayari.mcp_server
# or, after installing this project:
shayari-mcp
```

The server tools are:

- `search_shayari`
- `search_research_shayari` (explicitly non-active research inspection)
- `get_shayari`
- `audit_corpus`

The handlers call the same lexical service as the CLI and return Hindi-facing fields by default. An in-process JSON-RPC-compatible fallback is included for local clients that do not have the optional SDK installed.

Example Hermes MCP configuration:

```yaml
mcp_servers:
  shayari:
    command: python3
    args: ["-m", "shayari.mcp_server"]
    env:
      PYTHONPATH: "/absolute/path/to/Shayari/src"
```

## Development

```bash
PYTHONPATH=src python3 -m pytest -q
ruff check src scripts tests
ruff format --check src scripts tests
mypy --python-version 3.9 src scripts
python3 -m build --wheel --sdist
```

The downloaded `.cache/` archive is ignored. The versioned `corpus/generated/` snapshot can be regenerated from it. Do not add credentials, tokens, or environment files to the repository.