process_raw_data
Normalize, deduplicate, and upsert validated raw contest records into the primary contests collection.
Instructions
Read validated raw records from CHRawdata.rawdata for a given scraper
source, normalize, deduplicate, and upsert them into the primary
Contests collection (ContestHopperDb).
NOTE: By default, only records with validationStatus="validated" are processed. Run get_records_for_validation + submit_raw_validation first.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max records to process per call (default 100, max 1000) | |
| source | Yes | Scraper name (e.g. "contestwatchers", "opportunityDesk") | |
| auto_image | No | If True, automatically download, convert (WebP+AVIF), and upload images to R2 after upserting contest data | |
| dedupe_gate | No | If True (default), skip records whose title matches an existing live contest (normalized-title match from a different source, or reworded title from the same source). Set False to force-insert. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||