build_dataset
Generate a dataset from collected crawler data, halting the crawler before the full 7-day period ends. Use when enough data is gathered and you need results without waiting.
Instructions
Build a dataset from collected data before the 7-day task completion. Use this when you have enough data and don't want to wait for the full collection period.
WARNING: Building a dataset will STOP the crawler and de-register it from the network. The crawler will no longer collect new data after this operation.
Parameters:
crawler_id (str, REQUIRED): The ID of the crawler to build dataset from (Get this from get_gravity_task_status response - look for 'crawler_ids' field)
max_rows (int, default: 10000): Maximum number of rows to include in the dataset
email (str, optional): Email address for notification when dataset is ready
redirect_url (str, optional): URL to redirect to from the email notification
Returns:
dataset_id: Unique identifier to track the dataset build
Build status and progress information (10 steps total)
Example: build_dataset( crawler_id="crawler-0-multicrawler-9f518ae4-xxxx", max_rows=10000, email="user@example.com" )
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| No | |||
| max_rows | No | ||
| crawler_id | Yes | ||
| redirect_url | No | https://app.macrocosmos.ai/ |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |