Robots.txt Generator
crawler_generate_robots_txtGenerate a robots.txt using the InsightfulPipe crawler generator. Requires workspace_id, sitemap_url, and whether to block AI bots. Returns plaintext when successful so assistants can embed it directly.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"workspace_id": "<workspace_id>",
"sitemap_url": "<sitemap_url>"
},
"name": "crawler_generate_robots_txt"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| sitemap_url | Yes | ||
| workspace_id | Yes | ||
| block_ai_bots | No | ||
| platform_preset | No | ||
| ai_bots_to_block | No | ||
| additional_payload | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||