ingestUrls
Extract content and metadata from URLs using customizable scraping options. Supports chunking for efficient data management and integration with AI models.
Instructions
Ingests content from a list of URLs. Supports scraping options and metadata.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ingestConfig | Yes | ||
| namespaceId | No | ||
| tenantId | No |
Implementation Reference
- src/index.ts:252-267 (registration)Registers the 'ingestUrls' MCP tool with server.tool, providing description, input schema, and handler that creates SourceSyncApiClient and delegates to its ingestUrls method.'ingestUrls', 'Ingests content from a list of URLs. Supports scraping options and metadata.', IngestUrlsSchema.shape, async (params) => { return safeApiCall(async () => { const { namespaceId, tenantId, ingestConfig } = params // Create a client with the provided parameters const client = createClient({ namespaceId, tenantId }) // Direct passthrough to the API return await client.ingestUrls({ ingestConfig, }) }) },
- src/schemas.ts:202-214 (schema)Zod schema (IngestUrlsSchema) for input validation of the ingestUrls tool parameters, including namespaceId, ingestConfig with urls list and options, and tenantId.export const IngestUrlsSchema = z.object({ namespaceId: namespaceIdSchema.optional(), ingestConfig: z.object({ source: z.literal(SourceSyncIngestionSource.URLS_LIST), config: z.object({ urls: z.array(z.string()), scrapeOptions: ScrapeOptionsSchema.optional(), metadata: z.record(z.union([z.string(), z.array(z.string())])).optional(), }), chunkConfig: chunkConfigSchema.optional(), }), tenantId: tenantIdSchema, })
- src/sourcesync.ts:368-385 (handler)Core handler logic in SourceSyncApiClient.ingestUrls: sends POST request to SourceSync API /v1/ingest/urls endpoint with namespace-specific ingestConfig.public async ingestUrls({ ingestConfig, }: Omit< SourceSyncIngestUrlsRequest, 'namespaceId' >): Promise<SourceSyncIngestResponse> { return this.client .url('/v1/ingest/urls') .json({ namespaceId: this.namespaceId, ingestConfig: { ...ingestConfig, chunkConfig: SourceSyncApiClient.CHUNK_CONFIG, }, } satisfies SourceSyncIngestUrlsRequest) .post() .json<SourceSyncIngestResponse>() }