ingestUrls
Ingests content from multiple URLs, supporting custom scraping options and metadata extraction. Enables efficient organization and retrieval within knowledge bases for AI models.
Instructions
Ingests content from a list of URLs. Supports scraping options and metadata.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ingestConfig | Yes | ||
| namespaceId | No | ||
| tenantId | No |
Implementation Reference
- src/sourcesync.ts:368-385 (handler)Core handler function in SourceSyncApiClient that executes the ingestUrls logic by making a POST request to the SourceSync API endpoint /v1/ingest/urls.public async ingestUrls({ ingestConfig, }: Omit< SourceSyncIngestUrlsRequest, 'namespaceId' >): Promise<SourceSyncIngestResponse> { return this.client .url('/v1/ingest/urls') .json({ namespaceId: this.namespaceId, ingestConfig: { ...ingestConfig, chunkConfig: SourceSyncApiClient.CHUNK_CONFIG, }, } satisfies SourceSyncIngestUrlsRequest) .post() .json<SourceSyncIngestResponse>() }
- src/index.ts:250-268 (registration)MCP server registration of the 'ingestUrls' tool, including description, input schema, and thin wrapper handler that creates a SourceSync client and calls its ingestUrls method.// Add ingestUrls tool server.tool( 'ingestUrls', 'Ingests content from a list of URLs. Supports scraping options and metadata.', IngestUrlsSchema.shape, async (params) => { return safeApiCall(async () => { const { namespaceId, tenantId, ingestConfig } = params // Create a client with the provided parameters const client = createClient({ namespaceId, tenantId }) // Direct passthrough to the API return await client.ingestUrls({ ingestConfig, }) }) }, )
- src/schemas.ts:202-214 (schema)Zod schema defining the input parameters for the ingestUrls tool, including namespaceId, ingestConfig with URLs list, scrape options, metadata, chunkConfig, and tenantId.export const IngestUrlsSchema = z.object({ namespaceId: namespaceIdSchema.optional(), ingestConfig: z.object({ source: z.literal(SourceSyncIngestionSource.URLS_LIST), config: z.object({ urls: z.array(z.string()), scrapeOptions: ScrapeOptionsSchema.optional(), metadata: z.record(z.union([z.string(), z.array(z.string())])).optional(), }), chunkConfig: chunkConfigSchema.optional(), }), tenantId: tenantIdSchema, })
- src/sourcesync.types.ts:416-427 (schema)TypeScript type definition for the SourceSyncIngestUrlsRequest used by the API client implementation.export type SourceSyncIngestUrlsRequest = { namespaceId: string ingestConfig: { source: SourceSyncIngestionSource.URLS_LIST config: { urls: string[] scrapeOptions?: SourceSyncScrapeOptions metadata?: Record<string, any> } chunkConfig?: SourceSyncChunkConfig } }