ingestUrls
Extracts and processes content from specified URLs with configurable scraping options and metadata for knowledge management.
Instructions
Ingests content from a list of URLs. Supports scraping options and metadata.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| namespaceId | No | ||
| ingestConfig | Yes | ||
| tenantId | No |
Implementation Reference
- src/index.ts:251-268 (handler)MCP tool registration and handler implementation for 'ingestUrls'. Parses params, creates SourceSyncApiClient instance, and invokes its ingestUrls method via safeApiCall wrapper.
server.tool( 'ingestUrls', 'Ingests content from a list of URLs. Supports scraping options and metadata.', IngestUrlsSchema.shape, async (params) => { return safeApiCall(async () => { const { namespaceId, tenantId, ingestConfig } = params // Create a client with the provided parameters const client = createClient({ namespaceId, tenantId }) // Direct passthrough to the API return await client.ingestUrls({ ingestConfig, }) }) }, ) - src/schemas.ts:202-214 (schema)Zod schema (IngestUrlsSchema) defining the input structure and validation for the ingestUrls tool, including urls array, optional scrapeOptions, metadata, and chunkConfig.
export const IngestUrlsSchema = z.object({ namespaceId: namespaceIdSchema.optional(), ingestConfig: z.object({ source: z.literal(SourceSyncIngestionSource.URLS_LIST), config: z.object({ urls: z.array(z.string()), scrapeOptions: ScrapeOptionsSchema.optional(), metadata: z.record(z.union([z.string(), z.array(z.string())])).optional(), }), chunkConfig: chunkConfigSchema.optional(), }), tenantId: tenantIdSchema, }) - src/sourcesync.ts:368-385 (helper)SourceSyncApiClient.ingestUrls helper method: constructs and sends POST request to SourceSync API endpoint '/v1/ingest/urls' with namespaceId and ingestConfig (merging default chunkConfig).
public async ingestUrls({ ingestConfig, }: Omit< SourceSyncIngestUrlsRequest, 'namespaceId' >): Promise<SourceSyncIngestResponse> { return this.client .url('/v1/ingest/urls') .json({ namespaceId: this.namespaceId, ingestConfig: { ...ingestConfig, chunkConfig: SourceSyncApiClient.CHUNK_CONFIG, }, } satisfies SourceSyncIngestUrlsRequest) .post() .json<SourceSyncIngestResponse>() } - src/sourcesync.types.ts:416-427 (schema)TypeScript type definition for SourceSyncIngestUrlsRequest, used by the SourceSyncApiClient for type safety in the ingestUrls API call.
export type SourceSyncIngestUrlsRequest = { namespaceId: string ingestConfig: { source: SourceSyncIngestionSource.URLS_LIST config: { urls: string[] scrapeOptions?: SourceSyncScrapeOptions metadata?: Record<string, any> } chunkConfig?: SourceSyncChunkConfig } } - src/index.ts:10-19 (registration)Import statement registering IngestUrlsSchema for use in the ingestUrls tool definition.
import { validateApiKeySchema, createNamespaceSchema, listNamespacesSchema, getNamespaceSchema, updateNamespaceSchema, deleteNamespaceSchema, ingestTextSchema, IngestFileSchema, IngestUrlsSchema,