Skip to main content
Glama

ingestWebsite

Crawls and ingests website content recursively for knowledge base management, with configurable depth, path filtering, and chunking controls.

Instructions

Crawls and ingests content from a website recursively. Supports depth control and path filtering.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
namespaceIdNo
ingestConfigYes
tenantIdNo

Implementation Reference

  • src/index.ts:291-308 (registration)
    MCP tool registration for 'ingestWebsite' using server.tool, including inline handler that delegates to SourceSyncApiClient.ingestWebsite
    server.tool(
      'ingestWebsite',
      'Crawls and ingests content from a website recursively. Supports depth control and path filtering.',
      IngestWebsiteSchema.shape,
      async (params) => {
        return safeApiCall(async () => {
          const { namespaceId, ingestConfig, tenantId } = params
    
          // Create a client with the provided parameters
          const client = createClient({ namespaceId, tenantId })
    
          // Direct passthrough to the API
          return await client.ingestWebsite({
            ingestConfig,
          })
        })
      },
    )
  • Core handler logic in SourceSyncApiClient.ingestWebsite that sends POST request to SourceSync API /v1/ingest/website endpoint
    public async ingestWebsite({
      ingestConfig,
    }: Omit<
      SourceSyncIngestWebsiteRequest,
      'namespaceId'
    >): Promise<SourceSyncIngestResponse> {
      return this.client
        .url('/v1/ingest/website')
        .json({
          namespaceId: this.namespaceId,
          ingestConfig: {
            ...ingestConfig,
            chunkConfig: SourceSyncApiClient.CHUNK_CONFIG,
          },
        } satisfies SourceSyncIngestWebsiteRequest)
        .post()
        .json<SourceSyncIngestResponse>()
    }
  • Zod schema definition for ingestWebsite tool input validation (IngestWebsiteSchema)
    export const IngestWebsiteSchema = z.object({
      namespaceId: namespaceIdSchema.optional(),
      ingestConfig: z.object({
        source: z.literal(SourceSyncIngestionSource.WEBSITE),
        config: z.object({
          url: z.string(),
          maxDepth: z.number().optional(),
          maxLinks: z.number().optional(),
          includePaths: z.array(z.string()).optional(),
          excludePaths: z.array(z.string()).optional(),
          metadata: z.record(z.union([z.string(), z.array(z.string())])).optional(),
        }),
        chunkConfig: chunkConfigSchema.optional(),
      }),
      tenantId: tenantIdSchema,
    })
  • TypeScript type definition for the SourceSync API request (SourceSyncIngestWebsiteRequest) used by ingestWebsite
    export type SourceSyncIngestWebsiteRequest = {
      namespaceId: string
      ingestConfig: {
        source: SourceSyncIngestionSource.WEBSITE
        config: {
          url: string
          maxDepth?: number
          maxLinks?: number
          includePaths?: string[]
          excludePaths?: string[]
          scrapeOptions?: SourceSyncScrapeOptions
          metadata?: Record<string, any>
        }
        chunkConfig?: SourceSyncChunkConfig
      }
    }

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/pbteja1998/sourcesyncai-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server