fetch_url

Instructions

Implementation Reference

• src/index.ts:30-100 (handler)

  The core handler function that makes the HTTP call to the StripFeed API (/fetch) with the provided URL and optional parameters, processes the response (JSON or text), and returns clean content with metadata (tokens, cache, etc.).

  async function fetchUrl(params: FetchParams): Promise<{
    content: string;
    tokens: number;
    originalTokens: number;
    savingsPercent: string;
    cached: string;
    fetchMs: string;
    truncated: boolean;
    title?: string;
  }> {
    const apiKey = getApiKey();
    const searchParams = new URLSearchParams({ url: params.url });
  
    if (params.format) searchParams.set("format", params.format);
    if (params.selector) searchParams.set("selector", params.selector);
    if (params.model) searchParams.set("model", params.model);
    if (params.cache === false) searchParams.set("cache", "false");
    if (params.ttl !== undefined) searchParams.set("ttl", String(params.ttl));
    if (params.max_tokens !== undefined) searchParams.set("max_tokens", String(params.max_tokens));
  
    const response = await fetch(`${BASE_URL}/fetch?${searchParams}`, {
      headers: { Authorization: `Bearer ${apiKey}` },
    });
  
    if (!response.ok) {
      const body = await response.text();
      let message: string;
      try {
        message = JSON.parse(body).error;
      } catch {
        message = body;
      }
      throw new Error(`StripFeed API error ${response.status}: ${message}`);
    }
  
    const tokens = response.headers.get("X-StripFeed-Tokens") ?? "0";
    const originalTokens =
      response.headers.get("X-StripFeed-Original-Tokens") ?? "0";
    const savingsPercent =
      response.headers.get("X-StripFeed-Savings-Percent") ?? "0";
    const cached = response.headers.get("X-StripFeed-Cache") ?? "MISS";
    const fetchMs = response.headers.get("X-StripFeed-Fetch-Ms") ?? "0";
    const truncated = response.headers.get("X-StripFeed-Truncated") === "true";
  
    const contentType = response.headers.get("content-type") ?? "";
  
    if (contentType.includes("application/json")) {
      const json = await response.json();
      return {
        content: json.markdown,
        tokens: parseInt(tokens),
        originalTokens: parseInt(originalTokens),
        savingsPercent,
        cached,
        fetchMs,
        truncated,
        title: json.title,
      };
    }
  
    const content = await response.text();
    return {
      content,
      tokens: parseInt(tokens),
      originalTokens: parseInt(originalTokens),
      savingsPercent,
      cached,
      fetchMs,
      truncated,
    };
  }

• src/index.ts:20-28 (schema)

  TypeScript interface defining the input parameters for the fetch_url tool: url (required), format, selector, model, cache, ttl, and max_tokens (all optional).

  interface FetchParams {
    url: string;
    format?: string;
    selector?: string;
    model?: string;
    cache?: boolean;
    ttl?: number;
    max_tokens?: number;
  }

• src/index.ts:107-160 (registration)

  Registers the 'fetch_url' tool with the MCP server using the SDK's server.tool() method, including the Zod schema for parameter validation and the async handler that calls fetchUrl() and formats the result.

  server.tool(
    "fetch_url",
    "Convert any URL to clean, token-efficient Markdown. Strips ads, navigation, scripts, and noise. Returns clean content ready for LLM consumption.",
    {
      url: z.string().url().describe("The URL to fetch and convert to Markdown"),
      format: z
        .enum(["markdown", "json", "text", "html"])
        .optional()
        .describe("Output format: markdown (default), json, text, html"),
      selector: z
        .string()
        .optional()
        .describe(
          "CSS selector to extract specific elements (e.g. 'article', '.content', '#main')"
        ),
      model: z
        .string()
        .optional()
        .describe(
          "AI model ID for cost tracking (e.g. 'claude-sonnet-4-6', 'gpt-5')"
        ),
      cache: z
        .boolean()
        .optional()
        .describe("Set to false to bypass cache and force fresh fetch"),
      ttl: z
        .number()
        .optional()
        .describe("Cache TTL in seconds (default 3600, max 86400)"),
      max_tokens: z
        .number()
        .int()
        .positive()
        .optional()
        .describe("Truncate output to fit within this token budget"),
    },
    async (params) => {
      const result = await fetchUrl(params);
  
      const meta = [
        `Tokens: ${result.tokens.toLocaleString()} (saved ${result.savingsPercent}% from ${result.originalTokens.toLocaleString()})`,
        `Cache: ${result.cached}`,
        `Fetch: ${result.fetchMs}ms`,
      ];
      if (result.title) meta.unshift(`Title: ${result.title}`);
      if (result.truncated) meta.push("Truncated: yes");
  
      return {
        content: [
          { type: "text" as const, text: `${meta.join(" | ")}\n\n---\n\n${result.content}` },
        ],
      };
    }
  );

• src/index.ts:110-141 (schema)

  Zod schema definitions for runtime validation of the fetch_url tool's input parameters, including url (required URL), format (enum), selector, model, cache (boolean), ttl (number), and max_tokens (positive integer).

  {
    url: z.string().url().describe("The URL to fetch and convert to Markdown"),
    format: z
      .enum(["markdown", "json", "text", "html"])
      .optional()
      .describe("Output format: markdown (default), json, text, html"),
    selector: z
      .string()
      .optional()
      .describe(
        "CSS selector to extract specific elements (e.g. 'article', '.content', '#main')"
      ),
    model: z
      .string()
      .optional()
      .describe(
        "AI model ID for cost tracking (e.g. 'claude-sonnet-4-6', 'gpt-5')"
      ),
    cache: z
      .boolean()
      .optional()
      .describe("Set to false to bypass cache and force fresh fetch"),
    ttl: z
      .number()
      .optional()
      .describe("Cache TTL in seconds (default 3600, max 86400)"),
    max_tokens: z
      .number()
      .int()
      .positive()
      .optional()
      .describe("Truncate output to fit within this token budget"),