fetch_multiple

Instructions

Implementation Reference

• agentfetch/mcp/tools/fetch_multiple.py:9-29 (handler)

  Primary handler function for the fetch_multiple MCP tool. Validates input, delegates to fetch_many(), and returns {count, results}.

  def fetch_multiple(
      urls: List[str],
      max_tokens_each: Optional[int] = None,
      use_cache: bool = True,
  ) -> dict:
      """Fetch multiple URLs concurrently.
  
      Args:
          urls: List of URLs to fetch (recommend ≤20 per call).
          max_tokens_each: Optional cap on each URL's response size.
          use_cache: When true, prefer cached copies (≤6h old).
  
      Returns:
          `{count, results}` where each result is the same shape as `fetch_url`.
      """
      if not urls:
          return {"count": 0, "results": []}
      results = fetch_many(
          urls, max_tokens_each=max_tokens_each, use_cache=use_cache
      )
      return {"count": len(results), "results": results}

• agentfetch/mcp/server.py:136-161 (registration)

  MCP tool registration as @mcp.tool() decorator on fetch_multiple in the FastMCP server. This is where the tool is registered with the MCP framework under the name 'fetch_multiple'.

  @mcp.tool()
  def fetch_multiple(
      urls: List[str],
      max_tokens_each: Optional[int] = None,
      use_cache: bool = True,
  ) -> dict:
      """Fetch up to 20 URLs concurrently. Each result is the same shape as fetch_url.
  
      WHEN TO USE:
      - You have a list of URLs (search results, links from a doc, sitemap)
        and want them retrieved in parallel rather than one at a time.
  
      Args:
          urls: 1–20 URLs. Larger batches: split into multiple calls.
          max_tokens_each: Per-result cap. Apply this to keep total response
              inside your context budget — total ≈ len(urls) * max_tokens_each.
          use_cache: True for cache-aware fetching (default).
  
      Returns:
          {"count": int, "results": [<fetch_url shape>, ...]}
      """
      return _fetch_multiple(
          urls=urls,
          max_tokens_each=max_tokens_each,
          use_cache=use_cache,
      )

• agentfetch/core/pipeline.py:268-292 (helper)

  Core helper fetch_many() that calls fetch_pipeline() concurrently using a ThreadPoolExecutor. This is the actual workhorse invoked by the handler.

  def fetch_many(
      urls: List[str],
      *,
      max_tokens_each: Optional[int] = None,
      use_cache: bool = True,
      max_workers: int = 8,
  ) -> List[Dict[str, Any]]:
      """Fetch many URLs in parallel via a thread pool.
  
      Threads are fine here — each fetch is I/O-bound and httpx releases the GIL
      on socket reads.
      """
      if not urls:
          return []
      with ThreadPoolExecutor(max_workers=min(max_workers, len(urls))) as pool:
          futures = [
              pool.submit(
                  fetch_pipeline,
                  u,
                  max_tokens=max_tokens_each,
                  use_cache=use_cache,
              )
              for u in urls
          ]
          return [f.result() for f in futures]