mcp-server-webcrawl

mcp_server_webcrawl.crawlers.base.html•121 kB

<!DOCTYPE html> <html class="writer-html5" lang="en" data-content_root="./"> <head> <meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> <title>mcp_server_webcrawl.crawlers.base package — mcp-server-webcrawl documentation</title> <link rel="stylesheet" type="text/css" href="_static/pygments.css?v=80d5e7a1" /> <link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" /> <script src="_static/jquery.js?v=5d32c60e"></script> <script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script> <script src="_static/documentation_options.js?v=5929fcd5"></script> <script src="_static/doctools.js?v=888ff710"></script> <script src="_static/sphinx_highlight.js?v=dc90522c"></script> <script src="_static/js/theme.js"></script> <link rel="index" title="Index" href="genindex.html" /> <link rel="search" title="Search" href="search.html" /> <link rel="next" title="mcp_server_webcrawl.crawlers.archivebox package" href="mcp_server_webcrawl.crawlers.archivebox.html" /> <link rel="prev" title="mcp_server_webcrawl.crawlers package" href="mcp_server_webcrawl.crawlers.html" /> </head> <body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> <nav data-toggle="wy-nav-shift" class="wy-nav-side"> <div class="wy-side-scroll"> <div class="wy-side-nav-search" > <a href="index.html" class="icon icon-home"> mcp-server-webcrawl </a> <div role="search"> <form id="rtd-search-form" class="wy-form" action="search.html" method="get"> <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" /> <input type="hidden" name="check_keywords" value="yes" /> <input type="hidden" name="area" value="default" /> </form> </div> </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu"> Contents: <ul class="current"> <li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li> <li class="toctree-l1"><a class="reference internal" href="guides.html">Setup Guides</a></li> <li class="toctree-l1"><a class="reference internal" href="usage.html">Usage</a></li> <li class="toctree-l1"><a class="reference internal" href="prompts.html">Prompt Routines</a></li> <li class="toctree-l1"><a class="reference internal" href="interactive.html">Interactive Mode</a></li> <li class="toctree-l1 current"><a class="reference internal" href="modules.html">mcp_server_webcrawl</a><ul class="current"> <li class="toctree-l2 current"><a class="reference internal" href="mcp_server_webcrawl.html">mcp_server_webcrawl package</a></li> </ul> </li> </ul> </div> </div> </nav> <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" > <a href="index.html">mcp-server-webcrawl</a> </nav> <div class="wy-nav-content"> <div class="rst-content"> <div role="navigation" aria-label="Page navigation"> <ul class="wy-breadcrumbs"> <li><a href="index.html" class="icon icon-home" aria-label="Home"></a></li> <li class="breadcrumb-item"><a href="modules.html">mcp_server_webcrawl</a></li> <li class="breadcrumb-item"><a href="mcp_server_webcrawl.html">mcp_server_webcrawl package</a></li> <li class="breadcrumb-item"><a href="mcp_server_webcrawl.crawlers.html">mcp_server_webcrawl.crawlers package</a></li> <li class="breadcrumb-item active">mcp_server_webcrawl.crawlers.base package</li> <li class="wy-breadcrumbs-aside"> <a href="_sources/mcp_server_webcrawl.crawlers.base.rst.txt" rel="nofollow"> View page source</a> </li> </ul> <hr/> </div> <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> <div itemprop="articleBody"> <section id="mcp-server-webcrawl-crawlers-base-package"> <h1>mcp_server_webcrawl.crawlers.base package<a class="headerlink" href="#mcp-server-webcrawl-crawlers-base-package" title="Link to this heading"></a></h1> <section id="submodules"> <h2>Submodules<a class="headerlink" href="#submodules" title="Link to this heading"></a></h2> </section> <section id="module-mcp_server_webcrawl.crawlers.base.adapter"> <h2>mcp_server_webcrawl.crawlers.base.adapter module<a class="headerlink" href="#module-mcp_server_webcrawl.crawlers.base.adapter" title="Link to this heading"></a></h2> <dl class="py class"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexStatus"> class IndexStatus<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#IndexStatus">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexStatus" title="Link to this definition"></a></dt> <dd>Bases: <a class="reference external" href="https://docs.python.org/3/library/enum.html#enum.Enum" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">Enum</code></a> An enumeration. <dl class="py attribute"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.UNDEFINED"> UNDEFINED = ''<a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.UNDEFINED" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py attribute"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.IDLE"> IDLE = 'idle'<a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.IDLE" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py attribute"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.INDEXING"> INDEXING = 'indexing'<a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.INDEXING" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py attribute"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.PARTIAL"> PARTIAL = 'partial'<a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.PARTIAL" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py attribute"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.COMPLETE"> COMPLETE = 'complete'<a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.COMPLETE" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py attribute"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.REMOTE"> REMOTE = 'remote'<a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.REMOTE" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py attribute"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.FAILED"> FAILED = 'failed'<a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexStatus.FAILED" title="Link to this definition"></a></dt> <dd></dd></dl> </dd></dl> <dl class="py class"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexState"> class IndexState<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#IndexState">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState" title="Link to this definition"></a></dt> <dd>Bases: <a class="reference external" href="https://docs.python.org/3/library/functions.html#object" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">object</code></a> Shared state between crawler and manager for indexing progress <dl class="py attribute"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexState.status"> status: <a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexStatus" title="mcp_server_webcrawl.crawlers.base.adapter.IndexStatus">IndexStatus</a> = ''<a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState.status" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py attribute"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexState.processed"> processed: <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a> = 0<a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState.processed" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py attribute"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexState.time_start"> time_start: <a class="reference external" href="https://docs.python.org/3/library/datetime.html#datetime.datetime" title="(in Python v3.13)">datetime</a> | <a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)">None</a> = None<a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState.time_start" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py attribute"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexState.time_end"> time_end: <a class="reference external" href="https://docs.python.org/3/library/datetime.html#datetime.datetime" title="(in Python v3.13)">datetime</a> | <a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)">None</a> = None<a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState.time_end" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexState.set_status"> set_status(status)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#IndexState.set_status">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState.set_status" title="Link to this definition"></a></dt> <dd><dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd">status (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexStatus" title="mcp_server_webcrawl.crawlers.base.adapter.IndexStatus">IndexStatus</a>) – </dd> <dt class="field-even">Return type:</dt> <dd class="field-even">None </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexState.increment_processed"> increment_processed()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#IndexState.increment_processed">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState.increment_processed" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py property"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexState.duration"> property duration: <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState.duration" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexState.is_timeout"> is_timeout()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#IndexState.is_timeout">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState.is_timeout" title="Link to this definition"></a></dt> <dd>Check if the indexing operation has exceeded the timeout threshold <dl class="field-list simple"> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.13)">bool</a> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexState.to_dict"> to_dict()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#IndexState.to_dict">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState.to_dict" title="Link to this definition"></a></dt> <dd>Convert the IndexState to a dictionary representation <dl class="field-list simple"> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#dict" title="(in Python v3.13)">dict</a> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.IndexState.__init__"> __init__(status=IndexStatus.UNDEFINED, processed=0, time_start=None, time_end=None)<a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState.__init__" title="Link to this definition"></a></dt> <dd><dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>status (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexStatus" title="mcp_server_webcrawl.crawlers.base.adapter.IndexStatus">IndexStatus</a>) – </li> <li>processed (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – </li> <li>time_start (<a class="reference external" href="https://docs.python.org/3/library/datetime.html#datetime.datetime" title="(in Python v3.13)">datetime</a> | None) – </li> <li>time_end (<a class="reference external" href="https://docs.python.org/3/library/datetime.html#datetime.datetime" title="(in Python v3.13)">datetime</a> | None) – </li> </ul> </dd> <dt class="field-even">Return type:</dt> <dd class="field-even">None </dd> </dl> </dd></dl> </dd></dl> <dl class="py class"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.SitesGroup"> class SitesGroup<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#SitesGroup">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.SitesGroup" title="Link to this definition"></a></dt> <dd>Bases: <a class="reference external" href="https://docs.python.org/3/library/functions.html#object" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">object</code></a> Container class supports the searching of one or more sites at once. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>datasrc – site datasrc</li> <li>site_ids – site ids of the sites</li> <li>site_paths – paths to site contents (directories)</li> </ul> </dd> </dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.SitesGroup.__init__"> __init__(datasrc, site_ids, site_paths)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#SitesGroup.__init__">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.SitesGroup.__init__" title="Link to this definition"></a></dt> <dd>Container class supports the searching of one or more sites at once. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>datasrc (<a class="reference external" href="https://docs.python.org/3/library/pathlib.html#pathlib.Path" title="(in Python v3.13)">Path</a>) – site datasrc</li> <li>site_ids (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>]) – site ids of the sites</li> <li>site_paths (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/pathlib.html#pathlib.Path" title="(in Python v3.13)">Path</a>]) – paths to site contents (directories)</li> </ul> </dd> <dt class="field-even">Return type:</dt> <dd class="field-even">None </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.SitesGroup.get_sites"> get_sites()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#SitesGroup.get_sites">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.SitesGroup.get_sites" title="Link to this definition"></a></dt> <dd><dl class="field-list simple"> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#dict" title="(in Python v3.13)">dict</a>[<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>, <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>] </dd> </dl> </dd></dl> </dd></dl> <dl class="py class"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.SitesStat"> class SitesStat<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#SitesStat">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.SitesStat" title="Link to this definition"></a></dt> <dd>Bases: <a class="reference external" href="https://docs.python.org/3/library/functions.html#object" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">object</code></a> Some basic bookeeping, for troubleshooting <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>group – SitesGroup to track statistics for</li> <li>cached – whether the group was retrieved from cache</li> </ul> </dd> </dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.SitesStat.__init__"> __init__(group, cached)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#SitesStat.__init__">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.SitesStat.__init__" title="Link to this definition"></a></dt> <dd>Some basic bookeeping, for troubleshooting <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>group (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.adapter.SitesGroup" title="mcp_server_webcrawl.crawlers.base.adapter.SitesGroup">SitesGroup</a>) – SitesGroup to track statistics for</li> <li>cached (<a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.13)">bool</a>) – whether the group was retrieved from cache</li> </ul> </dd> <dt class="field-even">Return type:</dt> <dd class="field-even">None </dd> </dl> </dd></dl> </dd></dl> <dl class="py class"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.BaseManager"> class BaseManager<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#BaseManager">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.BaseManager" title="Link to this definition"></a></dt> <dd>Bases: <a class="reference external" href="https://docs.python.org/3/library/functions.html#object" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">object</code></a> Base class for managing web crawler data in in-memory SQLite databases. Provides connection pooling and caching for efficient access. Initialize the manager with statistics. <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.BaseManager.__init__"> __init__()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#BaseManager.__init__">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.BaseManager.__init__" title="Link to this definition"></a></dt> <dd>Initialize the manager with statistics. <dl class="field-list simple"> <dt class="field-odd">Return type:</dt> <dd class="field-odd">None </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.BaseManager.string_to_id"> static string_to_id(value)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#BaseManager.string_to_id">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.BaseManager.string_to_id" title="Link to this definition"></a></dt> <dd>Convert a string, such as a directory name, to a numeric ID suitable for a database primary key. Hash space and collision probability notes: - [:8] = 32 bits (4.29 billion values) - ~1% collision chance with 10,000 items - [:12] = 48 bits (280 trillion values) - ~0.0000001% collision chance with 10,000 items - [:16] = 64 bits (max safe SQLite INTEGER) - near-zero collision, 9.22 quintillion values - SQLite INTEGER type is 64-bit signed, with max value of 9,223,372,036,854,775,807. - The big problem with larger hashspaces is the length of the ids they generate for presentation. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd">value (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>) – Input string to convert to an ID </dd> <dt class="field-even">Returns:</dt> <dd class="field-even">Integer ID derived from the input string </dd> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.BaseManager.get_basic_headers"> static get_basic_headers(file_size, resource_type, path)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#BaseManager.get_basic_headers">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.BaseManager.get_basic_headers" title="Link to this definition"></a></dt> <dd>Generate basic HTTP headers for a resource. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>file_size (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – size of the file in bytes</li> <li>resource_type (<a class="reference internal" href="mcp_server_webcrawl.models.html#mcp_server_webcrawl.models.resources.ResourceResultType" title="mcp_server_webcrawl.models.resources.ResourceResultType">ResourceResultType</a>) – type of resource to generate headers for</li> <li>path (<a class="reference external" href="https://docs.python.org/3/library/pathlib.html#pathlib.Path" title="(in Python v3.13)">Path</a>) – file path used for MIME type detection</li> </ul> </dd> <dt class="field-even">Returns:</dt> <dd class="field-even">HTTP headers string with content type and length </dd> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.BaseManager.read_files"> static read_files(paths)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#BaseManager.read_files">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.BaseManager.read_files" title="Link to this definition"></a></dt> <dd>Read content from multiple files concurrently. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd">paths (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/pathlib.html#pathlib.Path" title="(in Python v3.13)">Path</a>]) – list of file paths to read </dd> <dt class="field-even">Returns:</dt> <dd class="field-even">dictionary mapping file paths to their content or None for binary/unreadable files </dd> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#dict" title="(in Python v3.13)">dict</a>[<a class="reference external" href="https://docs.python.org/3/library/pathlib.html#pathlib.Path" title="(in Python v3.13)">Path</a>, <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a> | None] </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.BaseManager.read_file_contents"> static read_file_contents(file_path, resource_type)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#BaseManager.read_file_contents">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.BaseManager.read_file_contents" title="Link to this definition"></a></dt> <dd>Read content from text files with better error handling and encoding detection. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>file_path (<a class="reference external" href="https://docs.python.org/3/library/pathlib.html#pathlib.Path" title="(in Python v3.13)">Path</a>) – path to the file to read</li> <li>resource_type (<a class="reference internal" href="mcp_server_webcrawl.models.html#mcp_server_webcrawl.models.resources.ResourceResultType" title="mcp_server_webcrawl.models.resources.ResourceResultType">ResourceResultType</a>) – type of resource to determine if content should be read</li> </ul> </dd> <dt class="field-even">Returns:</dt> <dd class="field-even">file content as string or None for binary/unreadable files </dd> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a> | None </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.BaseManager.decruft_path"> static decruft_path(path)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#BaseManager.decruft_path">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.BaseManager.decruft_path" title="Link to this definition"></a></dt> <dd>Very light touch cleanup of file naming, these tmps are creating noise and extensions are useful in classifying resources <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd">path (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>) – file path string to clean up </dd> <dt class="field-even">Returns:</dt> <dd class="field-even">cleaned path string with temp files and weird extensions normalized </dd> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.BaseManager.get_stats"> get_stats()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#BaseManager.get_stats">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.BaseManager.get_stats" title="Link to this definition"></a></dt> <dd><dl class="field-list simple"> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.adapter.SitesStat" title="mcp_server_webcrawl.crawlers.base.adapter.SitesStat">SitesStat</a>] </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.adapter.BaseManager.get_resources_for_sites_group"> get_resources_for_sites_group(sites_group, query, fields, sort, limit, offset, swap_values={})<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/adapter.html#BaseManager.get_resources_for_sites_group">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.adapter.BaseManager.get_resources_for_sites_group" title="Link to this definition"></a></dt> <dd>Get resources from directories using structured query parsing with SearchQueryParser. This method extracts types, fields, and statuses from the querystring instead of accepting them as separate arguments, using the new SearchSubquery functionality. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>sites_group (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.adapter.SitesGroup" title="mcp_server_webcrawl.crawlers.base.adapter.SitesGroup">SitesGroup</a>) – Group of sites to search in</li> <li>query (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>) – Search query string that can include field:value syntax for filtering</li> <li>fields (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>] | None) – resource fields to be returned by the API (Content, Headers, etc.)</li> <li>sort (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a> | None) – Sort order for results</li> <li>limit (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – Maximum number of results to return</li> <li>offset (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – Number of results to skip for pagination</li> <li>swap_values (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#dict" title="(in Python v3.13)">dict</a>) – per-field parameterized values to check for (and replace)</li> </ul> </dd> <dt class="field-even">Returns:</dt> <dd class="field-even">Tuple of (list of ResourceResult objects, total count, connection_index_state) </dd> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#tuple" title="(in Python v3.13)">tuple</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference internal" href="mcp_server_webcrawl.models.html#mcp_server_webcrawl.models.resources.ResourceResult" title="mcp_server_webcrawl.models.resources.ResourceResult">ResourceResult</a>], <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>, <a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState" title="mcp_server_webcrawl.crawlers.base.adapter.IndexState">IndexState</a>] </dd> </dl> Notes Returns empty results if sites is empty or not provided. If the database is being built, it will log a message and return empty results. This method extracts field-specific filters from the query string using SearchQueryParser: - type:html (to filter by resource type) - status:200 (to filter by HTTP status) Any fields present in the SearchSubquery will be included in the response. </dd></dl> </dd></dl> </section> <section id="module-mcp_server_webcrawl.crawlers.base.api"> <h2>mcp_server_webcrawl.crawlers.base.api module<a class="headerlink" href="#module-mcp_server_webcrawl.crawlers.base.api" title="Link to this heading"></a></h2> <dl class="py class"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.api.BaseJsonApiEncoder"> class BaseJsonApiEncoder<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/api.html#BaseJsonApiEncoder">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.api.BaseJsonApiEncoder" title="Link to this definition"></a></dt> <dd>Bases: <code class="xref py py-class docutils literal notranslate">JSONEncoder</code> Custom JSON encoder for BaseJsonApi objects and ResourceResultType enums. Constructor for JSONEncoder, with sensible defaults. If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, float or None. If skipkeys is True, such items are simply skipped. If ensure_ascii is true, the output is guaranteed to be str objects with all incoming non-ASCII characters escaped. If ensure_ascii is false, the output can contain non-ASCII characters. If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an OverflowError). Otherwise, no such check takes place. If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats. If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis. If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation. If specified, separators should be an (item_separator, key_separator) tuple. The default is (’, ‘, ‘: ‘) if indent is <code class="docutils literal notranslate">None</code> and (‘,’, ‘: ‘) otherwise. To get the most compact JSON representation, you should specify (‘,’, ‘:’) to eliminate whitespace. If specified, default is a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a <code class="docutils literal notranslate">TypeError</code>. <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.api.BaseJsonApiEncoder.default"> default(obj)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/api.html#BaseJsonApiEncoder.default">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.api.BaseJsonApiEncoder.default" title="Link to this definition"></a></dt> <dd>Override default encoder to handle custom types. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd">obj – Object to encode </dd> <dt class="field-even">Returns:</dt> <dd class="field-even">JSON serializable representation of the object </dd> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)">Any</a> </dd> </dl> </dd></dl> </dd></dl> <dl class="py class"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.api.BaseJsonApi"> class BaseJsonApi<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/api.html#BaseJsonApi">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.api.BaseJsonApi" title="Link to this definition"></a></dt> <dd>Bases: <a class="reference external" href="https://docs.python.org/3/library/functions.html#object" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">object</code></a> Base class for JSON API responses. Provides a standardized structure for API responses including metadata, results, and error handling. Construct with the arguments of creation (aoc), these will be echoed back in JSON response. This is an object that collapses into json on json dumps. This is done with everything within implementing to_dict. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>method – API method name</li> <li>args – Dictionary of API arguments</li> <li>index_state – indexing, complete, remote, etc.</li> </ul> </dd> </dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.__init__"> __init__(method, args, index_state=None)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/api.html#BaseJsonApi.__init__">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.__init__" title="Link to this definition"></a></dt> <dd>Construct with the arguments of creation (aoc), these will be echoed back in JSON response. This is an object that collapses into json on json dumps. This is done with everything within implementing to_dict. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>method (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>) – API method name</li> <li>args (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#dict" title="(in Python v3.13)">dict</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>, <a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)">Any</a>]) – Dictionary of API arguments</li> <li>index_state (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState" title="mcp_server_webcrawl.crawlers.base.adapter.IndexState">IndexState</a> | None) – indexing, complete, remote, etc.</li> </ul> </dd> </dl> </dd></dl> <dl class="py property"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.total"> property total: <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.total" title="Link to this definition"></a></dt> <dd>Returns the total number of results. <dl class="field-list simple"> <dt class="field-odd">Returns:</dt> <dd class="field-odd">Integer count of total results </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.get_results"> get_results()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/api.html#BaseJsonApi.get_results">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.get_results" title="Link to this definition"></a></dt> <dd>Returns list of results. <dl class="field-list simple"> <dt class="field-odd">Returns:</dt> <dd class="field-odd">Results of type SiteResult or ResourceResult </dd> <dt class="field-even">Return type:</dt> <dd class="field-even"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference internal" href="mcp_server_webcrawl.models.html#mcp_server_webcrawl.models.sites.SiteResult" title="mcp_server_webcrawl.models.sites.SiteResult">SiteResult</a> | <a class="reference internal" href="mcp_server_webcrawl.models.html#mcp_server_webcrawl.models.resources.ResourceResult" title="mcp_server_webcrawl.models.resources.ResourceResult">ResourceResult</a>] </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.set_results"> set_results(results, total, offset, limit)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/api.html#BaseJsonApi.set_results">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.set_results" title="Link to this definition"></a></dt> <dd>Set the results of the API response. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>results (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference internal" href="mcp_server_webcrawl.models.html#mcp_server_webcrawl.models.sites.SiteResult" title="mcp_server_webcrawl.models.sites.SiteResult">SiteResult</a> | <a class="reference internal" href="mcp_server_webcrawl.models.html#mcp_server_webcrawl.models.resources.ResourceResult" title="mcp_server_webcrawl.models.resources.ResourceResult">ResourceResult</a>]) – List of result objects</li> <li>total (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – Total number of results (including those beyond limit)</li> <li>offset (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – Starting position in the full result set</li> <li>limit (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – Maximum number of results to include</li> </ul> </dd> <dt class="field-even">Return type:</dt> <dd class="field-even">None </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.append_error"> append_error(message)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/api.html#BaseJsonApi.append_error">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.append_error" title="Link to this definition"></a></dt> <dd>Add an error to the JSON response, visible to the endpoint LLM. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd">message (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>) – Error message to add </dd> <dt class="field-even">Return type:</dt> <dd class="field-even">None </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.to_dict"> to_dict()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/api.html#BaseJsonApi.to_dict">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.to_dict" title="Link to this definition"></a></dt> <dd>Convert the object to a JSON-serializable dictionary. <dl class="field-list simple"> <dt class="field-odd">Returns:</dt> <dd class="field-odd">Dictionary representation of the API response </dd> <dt class="field-even">Return type:</dt> <dd class="field-even"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#dict" title="(in Python v3.13)">dict</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>, <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a> | <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a> | <a class="reference external" href="https://docs.python.org/3/library/functions.html#float" title="(in Python v3.13)">float</a> | <a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.13)">bool</a> | <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>] | <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>] | <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/functions.html#float" title="(in Python v3.13)">float</a>] | None] </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.to_json"> to_json()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/api.html#BaseJsonApi.to_json">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.api.BaseJsonApi.to_json" title="Link to this definition"></a></dt> <dd>Return a JSON serializable representation of this object. <dl class="field-list simple"> <dt class="field-odd">Returns:</dt> <dd class="field-odd">JSON string representation of the API response </dd> <dt class="field-even">Return type:</dt> <dd class="field-even"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a> </dd> </dl> </dd></dl> </dd></dl> </section> <section id="module-mcp_server_webcrawl.crawlers.base.crawler"> <h2>mcp_server_webcrawl.crawlers.base.crawler module<a class="headerlink" href="#module-mcp_server_webcrawl.crawlers.base.crawler" title="Link to this heading"></a></h2> <dl class="py class"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler"> class BaseCrawler<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler" title="Link to this definition"></a></dt> <dd>Bases: <a class="reference external" href="https://docs.python.org/3/library/functions.html#object" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">object</code></a> Base crawler class that implements MCP server functionality. This class provides the foundation for specialized crawlers to interact with the MCP server and handle tool operations for web resources. Initialize the BaseCrawler with a data source path and required adapter functions. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>datasrc – path to the data source</li> <li>get_sites_func – function to retrieve sites from the data source</li> <li>get_resources_func – function to retrieve resources from the data source</li> <li>resource_field_mapping – mapping of resource field names to display names</li> </ul> </dd> </dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.__init__"> __init__(datasrc, get_sites_func, get_resources_func, resource_field_mapping={'content': 'ResourcesFullText.Content', 'created': 'Resources.Created', 'fulltext': 'ResourcesFullText', 'headers': 'ResourcesFullText.Headers', 'id': 'ResourcesFullText.Id', 'modified': 'Resources.Modified', 'site': 'ResourcesFullText.Project', 'size': 'Resources.Size', 'status': 'Resources.Status', 'time': 'Resources.Time', 'type': 'ResourcesFullText.Type', 'url': 'ResourcesFullText.Url'})<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler.__init__">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.__init__" title="Link to this definition"></a></dt> <dd>Initialize the BaseCrawler with a data source path and required adapter functions. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>datasrc (<a class="reference external" href="https://docs.python.org/3/library/pathlib.html#pathlib.Path" title="(in Python v3.13)">Path</a>) – path to the data source</li> <li>get_sites_func (<a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Callable" title="(in Python v3.13)">Callable</a>) – function to retrieve sites from the data source</li> <li>get_resources_func (<a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Callable" title="(in Python v3.13)">Callable</a>) – function to retrieve resources from the data source</li> <li>resource_field_mapping (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#dict" title="(in Python v3.13)">dict</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>, <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>]) – mapping of resource field names to display names</li> </ul> </dd> <dt class="field-even">Return type:</dt> <dd class="field-even">None </dd> </dl> </dd></dl> <dl class="py property"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.datasrc"> property datasrc: <a class="reference external" href="https://docs.python.org/3/library/pathlib.html#pathlib.Path" title="(in Python v3.13)">Path</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.datasrc" title="Link to this definition"></a></dt> <dd></dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.mcp_list_prompts"> async mcp_list_prompts()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler.mcp_list_prompts">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.mcp_list_prompts" title="Link to this definition"></a></dt> <dd>List available prompts (currently none). <dl class="field-list simple"> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.mcp_list_resources"> async mcp_list_resources()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler.mcp_list_resources">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.mcp_list_resources" title="Link to this definition"></a></dt> <dd>List available resources (currently none). <dl class="field-list simple"> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.serve"> async serve(stdin, stdout)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler.serve">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.serve" title="Link to this definition"></a></dt> <dd>Launch the awaitable server. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>stdin (AsyncFile[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>] | None) – input stream for the server</li> <li>stdout (AsyncFile[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>] | None) – output stream for the server</li> </ul> </dd> <dt class="field-even">Returns:</dt> <dd class="field-even">The MCP server over stdio </dd> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#dict" title="(in Python v3.13)">dict</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>, <a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)">Any</a>] </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.get_initialization_options"> get_initialization_options()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler.get_initialization_options">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.get_initialization_options" title="Link to this definition"></a></dt> <dd>Get the MCP initialization object. <dl class="field-list simple"> <dt class="field-odd">Returns:</dt> <dd class="field-odd">Dictionary containing project information </dd> <dt class="field-even">Return type:</dt> <dd class="field-even">InitializationOptions </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.get_sites_api_json"> get_sites_api_json(**kwargs)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler.get_sites_api_json">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.get_sites_api_json" title="Link to this definition"></a></dt> <dd>Get sites API result as JSON. <dl class="field-list simple"> <dt class="field-odd">Returns:</dt> <dd class="field-odd">JSON string of sites API results </dd> <dt class="field-even">Return type:</dt> <dd class="field-even"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.get_resources_api_json"> get_resources_api_json(**kwargs)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler.get_resources_api_json">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.get_resources_api_json" title="Link to this definition"></a></dt> <dd>Get resources API result as JSON. <dl class="field-list simple"> <dt class="field-odd">Returns:</dt> <dd class="field-odd">JSON string of resources API results </dd> <dt class="field-even">Return type:</dt> <dd class="field-even"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.get_sites_api"> get_sites_api(ids=None, fields=None)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler.get_sites_api">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.get_sites_api" title="Link to this definition"></a></dt> <dd><dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>ids (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>] | None) – </li> <li>fields (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>] | None) – </li> </ul> </dd> <dt class="field-even">Return type:</dt> <dd class="field-even"><a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.api.BaseJsonApi" title="mcp_server_webcrawl.crawlers.base.api.BaseJsonApi">BaseJsonApi</a> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.get_resources_api"> get_resources_api(sites=None, query='', fields=None, sort=None, limit=20, offset=0, extras=None, extrasRegex=None, extrasXpath=None)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler.get_resources_api">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.get_resources_api" title="Link to this definition"></a></dt> <dd><dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>sites (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>] | None) – </li> <li>query (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>) – </li> <li>fields (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>] | None) – </li> <li>sort (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a> | None) – </li> <li>limit (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – </li> <li>offset (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – </li> <li>extras (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>] | None) – </li> <li>extrasRegex (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>] | None) – </li> <li>extrasXpath (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>] | None) – </li> </ul> </dd> <dt class="field-even">Return type:</dt> <dd class="field-even"><a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.api.BaseJsonApi" title="mcp_server_webcrawl.crawlers.base.api.BaseJsonApi">BaseJsonApi</a> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.mcp_list_tools"> async mcp_list_tools()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler.mcp_list_tools">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.mcp_list_tools" title="Link to this definition"></a></dt> <dd>List available tools. <dl class="field-list simple"> <dt class="field-odd">Returns:</dt> <dd class="field-odd">List of available tools </dd> <dt class="field-even">Raises:</dt> <dd class="field-even"><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#NotImplementedError" title="(in Python v3.13)">NotImplementedError</a> – This method must be implemented by subclasses </dd> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[Tool] </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.mcp_call_tool"> async mcp_call_tool(name, arguments)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler.mcp_call_tool">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.mcp_call_tool" title="Link to this definition"></a></dt> <dd>Handle tool execution requests. You can override this or super(), then tweak. Basically, it is a passthrough. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>name (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>) – name of the tool to call</li> <li>arguments (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#dict" title="(in Python v3.13)">dict</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>, <a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)">Any</a>] | None) – arguments to pass to the tool</li> </ul> </dd> <dt class="field-even">Returns:</dt> <dd class="field-even">List of content objects resulting from the tool execution </dd> <dt class="field-odd">Raises:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#ValueError" title="(in Python v3.13)">ValueError</a> – If the specified tool does not exist </dd> <dt class="field-even">Return type:</dt> <dd class="field-even"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[TextContent | ImageContent | EmbeddedResource] </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.get_thumbnails"> get_thumbnails(results)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/crawler.html#BaseCrawler.get_thumbnails">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler.get_thumbnails" title="Link to this definition"></a></dt> <dd><dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd">results (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference internal" href="mcp_server_webcrawl.models.html#mcp_server_webcrawl.models.resources.ResourceResult" title="mcp_server_webcrawl.models.resources.ResourceResult">ResourceResult</a>]) – </dd> <dt class="field-even">Return type:</dt> <dd class="field-even"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[ImageContent] </dd> </dl> </dd></dl> </dd></dl> </section> <section id="module-mcp_server_webcrawl.crawlers.base.indexed"> <h2>mcp_server_webcrawl.crawlers.base.indexed module<a class="headerlink" href="#module-mcp_server_webcrawl.crawlers.base.indexed" title="Link to this heading"></a></h2> <dl class="py class"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.indexed.IndexedManager"> class IndexedManager<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/indexed.html#IndexedManager">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.indexed.IndexedManager" title="Link to this definition"></a></dt> <dd>Bases: <a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.adapter.BaseManager" title="mcp_server_webcrawl.crawlers.base.adapter.BaseManager"><code class="xref py py-class docutils literal notranslate">BaseManager</code></a> Initialize the manager with statistics. <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.indexed.IndexedManager.__init__"> __init__()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/indexed.html#IndexedManager.__init__">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.indexed.IndexedManager.__init__" title="Link to this definition"></a></dt> <dd>Initialize the manager with statistics. </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.indexed.IndexedManager.get_connection"> get_connection(group)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/indexed.html#IndexedManager.get_connection">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.indexed.IndexedManager.get_connection" title="Link to this definition"></a></dt> <dd>Get database connection for sites in the group, creating if needed. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd">group (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.adapter.SitesGroup" title="mcp_server_webcrawl.crawlers.base.adapter.SitesGroup">SitesGroup</a>) – group of sites to connect to </dd> <dt class="field-even">Returns:</dt> <dd class="field-even"><dl class="simple"> <dt>Tuple of (SQLite connection to in-memory database with data loaded or None if building,</dt><dd>IndexState associated with this database) </dd> </dl> </dd> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#tuple" title="(in Python v3.13)">tuple</a>[<a class="reference external" href="https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection" title="(in Python v3.13)">Connection</a> | None, <a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.adapter.IndexState" title="mcp_server_webcrawl.crawlers.base.adapter.IndexState">IndexState</a>] </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.indexed.IndexedManager.get_sites_for_directories"> get_sites_for_directories(datasrc, ids=None, fields=None)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/indexed.html#IndexedManager.get_sites_for_directories">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.indexed.IndexedManager.get_sites_for_directories" title="Link to this definition"></a></dt> <dd>List site directories in the datasrc directory as sites. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>datasrc (<a class="reference external" href="https://docs.python.org/3/library/pathlib.html#pathlib.Path" title="(in Python v3.13)">Path</a>) – path to the directory containing site subdirectories</li> <li>ids (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>] | None) – optional list of site IDs to filter by</li> <li>fields (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>] | None) – optional list of fields to include in the response</li> </ul> </dd> <dt class="field-even">Returns:</dt> <dd class="field-even">List of SiteResult objects, one for each site directory </dd> <dt class="field-odd">Return type:</dt> <dd class="field-odd"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[<a class="reference internal" href="mcp_server_webcrawl.models.html#mcp_server_webcrawl.models.sites.SiteResult" title="mcp_server_webcrawl.models.sites.SiteResult">SiteResult</a>] </dd> </dl> Notes Returns an empty list if the datasrc directory doesn’t exist. </dd></dl> </dd></dl> <dl class="py class"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.indexed.IndexedCrawler"> class IndexedCrawler<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/indexed.html#IndexedCrawler">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.indexed.IndexedCrawler" title="Link to this definition"></a></dt> <dd>Bases: <a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler" title="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler"><code class="xref py py-class docutils literal notranslate">BaseCrawler</code></a> A crawler implementation for data sources that load into an in-memory sqlite. Shares commonality between specialized crawlers. Initialize the IndexedCrawler with a data source path and required adapter functions. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>datasrc – path to the data source</li> <li>get_sites_func – function to retrieve sites from the data source</li> <li>get_resources_func – function to retrieve resources from the data source</li> <li>resource_field_mapping – mapping of resource field names to display names</li> </ul> </dd> </dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.indexed.IndexedCrawler.__init__"> __init__(datasrc, get_sites_func, get_resources_func, resource_field_mapping={'content': 'ResourcesFullText.Content', 'created': 'Resources.Created', 'fulltext': 'ResourcesFullText', 'headers': 'ResourcesFullText.Headers', 'id': 'ResourcesFullText.Id', 'modified': 'Resources.Modified', 'site': 'ResourcesFullText.Project', 'size': 'Resources.Size', 'status': 'Resources.Status', 'time': 'Resources.Time', 'type': 'ResourcesFullText.Type', 'url': 'ResourcesFullText.Url'})<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/indexed.html#IndexedCrawler.__init__">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.indexed.IndexedCrawler.__init__" title="Link to this definition"></a></dt> <dd>Initialize the IndexedCrawler with a data source path and required adapter functions. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>datasrc (<a class="reference external" href="https://docs.python.org/3/library/pathlib.html#pathlib.Path" title="(in Python v3.13)">Path</a>) – path to the data source</li> <li>get_sites_func (<a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Callable" title="(in Python v3.13)">Callable</a>) – function to retrieve sites from the data source</li> <li>get_resources_func (<a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Callable" title="(in Python v3.13)">Callable</a>) – function to retrieve resources from the data source</li> <li>resource_field_mapping (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#dict" title="(in Python v3.13)">dict</a>[<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>, <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>]) – mapping of resource field names to display names</li> </ul> </dd> <dt class="field-even">Return type:</dt> <dd class="field-even">None </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.indexed.IndexedCrawler.mcp_list_tools"> async mcp_list_tools()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/indexed.html#IndexedCrawler.mcp_list_tools">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.indexed.IndexedCrawler.mcp_list_tools" title="Link to this definition"></a></dt> <dd>List available tools for this crawler. <dl class="field-list simple"> <dt class="field-odd">Returns:</dt> <dd class="field-odd">List of Tool objects </dd> <dt class="field-even">Return type:</dt> <dd class="field-even"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)">list</a>[Tool] </dd> </dl> </dd></dl> </dd></dl> </section> <section id="module-mcp_server_webcrawl.crawlers.base.tests"> <h2>mcp_server_webcrawl.crawlers.base.tests module<a class="headerlink" href="#module-mcp_server_webcrawl.crawlers.base.tests" title="Link to this heading"></a></h2> <dl class="py class"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests"> class BaseCrawlerTests<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/tests.html#BaseCrawlerTests">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests" title="Link to this definition"></a></dt> <dd>Bases: <code class="xref py py-class docutils literal notranslate">TestCase</code> Create an instance of the class that will use the named test method when executed. Raises a ValueError if the instance does not have a method with the specified name. <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.setUp"> setUp()<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/tests.html#BaseCrawlerTests.setUp">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.setUp" title="Link to this definition"></a></dt> <dd>Hook method for setting up the test fixture before exercising it. </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_search_tests"> run_pragmar_search_tests(crawler, site_id)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/tests.html#BaseCrawlerTests.run_pragmar_search_tests">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_search_tests" title="Link to this definition"></a></dt> <dd>Run a battery of database checks on the crawler and Boolean validation <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>crawler (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler" title="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler">BaseCrawler</a>) – </li> <li>site_id (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – </li> </ul> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_image_tests"> run_pragmar_image_tests(crawler, pragmar_site_id)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/tests.html#BaseCrawlerTests.run_pragmar_image_tests">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_image_tests" title="Link to this definition"></a></dt> <dd>Test InterroBot-specific image handling and thumbnails. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>crawler (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler" title="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler">BaseCrawler</a>) – </li> <li>pragmar_site_id (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – </li> </ul> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_sites_resources_tests"> run_sites_resources_tests(crawler, pragmar_site_id, example_site_id)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/tests.html#BaseCrawlerTests.run_sites_resources_tests">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_sites_resources_tests" title="Link to this definition"></a></dt> <dd><dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>crawler (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler" title="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler">BaseCrawler</a>) – </li> <li>pragmar_site_id (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – </li> <li>example_site_id (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – </li> </ul> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_tokenizer_tests"> run_pragmar_tokenizer_tests(crawler, site_id)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/tests.html#BaseCrawlerTests.run_pragmar_tokenizer_tests">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_tokenizer_tests" title="Link to this definition"></a></dt> <dd>fts hyphens and underscores are particularly challenging, thus have a dedicated test. these must be configured in multiple places including CREATE TABLE … tokenizer, as well as handled by the query parser. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>crawler (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler" title="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler">BaseCrawler</a>) – </li> <li>site_id (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – </li> </ul> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_site_tests"> run_pragmar_site_tests(crawler, site_id)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/tests.html#BaseCrawlerTests.run_pragmar_site_tests">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_site_tests" title="Link to this definition"></a></dt> <dd><dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>crawler (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler" title="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler">BaseCrawler</a>) – </li> <li>site_id (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – </li> </ul> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_sort_tests"> run_pragmar_sort_tests(crawler, site_id)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/tests.html#BaseCrawlerTests.run_pragmar_sort_tests">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_sort_tests" title="Link to this definition"></a></dt> <dd>Test sorting functionality with performance optimizations. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>crawler (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler" title="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler">BaseCrawler</a>) – </li> <li>site_id (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – </li> </ul> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_content_tests"> run_pragmar_content_tests(crawler, site_id, html_leniency)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/tests.html#BaseCrawlerTests.run_pragmar_content_tests">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_content_tests" title="Link to this definition"></a></dt> <dd><dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>crawler (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler" title="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler">BaseCrawler</a>) – </li> <li>site_id (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – </li> <li>html_leniency (<a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.13)">bool</a>) – </li> </ul> </dd> </dl> </dd></dl> <dl class="py method"> <dt class="sig sig-object py" id="mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_report"> run_pragmar_report(crawler, site_id, heading)<a class="reference internal" href="_modules/mcp_server_webcrawl/crawlers/base/tests.html#BaseCrawlerTests.run_pragmar_report">[source]</a><a class="headerlink" href="#mcp_server_webcrawl.crawlers.base.tests.BaseCrawlerTests.run_pragmar_report" title="Link to this definition"></a></dt> <dd>Generate a comprehensive report of all resources for a site. Returns a formatted string with counts and URLs by type. <dl class="field-list simple"> <dt class="field-odd">Parameters:</dt> <dd class="field-odd"><ul class="simple"> <li>crawler (<a class="reference internal" href="#mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler" title="mcp_server_webcrawl.crawlers.base.crawler.BaseCrawler">BaseCrawler</a>) – </li> <li>site_id (<a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)">int</a>) – </li> <li>heading (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)">str</a>) – </li> </ul> </dd> </dl> </dd></dl> </dd></dl> </section> <section id="module-mcp_server_webcrawl.crawlers.base"> <h2>Module contents<a class="headerlink" href="#module-mcp_server_webcrawl.crawlers.base" title="Link to this heading"></a></h2> </section> </section> </div> </div> <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer"> <a href="mcp_server_webcrawl.crawlers.html" class="btn btn-neutral float-left" title="mcp_server_webcrawl.crawlers package" accesskey="p" rel="prev"> Previous</a> <a href="mcp_server_webcrawl.crawlers.archivebox.html" class="btn btn-neutral float-right" title="mcp_server_webcrawl.crawlers.archivebox package" accesskey="n" rel="next">Next </a> </div> <hr/> <div role="contentinfo"> © Copyright 2025, pragmar. </div> Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. </footer> </div> </div> </section> </div> <script> jQuery(function () { SphinxRtdTheme.Navigation.enable(true); }); </script> </body> </html>

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/pragmar/mcp_server_webcrawl'

If you have feedback or need assistance with the MCP directory API, please join our Discord server