o
l�qh�U � @ sZ
d dl Z d dlZd dlZzd dlmZ e� W n
ey$ ed� Y nw d dlmZ d dl m
Z
d dlZd dl
Z
d dlZd dlmZ d dlmZmZmZmZmZ d dlZd dlZd dlZd dlZzd dlZdZW n eyt d ZY nw zd dlZdZW n ey� d ZY nw z
d d
lmZ dZ W n ey� d Z Y nw zd dl!Z"d dl#m$Z% d dl&m'Z' d dl(m)Z) d dl*Z+dZ,W n ey� d Z,ed� Y nw dZ-e j.�/d
d�Z0e j.�/dd�Z1e j.�/dd�Z2e j.�/dd�Z3e j.�/dd�Z4e j.�/dd�Z5e j.�/dd�Z6e j.�/de0�Z7e j.�/dd�Z8e j.�/dd�Z9g a:dd � Z;e0fd!e<d"ee< fd#d$�Z=e1fd!e<d"ee< fd%d&�Z>d'e<d(e<d"ee< fd)d*�Z?e@e j.�/d+d,��ZAe@e j.�/d-d.��ZBe@e j.�/d/d0��ZCd1d2� ZDd3d4� ZEd�d6e<d"e<fd7d8�ZFe5e6ddfd9e<d:e<d;e<d'e<d(e<d"e<fd<d=�ZGe8e9e7fd>e<d?e<d;e<d'e<d"eeeH f
d@dA�ZId�dBee dCee dDe<dEedFed"eJfdGdH�ZKd�dIe<dBee dCee dDe<dEedFed"eJfdJdK�ZLd�dBee dCee dDe<dEedFed"eJfdLdM�ZMd�dIe<dBee dCee dDe<dEedFed"eJfdNdO�ZNeAeBfd>e<dPe@dQe@d"ee< fdRdS�ZOdTee d"efdUdV�ZPdWe<d"e<fdXdY�ZQdBee dCee dDe<d"ee<e<f fdZd[�ZRd\e<dDe<d"e<fd]d^�ZSG d_d`� d`�ZTd�dadb�ZUd�dcdd�ZVd�dedf�ZWd�dgdh�ZXd�didj�ZYdkdl� ZZdmdn� Z[dodp� Z\dqdr� Z]ej^ej_�`� dsdt����Zae�bdu� e�bdv� e�bdw� e�c� �m ejddxdy�� ejeg dz�e6d{d|d}�ZfW d � n 1 �s�w Y ejddxdy�� ejgd~de0dd�d��ZhW d � n 1 �s�w Y ejddxdy�� ejgd�d�e3d�d d�d��ZiW d � n 1 �s�w Y W d � n 1 �s�w Y e6dk�r
e=e0�Zjne6d�k�re>e1�Zjng Zjej�se5gZje5ejv �r&e5nej�r-ejd ne5Zke�c� � ejlejekd�d�dd��Zmejnd�d�d��ZoW d � n 1 �sPw Y e�bd�� e�c� �I ejdd�dy�� ejgd�d�d�d�d��ZpW d � n 1 �szw Y ejddxdy�� ejqd�g d��d�d��ZrW d � n 1 �s�w Y W d � n 1 �s�w Y ejnd�d�d�d��Zse�c� �B ejddxdy�� ejtd�d��ZuW d � n 1 �s�w Y ejddxdy�� ejvd�d��ZwW d � n 1 �s�w Y W d � n 1 �sw Y e�c� � ejgd�d�d�d�d dd��ZxW d � n 1 �s"w Y e�bd�� e�c� �V e�d� �
e�bd�� W d � n 1 �sFw Y e�d� �
e�bd�� W d � n 1 �s`w Y e�d� �
e�bd�� W d � n 1 �szw Y W d � n 1 �s�w Y e�bd�� e�c� �p e�d� �
e�bd�� W d � n 1 �s�w Y e�d� �
e�bd�� W d � n 1 �s�w Y e�d� �
e�bd�� W d � n 1 �s�w Y e�d� �
e�bd�� W d � n 1 �s�w Y W d � n 1 �sw Y e�bd�� e�c� �� e�d� � e�bd�e6� d�e5� d�eAd��d��� W d � n 1 �s;w Y e�d� �% e�bd�eB� d�eCd k�rRd�ne<eC�� d�e,�r\d�nd�� d��� W d � n 1 �smw Y e�d� �" e�bd�e0� d�e1� d�e9� d�e8� d�e-�r�d�nd�� d��� W d � n 1 �s�w Y W d � n 1 �s�w Y e�bd�� ejyd�dde5ddgd�dde5ddggeperefemeheigeuewexge]d dčZzefj{eZefgeheigdō d�dDŽ Z|efj{e|efeheigemgdō ehj{e|efeheigemgdō eoj}e|efeheigemgdō esj}e]eperefemeheigeuewexgdō W d � n 1 �s&w Y e~d�k�r�edɃ ed�e6� �� ed�e5� �� ed�eAd��d͝� ed�eB� d͝� ed�eCd k�r\d�ne<eC�� �� ed�e,�rjd�nd�� �� ed�e0� �� ed�e1� �� ed�e-� �� ed�e� �� ed�e� �� e,�s�edփ ned׃ ed eajdddٍ dS dS )�� N)�load_dotenvzDpython-dotenv not installed. Install with: pip install python-dotenv)�
BeautifulSoup)�urlparse)�datetime)�Dict�List�Optional�Any�TupleTF)�
TextEmbedding)�to_hexz]Graph visualization libraries not available. Install networkx, matplotlib for SVG generation.�OLLAMA_BASE_URLzhttp://192.168.0.173:11434�LMSTUDIO_BASE_URL�http://localhost:1234�HOSTED_API_URL�https://api.openai.com�HOSTED_API_KEY� �HOSTED_MODELzgpt-4o-miniZLOCAL_MODELzdeepshr1t:latest�MODEL_PROVIDER�ollama�EMBEDDING_MODEL_URL�EMBEDDING_MODEL_NAMEz"mixedbread-ai/mxbai-embed-large-v1�EMBEDDING_PROVIDERc C sF t �� �d�}t�d|� d| � �� tt�dkrtdd � ad�t�S )N�%H:%M:%S�[�] �d i�����
)r �now�strftime�global_progress_log�append�len�join)�message� timestamp� r'