Skip to main content
Glama
rebots-online

Knowledge Graph Builder

app.cpython-310.pyc49.3 kB
o l�qh�U�@sZ ddlZddlZddlZz ddlmZe�Wn ey$ed�YnwddlmZddl m Z ddl Z ddl Z ddlZddlmZddlmZmZmZmZmZddlZddlZddlZddlZzddlZdZWn eytd ZYnwzddlZdZWn ey�d ZYnwz dd lmZdZ Wn ey�d Z Ynwzddl!Z"ddl#m$Z%ddl&m'Z'dd l(m)Z)ddl*Z+dZ,Wney�d Z,ed �YnwdZ-ej.�/d d�Z0ej.�/dd�Z1ej.�/dd�Z2ej.�/dd�Z3ej.�/dd�Z4ej.�/dd�Z5ej.�/dd�Z6ej.�/de0�Z7ej.�/dd�Z8ej.�/dd�Z9ga:dd �Z;e0fd!e<d"ee<fd#d$�Z=e1fd!e<d"ee<fd%d&�Z>d'e<d(e<d"ee<fd)d*�Z?e@ej.�/d+d,��ZAe@ej.�/d-d.��ZBe@ej.�/d/d0��ZCd1d2�ZDd3d4�ZEd�d6e<d"e<fd7d8�ZFe5e6ddfd9e<d:e<d;e<d'e<d(e<d"e<f d<d=�ZGe8e9e7fd>e<d?e<d;e<d'e<d"eeeHf d@dA�ZId�dBeedCeedDe<dEedFed"eJf dGdH�ZKd�dIe<dBeedCeedDe<dEedFed"eJfdJdK�ZLd�dBeedCeedDe<dEedFed"eJf dLdM�ZMd�dIe<dBeedCeedDe<dEedFed"eJfdNdO�ZNeAeBfd>e<dPe@dQe@d"ee<fdRdS�ZOdTeed"efdUdV�ZPdWe<d"e<fdXdY�ZQdBeedCeedDe<d"ee<e<ffdZd[�ZRd\e<dDe<d"e<fd]d^�ZSGd_d`�d`�ZTd�dadb�ZUd�dcdd�ZVd�dedf�ZWd�dgdh�ZXd�didj�ZYdkdl�ZZdmdn�Z[dodp�Z\dqdr�Z]e j^e j_�`�dsdt����Zae �bdu�e �bdv�e �bdw�e �c��me jddxdy��e jegdz�e6d{d|d}�ZfWd�n 1�s�wYe jddxdy��e jgd~de0dd�d��ZhWd�n 1�s�wYe jddxdy��e jgd�d�e3d�d d�d��ZiWd�n 1�s�wYWd�n 1�s�wYe6dk�r e=e0�Zjn e6d�k�re>e1�ZjngZjej�se5gZje5ejv�r&e5nej�r-ejdne5Zke �c��e jlejekd�d�dd��Zme jnd�d�d��ZoWd�n 1�sPwYe �bd��e �c��Ie jdd�dy��e jgd�d�d�d�d��ZpWd�n 1�szwYe jddxdy��e jqd�gd��d�d��ZrWd�n 1�s�wYWd�n 1�s�wYe jnd�d�d�d��Zse �c��Be jddxdy��e jtd�d��ZuWd�n 1�s�wYe jddxdy��e jvd�d��ZwWd�n 1�s�wYWd�n 1�swYe �c��e jgd�d�d�d�d dd��ZxWd�n 1�s"wYe �bd��e �c��Ve �d�� e �bd��Wd�n 1�sFwYe �d�� e �bd��Wd�n 1�s`wYe �d�� e �bd��Wd�n 1�szwYWd�n 1�s�wYe �bd��e �c��pe �d�� e �bd��Wd�n 1�s�wYe �d�� e �bd��Wd�n 1�s�wYe �d�� e �bd��Wd�n 1�s�wYe �d�� e �bd��Wd�n 1�s�wYWd�n 1�s wYe �bd��e �c���e �d��e �bd�e6�d�e5�d�eAd��d���Wd�n 1�s;wYe �d��%e �bd�eB�d�eCdk�rRd�ne<eC��d�e,�r\d�nd��d���Wd�n 1�smwYe �d��"e �bd�e0�d�e1�d�e9�d�e8�d�e-�r�d�nd��d�� �Wd�n 1�s�wYWd�n 1�s�wYe �bd��e jyd�dde5ddgd�dde5ddggeperefemeheigeuewexge]d dčZzefj{eZefgeheigdōd�dDŽZ|efj{e|efeheigemgdōehj{e|efeheigemgdōeoj}e|efeheigemgdōesj}e]eperefemeheigeuewexgdōWd�n 1�s&wYe~d�k�r�edɃed�e6���ed�e5���ed�eAd��d͝�ed�eB�d͝�ed�eCdk�r\d�ne<eC����ed�e,�rjd�nd����ed�e0���ed�e1���ed�e-���ed�e���ed�e���e,�s�edփned׃ed؃eajdddٍdSdS)��N)� load_dotenvzDpython-dotenv not installed. Install with: pip install python-dotenv)� BeautifulSoup)�urlparse)�datetime)�Dict�List�Optional�Any�TupleTF)� TextEmbedding)�to_hexz]Graph visualization libraries not available. Install networkx, matplotlib for SVG generation.�OLLAMA_BASE_URLzhttp://192.168.0.173:11434�LMSTUDIO_BASE_URL�http://localhost:1234�HOSTED_API_URL�https://api.openai.com�HOSTED_API_KEY�� HOSTED_MODELz gpt-4o-miniZ LOCAL_MODELzdeepshr1t:latest�MODEL_PROVIDER�ollama�EMBEDDING_MODEL_URL�EMBEDDING_MODEL_NAMEz"mixedbread-ai/mxbai-embed-large-v1�EMBEDDING_PROVIDERcCsFt���d�}t�d|�d|���tt�dkrtdd�ad�t�S)N�%H:%M:%S�[�] �di����� )r�now�strftime�global_progress_log�append�len�join)�message� timestamp�r'�:/home/robin/CascadeProjects/hKG-ontologizer-KGB-mcp/app.py�add_to_progress_logKs    r)�base_url�returnc CsLzgtr<|tkrtj|d�}|��}nt��}t|d�r&|j}dd�|D�WSt|t�r9|� dg�}dd�|D�WSWgSt j |�d�dd�}|j d kr]|� �}|� dg�}d d�|D�WSt d |j ���WgSt jjyzt d |���YgSt jjy�t d |���YgSty�}zt d|���WYd}~gSd}~ww)z#Fetch available models from Ollama.�Zhost�modelscSsg|] }t|d�r|j�qS)�model)�hasattrr.��.0r.r'r'r(� <listcomp>csz%get_ollama_models.<locals>.<listcomp>c S�(g|]}|r|�d|�dt|����qS��namer.��get�strr0r'r'r(r2g�(z /api/tags��Ztimeout��c Sr3r4r6r0r'r'r(r2nr9z!Ollama API returned status code: zCannot connect to Ollama at z Timeout connecting to Ollama at zError fetching Ollama models: N)�OLLAMA_AVAILABLEr r�Client�listr/r-� isinstance�dictr7�requests� status_code�json�print� exceptions�ConnectionError�Timeout� Exception)r*�clientZmodels_responseZ models_list�response�data�er'r'r(�get_ollama_modelsTs@     ��  �����rNc CszDtrtj|�d�dd�}|j��}dd�|jD�WStj|�d�dd�}|jd kr:|� �}d d�|�d g�D�WSt d |j���WgStj j yWt d |���YgStj j yit d|���YgSty�}zt d|���WYd}~gSd}~ww)z&Fetch available models from LM Studio.�/v1� lm-studio�r*�api_keycSsg|]}|j�qSr'��idr0r'r'r(r2�sz'get_lmstudio_models.<locals>.<listcomp>� /v1/modelsr:r;r<cS�g|]}|d�qSrSr'r0r'r'r(r2��rLz$LM Studio API returned status code: zCannot connect to LM Studio at z#Timeout connecting to LM Studio at z!Error fetching LM Studio models: N)�OPENAI_AVAILABLE�openai�OpenAIr-r?rLrBr7rCrDrErFrGrHrI)r*rJr-rKrLrMr'r'r(�get_lmstudio_modelsys2�  �����r[�api_urlrRc Cs�z*d|��dd�}tj|�d�|d�}|jdkr(|��}dd�|�d g�D�WSWgStyD}ztd |���WYd }~gSd }~ww) z=Fetch available models from a hosted API (OpenAI-compatible).zBearer zapplication/json)Z Authorizationz Content-TyperU��headersr<cSrVrSr'r0r'r'r(r2�rWz)get_hosted_api_models.<locals>.<listcomp>rLz"Error fetching hosted API models: N)rBr7rCrDrIrE)r\rRr^rKrLrMr'r'r(�get_hosted_api_models�s� ����r_� CHUNK_SIZEZ2000� CHUNK_OVERLAPZ200� MAX_CHUNKS�0c Cs�zCddi}tj||d�}|��t|jd�}|ddg�D]}|��q|��}dd�|��D�}d d�|D�}d �d d�|D��}|WSt y\}z d t |���WYd }~Sd }~ww)z$Extract text content from a web URL.z User-AgentzsMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36r]� html.parser�script�stylecss�|]}|��VqdS�N)�strip)r1�liner'r'r(� <genexpr>���z(extract_text_from_url.<locals>.<genexpr>css(�|]}|�d�D]}|��Vq qdS)z N)�splitrh)r1riZphraser'r'r(rj�s�&� css�|]}|r|VqdSrgr')r1�chunkr'r'r(rj�rkzError fetching URL: N) rBr7Zraise_for_statusr�content� decompose�get_text� splitlinesr$rIr8) �urlr^rK�soupre�text�lines�chunksrMr'r'r(�extract_text_from_url�s"�  ��rxc CsH�z�tj�|�d��}|dkr/t|ddd�� }|��Wd�WS1s'wYWdS|dkrRt|ddd�� }|��Wd�WS1sJwYWdS|dkr}t|ddd��}t�|�}tj|d d �Wd�WS1suwYWdS|d kr�zd dl }|� |�}|� �WWSt y�t|ddd��}|��Wd�YWS1s�wYYWdSw|d vr�t|ddd��#}|��}t |d�}|ddg�D]}|��q�|��Wd�WS1s�wYWdS|dk�r;z6d dl} t|d�� }| �|�} d} | jD] } | | ��7} �q | Wd�WWS1�s&wYWWdSt �y:YWdSw|dv�rjzd dl} | �|�}d} |jD] }| |jd7} �qO| WWSt �yiYWdSwt|ddd�� }|��Wd�WS1�s�wYWdSt�y�}z dt|���WYd}~Sd}~ww)z+Extract text content from an uploaded file.��.txt�r�utf-8��encodingN�.md�.json�)Zindent�.csvr)�.html�.htmrdrerf�.pdfZrbrz=Error: PyPDF2 not installed. Install with: pip install PyPDF2)�.docx�.docrzGError: python-docx not installed. Install with: pip install python-docxzError reading file: )�os�path�splitext�lower�open�readrD�load�dumpsZpandasZread_csvZ to_string� ImportErrorrrprq�PyPDF2Z PdfReaderZpagesZ extract_text�docxZDocumentZ paragraphsrurIr8)Z file_pathZfile_extension�frLZpdZdfrortrer��readerruZpager�ZdocZ paragraphrMr'r'r(�extract_text_from_file�s�(�(�  (�   .��  (�    .��   �*���r��kgb-mcp� namespacec Cs�tt��d�}|�d|�dt�d�����}t�|�����dd�}t |�}|dd@dB|d<|dd @dB|d<|��}|dd��d|dd ��d|d d��d|dd ��d|d d ��� S) zFGenerate a UUIDv8 for unified entity tracking across Neo4j and Qdrant.i��-�N�����?� �� ) �int�timer��urandom�hex�hashlibZsha256�encodeZdigest� bytearray)r�r&Z hash_inputZ hash_bytesZuuid_hexr'r'r(�generate_uuidv8�sHr��promptr.�providerc Csrz�|dkr?tr?|r |nt}|r)|tkr)tj|d�}|j|d|d�gddd�d�}ntj|d|d�gddd�d�}|d d WS|d krktrk|rI|nt}tj|d d d�}|jj j |d|d�gddd�}|j dj j WS|dkr�|r�|r�tr�tj|�d �r~|n|�d �|d�}|jj j |d|d�gddd�}|j dj j WSWdSty�}z dt|���WYd}~Sd}~ww)z0Call model via Ollama, LM Studio, or hosted API.rr,�user)Zrolerog�������?g�������?)� temperatureZtop_p)r.�messagesZoptionsr%ro�lmstudiorOrPrQi�)r.r�r�Z max_tokensr�hostedzGError: No compatible model provider available or missing configuration.zError calling model: N)r=r rr>ZchatrXrrYrZZ completions�create�choicesr%ro�endswithrIr8) r�r.r�r\rRr*rJrKrMr'r'r(�call_local_models\     � �  � �� ���r�ru� model_namec CsDz�|dkr3tr3|r |nt}|r!|tkr!tj|d�}|j||d�}ntj||d�}|�d|�dg��WS|dkrRtrRt|d�}t |� |g��}|rO|d� �WSWdS|d kr}t r}t j|�d �ra|n|�d �tj�d d �d �}|jj||d�}|jdjWStd|�d��WdSty�} ztd| ���WYd} ~ dSd} ~ ww)z>Generate embeddings for text using configured embedding model.rr,)r.r�� embedding� embeddings� fastembed)r�rrYrOZOPENAI_API_KEYrrQ)�inputr.zEmbedding provider z not available or not configuredNzError generating embeddings: )r=rr rr>r�r7�FASTEMBED_AVAILABLEr r?Zembed�tolistrXrYrZr�r��environr�rLr�rErI) rur�r�r\r*rJrKZembedding_modelr�rMr'r'r(�generate_embeddingsIs>      �  ����r��entities� relationships�uuid_v8�content_metadata�visualization_metadatac Cs�tstd�dSz�g}|D]�}d|��d|�dd���dt������g}|rT|�d|�d d ���d |�d d ���d|�dd���d|�dd ���d|�dd ���g�|r�|�d|�dd���d|�dd���d|�dd���d|�dd���dtr�t|d �nd ��g�|� |d!|d |d"��q |r�td#t |��d$|���g}|D]} |� | d%| d&| d'd(��q�|r�td#t |��d)|���Wd*St y�} ztd+| ���WYd,} ~ dSd,} ~ ww)-zfStore entities and relationships in Neo4j via MCP server with enhanced hKG and visualization metadata.z*MCP not available - skipping Neo4j storageF�UUID: z Description: � descriptionrz Extracted: zContent Length: �content_length�unknownzProcessing Method: �processing_method�singlez Chunk Count: � chunk_countryzModel: r.z Source Type: � source_typezVisualization Available: �visualization_availablezReal-Time Updates: �real_time_updateszIncremental Files: �incremental_files_savedrzSVG File Path: � svg_file_path�nonezEntity Color: �typer5)r5Z entityType� observationszStoring z entities in Neo4j with UUID �source�target� relationship)ZfromZtoZ relationTypez" relationships in Neo4j with UUID TzError storing in Neo4j: N) � MCP_AVAILABLErEr7rr� isoformat�extend�GRAPH_VIZ_AVAILABLE�get_entity_colorr"r#rI) r�r�r�r�r�Zneo4j_entities�entityr�Zneo4j_relations�relrMr'r'r(�store_in_neo4j_syncqs`�� � � ���r�roc Cs tstd�dSz�dd�|D�}dd�|D�}dd�|D�}d|�d|d d ��d d �|��d d �t|���dd�|��dt������ } |rp| d|�dt|���d|�dd��d|�dd��d|�dd��d|�dd��� 7} |r�| d|�dd��d |�d!d��d"|�d#d$��d%|�d&d'��d(t r�d �d)d�t|�D��nd*�� 7} t | �} | s�td+�d } n td,t| ��d-��td.|���td/t| ����td0t|��d1t|����td2|r�|�dd�nd���td3| r�d4nd5���Wd6St �y} ztd7| ���WYd } ~ dSd } ~ ww)8zeStore knowledge graph data in Qdrant with vector embeddings, hKG metadata, and visualization lineage.z+MCP not available - skipping Qdrant storageFcSrV�r5r'�r1rMr'r'r(r2�rWz(store_in_qdrant_sync.<locals>.<listcomp>cSrV�r�r'r�r'r'r(r2�rWcSs,g|]}|d�d|d�d|d���qS)r�rmr�r�r')r1r{r'r'r(r2�s,r�z Content: N��z Entities: z, z Entity Types: z Relationships: �; z Extracted: z Content Length: r�z Processing Method: r�r�z Chunk Count: r�ryz Model: r.r�z Source Type: r�z Visualization Available: r�z Real-Time Updates: r�z Incremental SVG Files: r�rz SVG File Path: r�r�z Entity Colors: cSsg|] }|�dt|����qS)�=�r��r1Zetr'r'r(r2�s� unavailablezLWarning: Could not generate embeddings for content. Using text-only storage.zGenerated embeddings with z dimensionsz,Storing knowledge graph in Qdrant with UUID zVector content length: zEntity count: z, Relationship count: zVisualization tracking: z Embeddings: u ✅ Generatedu ❌ FailedTzError storing in Qdrant: ) r�rEr$�setrrr�r7r#r�r�rI) ror�r�r�r�r�Z entity_names� entity_typesZrelationship_summariesZvector_contentr�rMr'r'r(�store_in_qdrant_sync�st �� �� �� � � � � � � � ����r�c�s�t|||||�Srg)r�)r�r�r�r�r�r'r'r(�store_in_neo4j�s�r�c�s�t||||||�Srg)r�)ror�r�r�r�r�r'r'r(�store_in_qdrant�s�r�� chunk_size�overlapc Cs�t|�|kr |gSg}d}|t|�krb||}|t|�krAg}t|d|�}t||�D] }||dvr8|�|�q+|rA|dd}|�|||��t|d||�}|t|�kr\ |S|t|�ks|S)z2Split text into overlapping chunks for processing.rr<z.!? �����ry)r#�max�ranger") rur�r�rw�start�endZ sentence_endsZ search_start�ir'r'r(� chunk_texts,     �   �r��resultsc Cs�i}g}|D]@}d|vrF|dD]5}|d|df}||vr#|||<q||�dd�}|�dd�}|rE||vrE|�d|���d�||d<qqt�}|D]&}d|vrr|dD]} | d| d | d f}||vrq|�|�|�| �qVqLt|���|d �S) z@Merge multiple extraction results into a single knowledge graph.r�r5r�r�rr�r�r�r�r�)r�r�)r7rhr��addr"r?�values) r�Zmerged_entitiesZmerged_relationships�resultr��keyZ existing_descZnew_descZrelationship_keysr�r'r'r(�merge_extraction_results%s2    �   � �r�� entity_typecCsddddddd�}|�|d�S) zGet color for entity type.z#FF6B6Bz#4ECDC4z#45B7D1z#96CEB4z#FFEAA7z#DDA0DD)ZPERSONZ ORGANIZATIONZLOCATIONZCONCEPTZEVENT�OTHER�#CCCCCC)r7)r�Z color_mapr'r'r(r�Gs� r�c!stsdS|sdS�z�t���|D]}�j|d|d|�dd�t|d�d�q|D]+}|dd d �|D�vrU|d d d �|D�vrU�j|d|d |d |�dd�d �q*t����dkrit ����ddi}n0t����dkr�t ����}|dd|ddi}nz tj �dddd�}Wn tj �dd�}Yt j dd�t �d�i}�jdd�D]J}|\} } } || \} } || \}}t j| |g| |gddddd �| |d| |d}}| �d d�}|r�t j|||d!d"d"td#d$d%dd&�dd'�q�|��D]M\}\}}�j|}|�d(d)�}|�dd*�}t j||fd+|d%dd,�}t ���|�t j||d-|d.d"d/d0d1d2�t j||d3d4|�d5�d!d"d/d6d7d1d8� q�t jd9|d:d!��d;�d<d0d=d>�g}t�fd?d@����D��}t|�D]}t|�}|�tj||dA���qm|�r�t j|dBdCdD�dEt|��dFt|���}t jdGdH|d"dId6dJ�t ���dK�t � �t!j"dLdM�}t j#|dNdOdPd$dQdR�t �$�t%|dSdTdU�� }|�&�}Wd:�n 1�s�wYt'�(|�|dVt|��dWt|��dX�fWSt)�y } zd:dYt*| ���fWYd:} ~ Sd:} ~ ww)Zz0Create SVG visualization of the knowledge graph.)Nz8Graph visualization not available - missing dependencies�NzNo entities to visualizer5r�r�r�r�r��colorr�cSrVr�r'r�r'r'r(r2jrWz.create_knowledge_graph_svg.<locals>.<listcomp>r�r��r�r�ryr�rrr��r�r�ryr��2�*��kZ iterations�seed�r �r�r��Zfigsize�offT�rL�gray�333333�?��alphaZ linewidth�zorderr��center� round,pad=0.3�white皙�����?�Zboxstyle� facecolorr� edgecolor��fontsize�ha�vaZbboxrr�r�r��333333�?�r�rr��?� �top�bold��rrr�weightr�ffffff�?�(�)�italic�ffffff�?�rrrrfrrz%Knowledge Graph Visualization\nUUID: N�...r�r��rr'Zpadc3�"�|] }�j|�dd�VqdS�r�r�N��nodesr7�r1�node��Gr'r(rj��� z-create_knowledge_graph_svg.<locals>.<genexpr>�r��label� upper right�ryry�ZhandlesZlocZbbox_to_anchor� Entities: � | Relationships: ��?�{�G�z�?r��rrrf�equal�.svg��suffix�svg�tight�,r���formatZ bbox_inchesZdpirrr{r|r}z"Successfully generated graph with � entities and � relationshipsz&Error generating graph visualization: )+r��nx�Graph�add_noder7r��add_edger#r3r?� spring_layout� random_layout�plt�figure�axis�edges�plotrurA�items�Circle�gca� add_patch�titler��sortedr"�patches�Patch�legend�figtext� set_aspect� tight_layout�tempfile�mktemp�savefig�closer�r�r��unlinkrIr8)!r�r�r�r�r��posr3Z edge_labels�edger�r�rL�x1�y1�x2�y2�mid_x�mid_yr�r5�x�y� node_datar�r��circle�legend_elementsr�� stats_text�svg_pathr�� svg_contentrMr'r6r(�create_knowledge_graph_svgSs�  �, ��        ��   ���   � �  ��rzryc Cs�|sdSz4d|dd��d�}tj�t��|�}t|ddd��}|�|�Wd�|WS1s1wY|WStyR}ztd|���WYd}~dSd}~ww) z/Save SVG content to a file and return the path.N�knowledge_graph_r�rD�wr|r}zError saving SVG file: )r�r�r$�getcwdr��writerIrE)ryr�Z svg_filenamerxr�rMr'r'r(� save_svg_file�s   ����rc@sreZdZdZdefdd�Zdedeeeffdd�Zdedeeeffd d �Z d ed efd d�Z defdd�Z dS)�RealTimeGraphVisualizerzDHandles real-time incremental graph visualization during processing.r�cCs||_g|_g|_g|_dSrg)r��current_entities�current_relationships� svg_history)�selfr�r'r'r(�__init__�s z RealTimeGraphVisualizer.__init__� progress_infor+c Cs�z<|d|_|d|_|�|�\}}|r8|j�|�t���d�}d|jdd��d|dd �d �}|� ||�||fWSt yW}zdd t |���fWYd}~Sd}~ww) z-Update the graph visualization with new data.r�r�z %Y%m%d_%H%M%Sr{Nr�Z_chunk_� chunk_numberZ04drDzError updating graph: ) r�r��create_incremental_svgr�r"rrr r��save_incremental_svgrIr8)r�r�ryr%r&Zincremental_filenamerMr'r'r(� update_graph�s   "  ��z$RealTimeGraphVisualizer.update_graphc*s�tsdS|d}|d}|d}|d}|d}|sdS�z1t���|D]}�j|d|d |�d d �t|d �d �q$|D]+}|d dd�|D�vri|ddd�|D�vri�j|d |d|d|�d d �d�q>t����dkr}t ����ddi} n0t����dkr�t ����} | dd| ddi} nz tj �dddd�} Wn tj �dd�} Yt j dd�t �d ��jd!d"�D]K} | \} } }| | \}}| | \}}t j||g||gd#d$ddd%�||d||d}}|�dd �}|�r t j|||d&d'd'td(d)d*d#d+�dd,�q�| ��D]N\}\}}�j|}|�d-d.�}|�d d/�}t j||fd0|d*dd1�}t ���|�t j||d2|d3d'd4d5d6d7�t j||d8d9|�d:�d&d'd4d;d<d6d=� �qd>|�d?|�d@|dA�dB|jdCd&��dD� }t j|dEd5dFdG�g}t�fdHdI����D��}t|�D]}t|�}|�tj||dJ���q�|�r�t j|dKdLdM�d*} dN}!dO}"dP}#t j|"|#dQt | d�d&dRdSdT�t |dU| d�}$t j|"|#dQ|$d&dVdSdT�dWt|��dXt|��dY|�d?|��}%t jdZd[|%d'd\d;d]�t ���!d^�t �"�t#j$d_d`�}&t j%|&dadbdcd)ddde�t �&�t'|&dfdgdh�� }'|'�(�}(WdC�n 1�s/wYt)�*|&�|(dit|��djt|��dk|�d?|�d:� fWSt+�yj})zdCdlt,|)���fWYdC})~)SdC})~)ww)mz)Create SVG for current incremental state.)Nz!Graph visualization not availabler�r�r�� total_chunks�progress_percentr�r5r�r�rr�r�cSrVr�r'r�r'r'r(r2rWzBRealTimeGraphVisualizer.create_incremental_svg.<locals>.<listcomp>r�r�rryrrr�rrrrrrr r r r Trrrrr�rrrrrrr�r�r�rr r!r"r#r$r%r&r(r)r*r+r,r-z.Knowledge Graph - Real-Time Processing\nChunk �/� (z.1fz% Complete)\nUUID: Nr.r�r�r/c3r0r1r2r4r6r'r(rj`r8zARealTimeGraphVisualizer.create_incremental_svg.<locals>.<genexpr>r9r;r<r=g���Q��?g�������?g�������?u█Z lightgrayZ monospace)rr�ZfamilyrZgreenr>r?z | Chunk: r@rAr�rBrCrDrErGrHrIr�rJr{r|r}zUpdated graph: � entities, z relationships (Chunk z*Error creating incremental visualization: )-r�rNrOrPr7r�rQr#r3r?rRrSrTrUrVrWrXrurArYrZr[r\r�r]r�r^r"r_r`rarbr�rcrdrerfrgrhr�r�r�rirIr8)*r�r�r�r�Z chunk_numr�r�r�r�rjr3rkr�r�rLrlrmrnrorprqr�r5rrrsrtr�r�ruZprogress_titlervr�Zprogress_bar_widthZprogress_bar_heightZ progress_xZ progress_yZ filled_widthrwrxr�ryrMr'r6r(r�s�  �, ��        ��   � �* ��$  � � ,��z.RealTimeGraphVisualizer.create_incremental_svgry�filenamec Cs�z/tj�t��|�}t|ddd�� }|�|�Wd�n1s!wYtd|���WdStyI}ztd|���WYd}~dSd}~ww)zSave incremental SVG file.r|r|r}NzSaved incremental graph: zError saving incremental SVG: )r�r�r$r}r�r~rErI)r�ryr�rxr�rMr'r'r(r��s ���z,RealTimeGraphVisualizer.save_incremental_svgcCs|jr|jdSdS)zGet the final complete SVG.r�N)r�)r�r'r'r(� get_final_svg�s z%RealTimeGraphVisualizer.get_final_svgN) �__name__� __module__� __qualname__�__doc__r8r�rr r�r�r�r�r'r'r'r(r��s r�c st|�tkr�tdt|��d��t|��tdkr1t��tkr1tdt�dt���d���dt��tdt���d ��g}g}g}t��D]z\�}td �d �d t���d t|��d����fdd�}t|||�}d|vr�|�|�t|�} |r��d t��| d| d�d t��dt|�d�} || �td�d �dt| d��dt| d��d��qEtd�d �d|d���qE|r�t|�} tdt| d��dt| d��d��| Sggdd�Sdd �} t||| �}|�rd|v�rd d |d|ddt|�d�} || �|S)!z|Use local model to extract entities and relationships from text, handling large content via chunking with real-time updates.zProcessing large content (z chars) in chunks...rz Limiting to z chunks (from z total)Nz Processing z chunks...zProcessing chunk ryr�r�z chars)...csHtd�d�dt���d|���td�d�dt���d|���dS)N�Chunk ryr��: )r)r#rE�r%�rwr�r'r(�chunk_progress_callback�s"&zCextract_entities_and_relationships.<locals>.chunk_progress_callback�errorr�r�r)r�r�r�r�r�Zcurrent_chunk_sizer�z completed. Running totals: r�rMzError in chunk r�zFinal results: z)Failed to process any chunks successfully�r�r�r�cSs td|���td|���dS)NzSingle chunk: )r)rEr�r'r'r(�single_progress_callback�szDextract_entities_and_relationships.<locals>.single_progress_callback) r#r`rEr�rb� enumerate�)extract_entities_and_relationships_singler"r�) ru�progress_callback� model_configZ all_entitiesZall_relationshipsZ chunk_resultsrnr�r�Zincremental_mergedr�Zmergedr�r'r�r(�"extract_entities_and_relationships�s`  (  �2&� �r�c Cszd|�d�}�zI|r|d�|r(t||�dt�|�dt�|�d�|�d�d�}nt|�}|r2|d �|�d �rG|r@|d |���gg|d �WS|}|ret|�d kr^|d|dd ��d��n|d|���d|vr�d|vr�ddl}|�d||j�}|r�|r�|� d�� �}|d|���|j dd||jd�� �}|r�|d�d|vr�d} d} |� | �} | dkr�| t| �7} |� | | �} | dkr�|| | �� �}n;|�d��r|� d��r|�d�} d} t| �d} tt| �ddd�D]}| |� �dkr�|} nq�d�| | | ��}|�r |d |���t�|�}|�r|d!�t|t��s"td"��d#|v�r+g|d#<d$|v�r4g|d$<|�rNt|d#�}t|d$�}|d%|�d&|�d'��|WStj�y�}z5d(t|���}|�rz|d)|���|d*d+t�v�ru|nd,���gg|d+t�v�r�|nd,d-�WYd}~Sd}~wt�y�}zd.t|���}|�r�|d/|���gg|d �WYd}~Sd}~ww)0z?Extract entities and relationships from a single chunk of text.a  Analyze the following text and extract key entities and their relationships. Return the result as a JSON object with this exact structure: { "entities": [ {"name": "entity_name", "type": "PERSON|ORGANIZATION|LOCATION|CONCEPT|EVENT|OTHER", "description": "brief description"} ], "relationships": [ {"source": "entity1", "target": "entity2", "relationship": "relationship_type", "description": "brief description"} ] } Text to analyze: z_ Please provide only the JSON response without any additional text or formatting. u"📤 Sending prompt to AI model...r.r�r\rR)r.r�r\rRu3📥 Received response from AI model, processing...zError:u❌ Model error: r�r�u$📝 AI Response (first 500 chars): Nr.u📝 AI Response: z<think>z</think>rz<think>(.*?)</think>ryu🤔 AI Thinking: z<think>.*?</think>r)�flagsu*🔧 Cleaning and parsing JSON response...z```jsonz```r�ru🔧 Cleaned JSON for parsing: u%✅ Successfully parsed JSON responsezResponse is not a JSON objectr�r�u📊 Extracted rLrMz&Failed to parse LLM response as JSON: u❌ JSON Parse Error: zRaw response: � response_textz No response)r�r�r�� raw_responsezError calling local model: u❌ Model Error: )r�r7� DEFAULT_MODELr� startswithr#�re�searchZDOTALL�grouprhZsub�findr�rlr�r$rD�loadsr@rA� ValueErrorZJSONDecodeErrorr8�localsrI)rur�r�Z entity_promptr�Zoriginal_responser�Z think_matchZ think_contentZ start_markerZ end_markerZ start_idxZend_idxrvr�r�� entity_count� rel_countrMZ error_msgr'r'r(r��s� �  � �    �  �      �����r�c�sn��z|s |s ddd�WSt�}|r*t|j�}|�d�r"|dd�WSd}|j}|}n?|rc|��rct|���}|jdvo=|j}|rZt|���}|�d�rQ|dd�WSd}|��}|}nd }d }|��}nddd�WSt |��d���fd d �} t || �} t |�t kr�d nd} t |�t kr�t t |��nd} t |�| | t�dt��|t����d�} t| d k| d kr�| nddttdd�| �dg�D���d�}d}d}t�r| �d��r| �d��rz!t| d| d|| |�IdH}t|| d| d|| |�IdH}Wnt�y}z td|���WYd}~nd}~ww||t |�dk�r$|dd�dn|d�| �dg�| �dg�t | �dg��t | �dg��d�t�dt��t |�|||t����| | t t|dd t�o_| d kd| d k�rg| ndt|du| d k||d!�d"�d#�d$�}d}d}t�r�| �d��r�z#��r��}n t| d| d|�\}}|�r�t||�}td%|���Wnt�y�}z td&|���WYd}~nd}~ww||t| d k| d k�r�| ndt�r�d'd(�td)d�| �dg�D��D�ni|�r�t����ndt�r�d*nd+d,�}||d-<d.| v�r| d.|d/<d0| v�r| d0|d1<|WSt�y6}zd2t|���dd�WYd}~Sd}~ww)3zHMain function to build knowledge graph from text, URL, or uploaded file.�2Please provide text, a valid URL, or upload a fileN�r��knowledge_graph�Error reading file�file�ZhttpZhttps�Error fetching URLrsru� direct_inputc�4tr��|�\}}|r|�td|���dSdSdS)NzReal-time graph updated: )r�r�rE�r�ryr%�� latest_svg�real_time_visualizerr'r(r����z0build_knowledge_graph.<locals>.progress_callback�chunkedr�ry�:�r�r�r�r.r�r&rZpendingcSrVr�r'r�r'r'r(r2�rWz)build_knowledge_graph.<locals>.<listcomp>r�)r�r�r�r�Zentity_types_presentFr�zError storing in databases: r<r.�r��valueZcontent_preview�r�r�r�Zrelationship_countTZ unlimited)Zreal_time_visualizationZsvg_files_generatedZentity_color_trackingZvisualization_lineageZincremental_updatesZneo4j_viz_metadataZqdrant_viz_metadata)r�r�r�Z chunk_overlapr�Zsupports_large_contentZmax_content_sizeZvisualization_integration)r.r��uuidZ neo4j_storedZ qdrant_storedr&Z hkg_metadata�r�r�ZmetadatazFinal SVG visualization saved: zError generating final SVG: cSsi|]}|t|��qSr'r�r�r'r'r(� <dictcomp>@sz)build_knowledge_graph.<locals>.<dictcomp>cSrVr�r'r�r'r'r(r2@rWznetworkx+matplotlibr�)ryr�r�r�r�Zentity_color_mappingZsvg_generation_timestampZvisualization_engine� visualizationr��extraction_errorr��raw_llm_responsezUnexpected error: ) r�r�r5r�rhr�scheme�netlocrxr�r�r#r`r�rr�rrr�r�r?r�r7r�r�r�rIrErarzrr8)� input_text� uploaded_filer��extracted_textr�r�ro�parsed�is_urlr��kg_datar�r�r�Zpreliminary_viz_metadataZ neo4j_successZqdrant_successrMrxr�� final_svg� svg_messageZfinal_viz_metadatar'r�r(�build_knowledge_graph�s*��  �    � �   � �  ���� �  �   ����)� ���, �     ���r�c Cs�ddl}z|�t||��}d}d|vr|ddr|dd}||fWStyC}zdt|���dd�}|dfWYd}~Sd}~ww)zBSynchronous wrapper for build_knowledge_graph with SVG extraction.rNr�ry�Error in async execution: r�)�asyncio�runr�rIr8)r�r�r�r�ryrM� error_resultr'r'r(�build_knowledge_graph_syncWs   ���r�cCs(td|�d|���g}|dkr+|r|��r|nt}t|�}tdt|��d|���nA|dkrJ|r7|��r7|nt}t|�}tdt|��d|���n"|dkrd|rd|rdt||�}tdt|��d |���ntd |�d ��|sxtg}td t���t|vr~tn|r�|d nt}td|���t j ||d�S)z1Update model dropdown based on selected provider.zUpdating models for provider: z, URL: rzFound z Ollama models: r�z LM Studio models: r�z hosted API models: z Provider z& not configured or missing credentialsz!No models found, using fallback: rzSetting default model to: )r�r�) rErhr rNr#rr[r_r��gr�Dropdown)r�r\rRr-r*Z default_valuer'r'r(�update_model_dropdownls( r�cCs�|dkrtjdddtdd�tjdtd�fS|dkr*tjdd d td d�tjd d d�fS|dkr?tjdddtdd�tjd d d�fStjd d�tjd d�fS)z5Show/hide API configuration fields based on provider.r�TzAPI URLrz%Base URL for hosted API (without /v1))�visibler:� placeholderr��info)r�r�r� Ollama URL�http://localhost:11434�URL of your Ollama serverFrr�z LM Studio URLrzURL of your LM Studio server)r�)r��updaterrr r)r�r'r'r(�toggle_api_fields�s � � �r�c s�||||dkr |ndd��g��fdd���d�ddfVddl}z4����fdd �}d}d} d } |�2z3dHW\} } } | } | durF| }| durL| } q56| || fVWdSty�}zd t|���dd �}�d t|����|dfVWYd}~dSd}~ww)z<Process with model configuration and yield progress updates.r�N�r�r.r\rRcs2��dt���d��d|���d��dd��S)Nrrrri����)r"rrr r$r�)� progress_logr'r(�update_progress�s z6process_with_config_streaming.<locals>.update_progress�+🚀 Starting knowledge graph extraction...rc s@��s�s�d�ddd�dfVdS�d�ddfVt�}�d|dd��d��ddfV�rk�d�j���ddfVt�j�}|�d �rU�d |���|dd�dfVdSd }�j}|}�d t|��d ��ddfVny�r׈��r�t����}|jdvo~|j}|r��d������ddfVt ����}|�d�r��d|���|dd�dfVdSd}���}|}�dt|��d ��ddfVn$d}d}���}�dt|��d ��ddfVn �d�ddd�dfVdSt |��d����fdd�}�fdd�}�d�d�d�d���ddfVt ||��} �d �ddfVt|�t k�r!d!nd"} t|�t k�r0tt |��nd#} t|�| | �d�d�d��|t����d$�} d} t�r�| �d%��r�z/�d&�ddfV��rd�} n t| d%| d'|�\} }| �r�t| |�}�d(|���d| fVWnt�y�}z�d)|���ddfVWYd}~nd}~ww||t|�d*k�r�|dd*�dn|d+�| �d%g�| �d'g�t| �d%g��t| �d'g��d,��d�d�d��t|�|t����d-�d.�}d/| v�r�| d/|d0<d1| v�r�| d1|d2<t| �d%g��}t| �d'g��}�d3|�d4|�d5��|| fVdS)6N�❌ No input providedr�r��🔧 Initializing processing...�🆔 Generated UUID: r�r.�📁 Processing uploaded file: r��❌ File error: r��✅ File processed: � charactersr��🌐 Fetching URL: r��❌ URL error: rs�✅ URL processed: rur��📝 Text input: �❌ No valid input providedc3s>�tr��|�\}}|r|��d|���d|fVdSdSdS�Nu📊 Graph updated: )r�r�r�)r�r�r�r'r(r��s��zQprocess_with_config_streaming.<locals>.run_with_config.<locals>.progress_callbackc3s��|�ddfVdSrgr'r�)r�r'r(�text_progress_callback�s�zVprocess_with_config_streaming.<locals>.run_with_config.<locals>.text_progress_callback�!🤖 Starting AI extraction with r�r�r.�✅ AI extraction completedr�r�ryr�r��&🎨 Generating final visualization...r��💾 SVG saved: �❌ SVG error: r<r�r��r.r�r�r&r�r�r�r�r��🎉 Completed! r�rM)r�r5r�r�r#rhrr�r�rxr�r�r`r�rrr�r�r7rzrrI)r�r�r�r�ror�r�r�r�r�r�r�r�r�r�rxrMr�r�r�)r�� text_inputr�r�r�r(�run_with_config�s����    ��     ���� $  � � �"�� �  � ��    "z6process_with_config_streaming.<locals>.run_with_configrr�r��❌ Fatal error: )r�rIr8)r�r�r�r.r\rRr�rZ final_resultr�Zfinal_progressZprogressr�rGrMr�r')r�r�r�r�r�r(�process_with_config_streaming�s@�� �� �(��rc �s��gatd�||||dkr|ndd��ddl}z���fdd�}|�|��}|d|dd �t�fWSty`} z!td t| ����d t| ���dd �} | dd �t�fWYd} ~ Sd} ~ ww) z!Process with model configuration.r�r�Nr�rc �s���s�std�ddd�dfStd�t�}td|dd��d���rWtd�j���t�j�}|�d �rEtd |���|dd�dfSd }�j}|}td t|��d ��ni�r����r�t����}|jdvoj|j }|r�td������t ����}|�d�r�td|���|dd�dfSd}���}|}tdt|��d ��nd}d}���}tdt|��d ��n td�ddd�dfSt |��d���fdd�}td�d�d�d���t ||��}td�t|�t kr�dnd } t|�t kr�tt|��nd!} t|�| | �d�d�d��|t����d"�} d} t�rb|�d#��rbz'td$���r+�} n t|d#|d%|�\} } | �rFt| |�}td&|���Wnt�ya}z td'|���WYd}~nd}~ww||t|�d(k�rs|dd(�dn|d)�|�d#g�|�d%g�t|�d#g��t|�d%g��d*��d�d�d��t|�|t����d+�d,�}d-|v�r�|d-|d.<d/|v�r�|d/|d0<t|�d#g��}t|�d%g��}td1|�d2|�d3��|| fS)4Nr�r�r�r�r�r�r.r�r�r�r�r�r�r�r�r�r�rsr�rur�r�r�cr�r�)r�r�r)r�r�r'r(r��r�zMprocess_with_config_async.<locals>.run_with_config.<locals>.progress_callbackr�r�r�r.r�r�r�ryr�r�r�r�r�r�r<r�r�r�r�r�r�r�r�r�r�rM)r)r�r5r�r�r#rhrr�r�rxr�r�r`r�rrr�r�r7rzrrI)r�r�r�r�ror�r�r�r�r�r�r�r�r�rxrMr�r�r��r�r�r�r�r(ris����  ��    ����  � � ��� �  � ��    z2process_with_config_async.<locals>.run_with_configryrrr�r�)r!r)r�r�r$rIr8) r�r�r�r.r\rRr�rr�rMr�r'rr(�process_with_config_asyncXs,��  ���rc CsRddl}z|��}Wnty|��}|�|�Ynw|�t||||||��S)z2Synchronous wrapper for process_with_config_async.rN)r�Zget_event_loop� RuntimeErrorZnew_event_loopZset_event_loopZrun_until_completer)r�r�r�r.r\rRr�Zloopr'r'r(�process_with_configs  ��ru🧠 Knowledge Graph Builder)Zthemer]u;# 🧠 Knowledge Graph Builder with Real-Time VisualizationzW**Build Knowledge Graphs with Local AI Models - Now with Real-Time SVG Visualization!**z## Model Configurationry)Zscale)rr�r�z AI ProviderzSelect your AI model provider)r�r�r:r�r�r�r�)r:r�r�r�r�zAPI Keyz Your API keyZpasswordzAPI key for hosted service)r:r�r�r�r�r�r�z Select ModelzChoose the AI model to use)r�r�r:r�� interactiveu🔄 Refresh ModelsZ secondary)�variantz## Inputr�zText or URL InputzGPaste any text paragraph or enter a web URL (e.g., https://example.com)r%r�)r:r�rv� max_linesz Upload File) rzrr�r�r�r�r�r�r��filepath)r:Z file_typesr�u🚀 Build Knowledge GraphZprimaryZlg)r�sizezKnowledge Graph Data)r:zGraph Visualizationz Real-Time Progress & AI Thinkingz#Processing logs will appear here...r"r�)r:r�rvr rZshow_copy_buttonz## Key Featuresu� **📥 Input Options:** • Text Input: Paste any text paragraph • URL Input: Extract from web pages • File Upload: Multiple formats supported • Large Content: 300MB+ handling u� **🎨 Visualization:** • Real-time graph updates • Color-coded entity types • SVG output with legend • Progress tracking u� **💾 Storage & Output:** • Neo4j graph database • Qdrant vector storage • JSON knowledge graph • Incremental SVG files z## Supported File Typesu📄 **Text Files:** .txt, .mdu 📊 **Data Files:** .json, .csvu🌐 **Web Files:** .html, .htmu%📖 **Documents:** .pdf, .docx, .docz## Current Configurationu8 **Model Setup:** • Provider: u • Model: u • Chunk Size: �,z chars u< **Processing:** • Chunk Overlap: u# chars • Max Chunks: Z Unlimitedu • Graph Viz: u ✅ Availableu!❌ Install matplotlib & networkxz u6 **Integration:** • Ollama: u • LM Studio: u • Embeddings: r�u • MCP: u❌ Not Availablez ## Examplesz�Apple Inc. was founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976. The company is headquartered in Cupertino, California.z5https://en.wikipedia.org/wiki/Artificial_intelligence)�examples�inputs�outputs�fnZcache_examples)rrrcCs t|||�S)z:Refresh the model dropdown based on current configuration.)r�)r�r\rRr'r'r(�refresh_models�s r�__main__uB🚀 Starting Knowledge Graph Builder with Real-Time Visualizationu📊 Model Provider: u 🤖 Model: u📈 Chunk Size: r�u🔄 Chunk Overlap: u📝 Max Chunks: u🎨 Graph Visualization: u🔗 Ollama URL: u🔗 LM Studio URL: u🔌 MCP Available: u🦙 Ollama Available: u🔧 OpenAI Client Available: uN⚠️ Graph visualization disabled. Install: pip install networkx matplotlibu(✅ Real-time SVG visualization enabled!u1📁 SVG files will be saved in current directory)Z mcp_serverZshare)r�)NNrg)�r�rDrBZdotenvrr�rEZbs4rZ urllib.parserZgradior�r�r�r�typingrrrr r r�reZmathZrandomrr=rYrXr�r r�ZnetworkxrNZmatplotlib.pyplotZpyplotrTZmatplotlib.patchesr_Zmatplotlib.colorsr ZnumpyZnpr�r�r�r7r rrrrr�rrrrr!r)r8rNr[r_r�r`rarbrxr�r�r��floatr��boolr�r�r�r�r�r�r�rzrr�r�r�r�r�r�r�rrrZBlocksZthemesZSoftZdemoZMarkdownZRowZColumnZRadioZprovider_radioZTextboxZ api_url_inputZ api_key_inputZinitial_modelsZinitial_default_valuer�Zmodel_dropdownZButtonZrefresh_models_btnr�ZFileZ file_inputZ submit_btnZJSONZ json_outputZHTMLZ viz_outputZprogress_outputZExamplesr ZchangerZclickr�Zlaunchr'r'r'r(�<module>sl    �    � �  �     � %=,.8,(0A,I0$$"* | A V  S <.    ��������      �� ������ ���� ��   �  �  ���    �  �  �  ���  ���� ��  �� ����  ���� �� ������� V  �

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/rebots-online/hKG-ontologizer-KGB-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server