knowledge_graph.cpython-310.pyc•19.9 kB
o
�}�g-~ � @ s� d Z ddlZddlZddlZddlZddlZddlmZm Z m
Z
m Z m Z m
Z
mZ ddlmZ ddlZddlZddlmZ ddlmZ ddlmZ ejejd � e�e �Z G d
d � d e e�Z G d d
� d
e e�Z G dd� de e�Z!G dd� d�Z"dS )z�
Knowledge Graph Module
Implements the core knowledge graph functionality for storing code components
and their relationships using LlamaIndex.
� N)�Dict�List�Optional�Any�Union�Set�Tuple)�Enum� )�config)�LlamaIndexManager)�Document)�levelc @ �$ e Zd ZdZdZdZdZdZdZdS )�
ComponentType�file�classZfunction�methodZvariable�moduleN) �__name__�
__module__� __qualname__ZFILEZCLASSZFUNCTIONZMETHODZVARIABLEZMODULE� r r �Z/Users/sparshdrolia/Sparsh personal/persistent-code-mcp/persistent_code/knowledge_graph.pyr � r c @ s e Zd ZdZdZdZdS )�ComponentStatusZplanned�partial� implementedN)r r r �PLANNED�PARTIAL� IMPLEMENTEDr r r r r ' s r c @ r )�RelationshipTypeZimportsZcallsZinherits�containsZ
depends_onZ
implementsN) r r r ZIMPORTSZCALLSZINHERITSZCONTAINS�
DEPENDS_ONZ
IMPLEMENTSr r r r r! , r r! c @ sD e Zd ZdZdKdedefdd�Zdedeeeeef fd d
�Z d e
ee f defd d
�Z de
jddfdededede
dede
ee f defdd�Z dLdedee dee
dee dee
ee f def dd�Z dKdeded ede
ee f def
d d �ZdMded ede
ee f fd!d"�Z #dNded$eee d%edee
ee f fd&d'�Z (dOd)ed*eee d+edee
ee f fd,d-�Z dPded.e
d/edefd0d1�Z dQd2e
ee f d3ee de
ee f fd4d5�Z 6 7dRd8ed+edee
ee f fd9d:�Z d;d<� Z d=d>� Z d?d@� Z dAdB� Z dCdD� Z!dEdF� Z"dGdH� Z#dIdJ� Z$dS )S�KnowledgeGraphzSA graph-based storage for code components and their relationships using LlamaIndex.N� project_name� storage_dirc C s� || _ |ptj�tj�tj�tj�t���d�| _tj| jdd� tj�| j|�| _ tj| j dd� tj�| j |� d��| _
tj�| j |� d��| _ t �
� | _t|| j t| jd�| _| j�� rc| jjnd| _| j�� ro| jjnd| _i | _| �� dS )z�Initialize a new knowledge graph for a project.
Args:
project_name: Name of the project
storage_dir: Directory to store the graph data (default: ./storage)
ZstorageT��exist_okz_nx_graph.jsonz_components.json)r% � project_dir�config_instanceZtriple_extract_fnN)r% �os�path�join�dirname�abspath�__file__r& �makedirsr) �
nx_graph_file�components_file�nx�DiGraph�graphr r* �_extract_triples�llama_index_manager� is_available�kg_index� embed_model�
components�load)�selfr% r& r r r �__init__7 s$ ,
� zKnowledgeGraph.__init__�document�returnc C s� g }|j }|�d�}|�d�}|�d�}|r|r|s |S |�|d|f� |�d�}|r3|�|d|f� |�dg �}|D ]
} |�|d| f� q;|�d �}
|
rU|�|d |
f� |�d
g �} | D ] } | �d |�}
| �d �}| �d
�}|r{|r{|�|
||f� q]|S )a Extract relationship triples for LlamaIndex KG from document metadata.
Args:
document: LlamaIndex document with component data in metadata
Returns:
List of (subject, predicate, object) triples
�id�name�typeZis_a�status�
has_status�basesZ
inherits_from� contained_in�
relationships�source�relation�target)�metadata�get�append)r> r@ ZtriplesrM � component_id�component_name�component_typerE rG �baserH rI ZrelrJ rK rL r r r r7 b s4
�z KnowledgeGraph._extract_triples�component_datac C s> |� dd�}|� dd�}|� d|� �}t|||� d�d�}|S )z�Convert a component to a LlamaIndex document.
Args:
component_data: Component data dictionary
Returns:
LlamaIndex Document
� code_text� � descriptionz
rB )�textrM �doc_id)rN r
)r> rT Zcomponent_textZcomponent_descZ full_text�docr r r �_component_to_document� s
�z%KnowledgeGraph._component_to_documentrV rC rR rU rE rW rM c
C s� t t�� �}||||||tj�� �� tj�� �� d|p i d�
}| jj|fi |�� || j|< | j �
� r>| � |�} | j � | � | �
� |S )a Add a new component to the graph.
Args:
name: Name of the component
component_type: Type of component (file, class, function, etc.)
code_text: The actual code text (optional)
status: Implementation status (planned, partial, implemented)
description: Semantic description of the component
metadata: Additional metadata as key-value pairs
Returns:
component_id: Unique identifier for the component
r
)
rB rC rD rE rW rU �
created_at�
last_modified�versionrM )�str�uuid�uuid4�datetime�now� isoformatr6 �add_noder< r8 r9 r[ Z add_document�save)
r> rC rR rU rE rW rM rP rT rZ r r r �
add_component� s& �
z KnowledgeGraph.add_componentrP c C s� || j vrdS t| j j| �}|dur||d< |dur ||d< |dur'||d< |dur2|d �|� tj�� �� |d< |d d 7 < | j j| �|� || j|< | j� � ra| �
|�}| j� |� | � � d
S ) a� Update an existing component in the graph.
Args:
component_id: ID of the component to update
code_text: New code text (if changed)
status: New implementation status (if changed)
description: New description (if changed)
metadata: New or updated metadata
Returns:
success: Whether the update was successful
FNrU rE rW rM r] r^ r
T)
r6 �dict�nodes�updaterb rc rd r< r8 r9 r[ Zupdate_documentrf )r> rP rU rE rW rM rT rZ r r r �update_component� s(
z KnowledgeGraph.update_component� source_id� target_id�relationship_typec
C s� || j vs
|| j vr dS tt�� �}||tj�� �� |p i d�}| j j||fi |�� | j� � rP| j j
| � d�}| j j
| � d�}t|�� � } | j�
|| |� | �� |S )a� Create a relationship between two components.
Args:
source_id: ID of the source component
target_id: ID of the target component
relationship_type: Type of relationship
metadata: Additional metadata for the relationship
Returns:
relationship_id: Unique identifier for the relationship
N)rB rD r\ rM rC )r6 r_ r` ra rb rc rd �add_edger8 r9 ri rN �lower�
add_triplerf )
r> rl rm rn rM Zrelationship_id� edge_dataZ source_nameZ target_namerK r r r �add_relationship" s �
z KnowledgeGraph.add_relationshipF�include_relatedc C s� || j vrdS t| j j| �}|rYg }| j j|dd�D ]\}}}t| j j| �}|�||d�� q||d< g } | j j|dd�D ]\}}
}t| j j|
�} | �| |d�� q?| |d< |S )a Retrieve a component by ID.
Args:
component_id: ID of the component
include_related: Whether to include related components
Returns:
component: Component data including code and metadata
NT��data)� componentZ relationshipZincoming_relationshipsZoutgoing_relationships)r6 rh ri �in_edgesrO � out_edges) r> rP rt rw Zincomingrl �_rr Z source_dataZoutgoingrm Z target_datar r r �
get_componentQ s*
�
�z KnowledgeGraph.get_componentr
�relationship_types�depthc
s� � �j vrg S �fdd���dkrCt� ��j j� dd�D ]\}}}�|�r*��|� q �j j� dd�D ]\}}}�|�rA��|� q3nt� �� �����fdd���� d� g }�D ]
} |�t�j j| �� qZ|S )af Find components related to a given component.
Args:
component_id: ID of the component
relationship_types: Types of relationships to consider (None for all)
depth: How many levels of relationships to traverse
Returns:
related_components: List of related components
c s � d u rdS | � d�� v S )NTrD )rN )rr )r| r r � matches_type� s z<KnowledgeGraph.find_related_components.<locals>.matches_typer
Tru c s� |�krd S �j j| dd�D ]\}}}�|�r'|� kr'��|� �||d � q�j j| dd�D ]\}}}�|�rI|� krI��|� �||d � q0d S )NTru r
)r6 ry �addrx )�node_idZ
current_depthrz rm rr rl )rP r} �dfsr~ � related_idsr> r r r� � s
�
��z3KnowledgeGraph.find_related_components.<locals>.dfs)r6 �setry r rx rO rh ri )
r> rP r| r} rz rm rr rl Zrelated_componentsZ
related_idr )rP r} r� r~ r� r| r> r �find_related_componentsx s*
�
��
z&KnowledgeGraph.find_related_components�
�query�component_types�limitc C s� g }t �� }|dkr |}| j�� rot�d|� �� | jj||d d�}|D ]&\}}|�d�} | rK| | jv rK| j| }
|rD|
�d�|vrDq%|� ||
f� q%|ro|j
dd� d d
� d d � |d
|� D �} t�dt | �� d�� | S t�d� | j j
d d�D ]S\} }
|r�|
�d�|vr�q{d}|�� }
|
�dd��� }|
|v r�d}|
|kr�d}||7 }|
|
�dd��� v r�|d7 }|
|
�dd��� v r�|d7 }|dkr�|� ||
f� q{|j
dd� d d
� dd � |d
|� D �S )a+ Search the codebase semantically.
Args:
query: Search query
component_types: Types of components to search
limit: Maximum number of results
Returns:
matching_components: Ranked list of matching components
� z Performing semantic search for: � )r� Zsimilarity_top_krB rD c S � | d S �Nr r ��xr r r �<lambda>� � z,KnowledgeGraph.search_code.<locals>.<lambda>T��key�reversec S � g | ]}|d �qS �r
r ��.0�mr r r �
<listcomp>� � z.KnowledgeGraph.search_code.<locals>.<listcomp>NzSemantic search found z matcheszUsing basic text searchru r rC rV � rW rU r
c S r� r� r r� r r r r� r� c S r� r� r r� r r r r� r� )r* Zget_similarity_top_kr8 r9 �logger�infoZsemantic_searchrN r< rO �sort�lenr6 ri rp )r> r� r� r� ZmatchesZ config_limitZsearch_resultsZscore�resultrY rT Zsemantic_resultsrP Z query_lowerrQ Z
name_scorer r r � search_code� sX
�
�
�zKnowledgeGraph.search_code�
new_status�notesc C s� || j vrdS || j j| d< d| j j| vr g | j j| d< | j j| d �|tj�� �� |d�� t| j j| �| j|< | j� � rZ| j j| �
d�}t |�� � }| j�
|d|� | �� dS )a; Update implementation status of a component.
Args:
component_id: ID of the component
new_status: New implementation status
notes: Optional notes about the status change
Returns:
success: Whether the update was successful
FrE Zstatus_history)rE Z timestampr� rC rF T)r6 ri rO rb rc rd rh r< r8 r9 rN r_ rp rq rf )r> rP r� r� rQ Z status_valuer r r �
update_status s"
�
z KnowledgeGraph.update_status�filters�groupingc C s6 t jdt jdt jdi}g }| jjdd�D ]6\}}|r6d}|�� D ]\}} ||v r2|| | kr2d} nq |r6q|�dt j�}
||
d7 < |�|� qt |�|dd�} | d dkrq|t j } |t j d }
| |
| d d
| d < |r�i }|D ]}|�|d �}||vr�g ||< || �|� qw|| d
< | S || d< | S )a( Retrieve implementation status across the project.
Args:
filters: Filters to apply to components
grouping: How to group the results (e.g., by type)
Returns:
status_summary: Summary of project implementation status
r Tru FrE r
)�total_components�
status_counts�implementation_percentager� g �?�d r� �unknownZgrouped_componentsr< )
r r r r r6 ri �itemsrN rO r� )r> r� r� r� Zfiltered_componentsrz rT Zskipr� �valuerE Zsummaryr r Zgroupedrw Z group_keyr r r �get_project_statusC sJ �� �
�z!KnowledgeGraph.get_project_status� dependenciesr� �
priority_typec C s� g }| j jdd�D ]\}}|�d�}|tjtjfv r!|�||f� q |dkr�g }|D ]K\}}g }| j j|dd�D ] \} }
} | �d�tj krU| j j|
�d�} |�|
| f� q8t
|�}
t dd� |D ��}|
dkrjd }n||
}|�||f� q*|j d
d � dd � d
d� |d|� D �}|S |dkr�g }|D ] \} }|�dd�}|�di ��dt
|��}|�||f� q�|j dd � d� dd� |d|� D �}|S ddl
}dd� |�|t|t
|���D �}|S )a/ Suggest logical next components to implement.
Args:
priority_type: How to prioritize tasks ("dependencies" or "complexity")
limit: Maximum number of tasks to suggest
Returns:
suggested_tasks: List of suggested next tasks
Tru rE r� rD c s s" � | ] \}}|t jkrd V qdS )r
N)r r )r� rz rE r r r � <genexpr>� s �
�z1KnowledgeGraph.find_next_tasks.<locals>.<genexpr>r g �?c S r� r� r r� r r r r� � r� z0KnowledgeGraph.find_next_tasks.<locals>.<lambda>r� c S � g | ]\}}|�qS r r �r� rz �compr r r r� � r� z2KnowledgeGraph.find_next_tasks.<locals>.<listcomp>N�
complexityrU rV rM c S r� r� r r� r r r r� � r� )r� c S r� r r r� r r r r� � r� c S r� r r r� r r r r� � r� )r6 ri rN r r r rO ry r! r# r� �sumr� �randomZsample�min)r> r� r� ZpendingrP rT rE Zsorted_pendingr� rz rm rr Z
target_statusZ
total_depsZimplemented_depsZ dep_scoreZ suggestedrU r� r� r r r �find_next_tasks� sH
� �� � z KnowledgeGraph.find_next_tasksc C s, | � � | �� | j�� r| j�� dS dS )z!Save the knowledge graph to disk.N)�_save_nx_graph�_save_componentsr8 r9 rf )r> r r r rf � s
�zKnowledgeGraph.savec
C sV | j tj�� �� g g d�}| jjdd�D ]0\}}i }|�� D ] \}}t|tt t
t t t
f�s1|du r6|||< q t|�||< q |d �|� q| jjdd�D ]5\}}} i }| �� D ] \}}t|tt t
t t t
f�sj|du ro|||< qWt|�||< qW|d �|||d�� qLt| jd��}
tj||
d d
� W d � n1 s�w Y t�d | j� �� dS ) z Save the NetworkX graph to disk.)r% Zsaved_atri �edgesTru Nri r� )rJ rL rv �wr� �ZindentzSaved NetworkX graph to )r% rb rc rd r6 ri r� �
isinstancer_ �int�float�bool�listrh rO r� �openr2 �json�dumpr� r� ) r> rv r� � node_dataZserializable_datar� r� rJ rL rr �fr r r r� � s6 �
� �z KnowledgeGraph._save_nx_graphc C sT t | jd��}tj| j|dd� W d � n1 sw Y t�d| j� �� dS )z'Save the components dictionary to disk.r� r� r� NzSaved components to )r� r3 r� r� r< r� r� �r> r� r r r r� s �z KnowledgeGraph._save_componentsc C sB t j�| jd�}t j|dd� | jjj|d� t� d|� �� dS )z,Save the LlamaIndex knowledge graph to disk.Z llama_indexTr' �� persist_dirzSaved LlamaIndex to N)
r+ r, r- r) r1 r: �storage_contextZpersistr� r� )r> r� r r r �_save_llama_index s z KnowledgeGraph._save_llama_indexc C sV t j�| j�r | �� t j�| j�r| �� | j�� }|r)| jj | _ | jj
| _
dS dS )z#Load the knowledge graph from disk.N) r+ r, �existsr2 �_load_nx_graphr3 �_load_componentsr8 r= r: r; )r> Zllama_index_loadedr r r r= s
�zKnowledgeGraph.loadc C s� t | jd��
}t�|�}W d � n1 sw Y t�� | _|�dg �D ]}|�d�}| jj |fi |�� q&|�dg �D ]}|d }|d }|d }| jj
||fi |�� q>t � d | j� �� dS )
z"Load the NetworkX graph from disk.�rNri rB r� rJ rL rv zLoaded NetworkX graph from )
r� r2 r� r= r4 r5 r6 rN �popre ro r� r� ) r> r� rv r� r� ZedgerJ rL rr r r r r� , s �
z KnowledgeGraph._load_nx_graphc C sN t | jd��}t�|�| _W d � n1 sw Y t�d| j� �� dS )z)Load the components dictionary from disk.r� NzLoaded components from )r� r3 r� r= r< r� r� r� r r r r� C s �z KnowledgeGraph._load_componentsc C s8 t j|d�}tj|| j| jdd�| _t�d|� �� dS )z�Load the LlamaIndex knowledge graph from disk.
Args:
persist_dir: Directory where index is persisted
r� T)r� r; Zkg_triple_extract_fnZinclude_embeddingszLoaded LlamaIndex from N) ZStorageContextZ
from_defaultsZKnowledgeGraphIndexZ from_storager; r7 r: r� r� )r> r� r� r r r �_load_llama_indexK s ��z KnowledgeGraph._load_llama_index)N)NNNN)F)Nr
)Nr� )rV )NN)r� r� )%r r r �__doc__r_ r? r
r r r7 r r r[ r r r rg r r� rk r! rs r{ r� r� r� r� r� r� rf r� r� r� r= r� r� r� r r r r r$ 4 s� +4 ������
�
�9������
�@����
�
� /)��
��
�F��
��
�[����
�0�
��
�E���
�F . r$ )#r� r+ r� rb Znetworkxr4 Ztempfile�typingr r r r r r r Zenumr r` Zloggingr r* r8 r Zllama_index.corer
Z basicConfigZINFOZ getLoggerr r� r_ r r r! r$ r r r r �<module> s& $