Skip to main content
Glama
search_and_index.cpython-313.pyc14.9 kB
� ���g{/� ��SrSSKrSSKrSSKrSSKrSSKrSSKJrJrJ r J r SSK J r SSK J r SSKJrJr SSKJr SSKJr SS KJr \R."\R0S S 9 \R2"\5r\ "5 SS \S \\S\S\\\\\44SjjrS \S\\\S\S\\4SjrSr \S:Xa\ "5 gg)z� Improved script to search GitHub for Move files containing specific keywords, download them, and index them for the MCP Server. �N)�List�Dict�Any�Tuple)� load_dotenv)�tqdm)�Github�RateLimitExceededException)�DocumentProcessor)�FAISSVectorStore)�index_move_filesz4%(asctime)s - %(name)s - %(levelname)s - %(message)s)�level�format�token�keywords� max_repos�returnc ���[U5n0nUH�n[RSU35 US-nURUSS5n[RSURSUS35 [ URU5n/XE'[ [U5SUS3S 9H]n Xyn XERU RU RU RU R/5 [R"S 5 M_ M� U$![a� [R!S 5 [R"S 5 Xyn XERU RU RU RU R/5 M�!["a1n [R%S U S['U 535 Sn A GMSn A ff=f["a0n [R%SU S['U 535 Sn A GMUSn A ff=f)a  Search GitHub for repositories using input keywords. Args: token: GitHub access token keywords: List of keywords to search for max_repos: Maximum number of repositories to retrieve per keyword Returns: Dictionary mapping keywords to lists of repository information zSearching GitHub for keyword: z+in:readme+in:description�stars�desc�Found z repositories for '�'zProcessing repos for '�r��-Rate limit exceeded. Sleeping for 60 seconds.�<zError retrieving repository �: N�Error processing repository )r �logger�info�search_repositories� totalCount�minr�range�append�name�html_url� description� full_name�time�sleepr �warning� Exception�error�str) rrr�github�results�keyword�query�search_results�count�i�repo�es �3/Users/kz/vcs/sui-ai/mcp_server/search_and_index.py�search_github_repositoriesr:s����E�]�F��G���� � �4�W�I�>�?��5�5���3�3�E�7�F�K��� � �f�^�6�6�7�7J�7�)�ST�U�V��N�-�-�y�9������e�E�l�+A�'��!�)L�M�A� K�%�(��� �'�'����D�M�M�4�CS�CS�UY�Uc�Uc�(d�e�� � �1� � N��: �N��.� O����N�O�� � �2��O�)�,�D��$�+�+�T�Y�Y�� � �t�GW�GW�Y]�Yg�Yg�,h�i�� �O��L�L�#?��s�"�S��V�H�!M�N�N��O��� K�� � �;�A�3�b��Q���I�J�J�� K�s>�AD�5G4�7AE=�= F8 �$F3 �+G4�3F8 �8 G4�$G/�/G4�repos� output_dirc ��[U5n/n[RS[U5S35 [ USS9GH�nUSn[RSU35 UR U5n[ RRX&RSS55n[ R"US S 9 URS 5S 4/n U (Ga U RS 5up�[U [5(dU /n U GH�n U RS :XaY[ RRX�R 5n URU R5nU R#X�45 MnU RS:XGa6U R R+S5(GaU R,R/S5nSUR15;a�U (a)[ RRX�R 5O U R n[ RRUU5n[ R"[ RR3U5S S 9 [5USSS9nUR7U5 SSS5 UR#U5 [R9SU35 GM�GM�GM�GM� [:R<"S5 U (aGMGM�GM� [RS[U5S35 U$![$a:n[R'SU RS[)U535 SnAGMESnAff=f!,(df  N�=f![$a:n[R'SU RS[)U535 SnAGM�SnAff=f![$a-n[R'S[)U535 SnAGM�SnAff=f![>a0 [R'S5 [:R<"S5 GM�[$a3n[RASUS S[)U535 SnAGMSnAff=f) a Find and download Move files from the provided repositories. Args: token: GitHub access token repos: List of repository information [name, url, description, full_name] output_dir: Directory to save downloaded files Returns: List of paths to downloaded Move files zSearching for Move files in z repositorieszProcessing repositoriesr�zChecking repository: �/�_T��exist_ok�r�dirzError accessing directory rN�filez.movezutf-8zuse sui�w)�encodingz Downloaded: zError downloading file zError processing content: g�?rrrz Downloaded z Move files)!r rr �lenr�get_repo�os�path�join�replace�makedirs� get_contents�pop� isinstance�list�typer&r%r-r,r/�endswith�decoded_content�decode�lower�dirname�open�write�debugr*r+r r.)rr;r<r0�downloaded_files� repo_info� repo_namer7�repo_dir�contents�current_contents� path_prefix�content� new_prefix� dir_contentsr8� file_content�rel_path� local_path�fs r9�find_move_files_in_reposrjLs����E�]�F��� �K�K�.�s�5�z�l�-�H�I��%�&?�@� �? R�!�!� �I� �K�K�/� �{�;� <��?�?�9�-�D��w�w�|�|�J�0A�0A�#�s�0K�L�H� �K�K��4� 0��*�*�2�.��3�4�H��08� � �Q��-� �"�"2�D�9�9�(8�'9�$�/�G�!N�"�<�<�5�0�)+�����k�<�<�)P�J�f�/3�/@�/@����/N� � (����0J� K�%�\�\�V�3�� � �8M�8M�g�8V�8V�c�/6�/F�/F�/M�/M�g�/V� �$-� �0B�0B�0D�#D�Ze�r�w�w�|�|�K���/V�kr�kw�kw�H�13�����h��1Q�J�%'�K�K������ �0K�VZ�$[�*.�j�#��)P�TU�()��� �(=�*Q�%5�$;�$;�J�$G�$*�L�L�<� �|�1L�$M�$E� 9W�3� 0�J� � �3��Y�(�A�D �K�K�+�c�"2�3�4�K�@�A� ���M$-�f� &���1K�G�L�L�>�Y[�\_�`a�\b�[c�/d� e� e��f��$*Q�)P�� $-�c� &���1H�����VX�Y\�]^�Y_�X`�/a� b� b��c��$�N����)C�C��F�8�'L�M�M��N�� *� � �N�N�J� K� �J�J�r�N�N�� R� �L�L�7� �!� �~�R��A��x�P� Q� Q�� R�s��B?O�?9N�9-L�&O�(2N�CM�#M�51M�&+O� M� .L?�8N�<O�?M�N� M �M� N�".N�N�O�N�N� O�)!O� O�O�O�5Q� Q�'Q � Qc � ^�[R"SS9nURSSSS9 URSSS 9 URS [R"S 5S S9 URS SSS9 URSSSS9 URS[ SSS9 URSSSS9 URSSSS9 URSSSS9 UR 5nUR(a2[R"5R[R5 UR(d[RS5 g [R"UR S!S"9 [#UR5nUR$(a�[R'S#UR$35 UR)UR$5mTR*TR,TR.TR0/nU/n[R'S$TR0S%TR2S&35 GO�UR8R;S*5Vs/sH)ofR=5(dMUR=5PM+ nnU(d[RS+5 g [?URXqR@5n/nURC5H;up�U H0m[EU4S,jU55(aMURGT5 M2 M= [R'S-[IU5S.35 URJ(a�[LRLRO5RQS/5n S0U S13n [SU S25n URC5H4up�U H)mU RUU S3TS4S3TS S3TS5S635 M+ M6 S)S)S)5 [R'S7U 35 [WURXAR 5nU(dKUR$(a$[RYS8UR$S935 g [RYS:5 g [R'S;5 [[UR UR\S<9n[R'S=US>35 [_5nURaUR\5 [R'S?[IURb5S@35 g4![4a8n[RS'UR$S([7U535 S)nAg S)nAff=fs snf!,(df  GN�=f![4a+n[RSA[7U535 S)nAg4S)nAff=f)BzEntry point for the script.zHSearch GitHub for Move files with keywords and index them for MCP Server)r(z --keywordszsui move,move frameworkzSComma-separated list of keywords to search for (default: 'sui move,move framework'))�default�helpz--repozHSpecific repository to search (format: owner/repo, e.g., MystenLabs/sui))rmz--token� GITHUB_TOKENzAGitHub personal access token (default: from GITHUB_TOKEN env var)z --output-dirzdocs/move_filesz=Directory to save downloaded files (default: docs/move_files)z --index-filezdata/faiss_index.binz=Path to save/load FAISS index (default: data/faiss_index.bin)z --max-repos�zCMaximum number of repositories to process per keyword (default: 20))rSrlrmz --new-index� store_truez7Create a new index instead of merging with existing one)�actionrmz --verbosezEnable verbose loggingz--output-resultszSave search results to a filezXGitHub token is required. Provide with --token or set GITHUB_TOKEN environment variable.rTrAzFetching specified repository: zUsing specified repository: z with z starszError fetching repository rN�,zGNo valid keywords provided. Use --keywords with comma-separated values.c3�8># �UHoSTS:Hv� M g7f)r>N�)�.0�rr7s �r9� <genexpr>�main.<locals>.<genexpr> s����>�I�q�Q�4�4��7�?�I�s�rz( unique repositories across all keywordsz%d%b%Y�github_search_results_z.txtrF� r�� zSaved search results to z"No Move files found in repository z with 'use sui' references.z:No Move files found or downloaded. Try different keywords.zIndexing downloaded files...)�docs_dir� index_filezSuccessfully indexed z document chunkszIndex verification: loaded z total documentszError verifying index: )2�argparse�ArgumentParser� add_argumentrJ�getenv�int� parse_args�verbose�logging� getLogger�setLevel�DEBUGrrr.rNr<r r7r rIr&r'r(r)�stargazers_countr-r/r�split�stripr:r�items�anyr%rH�output_results�datetime�now�strftimerYrZrjr,r r~r �load� documents)�parser�argsr0r]� all_reposr8�krr4r2r;� timestamp� results_file�f_outr\� num_indexed� vector_storer7s @r9�mainr��s���� � $� $�^��F� ����)� b���  ���� W���  ����� � �.�)� P���  ����!� L���  ����&� L���  ���� �� R� �� ����� F���  ����� %���  ����� ,��� � � � �D� �|�|�����$�$�W�]�]�3� �:�:�� � �o�p���K�K����$�/��D�J�J� �F� �y�y� � �K�K�9�$�)�)��E� F��?�?�4�9�9�-�D����D�M�M�4�3C�3C�T�^�^�T�I�"� �I� �K�K�6�t�~�~�6F�f�T�Mb�Mb�Lc�ci�j� k� (,�}�}�':�':�3�'?�M�'?�!�7�7�9�I�A�G�G�I�'?��M�� �L�L�b� c��4�D�J�J��.�.�Y��� �,�2�2�4�N�G����>�I�>�>�>��$�$�T�*��5� � � �f�S��^�,�,T�U�V� � � � �)�)�-�-�/�8�8��B�I�3�I�;�d�C�L��l�C�(�E�&4�&:�&:�&<�N�G� %��� � �w�i�r�$�q�'��"�T�!�W�I�R��Q��y�PR�$S�T�!&�'=�)� �K�K�2�<�.�A� B�0�� � �I���W�� � �9�9� �N�N�?�� � �{�Je�f� g�� �N�N�W� X�� �K�K�.�/�"�����?�?��K�  �K�K�'� �}�4D�E�F�9�'�)� ����$�/�/�*�� � �1�#�l�6L�6L�2M�1N�N^�_�`� ��}� � �L�L�5�d�i�i�[��3�q�6�(�K� L��� �� N�.)�(��> �9�� � �.�s�1�v�h�7�8�8� ��9�sK�BS0�T5� T5�6A T:�AU �0 T2�:.T-�-T2�: U � V�!U<�<V�__main__)ro)!�__doc__rJrr�r*r��typingrrrr�dotenvrrr0r r �#mcp_server.utils.document_processorr �mcp_server.models.vector_storer �mcp_server.index_move_filesr � basicConfig�INFOr��__name__rr/r�r:rjr�rt�r9�<module>r�s����  ��� ��)�)���5�A�;�8����'�,�,�Q�S� � � �8� $�� � �.�c�.�T�#�Y�.�3�.�X\�]`�bf�gk�lo�gp�bq�]q�Xr�.�`T�C�T��T�#�Y��T�S�T�UY�Z]�U^�T�lS �j �z���F�r�

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/ProbonoBonobo/sui-mcp-server'

If you have feedback or need assistance with the MCP directory API, please join our Discord server