extractors.cpython-313.pyc•7.13 kB
�
��7h � �X � S r SSKrSSKJrJr SSKJrJr SSKJ r SSK
r
" S S5 r g)zLink extraction utilities.� N)�List�Set)�urljoin�urlparse)�
BeautifulSoupc
�* � \ rS rSrSr\R " S\R 5 r\R " S\R 5 r SS\
S\
S\
S \ \
4S
jjr S\
S \
4S jr
SS \
S\
S \\
4S
jjrS\
S \\
4S jrS\
S \4S jrS\ \
S \ \
4S jrSrg)�
LinkExtractor� z(Extract links from HTML or text content.zfhttps?://(?:www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b(?:[-a-zA-Z0-9()@:%_\+.~#?&/=]*)z\[([^\]]+)\]\(([^)]+)\)N�content� content_type�base_url�returnc �r � US:X a U R U5 n[ 5 nUS:X a UR U R X5 5 UR U R U5 5 / nU H, nU R U5 ( d M UR
U5 M. [ [ [ U5 5 5 $ )a
Extract all links from content.
Args:
content: The content to extract links from
content_type: "html", "text", or "auto" (auto-detect)
base_url: Base URL for resolving relative links
Returns:
List of unique URLs found in the content
�auto�html) �_detect_content_type�set�update�_extract_html_links�_extract_text_links�_is_valid_link�append�sorted�list)�selfr r r
�links� valid_links�links �X/home/josh/Projects/reputation-and-validity-checker/url_reputation_checker/extractors.py�
extract_links�LinkExtractor.extract_links s� � � �6� !� �4�4�W�=�L� �%�� �6� !� �L�L��1�1�'�D� E� � � �T�-�-�g�6�7� � ��D��"�"�4�(�(��"�"�4�(� � �d�3�{�+�,�-�-� c �~ � SUR 5 ; d( SUR 5 ; d SUR 5 ; a gg)zAuto-detect content type.z<htmlz<bodyz<a hrefr �text)�lower)r r s r r �"LinkExtractor._detect_content_type: s6 � � �g�m�m�o� %��G�M�M�O�)C�y�T[�Ta�Ta�Tc�Gc��r" � html_contentc � � [ 5 n [ US5 nUR SSS9 H+ nUS nU( a [ X&5 nUR U5 M- UR SSS9 H+ nUS nU( a [ X&5 nUR U5 M- UR SSS9 H+ nUS nU( a [ X'5 nUR U5 M- UR S
SS9 H+ nUS nU( a [ X'5 nUR U5 M- UR S S S
0S9 H� nUR SS5 n[ R " SU[ R 5 n U ( d MD U R S5 R S5 n
U( a [ X*5 n
UR U
5 M� U$ ! [ a U$ f = f)z Extract links from HTML content.z html.parser�aT)�hrefr* r �img)�srcr, �script�metaz
http-equiv�refresh)�attrsr � z url=([^;]+)� z"') r r �find_allr �add�get�re�search�
IGNORECASE�group�strip� Exception) r r' r
r �soup�tagr* r, r �match�urls r r �!LinkExtractor._extract_html_linksA s� � ����+ � � �}�=�D� �}�}�S�t�}�4���6�{���"�8�2�D�� � �$�� 5� �}�}�V�$�}�7���6�{���"�8�2�D�� � �$�� 8� �}�}�U��}�5���%�j���!�(�0�C�� � �#�� 6� �}�}�X�4�}�8���%�j���!�(�0�C�� � �#�� 9� �}�}�V�L�)�3L�}�M�� �'�'�)�R�0��� � �.�'�2�=�=�I���5� �+�+�a�.�.�.�u�5�C� � %�h� 4���I�I�c�N� N� � �� � � �� � �s �EF3 �*AF3 �3
G� G� text_contentc �< � [ 5 nU R R U5 H# nUR UR S5 5 M% U R
R U5 HS nUR S5 nUR
S5 ( d UR
S5 ( d MB UR U5 MU [ R " S[ R 5 nUR U5 H# nUR UR S5 5 M% U$ )z*Extract links from plain text using regex.r � �http://�https://z"["\']?(https?://[^"\'\s<>]+)["\']?r2 )
r � URL_PATTERN�finditerr4 r9 �MARKDOWN_PATTERN�
startswithr6 �compiler8 )r rA r r> r? �quoted_url_patterns r r �!LinkExtractor._extract_text_linkst s� � ���� �%�%�.�.�|�<�E� �I�I�e�k�k�!�n� %� =� �*�*�3�3�L�A�E��+�+�a�.�C��~�~�i�(�(�C�N�N�:�,F�,F�� � �#�� B� �Z�Z� 1� �M�M�
�� (�0�0� �>�E� �I�I�e�k�k�!�n� %� ?� � r" r? c �4 � U( d g/ SQnUR 5 nU H nUR U5 ( d M g UR S5 ( a gUR S5 ( d UR S5 ( a [ R " U5 SL $ g)z/Check if a URL is valid and should be included.F) z javascript:zmailto:ztel:zftp:zfile:zdata:zabout:zchrome:zedge:�#rD rE T)r% rI �
validatorsr? )r r? �skip_protocols� url_lower�protocols r r � LinkExtractor._is_valid_link� s� � ���
��
�I�I�K� �&�H��#�#�H�-�-� � '�
�>�>�#� � �� �>�>�)� $� $����z�(B�(B� �>�>�#�&�$�.� .�r" �urlsc �� � [ 5 nU HK n [ U5 nUR ( a+ UR UR R 5 5 MK MM [ [
U5 5 $ ! Mh = f)z+Extract unique domains from a list of URLs.)r r �hostnamer4 r% r r )r rT �domainsr? �parseds r �extract_domains� LinkExtractor.extract_domains� sa � ��%���C�
�!�#� ���?�?��K�K���� 5� 5� 7�8� #� � �d�7�m�$�$��
��s �AA0�0A5� )r N)N)�__name__�
__module__� __qualname__�__firstlineno__�__doc__r6 rJ r8 rF rH �strr r r r r r �boolr rY �__static_attributes__r[ r" r r r s� � �2� �*�*� .�
�
�
��K� �z�z�"�
�
�
���
.�S� .�� .�PS� .�_c�dg�_h� .�>�C� �C� �1�� 1�s� 1�c�RU�h� 1�f�� ��C�� �0�#� �$� �4 %�D��I� %�$�s�)� %r" r ) r` r6 �typingr r � urllib.parser r �bs4r rO r r[ r" r �<module>rg s# �� � � � *� � �g%� g%r"