Fetches content from a URL.

Args: url: The URL to fetch timeout: Request timeout in seconds (default: 30) follow_redirects: Whether to follow redirects (default: True) user_agent: Custom user agent string (optional)

Returns: Dictionary containing status_code, content, headers, encoding, and error (if any)

Fetches a URL with custom headers.

Args: url: The URL to fetch headers: Dictionary of custom headers timeout: Request timeout in seconds (default: 30)

Returns: Dictionary containing status_code, content, headers, encoding, and error (if any)

Converts HTML content to Markdown format.

Args: html: HTML content to convert strip_tags: List of tags to strip (default: ['script', 'style', 'meta', 'link']) heading_style: Style for headings - "ATX" or "SETEXT" (default: "ATX")

Returns: Dictionary with converted markdown content

Extracts plain text from HTML content.

Args: html: HTML content to convert preserve_links: Whether to preserve link URLs in output (default: False)

Returns: Dictionary with extracted text

Cleans HTML by removing scripts, styles, and optionally limiting to allowed tags.

Args: html: HTML content to clean allowed_tags: List of allowed tags (None = all tags except script/style)

Returns: Dictionary with cleaned HTML

Extracts the main article content from a web page.

Args: html: HTML content url: Optional URL for resolving relative links

Returns: Dictionary with title, content, text_content, excerpt, and short_title

Extracts all links from HTML content.

Args: html: HTML content base_url: Base URL for resolving relative links internal_only: Only return links from the same domain (default: False)

Returns: Dictionary with list of links containing url, text, title, and rel attributes

Extracts metadata from HTML including title, description, and Open Graph tags.

Args: html: HTML content

Returns: Dictionary containing title, description, keywords, author, open_graph, twitter_card, etc.

Extracts all images from HTML content.

Args: html: HTML content base_url: Base URL for resolving relative image URLs min_width: Minimum image width filter min_height: Minimum image height filter

Returns: Dictionary with list of images containing src, alt, title, dimensions, etc.

Extracts structured data (JSON-LD, microdata) from HTML.

Args: html: HTML content

Returns: Dictionary containing json_ld and microdata if found

Converts JSON data to readable Markdown format.

Args: json_data: JSON string to convert

Returns: Dictionary with markdown representation of the JSON