content-core
This server extracts content from URLs and files and summarizes text using an LLM.
extract_content: Pull content from a URL (web pages, YouTube videos, PDF links, etc.) or a local file path, with optional engine overrides (auto, simple, firecrawl, jina, crawl4ai, docling). Most sources need no API key—only audio/video transcription does.
Docling options: Toggle OCR off (
no_ocr), extract formulas (formulas), or get image descriptions and chart data (pictures) when using the docling engine.summarize_content: Summarize provided text with an optional context hint (e.g., bullet points), returning plain text; requires an LLM provider key such as
OPENAI_API_KEY.
Exposes a set of compatible tools for Langchain framework, enabling extraction, cleaning, and summarization capabilities directly within Langchain agents and chains.
Enables right-click integration with macOS Finder through Services, allowing content extraction and summarization from any supported file with options for clipboard or TextEdit output.
Integrates with OpenAI services for transcription (Whisper) and content processing, allowing for AI-powered content extraction and summarization.
Provides a Python library for programmatic access to content extraction, cleaning, and summarization capabilities, with asynchronous functionality and customizable options.
Offers a Raycast extension with smart auto-detection commands for extracting and summarizing content from various sources, including URLs and files, with multiple output options and visual feedback.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@content-core extract the main points from this article: https://example.com/tech-news"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Content Core
Extract, process, and summarize content from URLs, files, and text through a unified async Python API, CLI, or MCP server.
Supported Formats
Category | Formats |
Web | URLs, HTML pages, YouTube videos, Reddit posts |
Documents | PDF, DOCX, PPTX, XLSX, ODT, ODS, ODP, EPUB, HTML, Markdown, plain text |
Media | MP3, WAV, M4A, FLAC, OGG (audio); MP4, AVI, MOV, MKV (video) |
Related MCP server: FreeCrawl MCP Server
Quick Start
pip install content-coreimport content_core
result = await content_core.extract_content(url="https://example.com")
print(result.content)Or with zero install:
uvx content-core extract "https://example.com"CLI Usage
Content Core provides a unified content-core command with subcommands for extraction, summarization, and MCP server.
Extract
# From a URL
content-core extract "https://example.com"
# From a file
content-core extract document.pdf
# With JSON output
content-core extract document.pdf --format json
# With a specific engine
content-core extract "https://example.com" --engine firecrawl
# From stdin
echo "some text" | content-core extractSummarize
# Summarize text
content-core summarize "Long article text here..."
# With context
content-core summarize "Long text" --context "bullet points"
# From stdin
cat article.txt | content-core summarize --context "explain to a child"MCP Server
content-core mcpConfiguration
# Set persistent config
content-core config set llm_provider anthropic
content-core config set llm_model claude-sonnet-5
# List current config
content-core config list
# Delete a config value
content-core config delete llm_providerConfig is stored in ~/.content-core/config.toml. Priority: command flags > env vars > config file > defaults.
Zero-Install with uvx
All commands work without installation using uvx:
uvx content-core extract "https://example.com"
uvx content-core summarize "text" --context "one sentence"
uvx content-core mcpPython API
Extraction
import content_core
# From a URL
result = await content_core.extract_content(url="https://example.com")
# From a file
result = await content_core.extract_content(file_path="document.pdf")
# From text
result = await content_core.extract_content(content="some text")
# With engine override
from content_core import ContentCoreConfig
config = ContentCoreConfig(url_engine="firecrawl")
result = await content_core.extract_content(url="https://example.com", config=config)Summarization
import content_core
summary = await content_core.summarize("long article text", context="bullet points")Configuration
from content_core import ContentCoreConfig
config = ContentCoreConfig(
url_engine="firecrawl",
document_engine="docling",
audio_concurrency=5,
)
result = await content_core.extract_content(url="https://example.com", config=config)MCP Integration
Content Core includes a Model Context Protocol (MCP) server for use with Claude Desktop and other MCP-compatible applications.
Add to your claude_desktop_config.json:
{
"mcpServers": {
"content-core": {
"command": "uvx",
"args": ["content-core", "mcp"],
"env": {
"OPENAI_API_KEY": "sk-..."
}
}
}
}The MCP server exposes two tools: extract_content and summarize_content. Both return plain text.
For detailed setup, see the MCP documentation.
Agent Skill (Claude Code & Codex)
Content Core ships an Agent Skill that teaches AI agents how to use it for extracting content from external sources. This repository is also a plugin marketplace, so the skill installs natively in both harnesses.
Claude Code — add the marketplace and install the plugin:
/plugin marketplace add lfnovo/content-core
/plugin install content-core@content-coreCodex — the repository carries a Codex plugin manifest (.codex-plugin/plugin.json) and marketplace catalog (.agents/plugins/marketplace.json) pointing at the same skill.
Manual fallback — copy the skill file directly into your project:
curl -o .claude/skills/content-core/SKILL.md --create-dirs \
https://raw.githubusercontent.com/lfnovo/content-core/main/skills/content-core/SKILL.mdOnce installed, the agent can use content-core to extract content from URLs, documents, and media files — either via CLI (uvx content-core) or MCP if configured.
AI Providers
Content Core uses Esperanto to support multiple LLM and STT providers. Switch providers by changing the config — no code changes needed:
# Use Anthropic for summarization
content-core config set llm_provider anthropic
content-core config set llm_model claude-sonnet-5
# Use Groq for transcription
content-core config set stt_provider groq
content-core config set stt_model whisper-large-v3Supported providers include OpenAI, Anthropic, Google, Groq, DeepSeek, Ollama, and more. See the Esperanto documentation for the full list.
Configuration
Content Core uses ContentCoreConfig powered by pydantic-settings. Settings are resolved in priority order: constructor args > env vars (CCORE_*) > config file (~/.content-core/config.toml) > defaults.
Environment Variables
Variable | Description | Default |
| URL extraction engine ( |
|
| Document extraction engine ( |
|
| Concurrent audio transcriptions (1-10) |
|
| Length of the segments long audio is split into; |
|
| Crawl4AI Docker API URL (omit for local browser mode) | - |
| Bearer token for the Crawl4AI Docker API (required by Crawl4AI >= 0.9.0) | - |
| Custom Firecrawl API URL for self-hosted instances or Firecrawl-compatible backends (e.g. fastCRW) | - |
| Firecrawl proxy mode ( |
|
| Wait time in ms before extraction |
|
| LLM provider for summarization | - |
| LLM model for summarization | - |
| Speech-to-text provider | - |
| Speech-to-text model | - |
| Speech-to-text timeout in seconds | - |
| Preferred YouTube transcript languages | - |
| Path to a Netscape | - |
| Proxy URL for YouTube transcripts, e.g. | - |
| OpenAI-compatible endpoint (vLLM, Ollama, LM Studio...) for Docling image descriptions; unset keeps Docling's inline model. Related: | - |
API keys for external services are set via their standard environment variables (e.g., OPENAI_API_KEY, FIRECRAWL_API_KEY, JINA_API_KEY).
Proxy Configuration
Content Core reads standard HTTP_PROXY / HTTPS_PROXY / NO_PROXY environment variables automatically. No additional configuration is needed.
YouTube on Blocked Networks
If YouTube extraction fails with IpBlocked/RequestBlocked on a flagged IP (cloud hosts, VPNs), either point CCORE_YOUTUBE_COOKIES_FILE at a cookies.txt exported from a logged-in browser (re-export it when the cookies expire), or set CCORE_YOUTUBE_PROXY to a residential proxy — datacenter proxies get blocked or CAPTCHA'd. See docs/usage.md.
Optional Dependencies
# Docling for advanced document parsing (PDF, DOCX, PPTX, XLSX)
# Required by document_engine="docling", which raises ConfigurationError without it.
# Use document_engine="auto" (default) or "simple" to proceed without Docling.
pip install content-core[docling]
# Crawl4AI for local browser-based URL extraction
pip install content-core[crawl4ai]
python -m playwright install --with-deps
# LangChain tool wrappers
pip install content-core[langchain]
# All optional features
pip install content-core[docling,crawl4ai,langchain]Using with LangChain
When installed with the langchain extra, Content Core provides LangChain-compatible tool wrappers:
from content_core.tools import extract_content_tool, summarize_content_tool
tools = [extract_content_tool, summarize_content_tool]Documentation
Usage Guide -- Python API details, configuration, and examples
Processors -- How content extraction works for each format
MCP Server -- Claude Desktop and MCP integration
Development
git clone https://github.com/lfnovo/content-core
cd content-core
uv sync --group dev
# Run tests
make test
# Lint
make ruffLicense
This project is licensed under the MIT License.
Contributing
Contributions are welcome! Please see our Contributing Guide for details.
Available Tools
2 toolsextract_contentExtract ContentA
Extract content from a URL or file. Does not require an API key for most sources (web pages, PDFs, documents, YouTube transcripts). API key is only needed for audio/video transcription.
Inputs: url, engine, no_ocr, formulas, +2 more. Returns: result
summarize_contentSummarize ContentB
Summarize content using LLM with optional context. Requires OPENAI_API_KEY (or another LLM provider key) to be configured.
Inputs: content (required), context. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 2 tools
The two tools target clearly distinct actions: extract_content pulls raw content from a source, while summarize_content condenses it via an LLM. An agent can easily tell which to call based on whether it needs raw text or a summary.
Both names follow the same verb_noun snake_case pattern (extract_content, summarize_content), with no deviations in convention or style.
Two tools is on the thin side for even a focused server, though each earns its place in a natural extract-then-summarize pipeline. A couple of supporting operations (e.g. format conversion or output saving) would round out the surface.
The extract and summarize pair covers the core content-processing lifecycle, including multiple input sources (URLs, PDFs, YouTube). Minor gaps exist around batch processing, output formatting, or chunking, but agents can work around these.
Maintenance
Related MCP Connectors
PDF, DOCX and image URLs to clean, LLM-ready Markdown or text, with OCR for scanned pages.
Any URL to clean markdown/JSON; PDF, XLSX, CSV, DOCX to text and tables. Free tier for agents.
Extract structured insights from videos, podcasts, articles, and PDFs with multi-model AI
Extract and parse web pages into clean HTML, links, or Markdown. Handle dynamic, complex, or block…
Related MCP Servers
- AlicenseCqualityBmaintenanceA powerful tool for fetching and extracting text content from web pages and APIs, supporting web scraping, REST API requests, and Google Custom Search integration.511MIT
- AlicenseNot gradedqualityDmaintenanceEnables web scraping and document processing with JavaScript execution, anti-detection measures, batch processing, and structured data extraction. Supports multiple formats including markdown, HTML, screenshots, and handles PDFs with OCR capabilities.4MIT
- AlicenseBqualityDmaintenanceParse any file or URL into structured text. Extract text from PDF, DOCX, YouTube, web pages, images, and 25+ formats via one API. Tools: parse_url, parse_file, get_youtube_transcript.37 npmMIT
- AlicenseNot gradedqualityDmaintenanceConverts any URL into clean, LLM-ready Markdown, text, or HTML with production-grade features like SSRF protection, rate limiting, retries, caching, and structured error handling.MIT