content-core
Content Core
Извлекайте, обрабатывайте и обобщайте контент из URL-адресов, файлов и текста с помощью унифицированного асинхронного Python API, CLI или сервера MCP.
Поддерживаемые форматы
Категория | Форматы |
Веб | URL, HTML-страницы, видео YouTube, посты Reddit |
Документы | PDF, DOCX, PPTX, XLSX, EPUB, Markdown, обычный текст |
Медиа | MP3, WAV, M4A, FLAC, OGG (аудио); MP4, AVI, MOV, MKV (видео) |
Related MCP server: FreeCrawl MCP Server
Быстрый старт
pip install content-coreimport content_core
result = await content_core.extract_content(url="https://example.com")
print(result.content)Или без установки:
uvx content-core extract "https://example.com"Использование CLI
Content Core предоставляет унифицированную команду content-core с подкомандами для извлечения, обобщения и работы с сервером MCP.
Извлечение
# From a URL
content-core extract "https://example.com"
# From a file
content-core extract document.pdf
# With JSON output
content-core extract document.pdf --format json
# With a specific engine
content-core extract "https://example.com" --engine firecrawl
# From stdin
echo "some text" | content-core extractОбобщение
# Summarize text
content-core summarize "Long article text here..."
# With context
content-core summarize "Long text" --context "bullet points"
# From stdin
cat article.txt | content-core summarize --context "explain to a child"Сервер MCP
content-core mcpКонфигурация
# Set persistent config
content-core config set llm_provider anthropic
content-core config set llm_model claude-sonnet-4-20250514
# List current config
content-core config list
# Delete a config value
content-core config delete llm_providerКонфигурация хранится в ~/.content-core/config.toml. Приоритет: флаги командной строки > переменные окружения > файл конфигурации > значения по умолчанию.
Использование без установки с помощью uvx
Все команды работают без установки с помощью uvx:
uvx content-core extract "https://example.com"
uvx content-core summarize "text" --context "one sentence"
uvx content-core mcpPython API
Извлечение
import content_core
# From a URL
result = await content_core.extract_content(url="https://example.com")
# From a file
result = await content_core.extract_content(file_path="document.pdf")
# From text
result = await content_core.extract_content(content="some text")
# With engine override
from content_core import ContentCoreConfig
config = ContentCoreConfig(url_engine="firecrawl")
result = await content_core.extract_content(url="https://example.com", config=config)Обобщение
import content_core
summary = await content_core.summarize("long article text", context="bullet points")Конфигурация
from content_core import ContentCoreConfig
config = ContentCoreConfig(
url_engine="firecrawl",
document_engine="docling",
audio_concurrency=5,
)
result = await content_core.extract_content(url="https://example.com", config=config)Интеграция с MCP
Content Core включает сервер Model Context Protocol (MCP) для использования с Claude Desktop и другими приложениями, совместимыми с MCP.
Добавьте в свой claude_desktop_config.json:
{
"mcpServers": {
"content-core": {
"command": "uvx",
"args": ["content-core", "mcp"],
"env": {
"OPENAI_API_KEY": "sk-..."
}
}
}
}Сервер MCP предоставляет два инструмента: extract_content и summarize_content. Оба возвращают обычный текст.
Подробную информацию по настройке см. в документации MCP.
Навык Claude Code
Content Core включает SKILL.md, который обучает ИИ-агентов использовать его для извлечения контента из внешних источников. Чтобы сделать его доступным в вашем проекте Claude Code, скопируйте его в директорию навыков:
# Download the skill
curl -o .claude/skills/content-core/SKILL.md --create-dirs \
https://raw.githubusercontent.com/lfnovo/content-core/main/SKILL.mdПосле установки Claude Code может использовать content-core для извлечения контента из URL-адресов, документов и медиафайлов — через CLI (uvx content-core) или MCP, если он настроен.
Провайдеры ИИ
Content Core использует Esperanto для поддержки нескольких провайдеров LLM и STT. Переключайте провайдеров, изменяя конфигурацию — изменения в коде не требуются:
# Use Anthropic for summarization
content-core config set llm_provider anthropic
content-core config set llm_model claude-sonnet-4-20250514
# Use Groq for transcription
content-core config set stt_provider groq
content-core config set stt_model whisper-large-v3Поддерживаемые провайдеры включают OpenAI, Anthropic, Google, Groq, DeepSeek, Ollama и другие. Полный список см. в документации Esperanto.
Конфигурация
Content Core использует ContentCoreConfig на базе pydantic-settings. Настройки разрешаются в порядке приоритета: аргументы конструктора > переменные окружения (CCORE_*) > файл конфигурации (~/.content-core/config.toml) > значения по умолчанию.
Переменные окружения
Переменная | Описание | По умолчанию |
| Движок извлечения URL ( |
|
| Движок извлечения документов ( |
|
| Параллельная транскрипция аудио (1-10) |
|
| URL API Crawl4AI Docker (оставьте пустым для локального режима браузера) | - |
| Пользовательский URL API Firecrawl для self-hosted экземпляров | - |
| Режим прокси Firecrawl ( |
|
| Время ожидания в мс перед извлечением |
|
| Провайдер LLM для обобщения | - |
| Модель LLM для обобщения | - |
| Провайдер преобразования речи в текст | - |
| Модель преобразования речи в текст | - |
| Тайм-аут преобразования речи в текст в секундах | - |
| Предпочитаемые языки транскриптов YouTube | - |
API-ключи для внешних сервисов устанавливаются через их стандартные переменные окружения (например, OPENAI_API_KEY, FIRECRAWL_API_KEY, JINA_API_KEY).
Конфигурация прокси
Content Core автоматически считывает стандартные переменные окружения HTTP_PROXY / HTTPS_PROXY / NO_PROXY. Дополнительная настройка не требуется.
Дополнительные зависимости
# Docling for advanced document parsing (PDF, DOCX, PPTX, XLSX)
pip install content-core[docling]
# Crawl4AI for local browser-based URL extraction
pip install content-core[crawl4ai]
python -m playwright install --with-deps
# LangChain tool wrappers
pip install content-core[langchain]
# All optional features
pip install content-core[docling,crawl4ai,langchain]Использование с LangChain
При установке с дополнением langchain, Content Core предоставляет обертки инструментов, совместимые с LangChain:
from content_core.tools import extract_content_tool, summarize_content_tool
tools = [extract_content_tool, summarize_content_tool]Документация
Руководство по использованию -- подробности Python API, конфигурация и примеры
Процессоры -- как работает извлечение контента для каждого формата
Сервер MCP -- интеграция с Claude Desktop и MCP
Разработка
git clone https://github.com/lfnovo/content-core
cd content-core
uv sync --group dev
# Run tests
make test
# Lint
make ruffЛицензия
Этот проект лицензирован по лицензии MIT.
Участие в разработке
Вклад приветствуется! Пожалуйста, ознакомьтесь с нашим Руководством по участию для получения подробной информации.
Available Tools
2 toolsextract_contentExtract ContentA
Extract content from a URL or file. Does not require an API key for most sources (web pages, PDFs, documents, YouTube transcripts). API key is only needed for audio/video transcription.
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | URL to extract content from (web page, YouTube video, PDF link, etc.) | |
| engine | No | Optional extraction engine override, routed by input type. With `url`: auto, simple, firecrawl, jina, crawl4ai. With `file_path`: auto, simple, docling — `docling` requires `pip install "content-core[docling]"` and fails with a configuration error when the extra is missing, in which case use `auto` or `simple`. Any other value is rejected with an error naming the accepted ones. | |
| no_ocr | No | Disable OCR in Docling (requires engine=docling) | |
| formulas | No | Enable formula extraction via Docling (requires engine=docling) | |
| pictures | No | Enable image description + chart data extraction via Docling (requires engine=docling) | |
| file_path | No | Local file path to extract content from |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden; it usefully discloses the auth profile (no key needed for most sources, key needed for audio/video). It is silent on other behaviors: what happens if neither url nor file_path is supplied (0 required params), and whether unsupported sources error or degrade.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences, front-loaded with the core action, and the auth caveat placed immediately after. No filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be described, and the auth context is a genuine addition. Still missing is guidance on the unusual 0-required-parameter shape (must one of url/file_path be given?) and any routing versus the sibling tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents engine routing, docling requirements, and the flag semantics in detail. The description adds only a loose mapping of source types, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Extract') and resource ('content from a URL or file') plus the supported source types. It does not explicitly differentiate itself from the sibling summarize_content, so an agent must infer that this returns raw content rather than a summary.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives conditional guidance about when an API key is required (audio/video transcription only), which is useful. However, it never says when to choose this over summarize_content, nor does it explain the url-vs-file_path choice, leaving usage largely implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
summarize_contentSummarize ContentB
Summarize content using LLM with optional context. Requires OPENAI_API_KEY (or another LLM provider key) to be configured.
| Name | Required | Description | Default |
|---|---|---|---|
| content | Yes | The text content to summarize | |
| context | No | Optional context to guide summarization (e.g., "summarize as bullet points") |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It usefully discloses a real prerequisite — an LLM provider key such as OPENAI_API_KEY must be configured — which is not in the schema. However it says nothing about cost, latency, or truncation behavior on long input, leaving gaps for a tool that calls an external LLM.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two compact sentences with no filler, and the core purpose is front-loaded ahead of the setup requirement. Slightly terse, but every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Output schema exists so return values need no explanation, and both parameters are schema-documented, making the definition largely adequate for a simple tool. The gaps are the absence of any routing guidance versus extract_content and the lack of cost/latency context for an external LLM call.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so both parameters are already fully documented, and the description adds nothing beyond restating that context is optional. Baseline 3 applies when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('summarize content') plus the mechanism ('using LLM'), which distinguishes it from the sibling extract_content at a high level. It does not explicitly name or contrast with that sibling, so it falls short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to choose this tool over extract_content, nor any exclusions or preconditions for use beyond the API-key note. The reader must infer that summarizing is the use case from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
2 tool updates
v2.1.1- Changed
extract_content6 fields changed- added
Input schema / properties / engine / descriptionAdded value: +"Optional extraction engine override, routed by input type.\nWith `url`: auto, simple, firecrawl, jina, crawl4ai.\nWith `file_path`: auto, simple, docling — `docling` requires\n`pip install \"content-core[docling]\"` and fails with a\nconfiguration error when the extra is missing, in which case use\n`auto` or `simple`.\nAny other value is rejected with an error naming the accepted ones." - added
Input schema / properties / file_path / descriptionAdded value: +"Local file path to extract content from" - added
Input schema / properties / formulas / descriptionAdded value: +"Enable formula extraction via Docling (requires engine=docling)" - added
Input schema / properties / no_ocr / descriptionAdded value: +"Disable OCR in Docling (requires engine=docling)" - added
Input schema / properties / pictures / descriptionAdded value: +"Enable image description + chart data extraction via Docling (requires engine=docling)" - added
Input schema / properties / url / descriptionAdded value: +"URL to extract content from (web page, YouTube video, PDF link, etc.)"
- Changed
summarize_content2 fields changed- added
Input schema / properties / content / descriptionAdded value: +"The text content to summarize" - added
Input schema / properties / context / descriptionAdded value: +"Optional context to guide summarization (e.g., \"summarize as bullet points\")"
2 tool updates
v2.0.4- Changed
extract_content9 fields changed- added
Input schema / additionalPropertiesAdded value: +false - added
Input schema / properties / engineAdded value: +{ + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null +} - added
Input schema / properties / formulasAdded value: +{ + "default": false, + "type": "boolean" +} - added
Input schema / properties / no_ocrAdded value: +{ + "default": false, + "type": "boolean" +} - added
Input schema / properties / picturesAdded value: +{ + "default": false, + "type": "boolean" +} - removed
Output schema / additionalPropertiesRemoved value: -true - added
Output schema / propertiesAdded value: +{ + "result": { + "type": "string" + } +} - added
Output schema / requiredAdded value: +[ + "result" +] - added
Output schema / x-fastmcp-wrap-resultAdded value: +true
- Added
summarize_content
1 tool update
v1.0.0- Changed
extract_content2 fields changed- removed
Input schema / properties / file_path / titleRemoved value: -"File Path" - removed
Input schema / properties / url / titleRemoved value: -"Url"
1 tool update
- First observed
extract_content
TDQS
Scored across 2 tools
The two tools target clearly distinct actions: extract_content pulls raw content from a source, while summarize_content condenses it via an LLM. An agent can easily tell which to call based on whether it needs raw text or a summary.
Both names follow the same verb_noun snake_case pattern (extract_content, summarize_content), with no deviations in convention or style.
Two tools is on the thin side for even a focused server, though each earns its place in a natural extract-then-summarize pipeline. A couple of supporting operations (e.g. format conversion or output saving) would round out the surface.
The extract and summarize pair covers the core content-processing lifecycle, including multiple input sources (URLs, PDFs, YouTube). Minor gaps exist around batch processing, output formatting, or chunking, but agents can work around these.
Maintenance
Related MCP Connectors
PDF, DOCX and image URLs to clean, LLM-ready Markdown or text, with OCR for scanned pages.
Any URL to clean markdown/JSON; PDF, XLSX, CSV, DOCX to text and tables. Free tier for agents.
Extract structured insights from videos, podcasts, articles, and PDFs with multi-model AI
Extract and parse web pages into clean HTML, links, or Markdown. Handle dynamic, complex, or block…
Related MCP Servers
- AlicenseCqualityBmaintenanceA powerful tool for fetching and extracting text content from web pages and APIs, supporting web scraping, REST API requests, and Google Custom Search integration.511MIT
- AlicenseNot gradedqualityDmaintenanceEnables web scraping and document processing with JavaScript execution, anti-detection measures, batch processing, and structured data extraction. Supports multiple formats including markdown, HTML, screenshots, and handles PDFs with OCR capabilities.4MIT
- AlicenseBqualityCmaintenanceParse any file or URL into structured text. Extract text from PDF, DOCX, YouTube, web pages, images, and 25+ formats via one API. Tools: parse_url, parse_file, get_youtube_transcript.37 npmMIT
- AlicenseNot gradedqualityDmaintenanceConverts any URL into clean, LLM-ready Markdown, text, or HTML with production-grade features like SSRF protection, rate limiting, retries, caching, and structured error handling.MIT