Skip to main content
Glama
content-core
by lfnovo

Content Core

License: MIT PyPI version Downloads Downloads GitHub stars GitHub forks GitHub issues Ruff

Extract, process, and summarize content from URLs, files, and text through a unified async Python API, CLI, or MCP server.

Supported Formats

Category

Formats

Web

URLs, HTML pages, YouTube videos, Reddit posts

Documents

PDF, DOCX, PPTX, XLSX, ODT, ODS, ODP, EPUB, HTML, Markdown, plain text

Media

MP3, WAV, M4A, FLAC, OGG (audio); MP4, AVI, MOV, MKV (video)

Related MCP server: FreeCrawl MCP Server

Quick Start

pip install content-core
import content_core

result = await content_core.extract_content(url="https://example.com")
print(result.content)

Or with zero install:

uvx content-core extract "https://example.com"

CLI Usage

Content Core provides a unified content-core command with subcommands for extraction, summarization, and MCP server.

Extract

# From a URL
content-core extract "https://example.com"

# From a file
content-core extract document.pdf

# With JSON output
content-core extract document.pdf --format json

# With a specific engine
content-core extract "https://example.com" --engine firecrawl

# From stdin
echo "some text" | content-core extract

Summarize

# Summarize text
content-core summarize "Long article text here..."

# With context
content-core summarize "Long text" --context "bullet points"

# From stdin
cat article.txt | content-core summarize --context "explain to a child"

MCP Server

content-core mcp

Configuration

# Set persistent config
content-core config set llm_provider anthropic
content-core config set llm_model claude-sonnet-5

# List current config
content-core config list

# Delete a config value
content-core config delete llm_provider

Config is stored in ~/.content-core/config.toml. Priority: command flags > env vars > config file > defaults.

Zero-Install with uvx

All commands work without installation using uvx:

uvx content-core extract "https://example.com"
uvx content-core summarize "text" --context "one sentence"
uvx content-core mcp

Python API

Extraction

import content_core

# From a URL
result = await content_core.extract_content(url="https://example.com")

# From a file
result = await content_core.extract_content(file_path="document.pdf")

# From text
result = await content_core.extract_content(content="some text")

# With engine override
from content_core import ContentCoreConfig
config = ContentCoreConfig(url_engine="firecrawl")
result = await content_core.extract_content(url="https://example.com", config=config)

Summarization

import content_core

summary = await content_core.summarize("long article text", context="bullet points")

Configuration

from content_core import ContentCoreConfig

config = ContentCoreConfig(
    url_engine="firecrawl",
    document_engine="docling",
    audio_concurrency=5,
)
result = await content_core.extract_content(url="https://example.com", config=config)

MCP Integration

Content Core includes a Model Context Protocol (MCP) server for use with Claude Desktop and other MCP-compatible applications.

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "content-core": {
      "command": "uvx",
      "args": ["content-core", "mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-..."
      }
    }
  }
}

The MCP server exposes two tools: extract_content and summarize_content. Both return plain text.

For detailed setup, see the MCP documentation.

Agent Skill (Claude Code & Codex)

Content Core ships an Agent Skill that teaches AI agents how to use it for extracting content from external sources. This repository is also a plugin marketplace, so the skill installs natively in both harnesses.

Claude Code — add the marketplace and install the plugin:

/plugin marketplace add lfnovo/content-core
/plugin install content-core@content-core

Codex — the repository carries a Codex plugin manifest (.codex-plugin/plugin.json) and marketplace catalog (.agents/plugins/marketplace.json) pointing at the same skill.

Manual fallback — copy the skill file directly into your project:

curl -o .claude/skills/content-core/SKILL.md --create-dirs \
  https://raw.githubusercontent.com/lfnovo/content-core/main/skills/content-core/SKILL.md

Once installed, the agent can use content-core to extract content from URLs, documents, and media files — either via CLI (uvx content-core) or MCP if configured.

AI Providers

Content Core uses Esperanto to support multiple LLM and STT providers. Switch providers by changing the config — no code changes needed:

# Use Anthropic for summarization
content-core config set llm_provider anthropic
content-core config set llm_model claude-sonnet-5

# Use Groq for transcription
content-core config set stt_provider groq
content-core config set stt_model whisper-large-v3

Supported providers include OpenAI, Anthropic, Google, Groq, DeepSeek, Ollama, and more. See the Esperanto documentation for the full list.

Configuration

Content Core uses ContentCoreConfig powered by pydantic-settings. Settings are resolved in priority order: constructor args > env vars (CCORE_*) > config file (~/.content-core/config.toml) > defaults.

Environment Variables

Variable

Description

Default

CCORE_URL_ENGINE

URL extraction engine (auto, simple, firecrawl, jina, crawl4ai)

auto

CCORE_DOCUMENT_ENGINE

Document extraction engine (auto, simple, docling) — docling raises ConfigurationError if the extra is not installed; auto falls back silently

auto

CCORE_AUDIO_CONCURRENCY

Concurrent audio transcriptions (1-10)

3

CCORE_AUDIO_SEGMENT_MINUTES

Length of the segments long audio is split into; 0 sends the file whole

10

CRAWL4AI_API_URL

Crawl4AI Docker API URL (omit for local browser mode)

-

CRAWL4AI_API_TOKEN

Bearer token for the Crawl4AI Docker API (required by Crawl4AI >= 0.9.0)

-

FIRECRAWL_API_URL

Custom Firecrawl API URL for self-hosted instances or Firecrawl-compatible backends (e.g. fastCRW)

-

CCORE_FIRECRAWL_PROXY

Firecrawl proxy mode (auto, basic, stealth)

auto

CCORE_FIRECRAWL_WAIT_FOR

Wait time in ms before extraction

3000

CCORE_LLM_PROVIDER

LLM provider for summarization

-

CCORE_LLM_MODEL

LLM model for summarization

-

CCORE_STT_PROVIDER

Speech-to-text provider

-

CCORE_STT_MODEL

Speech-to-text model

-

CCORE_STT_TIMEOUT

Speech-to-text timeout in seconds

-

CCORE_YOUTUBE_LANGUAGES

Preferred YouTube transcript languages

-

CCORE_YOUTUBE_COOKIES_FILE

Path to a Netscape cookies.txt for YouTube transcripts (missing file raises ConfigurationError)

-

CCORE_YOUTUBE_PROXY

Proxy URL for YouTube transcripts, e.g. http://user:pass@host:port

-

CCORE_DOCLING_VISION_URL

OpenAI-compatible endpoint (vLLM, Ollama, LM Studio...) for Docling image descriptions; unset keeps Docling's inline model. Related: CCORE_DOCLING_VISION_MODEL, _API_KEY, _PROMPT, _MAX_TOKENS, _TIMEOUT, _TOKEN_PARAM (details)

-

API keys for external services are set via their standard environment variables (e.g., OPENAI_API_KEY, FIRECRAWL_API_KEY, JINA_API_KEY).

Proxy Configuration

Content Core reads standard HTTP_PROXY / HTTPS_PROXY / NO_PROXY environment variables automatically. No additional configuration is needed.

YouTube on Blocked Networks

If YouTube extraction fails with IpBlocked/RequestBlocked on a flagged IP (cloud hosts, VPNs), either point CCORE_YOUTUBE_COOKIES_FILE at a cookies.txt exported from a logged-in browser (re-export it when the cookies expire), or set CCORE_YOUTUBE_PROXY to a residential proxy — datacenter proxies get blocked or CAPTCHA'd. See docs/usage.md.

Optional Dependencies

# Docling for advanced document parsing (PDF, DOCX, PPTX, XLSX)
# Required by document_engine="docling", which raises ConfigurationError without it.
# Use document_engine="auto" (default) or "simple" to proceed without Docling.
pip install content-core[docling]

# Crawl4AI for local browser-based URL extraction
pip install content-core[crawl4ai]
python -m playwright install --with-deps

# LangChain tool wrappers
pip install content-core[langchain]

# All optional features
pip install content-core[docling,crawl4ai,langchain]

Using with LangChain

When installed with the langchain extra, Content Core provides LangChain-compatible tool wrappers:

from content_core.tools import extract_content_tool, summarize_content_tool

tools = [extract_content_tool, summarize_content_tool]

Documentation

  • Usage Guide -- Python API details, configuration, and examples

  • Processors -- How content extraction works for each format

  • MCP Server -- Claude Desktop and MCP integration

Development

git clone https://github.com/lfnovo/content-core
cd content-core

uv sync --group dev

# Run tests
make test

# Lint
make ruff

License

This project is licensed under the MIT License.

Contributing

Contributions are welcome! Please see our Contributing Guide for details.

Available Tools

2 tools
extract_contentExtract ContentA

Extract content from a URL or file. Does not require an API key for most sources (web pages, PDFs, documents, YouTube transcripts). API key is only needed for audio/video transcription.

Inputs: url, engine, no_ocr, formulas, +2 more. Returns: result

summarize_contentSummarize ContentB

Summarize content using LLM with optional context. Requires OPENAI_API_KEY (or another LLM provider key) to be configured.

Inputs: content (required), context. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.6/5.0

    Scored across 2 tools

    Disambiguation5/5

    The two tools target clearly distinct actions: extract_content pulls raw content from a source, while summarize_content condenses it via an LLM. An agent can easily tell which to call based on whether it needs raw text or a summary.

    Naming Consistency5/5

    Both names follow the same verb_noun snake_case pattern (extract_content, summarize_content), with no deviations in convention or style.

    Tool Count3/5

    Two tools is on the thin side for even a focused server, though each earns its place in a natural extract-then-summarize pipeline. A couple of supporting operations (e.g. format conversion or output saving) would round out the surface.

    Completeness4/5

    The extract and summarize pair covers the core content-processing lifecycle, including multiple input sources (URLs, PDFs, YouTube). Minor gaps exist around batch processing, output formatting, or chunking, but agents can work around these.

    Maintenance

    ActivityMaintained
    ResponsivenessLimited data

    Related MCP Connectors

    Related MCP Servers