Skip to main content
Glama
lfnovo
by lfnovo

Content Core

License: MIT PyPI version Downloads Downloads GitHub stars GitHub forks GitHub issues Ruff

Extract, process, and summarize content from URLs, files, and text through a unified async Python API, CLI, or MCP server.

Supported Formats

Category

Formats

Web

URLs, HTML pages, YouTube videos, Reddit posts

Documents

PDF, DOCX, PPTX, XLSX, EPUB, Markdown, plain text

Media

MP3, WAV, M4A, FLAC, OGG (audio); MP4, AVI, MOV, MKV (video)

Related MCP server: FreeCrawl MCP Server

Quick Start

pip install content-core
import content_core

result = await content_core.extract_content(url="https://example.com")
print(result.content)

Or with zero install:

uvx content-core extract "https://example.com"

CLI Usage

Content Core provides a unified content-core command with subcommands for extraction, summarization, and MCP server.

Extract

# From a URL
content-core extract "https://example.com"

# From a file
content-core extract document.pdf

# With JSON output
content-core extract document.pdf --format json

# With a specific engine
content-core extract "https://example.com" --engine firecrawl

# From stdin
echo "some text" | content-core extract

Summarize

# Summarize text
content-core summarize "Long article text here..."

# With context
content-core summarize "Long text" --context "bullet points"

# From stdin
cat article.txt | content-core summarize --context "explain to a child"

MCP Server

content-core mcp

Configuration

# Set persistent config
content-core config set llm_provider anthropic
content-core config set llm_model claude-sonnet-5

# List current config
content-core config list

# Delete a config value
content-core config delete llm_provider

Config is stored in ~/.content-core/config.toml. Priority: command flags > env vars > config file > defaults.

Zero-Install with uvx

All commands work without installation using uvx:

uvx content-core extract "https://example.com"
uvx content-core summarize "text" --context "one sentence"
uvx content-core mcp

Python API

Extraction

import content_core

# From a URL
result = await content_core.extract_content(url="https://example.com")

# From a file
result = await content_core.extract_content(file_path="document.pdf")

# From text
result = await content_core.extract_content(content="some text")

# With engine override
from content_core import ContentCoreConfig
config = ContentCoreConfig(url_engine="firecrawl")
result = await content_core.extract_content(url="https://example.com", config=config)

Summarization

import content_core

summary = await content_core.summarize("long article text", context="bullet points")

Configuration

from content_core import ContentCoreConfig

config = ContentCoreConfig(
    url_engine="firecrawl",
    document_engine="docling",
    audio_concurrency=5,
)
result = await content_core.extract_content(url="https://example.com", config=config)

MCP Integration

Content Core includes a Model Context Protocol (MCP) server for use with Claude Desktop and other MCP-compatible applications.

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "content-core": {
      "command": "uvx",
      "args": ["content-core", "mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-..."
      }
    }
  }
}

The MCP server exposes two tools: extract_content and summarize_content. Both return plain text.

For detailed setup, see the MCP documentation.

Agent Skill (Claude Code & Codex)

Content Core ships an Agent Skill that teaches AI agents how to use it for extracting content from external sources. This repository is also a plugin marketplace, so the skill installs natively in both harnesses.

Claude Code — add the marketplace and install the plugin:

/plugin marketplace add lfnovo/content-core
/plugin install content-core@content-core

Codex — the repository carries a Codex plugin manifest (.codex-plugin/plugin.json) and marketplace catalog (.agents/plugins/marketplace.json) pointing at the same skill.

Manual fallback — copy the skill file directly into your project:

curl -o .claude/skills/content-core/SKILL.md --create-dirs \
  https://raw.githubusercontent.com/lfnovo/content-core/main/skills/content-core/SKILL.md

Once installed, the agent can use content-core to extract content from URLs, documents, and media files — either via CLI (uvx content-core) or MCP if configured.

AI Providers

Content Core uses Esperanto to support multiple LLM and STT providers. Switch providers by changing the config — no code changes needed:

# Use Anthropic for summarization
content-core config set llm_provider anthropic
content-core config set llm_model claude-sonnet-5

# Use Groq for transcription
content-core config set stt_provider groq
content-core config set stt_model whisper-large-v3

Supported providers include OpenAI, Anthropic, Google, Groq, DeepSeek, Ollama, and more. See the Esperanto documentation for the full list.

Configuration

Content Core uses ContentCoreConfig powered by pydantic-settings. Settings are resolved in priority order: constructor args > env vars (CCORE_*) > config file (~/.content-core/config.toml) > defaults.

Environment Variables

Variable

Description

Default

CCORE_URL_ENGINE

URL extraction engine (auto, simple, firecrawl, jina, crawl4ai)

auto

CCORE_DOCUMENT_ENGINE

Document extraction engine (auto, simple, docling)

auto

CCORE_AUDIO_CONCURRENCY

Concurrent audio transcriptions (1-10)

3

CRAWL4AI_API_URL

Crawl4AI Docker API URL (omit for local browser mode)

-

FIRECRAWL_API_URL

Custom Firecrawl API URL for self-hosted instances

-

CCORE_FIRECRAWL_PROXY

Firecrawl proxy mode (auto, basic, stealth)

auto

CCORE_FIRECRAWL_WAIT_FOR

Wait time in ms before extraction

3000

CCORE_LLM_PROVIDER

LLM provider for summarization

-

CCORE_LLM_MODEL

LLM model for summarization

-

CCORE_STT_PROVIDER

Speech-to-text provider

-

CCORE_STT_MODEL

Speech-to-text model

-

CCORE_STT_TIMEOUT

Speech-to-text timeout in seconds

-

CCORE_YOUTUBE_LANGUAGES

Preferred YouTube transcript languages

-

API keys for external services are set via their standard environment variables (e.g., OPENAI_API_KEY, FIRECRAWL_API_KEY, JINA_API_KEY).

Proxy Configuration

Content Core reads standard HTTP_PROXY / HTTPS_PROXY / NO_PROXY environment variables automatically. No additional configuration is needed.

Optional Dependencies

# Docling for advanced document parsing (PDF, DOCX, PPTX, XLSX)
pip install content-core[docling]

# Crawl4AI for local browser-based URL extraction
pip install content-core[crawl4ai]
python -m playwright install --with-deps

# LangChain tool wrappers
pip install content-core[langchain]

# All optional features
pip install content-core[docling,crawl4ai,langchain]

Using with LangChain

When installed with the langchain extra, Content Core provides LangChain-compatible tool wrappers:

from content_core.tools import extract_content_tool, summarize_content_tool

tools = [extract_content_tool, summarize_content_tool]

Documentation

  • Usage Guide -- Python API details, configuration, and examples

  • Processors -- How content extraction works for each format

  • MCP Server -- Claude Desktop and MCP integration

Development

git clone https://github.com/lfnovo/content-core
cd content-core

uv sync --group dev

# Run tests
make test

# Lint
make ruff

License

This project is licensed under the MIT License.

Contributing

Contributions are welcome! Please see our Contributing Guide for details.

Available Tools

2 tools
extract_contentA

Extract content from a URL or file. Does not require an API key for most sources (web pages, PDFs, documents, YouTube transcripts). API key is only needed for audio/video transcription.

Args: url: URL to extract content from (web page, YouTube video, PDF link, etc.) file_path: Local file path to extract content from engine: Optional extraction engine override (firecrawl, jina, crawl4ai, simple, docling) formulas: Enable formula extraction via Docling (requires engine=docling) pictures: Enable image description + chart data extraction via Docling (requires engine=docling) no_ocr: Disable OCR in Docling (requires engine=docling)

Returns: Extracted text content

ParametersJSON Schema
NameRequiredDescriptionDefault
urlNo
engineNo
no_ocrNo
formulasNo
picturesNo
file_pathNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It discloses that API key is needed for some sources, describes engine override options, and explains the effects of flags like formulas and pictures. It does not mention rate limits or error handling, but still adds significant value.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a clear intro, bulleted argument list, and return type. It is relatively concise, though the argument list could be slightly more compact without losing clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 6 parameters, no required ones, and an output schema, the description covers all parameters and return type, and provides context on API key requirements. It does not document error cases or additional behavior, but the output schema likely covers structure.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It explains all 6 parameters (url, file_path, engine, formulas, pictures, no_ocr) with details on their usage and effects, going far beyond what the JSON schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool extracts content from a URL or file, specifies supported sources (web pages, PDFs, YouTube transcripts), and distinguishes itself from the sibling tool 'summarize_content' by its action of extraction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context on when to use the tool, noting that an API key is not needed for most sources but is required for audio/video transcription. However, it does not explicitly state when not to use it or how it compares to the sibling tool beyond the action.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

summarize_contentA

Summarize content using LLM with optional context. Requires OPENAI_API_KEY (or another LLM provider key) to be configured.

Args: content: The text content to summarize context: Optional context to guide summarization (e.g., "summarize as bullet points")

Returns: Summarized text

ParametersJSON Schema
NameRequiredDescriptionDefault
contentYes
contextNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description bears the full burden of behavioral disclosure. It notes the requirement for an LLM provider API key and indicates that the tool uses an LLM for summarization. However, it does not disclose potential rate limits, costs, or failure modes, leaving some behavioral aspects opaque.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description uses a docstring format with sections (Args, Returns), making it structured but slightly verbose. It front-loads the core purpose but adds extra formatting that could be trimmed. It is not overly long but could be more concise.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the low schema coverage and absence of annotations, the description provides the essential parameter meanings and return type. It also mentions the critical API key dependency. However, it lacks constraints like maximum content length or edge cases, and the output schema existence lightens the burden but doesn't fully compensate for missing details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% (no descriptions in input schema), but the description adds meaningful explanations for both parameters: 'content' is the text to summarize, and 'context' is optional guidance with an example ('summarize as bullet points'). This compensates well for the missing schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Summarize content using LLM with optional context.' It specifies a specific verb-resource relationship and distinguishes from the sibling tool 'extract_content' which serves a different purpose.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description mentions a prerequisite (API key configuration) but provides no guidance on when to use this tool versus alternatives, such as the sibling 'extract_content'. No explicit when-to-use or when-not-to-use guidance is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

TDQS

A3.9/5.0
Disambiguation5/5

The two tools have clearly distinct purposes: extract_content retrieves raw content from various sources, while summarize_content processes text to produce a summary. No overlap in functionality.

Naming Consistency5/5

Both tools follow a consistent verb_noun pattern (extract_content, summarize_content), using snake_case and descriptive verbs.

Tool Count3/5

With only 2 tools, the server feels minimally scoped. While it covers the basic extract-summarize workflow, the small count suggests a limited surface for a domain that could benefit from additional tools like formatting or source management.

Completeness3/5

The pair covers core extraction and summarization operations, but notable gaps exist: no tool for listing supported engines, no content comparison or conversion, and no error-handling utilities. The surface is functional but incomplete for advanced content processing.

Maintenance

ActivitySlowing
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/lfnovo/content-core'

If you have feedback or need assistance with the MCP directory API, please join our Discord server