Skip to main content
Glama
content-core
by lfnovo

Content Core

License: MIT PyPI version Downloads Downloads GitHub stars GitHub forks GitHub issues Ruff

통합 비동기 Python API, CLI 또는 MCP 서버를 통해 URL, 파일 및 텍스트에서 콘텐츠를 추출, 처리 및 요약합니다.

지원 형식

카테고리

형식

웹

URL, HTML 페이지, YouTube 비디오, Reddit 게시물

문서

PDF, DOCX, PPTX, XLSX, EPUB, Markdown, 일반 텍스트

미디어

MP3, WAV, M4A, FLAC, OGG (오디오); MP4, AVI, MOV, MKV (비디오)

Related MCP server: FreeCrawl MCP Server

빠른 시작

pip install content-core
import content_core

result = await content_core.extract_content(url="https://example.com")
print(result.content)

또는 설치 없이 사용:

uvx content-core extract "https://example.com"

CLI 사용법

Content Core는 추출, 요약 및 MCP 서버를 위한 하위 명령어가 포함된 통합 content-core 명령어를 제공합니다.

추출

# From a URL
content-core extract "https://example.com"

# From a file
content-core extract document.pdf

# With JSON output
content-core extract document.pdf --format json

# With a specific engine
content-core extract "https://example.com" --engine firecrawl

# From stdin
echo "some text" | content-core extract

요약

# Summarize text
content-core summarize "Long article text here..."

# With context
content-core summarize "Long text" --context "bullet points"

# From stdin
cat article.txt | content-core summarize --context "explain to a child"

MCP 서버

content-core mcp

구성

# Set persistent config
content-core config set llm_provider anthropic
content-core config set llm_model claude-sonnet-4-20250514

# List current config
content-core config list

# Delete a config value
content-core config delete llm_provider

설정은 ~/.content-core/config.toml에 저장됩니다. 우선순위: 명령 플래그 > 환경 변수 > 설정 파일 > 기본값.

uvx를 사용한 무설치 실행

모든 명령어는 uvx를 사용하여 설치 없이 작동합니다:

uvx content-core extract "https://example.com"
uvx content-core summarize "text" --context "one sentence"
uvx content-core mcp

Python API

추출

import content_core

# From a URL
result = await content_core.extract_content(url="https://example.com")

# From a file
result = await content_core.extract_content(file_path="document.pdf")

# From text
result = await content_core.extract_content(content="some text")

# With engine override
from content_core import ContentCoreConfig
config = ContentCoreConfig(url_engine="firecrawl")
result = await content_core.extract_content(url="https://example.com", config=config)

요약

import content_core

summary = await content_core.summarize("long article text", context="bullet points")

구성

from content_core import ContentCoreConfig

config = ContentCoreConfig(
    url_engine="firecrawl",
    document_engine="docling",
    audio_concurrency=5,
)
result = await content_core.extract_content(url="https://example.com", config=config)

MCP 통합

Content Core는 Claude Desktop 및 기타 MCP 호환 애플리케이션에서 사용할 수 있는 MCP(Model Context Protocol) 서버를 포함합니다.

claude_desktop_config.json에 추가하세요:

{
  "mcpServers": {
    "content-core": {
      "command": "uvx",
      "args": ["content-core", "mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-..."
      }
    }
  }
}

MCP 서버는 extract_content와 summarize_content라는 두 가지 도구를 제공합니다. 둘 다 일반 텍스트를 반환합니다.

자세한 설정은 MCP 문서를 참조하세요.

Claude Code 스킬

Content Core는 AI 에이전트가 외부 소스에서 콘텐츠를 추출하는 방법을 학습할 수 있는 SKILL.md를 포함합니다. Claude Code 프로젝트에서 사용하려면 스킬 디렉토리에 복사하세요:

# Download the skill
curl -o .claude/skills/content-core/SKILL.md --create-dirs \
  https://raw.githubusercontent.com/lfnovo/content-core/main/SKILL.md

설치가 완료되면 Claude Code는 CLI(uvx content-core) 또는 구성된 경우 MCP를 통해 content-core를 사용하여 URL, 문서 및 미디어 파일에서 콘텐츠를 추출할 수 있습니다.

AI 제공업체

Content Core는 Esperanto를 사용하여 여러 LLM 및 STT 제공업체를 지원합니다. 설정을 변경하여 제공업체를 전환하세요. 코드 변경은 필요하지 않습니다:

# Use Anthropic for summarization
content-core config set llm_provider anthropic
content-core config set llm_model claude-sonnet-4-20250514

# Use Groq for transcription
content-core config set stt_provider groq
content-core config set stt_model whisper-large-v3

지원되는 제공업체에는 OpenAI, Anthropic, Google, Groq, DeepSeek, Ollama 등이 포함됩니다. 전체 목록은 Esperanto 문서를 참조하세요.

구성

Content Core는 pydantic-settings 기반의 ContentCoreConfig를 사용합니다. 설정은 우선순위 순으로 결정됩니다: 생성자 인수 > 환경 변수 (CCORE_*) > 설정 파일 (~/.content-core/config.toml) > 기본값.

환경 변수

변수

설명

기본값

CCORE_URL_ENGINE

URL 추출 엔진 (auto, simple, firecrawl, jina, crawl4ai)

auto

CCORE_DOCUMENT_ENGINE

문서 추출 엔진 (auto, simple, docling)

auto

CCORE_AUDIO_CONCURRENCY

동시 오디오 전사 (1-10)

3

CRAWL4AI_API_URL

Crawl4AI Docker API URL (로컬 브라우저 모드 시 생략)

-

FIRECRAWL_API_URL

자체 호스팅 인스턴스를 위한 사용자 지정 Firecrawl API URL

-

CCORE_FIRECRAWL_PROXY

Firecrawl 프록시 모드 (auto, basic, stealth)

auto

CCORE_FIRECRAWL_WAIT_FOR

추출 전 대기 시간 (ms)

3000

CCORE_LLM_PROVIDER

요약을 위한 LLM 제공업체

-

CCORE_LLM_MODEL

요약을 위한 LLM 모델

-

CCORE_STT_PROVIDER

음성-텍스트 변환 제공업체

-

CCORE_STT_MODEL

음성-텍스트 변환 모델

-

CCORE_STT_TIMEOUT

음성-텍스트 변환 시간 제한 (초)

-

CCORE_YOUTUBE_LANGUAGES

선호하는 YouTube 자막 언어

-

외부 서비스에 대한 API 키는 표준 환경 변수(예: OPENAI_API_KEY, FIRECRAWL_API_KEY, JINA_API_KEY)를 통해 설정됩니다.

프록시 구성

Content Core는 표준 HTTP_PROXY / HTTPS_PROXY / NO_PROXY 환경 변수를 자동으로 읽습니다. 추가 구성은 필요하지 않습니다.

선택적 종속성

# Docling for advanced document parsing (PDF, DOCX, PPTX, XLSX)
pip install content-core[docling]

# Crawl4AI for local browser-based URL extraction
pip install content-core[crawl4ai]
python -m playwright install --with-deps

# LangChain tool wrappers
pip install content-core[langchain]

# All optional features
pip install content-core[docling,crawl4ai,langchain]

LangChain과 함께 사용

langchain 추가 기능과 함께 설치하면 Content Core는 LangChain 호환 도구 래퍼를 제공합니다:

from content_core.tools import extract_content_tool, summarize_content_tool

tools = [extract_content_tool, summarize_content_tool]

문서

개발

git clone https://github.com/lfnovo/content-core
cd content-core

uv sync --group dev

# Run tests
make test

# Lint
make ruff

라이선스

이 프로젝트는 MIT 라이선스에 따라 라이선스가 부여됩니다.

기여

기여를 환영합니다! 자세한 내용은 기여 가이드를 참조하세요.

Available Tools

2 tools
extract_contentExtract ContentA

Extract content from a URL or file. Does not require an API key for most sources (web pages, PDFs, documents, YouTube transcripts). API key is only needed for audio/video transcription.

NameRequiredDescriptionDefault
urlNoURL to extract content from (web page, YouTube video, PDF link, etc.)
engineNoOptional extraction engine override, routed by input type. With `url`: auto, simple, firecrawl, jina, crawl4ai. With `file_path`: auto, simple, docling — `docling` requires `pip install "content-core[docling]"` and fails with a configuration error when the extra is missing, in which case use `auto` or `simple`. Any other value is rejected with an error naming the accepted ones.
no_ocrNoDisable OCR in Docling (requires engine=docling)
formulasNoEnable formula extraction via Docling (requires engine=docling)
picturesNoEnable image description + chart data extraction via Docling (requires engine=docling)
file_pathNoLocal file path to extract content from

Output Schema

NameRequiredDescription
resultYes

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden; it usefully discloses the auth profile (no key needed for most sources, key needed for audio/video). It is silent on other behaviors: what happens if neither url nor file_path is supplied (0 required params), and whether unsupported sources error or degrade.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences, front-loaded with the core action, and the auth caveat placed immediately after. No filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be described, and the auth context is a genuine addition. Still missing is guidance on the unusual 0-required-parameter shape (must one of url/file_path be given?) and any routing versus the sibling tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents engine routing, docling requirements, and the flag semantics in detail. The description adds only a loose mapping of source types, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Extract') and resource ('content from a URL or file') plus the supported source types. It does not explicitly differentiate itself from the sibling summarize_content, so an agent must infer that this returns raw content rather than a summary.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives conditional guidance about when an API key is required (audio/video transcription only), which is useful. However, it never says when to choose this over summarize_content, nor does it explain the url-vs-file_path choice, leaving usage largely implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

summarize_contentSummarize ContentB

Summarize content using LLM with optional context. Requires OPENAI_API_KEY (or another LLM provider key) to be configured.

NameRequiredDescriptionDefault
contentYesThe text content to summarize
contextNoOptional context to guide summarization (e.g., "summarize as bullet points")

Output Schema

NameRequiredDescription
resultYes

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It usefully discloses a real prerequisite — an LLM provider key such as OPENAI_API_KEY must be configured — which is not in the schema. However it says nothing about cost, latency, or truncation behavior on long input, leaving gaps for a tool that calls an external LLM.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two compact sentences with no filler, and the core purpose is front-loaded ahead of the setup requirement. Slightly terse, but every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Output schema exists so return values need no explanation, and both parameters are schema-documented, making the definition largely adequate for a simple tool. The gaps are the absence of any routing guidance versus extract_content and the lack of cost/latency context for an external LLM call.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both parameters are already fully documented, and the description adds nothing beyond restating that context is optional. Baseline 3 applies when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('summarize content') plus the mechanism ('using LLM'), which distinguishes it from the sibling extract_content at a high level. It does not explicitly name or contrast with that sibling, so it falls short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to choose this tool over extract_content, nor any exclusions or preconditions for use beyond the API-key note. The reader must infer that summarizing is the use case from the name alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 2 tool updatesv2.1.1
    • Changedextract_content6 fields changed
      • addedInput schema / properties / engine / description
        Added value: +"Optional extraction engine override, routed by input type.\nWith `url`: auto, simple, firecrawl, jina, crawl4ai.\nWith `file_path`: auto, simple, docling — `docling` requires\n`pip install \"content-core[docling]\"` and fails with a\nconfiguration error when the extra is missing, in which case use\n`auto` or `simple`.\nAny other value is rejected with an error naming the accepted ones."
      • addedInput schema / properties / file_path / description
        Added value: +"Local file path to extract content from"
      • addedInput schema / properties / formulas / description
        Added value: +"Enable formula extraction via Docling (requires engine=docling)"
      • addedInput schema / properties / no_ocr / description
        Added value: +"Disable OCR in Docling (requires engine=docling)"
      • addedInput schema / properties / pictures / description
        Added value: +"Enable image description + chart data extraction via Docling (requires engine=docling)"
      • addedInput schema / properties / url / description
        Added value: +"URL to extract content from (web page, YouTube video, PDF link, etc.)"
    • Changedsummarize_content2 fields changed
      • addedInput schema / properties / content / description
        Added value: +"The text content to summarize"
      • addedInput schema / properties / context / description
        Added value: +"Optional context to guide summarization (e.g., \"summarize as bullet points\")"
  2. 2 tool updatesv2.0.4
    • Changedextract_content9 fields changed
      • addedInput schema / additionalProperties
        Added value: +false
      • addedInput schema / properties / engine
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null
        +}
      • addedInput schema / properties / formulas
        Added value: +{
        +  "default": false,
        +  "type": "boolean"
        +}
      • addedInput schema / properties / no_ocr
        Added value: +{
        +  "default": false,
        +  "type": "boolean"
        +}
      • addedInput schema / properties / pictures
        Added value: +{
        +  "default": false,
        +  "type": "boolean"
        +}
      • removedOutput schema / additionalProperties
        Removed value: -true
      • addedOutput schema / properties
        Added value: +{
        +  "result": {
        +    "type": "string"
        +  }
        +}
      • addedOutput schema / required
        Added value: +[
        +  "result"
        +]
      • addedOutput schema / x-fastmcp-wrap-result
        Added value: +true
    • Addedsummarize_content
  3. 1 tool updatev1.0.0
    • Changedextract_content2 fields changed
      • removedInput schema / properties / file_path / title
        Removed value: -"File Path"
      • removedInput schema / properties / url / title
        Removed value: -"Url"
  4. 1 tool update
    • First observedextract_content

TDQS

A3.6/5.0

Scored across 2 tools

Disambiguation5/5

The two tools target clearly distinct actions: extract_content pulls raw content from a source, while summarize_content condenses it via an LLM. An agent can easily tell which to call based on whether it needs raw text or a summary.

Naming Consistency5/5

Both names follow the same verb_noun snake_case pattern (extract_content, summarize_content), with no deviations in convention or style.

Tool Count3/5

Two tools is on the thin side for even a focused server, though each earns its place in a natural extract-then-summarize pipeline. A couple of supporting operations (e.g. format conversion or output saving) would round out the surface.

Completeness4/5

The extract and summarize pair covers the core content-processing lifecycle, including multiple input sources (URLs, PDFs, YouTube). Minor gaps exist around batch processing, output formatting, or chunking, but agents can work around these.

Maintenance

ActivityNo data
ResponsivenessSyncing

Related MCP Connectors

Related MCP Servers