Skip to main content
Glama
V2-Digital

V2.ai Insights Scraper MCP

by V2-Digital

V2.ai Insights Scraper MCP

A Model Context Protocol (MCP) server that scrapes blog posts from V2.ai Insights, extracts content, and provides AI-powered summaries using OpenAI's GPT-4. Currently supports Contentful CMS integration with search capabilities.

📋 Strategic Vision: This project is evolving into a comprehensive AI intelligence platform. See STRATEGIC_VISION.md for the complete roadmap from content API to strategic intelligence platform.

Features

  • 🔍 Multi-Source Content: Fetches from Contentful CMS and V2.ai web scraping

  • 📝 Content Extraction: Extracts title, date, author, and content with intelligent fallbacks

  • 🔎 Full-Text Search: Search across all blog content with Contentful's search API

  • 🤖 AI Summarization: Generates summaries using OpenAI GPT-4

  • 🔧 MCP Integration: Exposes tools for Claude Desktop integration

Related MCP server: MCP Web Tools Server

Tools Available

  • get_latest_posts() - Retrieves blog posts with metadata (Contentful + V2.ai fallback)

  • get_contentful_posts(limit) - Fetch posts directly from Contentful CMS

  • search_blogs(query, limit) - NEW - Search across all blog content

  • summarize_post(index) - Returns AI-generated summary of a specific post

  • get_post_content(index) - Returns full content of a specific post

Setup

Prerequisites

  • Python 3.12+

  • uv package manager

  • OpenAI API key

  • Contentful CMS credentials (optional, for enhanced functionality)

Installation

  1. Clone and navigate to project:

    cd v2-ai-mcp
  2. Install dependencies:

    uv add fastmcp beautifulsoup4 requests openai
  3. Set up environment variables:

    Create a .env file based on .env.example:

    cp .env.example .env

    Edit .env with your credentials:

    # Required
    OPENAI_API_KEY=your-openai-api-key-here
    
    # Optional (for Contentful integration)
    CONTENTFUL_SPACE_ID=your-contentful-space-id
    CONTENTFUL_ACCESS_TOKEN=your-contentful-access-token
    CONTENTFUL_CONTENT_TYPE=pageBlogPost

Running the Server

uv run python -m src.v2_ai_mcp.main

The server will start and be available for MCP connections.

Testing the Scraper

Test individual components:

# Test scraper
uv run python -c "from src.v2_ai_mcp.scraper import fetch_blog_posts; print(fetch_blog_posts()[0]['title'])"

# Test with summarizer (requires OpenAI API key)
uv run python -c "from src.v2_ai_mcp.scraper import fetch_blog_posts; from src.v2_ai_mcp.summarizer import summarize; post = fetch_blog_posts()[0]; print(summarize(post['content'][:1000]))"

# Run unit tests
uv run pytest tests/ -v --cov=src

Claude Desktop Integration

Configuration

  1. Install Claude Desktop (if not already installed)

  2. Configure MCP in Claude Desktop:

    Add to your Claude Desktop MCP configuration:

    {
      "mcpServers": {
        "v2-insights-scraper": {
          "command": "/path/to/uv",
          "args": ["run", "--directory", "/path/to/your/v2-ai-mcp", "python", "-m", "src.v2_ai_mcp.main"],
          "env": {
            "OPENAI_API_KEY": "your-api-key-here",
            "CONTENTFUL_SPACE_ID": "your-contentful-space-id",
            "CONTENTFUL_ACCESS_TOKEN": "your-contentful-access-token",
            "CONTENTFUL_CONTENT_TYPE": "pageBlogPost"
          }
        }
      }
    }
  3. Restart Claude Desktop to load the MCP server

Using the Tools

Once configured, you can use these tools in Claude Desktop:

  • Get latest posts: get_latest_posts() (intelligent Contentful + V2.ai fallback)

  • Get Contentful posts: get_contentful_posts(10) (direct CMS access)

  • Search blogs: search_blogs("AI automation", 5) (NEW - full-text search)

  • Summarize post: summarize_post(0) (index 0 for first post)

  • Get full content: get_post_content(0)

Example Usage

🔍 Search for AI-related content:
search_blogs("artificial intelligence", 3)

📚 Get latest posts with automatic source selection:
get_latest_posts()

🤖 Get AI summary of specific post:
summarize_post(0)

Project Structure

v2-ai-mcp/
├── src/
│   └── v2_ai_mcp/
│       ├── __init__.py      # Package initialization
│       ├── main.py          # FastMCP server with tool definitions
│       ├── scraper.py       # Web scraping logic
│       └── summarizer.py    # OpenAI GPT-4 integration
├── tests/
│   ├── __init__.py          # Test package initialization
│   ├── test_scraper.py      # Unit tests for scraper
│   └── test_summarizer.py   # Unit tests for summarizer
├── .github/
│   └── workflows/
│       └── ci.yml           # GitHub Actions CI/CD pipeline
├── pyproject.toml           # Project dependencies and config
├── .env.example             # Environment variables template
├── .gitignore               # Git ignore patterns
└── README.md                # This file

Current Implementation

The scraper currently targets this specific blog post:

  • URL: https://www.v2.ai/insights/adopting-AI-assistants-while-balancing-risks

Extracted Data

  • Title: "Adopting AI Assistants while Balancing Risks"

  • Author: "Ashley Rodan"

  • Date: "July 3, 2025"

  • Content: ~12,785 characters of main content

Development

Adding More Blog Posts

To scrape multiple posts or different URLs, modify the fetch_blog_posts() function in scraper.py:

def fetch_blog_posts() -> list:
    urls = [
        "https://www.v2.ai/insights/post1",
        "https://www.v2.ai/insights/post2",
        # Add more URLs
    ]
    return [fetch_blog_post(url) for url in urls]

Improving Content Extraction

The scraper uses multiple fallback strategies for extracting content. You can enhance it by:

  1. Inspecting V2.ai's HTML structure

  2. Adding more specific CSS selectors

  3. Improving date/author extraction patterns

Troubleshooting

Common Issues

  1. OpenAI API Key Error: Ensure your API key is set in environment variables

  2. Import Errors: Run uv sync to ensure all dependencies are installed

  3. Scraping Issues: Check if the target URL is accessible and the HTML structure hasn't changed

Testing Components

# Test scraper only
uv run python -c "from src.v2_ai_mcp.scraper import fetch_blog_posts; posts = fetch_blog_posts(); print(f'Found {len(posts)} posts')"

# Run full test suite
uv run pytest tests/ -v --cov=src

# Test MCP server startup
uv run python -m src.v2_ai_mcp.main

Development

Running Tests

# Run all tests
uv run pytest

# Run with coverage
uv run pytest --cov=src --cov-report=html

# Run specific test file
uv run pytest tests/test_scraper.py -v

Code Quality

# Format code
uv run ruff format src tests

# Lint code
uv run ruff check src tests

# Fix auto-fixable issues
uv run ruff check --fix src tests

License

This project is for educational and development purposes.

Available Tools

5 tools
get_contentful_postsC

Fetch posts directly from Contentful CMS (if configured)

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo

TDQS

C2.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions fetching posts and a configuration prerequisite, but does not cover critical aspects such as authentication needs, rate limits, error handling, or what the output looks like (e.g., post format, pagination). This leaves significant gaps for an agent to understand how to invoke it safely and effectively.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is front-loaded with the main action ('Fetch posts'), though the conditional 'if configured' adds minor complexity. Overall, it is appropriately sized for a simple tool but could be more informative.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has no annotations, no output schema, and low parameter coverage, the description is incomplete. It does not explain the return values, error conditions, or behavioral details needed for an agent to use it correctly. The conditional 'if configured' is vague, and without sibling differentiation, the context is insufficient for reliable tool selection.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has one parameter ('limit') with 0% description coverage, and the tool description does not mention any parameters or their semantics. This fails to compensate for the lack of schema documentation, leaving the agent without guidance on what 'limit' controls (e.g., number of posts, pagination) or how it interacts with the fetching behavior.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states the tool 'fetch[es] posts directly from Contentful CMS', which provides a clear verb ('fetch') and resource ('posts from Contentful CMS'). However, it does not differentiate from sibling tools like 'get_latest_posts' or 'search_blogs', leaving ambiguity about its specific scope or filtering capabilities beyond the basic action.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description includes a conditional clause 'if configured', which hints at a prerequisite but does not explain what configuration is needed or when to use this tool versus alternatives like 'get_latest_posts' or 'search_blogs'. There is no explicit guidance on when to choose this tool over siblings, making usage unclear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_latest_postsB

Retrieves the latest blog posts with metadata

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It states it 'retrieves' posts, implying a read-only operation, but doesn't specify aspects like rate limits, authentication needs, pagination, or what 'latest' means (e.g., time frame, sorting). This leaves significant gaps for a tool with no structured safety hints.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with no wasted words. It front-loads the key action and resource, making it easy to parse quickly, which is ideal for a simple tool with no parameters.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 0 parameters and no output schema, the description is minimally adequate but incomplete. It lacks details on behavioral traits (e.g., how 'latest' is defined, metadata structure) and doesn't differentiate from siblings, which could confuse an agent in a server with multiple post-related tools.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description adds value by specifying 'latest blog posts with metadata', which clarifies the scope and output content beyond what the empty schema provides, earning a baseline score above 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Retrieves') and resource ('latest blog posts with metadata'), making the purpose understandable. However, it doesn't differentiate from sibling tools like 'get_contentful_posts' or 'search_blogs', which likely retrieve similar content but with different scopes or filters.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like 'search_blogs' or 'get_contentful_posts'. It implies usage for retrieving latest posts but doesn't specify contexts, exclusions, or prerequisites, leaving the agent to guess based on tool names alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_post_contentC

Returns the full content of the blog post at the specified index

ParametersJSON Schema
NameRequiredDescriptionDefault
indexYes

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool returns content, implying a read-only operation, but does not cover aspects like error handling (e.g., invalid index), performance, or output format. This is a significant gap for a tool with zero annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with zero waste. It is front-loaded with the core purpose and includes necessary details without redundancy, making it appropriately sized and easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has no annotations, no output schema, and low schema coverage, the description is incomplete. It lacks details on return values, error cases, and behavioral traits, which are crucial for a read operation. The description does not adequately compensate for the missing structured data.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 0%, so the description must compensate. It adds meaning by specifying that 'index' refers to a blog post index, which is useful beyond the schema's basic type. However, it does not explain index range, format, or examples, leaving some ambiguity, so it partially compensates but not fully.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Returns') and resource ('full content of the blog post'), specifying what the tool does. However, it does not explicitly differentiate from siblings like 'get_contentful_posts' or 'get_latest_posts', which may also retrieve post content, so it lacks sibling distinction for a perfect score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like 'search_blogs' or 'summarize_post'. It mentions 'at the specified index', which implies usage for a known post index, but does not clarify context, exclusions, or prerequisites, leaving the agent without explicit usage instructions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_blogsC

Search blog posts across all content using text query. Searches titles, content, authors, and other fields.

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYes
limitNo

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool searches across multiple fields but does not disclose critical traits like whether it's read-only, how results are ordered, if there's pagination, rate limits, or error handling. For a search tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately sized with two concise sentences that directly address the tool's function. It is front-loaded with the core purpose and avoids unnecessary details. However, it could be slightly more structured by explicitly separating scope from field details.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (search functionality with 2 parameters), lack of annotations, and no output schema, the description is incomplete. It does not explain what the tool returns (e.g., list of posts, metadata), how results are formatted, or any limitations. This leaves the agent with insufficient context to use the tool effectively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description adds minimal meaning beyond the input schema. It implies the 'query' parameter searches text across fields like titles and content, but with 0% schema description coverage, the schema provides no details on parameter semantics. The description does not explain the 'limit' parameter or provide syntax examples, so it only partially compensates for the coverage gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Search blog posts across all content using text query' specifies the verb (search), resource (blog posts), and scope (across all content). It distinguishes from siblings like 'get_contentful_posts' or 'get_latest_posts' by emphasizing search functionality rather than retrieval by specific criteria.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It mentions searching 'titles, content, authors, and other fields' but does not specify when this is preferable over siblings like 'get_latest_posts' for recent content or 'summarize_post' for summaries. No exclusions or prerequisites are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

summarize_postC

Returns a summary of the blog post at the specified index

ParametersJSON Schema
NameRequiredDescriptionDefault
indexYes

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It states this is a read operation ('Returns a summary'), which implies it's non-destructive, but doesn't cover aspects like error handling (e.g., what happens if the index is invalid), performance characteristics, or authentication needs. This leaves significant gaps for an agent to understand how to use it effectively.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, direct sentence with zero wasted words. It front-loads the core purpose ('Returns a summary') and efficiently specifies the resource and parameter, making it easy to parse quickly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity of summarizing content, lack of annotations, and no output schema, the description is incomplete. It doesn't explain what the summary includes (e.g., length, format), potential limitations, or how it interacts with sibling tools. For a tool that likely involves processing blog posts, more context is needed to guide an agent effectively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0% description coverage, so the description must compensate. It adds meaning by specifying that the 'index' parameter refers to a blog post index, which clarifies beyond the schema's generic 'integer' type. However, it doesn't provide details like valid index ranges or how indices are assigned, leaving some ambiguity. Given the single parameter and low schema coverage, this is adequate but not comprehensive.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose with a specific verb ('Returns a summary') and resource ('blog post at the specified index'), making it immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'get_post_content' or 'search_blogs', which might also retrieve post information, so it falls short of a perfect score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives such as 'get_post_content' or 'search_blogs'. It mentions a specific index parameter but doesn't explain prerequisites like needing to know the index beforehand or how it relates to other tools that might list posts.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 5 tool updatesv0.1.0
    • First observedget_contentful_posts
    • First observedget_latest_posts
    • First observedget_post_content
    • First observedsearch_blogs
    • First observedsummarize_post

TDQS

B3.2/5.0

Scored across 5 tools

Disambiguation4/5

Most tools have distinct purposes, but there is some potential overlap between get_latest_posts and get_contentful_posts, as both retrieve posts, though the latter is conditional on Contentful configuration. The other tools (get_post_content, search_blogs, summarize_post) are clearly differentiated by their specific actions on blog content.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern with snake_case, such as get_contentful_posts, get_latest_posts, get_post_content, search_blogs, and summarize_post. This uniformity makes the tool set predictable and easy to understand.

Tool Count5/5

With 5 tools, the server is well-scoped for its purpose of scraping and analyzing blog content. Each tool serves a specific function in the workflow, from fetching and searching to summarizing posts, without being overly sparse or bloated.

Completeness4/5

The tool set covers core operations for blog content retrieval and analysis, including fetching, searching, and summarizing. However, there is a minor gap in update or delete operations, which might be outside the scraper's scope, but could limit full lifecycle management if needed.

Maintenance

ActivityInactive
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    D
    maintenance
    A production-ready Model Context Protocol server that enables language models to leverage AI-powered web scraping capabilities, offering tools for transforming webpages to markdown, extracting structured data, and executing AI-powered web searches.
    8
    120 PyPI
    110
    MIT
  • A
    license
    B
    quality
    D
    maintenance
    A Model Context Protocol server that intelligently fetches and processes web content, transforming websites and documentation into clean, structured markdown with nested URL crawling capabilities.
    2
    6 npm
    9
    MIT
  • A
    license
    B
    quality
    D
    maintenance
    A comprehensive Model Context Protocol server for content summarization that supports web scraping, file reading, content summarization, and topic-based summarization features.
    7
    11
    Apache 2.0