Skip to main content
Glama
hakohli

mcp-streaming-msk-bedrock

by hakohli
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Hybrid

MCP Streaming Demo — Amazon MSK + Bedrock Agents

A working demo of Streaming MCP (Model Context Protocol) on AWS — pushing real-time context to a Bedrock-powered AI agent using Amazon MSK, WebSockets, and IAM authorization.

Based on mcp-streaming-demo, adapted to use managed AWS services.

Architecture

log_simulator.py → Amazon MSK (app-logs) → streaming_mcp_server.py → bedrock_agent.py
                                                   ↕ WebSocket              ↕
                                          subscribe / get_anomalies    Bedrock Claude
                                                                      (invoke_model)

Lambda (Action Group) ← Bedrock Agent (managed) → MSK

Related MCP server: CloudWatch Log Analyst MCP

Components

File

Description

src/streaming_mcp_server.py

Core server — MSK consumer (IAM auth) + WebSocket push

src/bedrock_agent.py

AI agent — subscribes to stream, invokes Bedrock for analysis

src/log_simulator.py

Produces realistic log events to MSK

src/mock_kafka.py

Patches for local dev without IAM auth

lambda/handler.py

Bedrock Agent Action Group Lambda

lambda/api_schema.json

OpenAPI schema for action group

infra/template.yaml

CloudFormation — MSK Serverless + Bedrock Agent + Lambda

docker-compose.yml

Local Kafka for development

Prerequisites

  • Python 3.9+

  • AWS account with Bedrock model access (Claude 3 Sonnet)

  • AWS CLI configured with appropriate credentials

  • Docker (for local development)

Quick Start (Local Dev)

# 1. Start local Kafka
docker compose up -d

# 2. Install dependencies
pip install -r requirements.txt

# 3. Start server (local mode)
MSK_BOOTSTRAP=localhost:9092 python src/streaming_mcp_server.py

# 4. Start log simulator (separate terminal)
MSK_BOOTSTRAP=localhost:9092 python src/log_simulator.py

# 5. Start Bedrock agent (separate terminal)
python src/bedrock_agent.py

Deploy to AWS

# 1. Deploy infrastructure
aws cloudformation deploy \
  --template-file infra/template.yaml \
  --stack-name mcp-streaming-demo \
  --parameter-overrides VpcId=vpc-xxx SubnetIds=subnet-aaa,subnet-bbb \
  --capabilities CAPABILITY_NAMED_IAM

# 2. Get MSK bootstrap servers
aws kafka get-bootstrap-brokers --cluster-arn <ClusterArn from outputs>

# 3. Store bootstrap in SSM (used by Lambda)
aws ssm put-parameter --name /mcp-demo/msk-bootstrap --value "<bootstrap-servers>" --type String

# 4. Run server and simulator with MSK endpoint
export MSK_BOOTSTRAP="<bootstrap-servers>"
python src/streaming_mcp_server.py
python src/log_simulator.py
python src/bedrock_agent.py

Key Differences from Original

Original

This Version

Local Kafka (Docker)

Amazon MSK Serverless

Custom Python agent

Bedrock Agent + direct invoke_model client

No auth

IAM OAUTHBEARER (MSK) + IAM roles (Bedrock)

Manual remediation mapping

Claude-powered root cause analysis

docker-compose only

CloudFormation IaC

What You'll See

  1. Normal log traffic flowing through the agent in real-time

  2. Every ~30 seconds, an error spike hits

  3. The agent detects the spike (>30% error rate in sliding window)

  4. Bedrock Claude analyzes the errors and provides root cause + remediation

Teardown

# Local
docker compose down -v

# AWS
aws cloudformation delete-stack --stack-name mcp-streaming-demo

This server cannot be deployed

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Maintenance

ActivityInactive
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • KafkaIQ

    ojhaayush03
    F
    license
    Not graded
    quality
    D
    maintenance
    Enables AI assistants to manage and monitor Apache Kafka clusters through natural language, providing real-time operations, health monitoring, consumer lag analysis, and temporal trend detection for intelligent cluster management.
    -

  • AWS_CloudGuardMCP

    madhurprash
    F
    license
    Not graded
    quality
    D
    maintenance
    An intelligent AWS monitoring and incident response solution using Anthropic's Model Context Protocol (MCP). Enables users to monitor AWS resources, analyze CloudWatch logs and metrics, and automatically create Jira tickets with remediation steps.
    3
    -

  • mcp-kafka-observer

    Rushi264
    A
    license
    Not graded
    quality
    D
    maintenance
    An MCP server that gives AI agents real-time observability into Apache Kafka clusters, enabling natural language queries for broker health, consumer lag, and diagnostics.
    MIT