DataFlow MCP Server
Provides CRUD operations, filtering, pagination, sorting, health checks, and secure MongoDB integration for document management.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@DataFlow MCP Server show me the first 10 active users"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
DataFlow MCP Server - Production Grade
A secure, production-ready Model Context Protocol (MCP) server with MongoDB integration, featuring comprehensive security controls, CRUD operations, logging, and monitoring.
๐ Features
Security
โ Input Validation & Sanitization - Prevents NoSQL injection attacks
โ MongoDB SSL/TLS Support - Secure cloud deployments
โ Rate Limiting - Protects against abuse (100 req/min default)
โ Connection Pooling - Optimized for performance
โ Document Size Limits - Prevents resource exhaustion
โ Field Name Validation - Blacklists dangerous operators
Operations
โ CRUD Operations - Create, Read, Update, Delete documents
โ Filtering & Pagination - Flexible data retrieval with limits
โ Sorting Support - Sort by any field (ascending/descending)
โ Bulk Operations Ready - Extensible architecture
Monitoring & Observability
โ Comprehensive Logging - File & console with rotation
โ Health Checks - Service health status endpoint
โ Metrics Tracking - Request counts, success rates
โ Error Handling - Detailed error reporting
Production Ready
โ Security First - SSL/TLS support, input validation
โ Environment Config - 12-factor app ready
โ Graceful Shutdown - Proper resource cleanup
Related MCP server: Kroki MCP
๐ Quick Start
Prerequisites
Python 3.12+
Docker & Docker Compose (optional)
MongoDB (or use Docker Compose)
Local Development
Clone and setup:
cd dataflow_mcp
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -e .Configure environment:
cp .env.example .env
# Edit .env with your MongoDB connectionRun the server:
python main.py๐ก API Tools
Health Check
Get server status and metrics.
{
"status": "healthy",
"uptime_seconds": 123.45,
"metrics": {
"total_requests": 42,
"successful_requests": 40,
"failed_requests": 2,
"success_rate": 95.24
}
}Read Collection
Retrieve documents with filtering, pagination, and sorting.
Parameters:
collection_name(required): Collection namefilter_query: JSON string with MongoDB filterlimit: Max documents (default: 100, max: 1000)skip: Skip N documents (default: 0)sort_by: Field to sort by
Example:
{
"collection_name": "users",
"filter_query": "{\"status\": \"active\"}",
"limit": 10,
"skip": 0,
"sort_by": "created_at"
}Get Document
Retrieve a single document by ID.
Parameters:
collection_name: Collection namedocument_id: MongoDB ObjectId as string
Create Document
Create a new document in a collection.
Parameters:
collection_name: Collection namedocument_json: JSON string representing the document
Example:
{
"collection_name": "users",
"document_json": "{\"name\": \"John\", \"email\": \"john@example.com\", \"status\": \"active\"}"
}Update Document
Update an existing document.
Parameters:
collection_name: Collection namedocument_id: MongoDB ObjectId as stringupdate_json: JSON with fields to update
Example:
{
"collection_name": "users",
"document_id": "65f8a1b2c3d4e5f6g7h8i9j0",
"update_json": "{\"status\": \"inactive\", \"updated_at\": \"2024-01-01T12:00:00Z\"}"
}Delete Document
Delete a document from a collection.
Parameters:
collection_name: Collection namedocument_id: MongoDB ObjectId as string
๐ Security Features
Input Validation
Collection names: Alphanumeric, dash, underscore only
Field names: Prevents dangerous operators ($where, $function, etc.)
Filters: Maximum 10KB, blacklist dangerous operations
Documents: Maximum 1MB, enforced size limits
MongoDB Security
Connection Options:
Connection pooling (default: 10 connections)
Retry writes enabled
Write concern: majority
Journaling enabled
SSL/TLS for cloud deployments
Environment Variables:
MONGO_USE_TLS=true MONGO_CA_CERT_PATH=/path/to/ca.pem MONGO_ALLOW_INVALID_CERTS=false
Rate Limiting
100 requests per 60 seconds (configurable)
Per-client tracking
Returns clear error on limit exceeded
Error Handling
Safe error messages (no sensitive data leaks)
Detailed internal logging
Graceful degradation
๐ Environment Variables
Required
MONGO_URI=mongodb://user:password@host:port/database
MONGO_DB_NAME=dataflowOptional (with defaults)
MONGO_TIMEOUT=5000 # Connection timeout (ms)
MONGO_POOL_SIZE=10 # Connection pool size
MONGO_MAX_IDLE_TIME=45000 # Max idle time (ms)
MONGO_USE_TLS=false # Enable TLS
MONGO_CA_CERT_PATH= # CA certificate path
LOGS_DIR=./logs # Log directory
LOG_LEVEL=INFO # Logging level๐ Project Structure
dataflow_mcp/
โโโ core.py # FastMCP instance, rate limiter, metrics, prompt loading, normalization
โโโ server.py # tool registration + mcp.run()
โโโ tools/
โ โโโ health.py # health_check, database_status
โ โโโ crud.py # generic MongoDB CRUD tools
โ โโโ images.py # contest banner pipeline (missing/broken images, cover prompts)
โ โโโ migration.py # v4.0 schema migration/backfill tools
โ โโโ contests.py # structuring + full generation + detail generation
โ โโโ events.py # events pipeline (fetch โ structure โ submit โ query)
โ โโโ raw_data.py # raw scraped data bridge + overview
โ โโโ validation.py # chatbot-driven web validation pipeline
โ โโโ audit.py # duplicate audit + discrepancy flagging
config/
โโโ mongodb.py # MongoDB connection with pooling
โโโ security.py # Validation and rate limiting
โโโ logging_config.py # Logging setup
tools/ # service layer (DataManager, generators, dedup gate, validators)
prompts/ # prompt files (descriptive names + Prompts*.txt aliases)
main.py # thin entry point โ dataflow_mcp.server
โโโ pyproject.toml # Dependencies and config (console script: dataflow-mcp)
โโโ .env.example # Environment template๐ช Events Pipeline
The MCP server includes a full events pipeline so AI chatbots can harvest and structure participatory events (conferences, summits, workshops, webinars, meetups, trainings, โฆ):
1. get_records_for_events(source=..., limit=10) โ raw URLs + events-v1.1 prompt
2. [chatbot researches each URL and outputs event JSON]
3. submit_structured_events(events_json) โ persists to the Events collection
4. get_events(event_type=..., upcoming_only=true) โ read structured events back
5. get_events_overview() โ counts by type/status
6. get_events_for_detail_generation(batch_size=10) โ events + event-details-v1.0.txt prompt
7. [chatbot researches and writes event details]
8. submit_event_details(event_id, details_json) โ versioned event_details saved
9. get_event_detail_status() โ coverage metrics (remaining events to generate)Event detail pages mirror the contest detail flow: EventDetailGenerator
(tools/event_detail_generator.py) provides the priority queue, quality
validation, and versioned event_details storage.
Prompt files were renamed to descriptive names (contest-structuring-v4.0.txt,
event-structuring-v1.1.txt, โฆ) with the old Prompts*.txt names kept as
aliases. See TOOLS_REFERENCE.md for the full tool reference.
๐ง Configuration for Cloud Deployment
AWS Deployment
MONGO_URI=mongodb+srv://user:password@cluster.mongodb.net/dataflow
MONGO_USE_TLS=true
MONGO_ALLOW_INVALID_CERTS=falseAzure Deployment
MONGO_URI=mongodb://user:password@host.mongo.cosmos.azure.com:10255/database
MONGO_USE_TLS=true
MONGO_CA_CERT_PATH=/etc/ssl/certs/ca-certificates.crtGCP Deployment
MONGO_URI=mongodb://user:password@instance:27017/database
MONGO_USE_TLS=true๐จ Production Checklist
MongoDB backups configured
SSL/TLS certificates installed
Environment variables set securely (not in code)
Logs redirected to centralized logging
Health checks configured in load balancer
Rate limits adjusted for your use case
MongoDB indexes optimized
Connection pool size tuned
Monitoring/alerting setup
Graceful shutdown tested
๐ Performance Optimization
MongoDB Indexes
Pre-created indexes in scripts/mongo-init.js:
User email: unique constraint
Timestamps: for sorting and TTL
Status: for filtering
Connection Pooling
Default pool size: 10 (adjust via
MONGO_POOL_SIZE)Min connections: 2 (automatically maintained)
Max idle time: 45 seconds
Request Limits
Max filter size: 10KB
Max document size: 1MB
Max page size: 1000 documents
Rate limit: 100 req/min
๐งช Testing & Development
Install dev dependencies:
pip install -e ".[dev]"Run tests:
pytest --cov=tools --cov=configCode formatting:
black .
flake8 .
mypy .๐ Logging
Logs are written to:
File:
./logs/mcp_server_YYYYMMDD.log(rotated daily, max 10MB)Console: Real-time output
Log levels:
DEBUG- Detailed diagnostic infoINFO- General eventsWARNING- Warning messagesERROR- Error events
๐ Troubleshooting
MongoDB Connection Failed
Check MONGO_URI and credentials
Verify MongoDB is running: mongosh "mongodb://..."
Check network connectivity and firewallRate Limit Exceeded
Default: 100 requests per 60 seconds
Increase MONGO_POOL_SIZE and optimize queries
Implement request queuing on clientHigh Memory Usage
Reduce MONGO_POOL_SIZE
Lower MONGO_MAX_IDLE_TIME
Check for large result sets (use pagination)๐ References
๐ License
MIT License - See LICENSE file for details
๐ค Support
For issues and questions:
Check troubleshooting section
Review logs in
./logs/Check MongoDB connection
Verify environment variables
Built for production-grade data operations with security-first design.
dataflow_mcp
Available Tools
41 toolsapply_migration_patchA
Apply a validated normalized patch to update a single contest.
All patches go through 4 validations before writing:
Field whitelist โ only allowed fields may be patched
Schema compliance โ types, enums, formats checked
Destructive write protection โ populated fields not overwritten with null
Cross-field consistency โ no contradictory values
Inputs: contest_id (required), patch_json (required), force. Returns: object
bulk_apply_migrationsA
Apply multiple migration patches in one batch (with validation).
All patches go through 4 validations before writing.
Inputs: migrations_json (required), force. Returns: object
create_documentB
Create a new document in a collection.
Inputs: document_json (required), collection_name (required). Returns: object
database_statusA
Check MongoDB connectivity separately from server health.
Returns: Database connection status and any connection error message
Inputs: no named parameters. Returns: object
delete_documentB
Delete a document from a collection.
Inputs: document_id (required), collection_name (required). Returns: object
find_duplicate_contestsA
Read-only audit: find groups of LIVE contests that share the same normalized title (case/punctuation-insensitive, word-order-insensitive).
These are the records the duplicate-title gate would block on ingestion. Same-source exact-title duplicates are also shown here (they only update in place during ingestion, but if two records share a source + title the older one is effectively shadowed).
Inputs: min_live. Returns: object
flag_contest_discrepancyA
Flag a factual discrepancy found in CONTEST DATA during AI research.
When a chatbot discovers a concrete, verifiable error in the Contests collection while doing research (e.g. the prize on the official page differs from what's stored), it can call this tool to save the finding to the flagged_discrepancies collection for human review.
This tool does NOT modify the Contests collection โ it only records the finding. A human should review and resolve via the appropriate pipeline (apply_migration_patch, etc.).
Inputs: contest_id (required), discrepancies_json (required), source, flagged_by. Returns: object
generate_cover_prompt_for_contestA
Generate a premium image-generation prompt for one contest by ID.
This returns a single prompt string that can be fed into any image model when the original contest banner is missing.
Inputs: contest_id (required). Returns: object
get_contests_for_detail_generationA
Return contests needing AI-generated detail pages, sorted by priority.
The response includes both the prompt text (contest-details-v1.0.txt) and the contest documents. Send both to Mistral so it can research and generate structured contest details.
Priority order: trending > open > high view velocity > recently added > prize value.
Inputs: skip, batch_size. Returns: object
get_contests_for_migrationA
Get a batch of existing contests that need migration to v4.0 schema.
Returns contests missing key fields like category, prizeSummary, or feeConfidence. Use pagination to process in batches.
Inputs: skip, batch_size. Returns: object
get_contests_missing_imagesA
Fetch contests where the primary image URL is missing or empty.
Use this to identify documents that need AI-generated replacement banners.
Inputs: skip, batch_size. Returns: object
get_contests_with_broken_imagesA
Fetch contests whose image.primary.status is marked as 'broken'.
Returns the contests so the chatbot can display or re-generate banners.
Inputs: skip, batch_size. Returns: object
get_documentB
Get a single document by ID.
Inputs: document_id (required), collection_name (required). Returns: object
get_event_detail_statusA
Get coverage metrics for the event detail generation pipeline.
Surfaces EventDetailGenerator.get_status: how many live events exist in the Events collection, how many already have event_details documents (broken down by detail status), how many still need generation, and the overall coverage percentage.
Use this to see how much event-detail work remains before deciding how many batches of get_events_for_detail_generation to run.
Returns: Dictionary with total_events, total_with_details, total_without_details, by_status, and coverage_pct.
Inputs: no named parameters. Returns: object
get_eventsA
Read structured events from the Events collection with filters.
Use this to query events harvested by the pipeline โ e.g. all upcoming conferences, or everything in draft status awaiting review.
Inputs: skip, limit, status, event_type, +1 more. Returns: object
get_events_for_detail_generationA
Return events needing AI-generated detail pages, sorted by priority.
The response includes both the prompt text (event-details-v1.0.txt) and the event documents. Send both to the LLM so it can research and generate structured event details (whyAttend, whoShouldAttend, benefits, tips, agenda highlights, FAQ, SEO).
Priority order: upcoming > published > registration open > has speakers/ agenda > recently added.
Inputs: skip, batch_size. Returns: object
get_events_overviewA
Get a quick overview of the Events collection.
Returns counts by eventType and status, plus the number of upcoming events. Use this to decide what to review or process next.
Returns: Dictionary with overview statistics.
Inputs: no named parameters. Returns: object
get_migration_statusA
Get overall migration progress statistics for the 810 contests.
Shows how many contests have been migrated to v4.0 schema, how many are pending, and what fields are missing.
Returns: Dictionary with migration progress and breakdown by field
Inputs: no named parameters. Returns: object
get_prompted_contestsA
Return the prompt text together with contest documents for AI processing.
Use this when Claude or ChatGPT needs both the instructions and the raw MongoDB contests in a single response so it can normalize them locally.
Inputs: skip, batch_size, prompt_name. Returns: object
get_raw_data_statusA
Return a summary of what raw scraped data is available in CHRawdata.rawdata.
If source is provided (e.g. "contestwatchers", "opportunityDesk"), only
records from that scraper are considered.
Inputs: source. Returns: object
get_records_for_contest_validationA
Claim a batch of unvalidated contest documents and return them with a validation prompt for a chatbot.
Similar to get_records_for_validation but works on the Contests collection. Use this for Stage 2 validation before LLM normalization.
The chatbot uses its OWN web search to verify each contest's key fields (title, deadline, prize, eligibility) against the source page.
Inputs: batch_size, chatbot_id. Returns: object
get_records_for_eventsA
Fetch raw records + the Events prompt (event-structuring-v1.1.txt, events-v1.1 schema) so a chatbot can structure participatory events (conferences, workshops, meetups, webinars, summits, trainings).
Use this when you want to go from raw event URLs/titles to a structured event document in one AI round-trip. The chatbot should:
Read the event prompt_text (events-v1.1 schema and rules)
For each record, pin the target event (name, edition, location, domain) and hunt the official site's subpages (speakers, agenda, pricing, venue)
Extract fields following the events-v1.1 schema
Return ONE event JSON per record via submit_structured_events
Inputs: limit, source, collection_name. Returns: object
get_records_for_full_generationA
Fetch raw scraped records + BOTH prompts (contest-structuring-v4.0.txt + contest-details-v1.0.txt) so a chatbot can structure AND generate contest details in one pass.
Use this when you want to go from raw scraped data to published contest details in a single AI round-trip. The chatbot should:
Read BOTH prompt texts (structuring schema + detail generation rules)
For each record, use its URL (or title) to search the web and find the actual contest page
Extract structured fields following the v4.0 schema
Research and generate contest details following contest-details-v1.0.txt
Return both via submit_full_generation
Inputs: limit, source, require_validated. Returns: object
get_records_for_structuringA
Fetch raw scraped records + the structuring prompt (contest-structuring-v4.0.txt, v4.0 schema) so a chatbot can structure them into the normalized Contests format.
Inputs: limit, source, require_validated. Returns: object
get_records_for_validationA
Claim a batch of unvalidated raw records and return them with a validation prompt for a chatbot.
The chatbot uses its OWN web search capability to verify each record by visiting the source URL or searching the web for the contest title.
Records are atomically marked as 'in_progress' for this chatbot_id, preventing other chatbots from claiming the same records.
Inputs: source (required), limit, chatbot_id. Returns: object
get_scraped_overviewA
Get a quick, actionable overview of what raw scraped records are available.
Use this to see what's in the pipeline before deciding which source to work on. Returns counts by source, validation status breakdown, total records, newest/oldest record dates, and a few sample titles.
This is designed for AI agents (ChatGPT, Mistral, Claude) to quickly understand what data is available and decide what to work on next.
Inputs: source. Returns: object
get_validation_promptA
Preview the validation prompt without claiming any records.
Use this to see the instructions that will be sent to the chatbot before starting the validation workflow.
Inputs: record_type. Returns: object
get_validation_statusA
Get an overview of the validation pipeline status.
Shows how many raw records are pending, in progress, validated, failed, or skipped โ broken down by source if specified.
Use this to monitor progress across multiple chatbots and decide when to run process_raw_data.
Inputs: source. Returns: object
health_checkA
Check the health status of the MCP server process.
Returns: Dictionary with health status and metrics
Inputs: no named parameters. Returns: object
process_raw_dataA
Read validated raw records from CHRawdata.rawdata for a given scraper
source, normalize, deduplicate, and upsert them into the primary
Contests collection (ContestHopperDb).
NOTE: By default, only records with validationStatus="validated" are processed. Run get_records_for_validation + submit_raw_validation first.
Inputs: source (required), limit, auto_image, dedupe_gate. Returns: object
read_collectionB
Read documents from a MongoDB collection with filtering and pagination.
Inputs: collection_name (required), skip, limit, sort_by, +1 more. Returns: object
read_raw_collectionA
Read documents from the CHRawdata database (raw scraped data) with filtering and pagination.
Inputs: collection_name (required), skip, limit, sort_by, +1 more. Returns: object
submit_contest_detailsA
Submit AI-generated contest details (from Mistral) for validation and storage.
The details are validated for quality, then saved to the contest_details collection with automatic versioning.
Inputs: contest_id (required), details_json (required). Returns: object
submit_contest_validationA
Submit validation results from a chatbot for existing contest documents.
Same as submit_raw_validation but updates the Contests collection. Use this for Stage 2 validation before LLM normalization.
Inputs: chatbot_id (required), validation_json (required). Returns: object
submit_event_detailsA
Submit AI-generated event details (from the LLM) for validation and storage.
The details are validated for quality (minimum word count, honest readingTime, no first-person, no hallucinated URLs), then saved to the event_details collection with automatic versioning.
Inputs: event_id (required), details_json (required). Returns: object
submit_full_generationA
Submit a full generation result that includes BOTH structured contest data AND contest details in one call.
Use this after get_records_for_full_generation. The JSON must contain an 'items' array, where each item has:
record: Structured contest data following the v4.0 schema
details: Contest details following contest-details-v1.0.txt schema
Inputs: generation_json (required), dedupe_gate. Returns: object
submit_raw_validationA
Submit validation results from a chatbot for raw scraped records.
The chatbot should have received records via get_records_for_validation,
validated them using its own web search, and returned a JSON response.
This tool processes that JSON and updates each record's validation status
in the database.
Inputs: chatbot_id (required), validation_json (required). Returns: object
submit_structured_eventsA
Submit structured event records (following the event-structuring-v1.1.txt events-v1.1 schema) produced by a chatbot and persist them to the Events collection.
This is the event counterpart of submit_structured_records:
Deduplication key: source.name + title (upsert-in-place on re-submit).
Optional duplicate-title GATE (on by default): a record whose title matches an existing LIVE event โ same normalized title from a different source, or a reworded title from the same source โ is SKIPPED and reported under "duplicates".
events-v1.1 defaults are applied (type="event", status="draft", visibility="public", featured=false, analytics=0) and off-schema enum values are downgraded to null with a warning (see details).
The audit
metadatablock is stripped unless keep_metadata=True.
Inputs: events_json (required), dedupe_gate, keep_metadata. Returns: object
submit_structured_recordsA
Submit structured contest records (following the contest-structuring-v4.0.txt schema) produced by a chatbot. Validates required fields and upserts into the Contests collection.
Deduplication key: source.name + title (same as process_raw_data). Plus an optional duplicate-title GATE (on by default): any record whose title matches an existing LIVE contest โ same normalized title from a different source, or a reworded title from the same source โ is SKIPPED and reported under "duplicates". The same-source exact-title match is the intended update path and still updates in place.
Inputs: records_json (required), dedupe_gate. Returns: object
update_documentC
Update an existing document in a collection.
Inputs: document_id (required), update_json (required), collection_name (required). Returns: object
verify_image_urlsA
Verify image URLs for contests and mark broken images in the database.
This tool scans contests that have an image.primary.url, performs a
lightweight HTTP HEAD/GET to verify reachability, and updates
image.primary.status to 'active' or 'broken'. It returns a report.
Inputs: skip, batch_size, user_agent. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelogโฆ
TDQS
Scored across 41 tools
Many tools follow similar 'get_records_for_X' and 'submit_X' patterns, making it difficult to distinguish between validation, structuring, detail generation, and migration workflows. While descriptions are detailed, the overlapping purposes and subtle differences (e.g., get_records_for_structuring vs get_records_for_full_generation) create ambiguity.
Most tools use a consistent get_/submit_/create_/update_ verb-noun convention, with minor deviations like database_status and health_check. The get_records_for_* and get_contests_for_* patterns are predictable, though read_collection vs get_document is slightly inconsistent.
41 tools is far beyond the typical well-scoped range. Even for a complex data pipeline, many tools are narrow pipeline stages (e.g., get_records_for_validation, get_records_for_contest_validation, get_records_for_structuring) that could be consolidated. The tool surface feels bloated.
The pipeline covers raw data ingestion, validation, structuring, detail generation, migration, and image verification, which is fairly comprehensive. However, there are gaps such as no explicit image update/replacement tool after generating cover prompts, and events lack migration tools. Generic CRUD tools partially fill gaps.
Maintenance
Related MCP Connectors
Hosted MCP server for AI-driven data ops. Create apps, manage schemas, and CRUD structured data.
The official MCP Server from Mia-Platform to interact with Mia-Platform Console
MCP server for Product Management
MCP server for InsForge BaaS โ database, storage, edge functions, and deployments
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceA powerful Model Context Protocol (MCP) server implementation that provides standardized interaction with MongoDB databases, supporting complete CRUD operations, async patterns, and real-time updates via SSE.MIT
- AlicenseNot gradedqualityDmaintenanceA production-ready MCP server scaffold that features built-in authentication, Docker support, and a comprehensive CI/CD release pipeline. It provides a standardized template for deploying servers with multi-transport support and configurable read-only modes.MIT
- AlicenseNot gradedqualityDmaintenanceA lightweight MCP server for MongoDB 3.6+ providing read, write, metadata, and management tools.12 npm1MIT

NetADX AI-CORE MCP APIofficial
AlicenseNot gradedqualityDmaintenanceA minimal, production-ready MCP server boilerplate for building AI-powered backend services with TypeScript, MongoDB, and JWT authentication.1MIT