PM Counter Monitoring System

A comprehensive system for monitoring telecom performance monitoring (PM) counters from remote SFTP locations, storing them in a time-series database, and providing access through API endpoints and a Streamlit chat interface.

Architecture

Components

1. SFTP Client (sftp_client.py) - Handles file downloads from remote SFTP server

2. Job Server (job_server.py) - Periodically fetches and processes XML files

3. XML Parser (xml_parser.py) - Parses PM counter XML files

4. Database (database.py) - PostgreSQL schema and models

5. Data Storage (data_storage.py) - Saves parsed data to database

6. API Server (api_server.py) - FastAPI REST endpoints

7. MCP Server (mcp_server.py) - Model Context Protocol server

8. Streamlit Frontend (streamlit_app.py) - Chat bot interface

Quick Start with Docker (Recommended)

The easiest way to run the entire system is using Docker Compose:

The Docker setup includes:

• PostgreSQL database

• SFTP server (for testing, with example XML files)

• Job server (fetches files every hour)

• API server

• MCP server

• Streamlit frontend

All services are automatically configured to work together.

Manual Setup (Without Docker)

1. Install Dependencies

2. Configure Environment

Copy .env.example to .env and update with your settings:

Edit .env with your database and SFTP credentials.

3. Setup Remote Location (SFTP Server)

The remote location is where your XML files are stored. You have several options:

Option A: Use Local Files for Testing (Easiest)

Option B: Set Up Local SFTP Server See SETUP_REMOTE.md for detailed instructions on setting up a local SFTP server.

Option C: Use Existing Remote SFTP Server Update .env with your remote SFTP server credentials:

For more details, see SETUP_REMOTE.md.

4. Setup PostgreSQL Database

5. Initialize Database Schema

Or run:

Running the System

With Docker (Recommended)

Without Docker

1. Start Job Server

The job server fetches files from SFTP at configured intervals:

2. Start API Server

Or using uvicorn:

3. Start MCP Server

Or using uvicorn:

4. Start Streamlit Frontend

Docker Commands

Use the Makefile for convenient commands:

Or use docker-compose directly:

Configuration

Changing Fetch Interval

The fetch interval can be configured in two ways:

1. Environment Variable: Set FETCH_INTERVAL_HOURS in .env file (for Docker) or environment

2. Docker Compose: Update FETCH_INTERVAL_HOURS in docker-compose.yml or .env file

For Docker, update the environment variable and restart the job server:

For non-Docker, update Config.FETCH_INTERVAL_HOURS in config.py or set environment variable.

API Endpoints

Main API (Port 8000)

• GET / - API information

• GET /network-elements - List all network elements

• GET /interfaces/{interface_name}/counters - Get interface counters

• GET /system/counters - Get system counters

• GET /cpu/utilization - Get CPU utilization

• GET /memory/utilization - Get memory utilization

• GET /bgp/peers - List BGP peers

• GET /bgp/peers/{peer_address}/counters - Get BGP peer counters

• GET /files/processed - List processed files

• GET /stats/summary - Get summary statistics

MCP Server (Port 8001)

• POST /mcp - MCP protocol endpoint

• GET /mcp/methods - List available MCP methods

MCP Methods:

• get_interface_counters - Get interface counters

• get_system_counters - Get system counters

• get_cpu_utilization - Get CPU utilization

• get_memory_utilization - Get memory utilization

• get_latest_metrics - Get latest metrics summary

Streamlit Chat Interface

The Streamlit frontend provides a chat bot interface where you can ask questions like:

• "What is the current CPU utilization?"

• "Show me memory usage for the last 12 hours"

• "Get interface counters for GigabitEthernet1/0/1"

• "What are the latest metrics?"

• "Show me system statistics"

Database Schema

The system stores data in the following tables:

• file_records - Track downloaded XML files

• network_elements - Network element information

• measurement_intervals - Time intervals for measurements

• interface_counters - Interface performance counters

• ip_counters - IP layer counters

• tcp_counters - TCP layer counters

• system_counters - System performance counters

• bgp_counters - BGP peer counters

• threshold_alerts - Threshold alerts from XML files

Testing with Local Files

For testing without a real SFTP server, you can:

1. Use the existing example_1.xml and example_2.xml files

2. Modify the job server to process local files directly

3. Use a local SFTP server like openssh-server for testing

Troubleshooting

1. Database Connection Issues: Ensure PostgreSQL is running and credentials are correct

2. SFTP Connection Issues: Verify SFTP server is accessible and credentials are correct

3. API Not Responding: Check if services are running on correct ports

4. No Data: Ensure job server has processed files and data is in the database

License

MIT License