🌐 Nexlink Telecom NOC — Autonomous Agent Operations Platform

A complete, enterprise-grade Network Operations Center (NOC) autonomous agent operations platform built for Nexlink Telecom. The platform integrates an MCP protocol server, long-term memory & hybrid RAG, task decomposition & multi-path planning algorithms, durable state graphs with persistent checkpointing, Human-in-the-Loop (HITL) approval gates, an unplanned failure ticket recovery system, and a full-stack web operations platform.

🏛️ Comprehensive System Architecture

flowchart TD
    subgraph UI ["🖥️ Platform Product Surface (Web Dashboard)"]
        UserChat["💬 Multi-Agent Chat Console\n(Switch Agents & Stream Responses)"]
        AdminTools["🛠️ Agent & Tool Registry\n(Dynamic Runtime Toggles)"]
        AdminRAG["📚 RAG Document Manager\n(Live Corpus Upload & Re-indexing)"]
        AdminHITL["🚦 HITL Approvals Queue\n(Policy Gates & State Resumption)"]
        AdminTickets["🎫 Failure Ticket Dashboard\n(Error Inspection & Checkpoint Recovery)"]
    end

    subgraph Agents ["🤖 Autonomous Agent Fleet"]
        SG_Agent["State Graph Agent\n(3 Stateful Workflows)"]
        MR_Agent["Memory & RAG Agent\n(STM, Semantic Store, Self-RAG)"]
        Plan_Agent["Decomposition & Planning Agent\n(TaskDAG, ToT, LATS, Self-Refine)"]
    end

    subgraph StateGraphEngine ["🔄 State Graph & Checkpointing Engine (state_graph/)"]
        Checkpointer[("💾 SQLite Checkpointer\n(db/state_checkpoints.db)")]
        HITL_Node["⏸️ HITL Gate Node\n(Policy-driven Pause)"]
        Ticket_Handler["🚨 Unplanned Failure Boundary\n(Persisted Ticket Creation)"]
        WF1["Disaster Recovery & Traffic Migration\n(Task Decomp + LATS)"]
        WF2["Chronic Fiber Maintenance\n(RAG + Constrained ReAct)"]
        WF3["Enterprise SLA Dispute Settlement\n(Tree of Thoughts + ReAct)"]
    end

    subgraph MCP ["🔌 Model Context Protocol Server (mcp_server/)"]
        Tools["MCP Tools (11 Scoped Endpoints)"]
        Resources["MCP Resources (SLA Policy & Runbook)"]
        Prompts["MCP Parameterized Prompts"]
        Notifications["tools/list_changed Push"]
        Sampling["LLM Protocol Sampling"]
        Elicitation["ctx.elicit() Mid-Call Confirmation"]
    end

    subgraph DB ["🗄️ Database & Storage Layer"]
        NexlinkDB[("db/nexlink.db\n(Customers, Nodes, Services, Logs)")]
        MemoryDB[("db/memory.db\n(Episodic & Semantic Stores)")]
        PlatformDB[("db/platform.db\n(HITL Tasks, Tickets, Tool Registry)")]
        VectorStore[("ChromaDB Vector Store\n(HNSW Dense Embeddings + BM25)")]
    end

    UI --> Agents
    SG_Agent --> StateGraphEngine
    StateGraphEngine --> Checkpointer
    StateGraphEngine --> HITL_Node
    StateGraphEngine --> Ticket_Handler
    HITL_Node --> AdminHITL
    Ticket_Handler --> AdminTickets
    AdminTools --> Tools
    AdminRAG --> VectorStore

    Agents --> MCP
    MCP --> NexlinkDB
    MR_Agent --> MemoryDB
    MR_Agent --> VectorStore

Related MCP server: agentic-ops-builder

📑 Complete Table of Contents

1. Lab 1: MCP Server Protocol & Database Layer

2. Lab 2: Memory Subsystem & Grounded Knowledge (RAG)

3. Lab 3: Decomposition & Planning Engine

4. Lab 4 / Final Project: State Graphs, HITL, Ticket Recovery & Web Platform

5. Cross-Lab Comparison & Benchmark Tables

6. Quickstart & Verification Guide

🔌 Lab 1: MCP Server Protocol & Database Layer

Database Schema & ERD

The system operates over db/nexlink.db with strict foreign keys and WAL mode:

erDiagram
    users {
        int id PK
        string username UK
        string role "NOC_Admin | NOC_Engineer | Guest"
        string api_token UK
    }

    customers {
        int id PK
        string name
        string industry
        string sla_tier "VIP | Enterprise | Standard"
        string contact_email
    }

    network_nodes {
        int id PK
        string name
        string type "Fiber | 5G Core | Edge Router | Satellite"
        float max_capacity_gbps
        float current_load_gbps
        string status "Healthy | Congested | Down | Maintenance"
        string location
    }

    services {
        int id PK
        int customer_id FK
        int node_id FK
        float allocated_bandwidth_gbps
        string status "Active | Suspended | Pending"
    }

    audit_logs {
        int id PK
        datetime timestamp
        int user_id
        string action
        string details
    }

    customers ||--o{ services : "subscribes to"
    network_nodes ||--o{ services : "hosts"
    users ||--o{ audit_logs : "triggers"

8 MCP Protocol Concerns Implementation

🧠 Lab 2: Memory Subsystem & Grounded Knowledge (RAG)

Architecture

• Short-Term Memory & Scratchpad (memory/short_term.py): Rolling FIFO buffer preserving transient dialog distinct from the persistent working scratchpad.

• Promote-or-Drop Router (memory/router.py): Filters aging messages on overflow using an importance threshold ($\ge 0.40$), logging reasoning to memory/routing_log.jsonl.

• Semantic Memory Consolidation (memory/consolidation.py): Periodic offline pass over episodic memory that resolves contradictions, handles versioning (version = old + 1), and flags stale facts.

• Hybrid Vector + Keyword RAG (rag/hybrid_rag.py): Dense vector search via ChromaDB (HNSW index) + BM25 keyword matching fused via Reciprocal Rank Fusion (RRF): [ \text{RRF_Score}(d) = \sum_{m \in {\text{vector}, \text{bm25}}} \frac{1}{60 + \text{rank}_m(d)} ]

• Self-RAG Verification (rag/self_rag.py): Reflection checks verifying retrieved chunk relevance and answer grounding before dispatch.

📐 Lab 3: Decomposition & Planning Engine

TaskDAG & Algorithms (planning/)

• TaskDAG (planning/dag.py): Strict acyclicity enforcement during construction via DFS back-edge detection and topological scheduling via Kahn's algorithm.

• Decomposition-First vs. Dynamic Decomposition:

  • Decompose-First (planning/decompose_first.py): Upfront full DAG generation in one shot.

  • Dynamic Interleaved (planning/decompose_dynamic.py): Step-by-step re-planning after observing live execution results.

• Three Planning Algorithms:

  • Plan-and-Solve (planning/plan_and_solve.py): Single-pass sequential plan generation and execution.

  • Tree of Thoughts (planning/tree_of_thoughts.py): BFS lookahead search ($b=3, d=2$) with scoring and branch pruning ($\ge 6/10$).

  • Language Agent Tree Search - LATS (planning/lats.py): MCTS search guided by real database tool feedback with verbal failure reflection accumulation.

• Self-Correction & Grounded Critique (planning/critique.py): Grounded critique querying real database status (catches maintenance nodes and SLA limits that ungrounded LLM self-evaluation misses).

🚀 Lab 4 / Final Project: State Graphs, HITL, Ticket Recovery & Web Platform

Three Genuinely Stateful Telecom Problems

stateDiagram-v2
    direction LR

    subgraph Problem 1: Disaster Recovery & Traffic Migration
        [*] --> Triage
        Triage --> LATS_Selection : Node Outage
        LATS_Selection --> Failover_Plan : Task Decomp
        Failover_Plan --> Awaiting_Clearance : Wait Field Crew
        Awaiting_Clearance --> HITL_Reroute_Gate : Bandwidth > 3Gbps / VIP
        HITL_Reroute_Gate --> Execute_Migration : Admin Approved
        Execute_Migration --> Verify_Health
        Verify_Health --> [*] : Healthy
    end

1. Disaster Recovery & VIP Traffic Failover (state_graph/disaster_recovery_graph.py)

• Real World Stakes: Core fiber break (e.g. Cairo Metro Line 3 sever) requires rerouting multi-Gbps live enterprise traffic. Rerouting blindly without human oversight risks violating banking/hospital SLAs or overloading adjacent nodes.

• Why it's a State Graph: Spans multiple stages, includes an external wait for field crew clearance, requires policy-gated human sign-off, and must recover cleanly from mid-migration failures.

• Two LLM Additions: Task Decomposition (cutover sequence planning) + LATS (MCTS target node search with live DB capacity validation).

2. Chronic Fiber Degradation & Maintenance Scheduling (state_graph/fiber_maintenance_graph.py)

• Real World Stakes: Intermittent optical attenuation and packet loss on high-load nodes require scheduling physical maintenance windows without disrupting peak enterprise operations.

• Why it's a State Graph: Involves external parts availability checks, maintenance window drafting, and mandatory admin approval for core backbone nodes.

• Two LLM Additions: RAG (queries historical splicing notes & vendor bulletins) + Constrained ReAct (safe execution of maintenance dispatch tools).

3. Enterprise SLA Breach Dispute & Financial Credit Settlement (state_graph/sla_dispute_graph.py)

• Real World Stakes: VIP customers claim SLA financial penalties following downtime. Negotiating compensation balances financial liability against customer churn.

• Why it's a State Graph: Reconciles telemetry across database records, explores settlement packages, waits for finance ledger sync, and requires Finance Director authorization for credits > $5,000.

• Two LLM Additions: Tree of Thoughts (explores multi-branch goodwill vs. cash credit packages) + Constrained ReAct (applies authorized credit notes and service adjustments).

Human-in-the-Loop (HITL) Policy Rules

Unplanned Failure Ticket Recovery System

When an unexpected exception occurs mid-node (database lock, tool network timeout, schema validation error):

1. Error Boundary: Caught immediately by StateGraph exception handler.

2. Durable Snapshot: State snapshot and error traceback are saved to db/state_checkpoints.db.

3. Failure Ticket: Persisted in db/platform.db with status open, inspectable on the web platform.

4. Resumption from Checkpoint: Admin inspects error, edits state variables if necessary, and clicks "Resume from Checkpoint" to continue execution without restarting from scratch.

Crash-and-Resume Proof (Surviving Process Death)

The system utilizes SQLiteCheckpointer (db/state_checkpoints.db) to record every node transition. If the backend process is killed mid-run (kill -9 / power failure):

1. The thread state remains fully intact in SQLite.

2. Upon restart, calling resume_hitl(thread_id, ...) or resume_ticket(thread_id, ...) reloads the exact node checkpoint.

3. Execution resumes with 0 duplicate steps and 0 lost state.

The Platform Web UI (web_platform/ & platform/)

Run the full-stack web platform with:

python platform/run_platform.py

Open http://localhost:8000 in your browser to access:

• 💬 Multi-Agent Chat Console: Switch seamlessly between State Graph Agent, Memory & RAG Agent, and Planning Agent with real-time response rendering and execution telemetry.

• 🛠️ Agent & Tool Registry (Admin): Dynamically enable or disable MCP tools per agent in real time with immediate server-side enforcement.

• 📚 RAG Knowledge Base (Admin): Upload new .txt domain documents, delete documents, and trigger live ChromaDB vector re-indexing.

• 🚦 HITL Approvals Queue (Admin): Live task queue with parameter inspector, Approve / Reject / Modify controls, and instant graph resumption.

• 🎫 Failure Tickets & Recovery (Admin): Inspect open failure tickets, view error tracebacks, edit state snapshots, and resume execution.

• 📈 Checkpoint Timeline: Visual timeline of all execution threads and durable state snapshots.

📊 Cross-Lab Comparison & Benchmark Tables

1. Planning Subsystem Benchmark (12 Real-World Scenarios)

Key Takeaway: Grounded LATS achieves 100% accuracy by validating candidate migrations against real database capacity, completely eliminating false positives from ungrounded self-evaluation.

2. Context Window Management Benchmark (40-Turn Diagnostic Transcript)

Selected Strategy: Observation Masking ships as default for highest recall (90%) and lowest overhead (0.9s latency).

3. Retrieval Architecture Benchmark (12 Domain Questions)

🛠️ Quickstart & Verification Guide

1. Environment Setup

# Clone and enter directory
cd Nexlink-Telecom-B-

# Activate virtual environment
.venv\Scripts\activate

# Install all dependencies
pip install -r requirements.txt

2. Run the Full 4-Lab Audit Suite

python run_all_tests.py

Executes all 4 audit stages covering Database, Memory & RAG, Planning DAG, and State Graphs + Web Platform APIs.

3. Run Automated State Graph Demonstrations

python demo_state_graphs.py

Verifies HITL pause/resumption, failure ticket recovery, and crash-and-resume.

4. Launch Interactive CLI Clients

# State Graph Agent CLI
python agent/state_graph_client.py

# Decomposition & Planning Agent CLI
python agent/planning_client.py

# Memory & RAG Agent CLI
python agent/memory_rag_client.py

5. Launch the Web Platform

python platform/run_platform.py

Navigate to http://localhost:8000 in your browser.