Agentic-Ops Control Plane

Language / Langue : English (Main Documentation) | Lire la documentation complète en Français (README_FR.md)

Agentic-Ops is a personal MLOps control-plane project for exploring how AI agents (LLMs) can help investigate operational signals such as Data Drift, Model Staleness, and OOMKilled Pods.

Unlike frameworks that prioritize agent autonomy, it explores SRE-inspired controls: deterministic policies, observability, fail-closed authorization, asynchronous human review, and cryptographic auditability. These features are engineering work in progress, not a claim of production qualification or EU AI Act compliance.

Current release and honest qualification status

• The Python package is distributed on PyPI; multi-architecture container images (linux/amd64, linux/arm64) are distributed on GHCR (ghcr.io/nolll77/agentic-ops) and Docker Hub (nolll78/agentic-ops).

• A local Docker/kind exercise validated an alert webhook through the worker to a simulated investigation episode, and confirmed that its state survived a pod restart on the same PVC. This is local single-node evidence, not a backup-restore or cluster-loss test.

• Release automation tests and version-checks a GitHub release, publishes Python distributions to PyPI, then builds and publishes the matching AMD64/ARM64 container to both GHCR and Docker Hub. See docs/CONTAINER_IMAGE_PUBLICATION.md.

• Release v0.1.4 adds a fail-closed local inventory, SQLite backup/restore commands, and a synthetic load/crash-recovery benchmark. These tools do not themselves qualify a production deployment.

• Release v0.1.5 improves protected console/LLM-probe authentication UX. Render must still have a private ADMIN_API_KEY configured for these protected actions.

• Release v0.1.6 adds verified multi-database/evidence backup sets, a simulation-only approval continuation on the original episode, expanded worker crash and inventory lifecycle tests, and a size-matrix benchmark. It does not certify an off-host restore or enable live actions.

• Production qualification is not claimed. Entra/SCIM, multi-node HA, deployment-target backup restoration, recovery objectives, synchronization of local inventory with a live source, and action-by-action live execution remain unqualified.

• The detailed repository-to-production mapping, missing target-specific values, owners, validation evidence, and promotion gates are documented in docs/INDUSTRIAL_PRODUCTION_BLUEPRINT.md.

• Render console/chat probe setup and the exact ADMIN_API_KEY prerequisite are documented in docs/RENDER_CONSOLE_CHAT_TROUBLESHOOTING.md.

• A solo, single-node observation/simulation profile does not require an Entra tenant. Its scope and remaining self-operated qualification work are in docs/SELF_HOSTED_PRODUCTION_PATH.md. This is distinct from enterprise SSO, HA and independent audit anchoring.

For implemented behavior versus open production gates, see docs/INDUSTRIAL_IMPLEMENTATION_STATUS.md and docs/PRODUCTION_READINESS_ROADMAP_AND_GAPS.md.

Related MCP server: dingdawg-governance

Project Origin: Extracted from nolll77/car_accident into an MLOps Control Plane

Originally, Agentic-Ops was developed as an SRE governance and self-healing component inside the road-safety MLOps platform nolll77/car_accident (BAAC data warehouse, Feast feature store, MLflow, and FastAPI inference).

It was extracted into this standalone repository to explore a reusable MLOps control plane:

• Zero Domain Coupling: All dataset-specific logic (car_accident) was removed in favor of a Hexagonal Architecture (Ports & Adapters) driven by declarative YAML policies (autonomy_policy.yaml, action_catalog.yaml, risk_register.yaml).

• Designed for adapters: The core engine (Orchestrator FSM, PolicyEngine, Trust Ledger) uses canonical Signal and SystemFacts contracts. Connecting it to another system still requires a suitable adapter, security review, and qualification for that target.

Live Demonstration & Validation Reports

The following public Render instance and reports demonstrate selected features. They are not evidence of production qualification or an availability guarantee:

1. Live Serverless Auto-Wake Production Instance (Render + Tailscale + cron-job.org):

   • Live Control Plane Status: https://agentic-ops-5ebv.onrender.com/

   • Live Interactive SRE Web Console (/console): https://agentic-ops-5ebv.onrender.com/console

   • 💬 Live Interactive Web Chat Assistant (/chat): https://agentic-ops-5ebv.onrender.com/chat  

   • Live MCP (Model Context Protocol) Manifest (/mcp/manifest): https://agentic-ops-5ebv.onrender.com/mcp/manifest *(Full guide: docs/MCP_SERVER_GUIDE.md)*

   • Live Tailscale Mesh Status (100.85.***.*** anonymized): https://agentic-ops-5ebv.onrender.com/api/tailscale/status

   • Live Read-Only Pull Snapshot (jakatt/cac_mlops): https://agentic-ops-5ebv.onrender.com/api/cac/snapshot

2. Cloud Sandbox Validation Report (Docker Agentic Platform): End-to-end deployment on an isolated micro-VM and line-by-line technical interpretation of all 30 unit, adversarial, and cryptographic security tests:
   Read the Cloud Sandbox Deployment & Validation Report (DOCKER_SANDBOX_VALIDATION.md)

3. Tailscale Pull Interconnection Report with Target Platform jakatt/cac_mlops: Userspace Tailscale tunneling (--tun=userspace-networking / SOCKS5 proxy / DERP relays), strict read-only ingestion across 5 observability sources (Loki, Prometheus, MLflow, Prefect, Evidently), and validation via 13 unit tests:
   Read the Tailscale Pull Connector Architecture & Validation Report (CAC_TAILSCALE_PULL_CONNECTOR.md)

4. Target Outage & Scaleway/Tailscale Degradation Resilience Guide: Explains why the Control Plane never crashes if the supervised VPS goes down, detailing RemoteServiceStatus.REMOTE_TARGET_OFFLINE error absorption, Fail-Closed governance, autonomous local test execution, and 1-minute self-healing recovery:
   Read the Target Outage & Resilience Guide (docs/TARGET_OUTAGE_AND_RESILIENCE_GUIDE.md)

5. Official Thesis Defense Report (PDF, 40 Pages): Comprehensive SRE & MLOps thesis defense report on governance, multi-layer observability, deterministic FSM, and interconnection with jakatt/cac_mlops:
   Read the Official Defense Report (docs/Rapport_soutenance_Mar25_MLOps_Agentic_Car_Accident.pdf)

Presentation Slides Explaining this Repository

• Online Google Slides Presentation: https://docs.google.com/presentation/d/1fi5rEGww1CexT4OUcDj1d9W8YQhkL5GoXZwWDtCjLA0/edit?usp=sharing

Foundational Knowledge Engineering: The MLOps Failure Taxonomy & Incident War-Room Scenarios Registry

An AI Agent cannot safely diagnose or govern a production MLOps platform without a formal, structured ontology of how machine learning systems fail in the real world. Before writing a single line of orchestration code, Agentic-Ops was grounded in two foundational engineering registries that serve as the cognitive, observability, and decision-making reference frame for the entire repository:

Why These Two Referentials Are Central to the LLM & Agentic Architecture

1. Guiding Observability (What to Measure & Why): Rather than scraping arbitrary logs, CacPullCollector and the Signal contracts monitor the exact symptom vectors identified in the Taxonomy (Data Drift Share, LogQL error patterns, OOMKilled pod restarts, latency P95, MLflow @Production staleness) mapped to explicit risk_id identifiers (ML-014, DATA-042, INFRA-001).

2. Empowering Root-Cause Comprehension (Distinguishing Symptoms from Causes): As formalized in Scenario 1 (Silent Cascade) and Scenario 2 (Multi-Fail Catastrophe) of docs/mlops_simulation_scenarios_registry.md, a production ML model can return HTTP 200 OK while serving corrupted predictions, or exhibit an API latency spike whose true origin lies in an upstream data pipeline (data_pipeline vs. infrastructure). The Runbook RAG (agentic_ops/knowledge_base/runbooks/) and specialist agents (data_specialist_llm, infra_specialist_llm) draw directly from these RCA patterns to avoid misdiagnosing upstream data corruption as an infrastructure failure.

3. Anchoring Deterministic Decision-Making & Guardrails (PolicyEngine & Phase D): Every veto rule in autonomy_policy.yaml—such as Rule R-001 (which blocks restart_pod, scale_up, or blind retrain_model when upstream data anomalies are present) and Catalog Rules CAT-001 / CAT-002—was derived from the failure cascades documented in these two registries and verified live via the 10 Phase D scenarios (SC-101 to SC-005, forbidden_executed = 0).

Direct Code Lineage & Mapping Keys (From Theoretical Ontology to Executable Agentic Code)

To prove that these two documents directly shaped the Agentic-Ops codebase rather than sitting as passive documentation, every executable scenario in tests/scenarios/data/ embeds explicit source (SIM-*) and trigger.risk_id (DATA-*, ML-*, INFRA-*) foreign keys pointing back to the Taxonomy and Scenarios Registry:

Scope Transparency (Research Ontology vs. Industrialized Core): The 435 failure modes in docs/mlops_failure_modes_taxonomy.md and 36 War-Room scenarios in docs/mlops_simulation_scenarios_registry.md constitute the exhaustive research ontology of MLOps failure dynamics. From this broad cartography, the critical core (DATA-042, ML-014, INFRA-001, and the 7 operational/adversarial scenarios mapped above) was extracted and industrialized into the runtime FSM and PolicyEngine to prove end-to-end deterministic containment (forbidden_executed = 0).

In-Memory Deterministic RAG Engine (Zero-Token & Zero-Cost Grounding for Chat & MCP)

To enable live interrogation of this foundational knowledge base without incurring external API costs, vector DB overhead, or token hallucinations, Agentic-Ops includes a Deterministic In-Memory RAG Engine (agentic_ops/core/deterministic_rag.py):

1. Sub-4ms In-Memory Cold Start (Zero External Vector DB):

   • Indexes all 435 Failure Modes from docs/mlops_failure_modes_taxonomy.md (IDs DATA-042, INFRA-001, SEC-054, calibrated RPN scores, SLO impacts, automated mitigations, and chaos scripts).

   • Indexes all 36 Simulation Scenarios from docs/mlops_simulation_scenarios_registry.md (SIM-001 to SIM-036, timelines $T_0 \dots T_4$, Amazon 5-Whys RCA, and ASCII topologies).

   • Indexes all Declarative YAML contracts from tests/scenarios/data/ (SC-001 to SC-005, adversarial SC-101 to SC-103).

2. Dual-Mode Execution (Online LLM Grounding + Offline Zero-Token Fallback):

   • Online Mode (OpenRouter / Gemma 4 / Nemotron): When a user queries /chat about an incident ID, RPN score, or scenario name, the engine extracts the exact authoritative Markdown card and injects it into the LLM system prompt as [CONTEXTE DE RÉFÉRENCE MLOPS], guaranteeing 0 hallucination.

   • Offline Degraded Mode (Zero-Token / 0 EUR): If cloud LLMs are unreachable or OPENROUTER_API_KEY is omitted, the engine returns the complete technical card directly in under 1 millisecond with 0 tokens consumed.

3. Interactive Web Chat Exploration (/chat):

   • Two dedicated header modals: 🎯 RAG Taxonomie (435) and 🎬 RAG Scénarios (36) provide one-click interactive test queries directly within the browser interface.

4. Governed MCP Tool Access:

   • External IDEs and AI clients (Cursor, Claude Desktop) can query this engine via MCP tools search_failure_taxonomy(query, max_results) and get_scenario_blueprint(scenario_id).

Global Architecture (Hexagonal)

The system is built on a Hexagonal Architecture coupled with a strict Finite State Machine (FSM).

1. Orchestrator (FSM): Coordinates the strict lifecycle of an incident (DETECTED -> INVESTIGATING -> DIAGNOSED -> PLANNED -> AWAITING_APPROVAL / EXECUTING -> RESOLVED / DENIED).

2. Policy Engine: The governance brain. Evaluates every proposed action against a declarative YAML manifest (autonomy_policy.yaml) to determine the permitted level of autonomy.

3. Trust Ledger (Evidence Store): Dual-tier immutable cryptographic registry (WORM). Combines an append-only linear hash chain (previous_hash + HMAC-SHA512) with an RFC 6962 Binary Merkle Tree providing 32-byte Merkle root condensation and $\mathcal{O}(\log N)$ Selective Disclosure Audit Proofs for EU AI Act compliance.

4. Executor & Model Registry (Simulated / Adapter): Executes approved actions and strictly governs model lifecycle transitions (ModelRegistryStatusEnum, ModelDeploymentStatusEnum).

5. Agent Tools (Sandboxing): Tools exposed to LLMs, confined at the OS process level with an enforced hardware/OS timeout.

Project Structure (ASCII Tree)

The application architecture is reflected in this formal directory layout, where each module enforces a critical link in the security chain:

agentic-ops-core/
├── agentic_ops/
│   ├── core/
│   │   ├── orchestrator.py       # Main FSM; durable approval resumes the original episode in simulation only.
│   │   ├── policy.py             # Governance engine (Pydantic), evaluates permission manifests.
│   │   ├── executor.py           # Last-mile actuator. Mutates the Model Registry. Returns FAILED on business rule rejection.
│   │   ├── merkle.py             # RFC 6962 Binary Merkle Tree. Domain separators (0x00/0x01), O(log N) audit proofs, second-preimage attack resistance.
│   │   ├── evidence.py           # WORM Trust Ledger. Canonical JSON serialization, block chaining (previous_hash), Merkle tree integration.
│   │   ├── distributed.py        # Idempotency and Rate Limiting via Redis (Atomic pipelines & distributed locks).
│   │   └── circuit_breaker.py    # FinOps protection against Monitoring Storms (automatic degraded mode fallback).
│   ├── adapters/
│   │   ├── cac_pull_collector.py # Tailscale Pull Connector (GET-only) targeting jakatt/cac_mlops (Loki, Prometheus, MLflow, Prefect).
│   │   └── registry_adapter.py   # Protocol and Gatekeeper simulation (e.g., automatic rejection if F1 < 0.85).
│   ├── governance/
│   │   ├── autonomy_policy.yaml  # Strict production MLOps manifest (approval requirements).
│   │   └── shadow_policy.yaml    # Security policy A/B testing (divergence analysis).
│   ├── cli/
│   │   ├── main.py               # FastAPI server exposing /, /healthz, /ready, /api/tailscale/status, /api/cac/snapshot, /slack/interactive.
│   │   └── trust_ledger.py       # Trust Ledger CLI: list, inspect, merkle-root, and merkle-proof <episode_id>.
│   └── entrypoints/
│       ├── mcp_server.py         # Model Context Protocol (MCP) Governance Gateway (8 Tools, 5 Resources, 20 Prompts).
│       └── slack_webhook.py      # HITL: Human approval ingestion (HMAC-SHA256 verification, 5m anti-replay window).
├── scripts/
│   └── entrypoint.sh             # Zero-Click Auto-Wake boot script (CSPRNG session key + Redis :6379 + Tailscale :1055 + FastAPI).
├── docs/
│   ├── WORM_TRUST_LEDGER_GUIDE.md           # Comprehensive WORM Trust Ledger & RFC 6962 Binary Merkle Tree guide.
│   ├── PRODUCTION_READINESS_ROADMAP_AND_GAPS.md # 7-Pillar enterprise production hardening roadmap.
│   ├── MCP_SERVER_GUIDE.md                  # Complete MCP Server guide, STDIO/HTTP connection configs, and 20 Starter Prompts.
│   ├── mlops_failure_modes_taxonomy.md      # 435 MLOps failure modes ontology across 10 dimensions.
│   └── mlops_simulation_scenarios_registry.md # MLOps crisis simulation scenario registry.
├── tests/
│   ├── adapters/
│   │   ├── test_cac_pull_collector.py # 13 unit tests (JSON parsing, offline resilience, strict read-only enforcement).
│   │   └── test_llm_provider_registry.py # 16 tests for multi-LLM providers, failover chains, and API key masking.
│   ├── entrypoints/
│   │   ├── test_mcp_server.py         # 24 unit & E2E tests for the MCP Governance Gateway.
│   │   ├── test_tailscale_endpoint.py # 4 tests for mesh status and PII redaction.
│   │   └── test_chat_endpoint.py      # 10 tests for live SSE streaming and chat assistant endpoints.
│   ├── adversarial/
│   │   ├── test_merkle_tree.py        # 13 tests for RFC 6962 Merkle Tree, odd/even leaves, O(log N) proofs & tamper resistance.
│   │   ├── test_evidence_integrity.py # 4 tests for WORM linear hash chain tampering and production fail-closed.
│   │   ├── test_prompt_hygiene.py     # Prompt injection fencing (<untrusted_telemetry>).
│   │   └── test_idempotency_and_rate.py # Redis distributed locks & token-bucket rate limiting.
│   ├── policy/
│   │   └── test_policy_engine.py      # Autonomy matrix and YAML rule verification.
│   └── scenarios/
│       └── test_scenarios.py          # 10 production incident cascades (SC-101 to SC-005, 0 forbidden executed).
├── Dockerfile                         # Production Auto-Wake container image (Render / Koyeb / Self-Hosted, Port 8001).
├── Dockerfile.hf                      # Hugging Face Spaces Auto-Wake container image (Port 7860, non-root UID 1000).
├── DOCKER_SANDBOX_VALIDATION.md       # Deployment and validation report on Docker Cloud Sandbox.
├── CAC_TAILSCALE_PULL_CONNECTOR.md    # Architecture and validation of the Tailscale Pull Connector to CAC MLOps.
├── README.md                          # Main documentation (English).
├── README_FR.md                       # Complete documentation (French).
└── pyproject.toml                     # Dependency manifest (Pydantic, FastAPI, Redis, httpx[socks], etc.)

Live Production Interconnection: Plugging Agentic-Ops into CAC MLOps (jakatt/cac_mlops)

Agentic-Ops is not another model-training platform: it is an agnostic "Control Plane" (an SRE governance brain) that plugs directly on top of an existing production MLOps execution platform.

In our reference deployment, Agentic-Ops runs both in an isolated Docker Agentic Platform micro-VM (Tailscale IP 100.102.***.***) and as a live 24/7 Auto-Wake PaaS Web Service on Render (https://agentic-ops-5ebv.onrender.com, anonymized Tailscale IP 100.85.***.***, hostname agentic-ops-render) connected to the target production platform CAC MLOps:

• Target Platform GitHub Repository: https://github.com/jakatt/cac_mlops (Road accident severity prediction platform).

• Target Production Server (Scaleway VPS): scw-jovial-dubinsky — Tailscale IP: 100.117.99.62 — Public domain: https://mlops.jakat-inc.fr.

• Live Agentic-Ops Control Plane (Render): https://agentic-ops-5ebv.onrender.com — Tailscale IP: 100.85.***.*** (anonymized).

• Dedicated Technical Connector Documentation: CAC_TAILSCALE_PULL_CONNECTOR.md.

A. End-to-End Architectural Connection Diagram (Agentic-Ops <-> jakatt/cac_mlops <-> Slack HITL)

The ASCII architecture diagram below illustrates all network connection points, the strict boundary between read-only observability endpoints and the sanctuarized write zone, as well as the human interaction loop via Slack:

+=======================================================================================================+
| TARGET PRODUCTION MLOPS PLATFORM : jakatt/cac_mlops (Scaleway VPS scw-jovial-dubinsky)                |
| Repo : https://github.com/jakatt/cac_mlops  |  Tailscale IP : 100.117.99.62                           |
+=======================================================================================================+
|                                                                                                       |
|  [ ZONE 1 : OBSERVABILITY CONNECTION POINTS (AUTHORIZED - STRICT HTTP GET READ-ONLY) ]                |
|  +-------------------------------------------------------------------------------------------------+  |
|  | 1. Grafana Loki   (:3100) -> GET /loki/api/v1/query_range    (Alert logs, rollbacks, CI/CD)     |  |
|  | 2. Prometheus     (:9090) -> GET /api/v1/query               (cac_mlops_drift_share, model_info)|  |
|  | 3. MLflow Server  (:5001) -> GET /api/2.0/mlflow/...         (Alias @Production, F1-score, run) |  |
|  | 4. Prefect Server (:4200) -> GET /api/flow_runs              (Status of last 10 pipeline runs)  |  |
|  | 5. Evidently Web  (:443)  -> GET https://mlops.jakat-inc.fr/reports/drift/latest_summary.json   |  |
|  +------------------------------------------------+------------------------------------------------+  |
|                                                   |                                                   |
|  [ ZONE 2 : SANCTUARIZED ZONE (FORBIDDEN IN PULL MODE - PROTECTED BY LOGICAL FIREWALL) ]              |
|  +------------------------------------------------v------------------------------------------------+  |
|  | [X] POST /predict (Inference API)        -> Forbidden (would pollute prod DB & drift metrics)   |  |
|  | [X] POST /loki/api/v1/push               -> Forbidden (would inject fake alerts into prod)      |  |
|  | [X] POST/DELETE .../registered-models    -> Forbidden outside approved Executor (model mutation)|  |
|  | [X] POST /api/deployments/.../flow_run   -> Forbidden outside approved Executor (reset risk)    |  |
|  | [X] Port :7860 (Gradio Admin Cockpit)    -> Forbidden (triggers direct un-audited actions)      |  |
|  +-------------------------------------------------------------------------------------------------+  |
+===================================================|===================================================+
                                                    |
                                                    | Encrypted WireGuard Mesh Network (Tailscale)
                                                    | Firewall traversal via DERP Relays (Paris <-> Seattle)
                                                    | + Direct HTTPS stream for public Evidently drift report
                                                    v
+=======================================================================================================+
| GOVERNANCE CONTROL PLANE : AGENTIC-OPS (Docker Cloud Sandbox - Tailscale IP : 100.102.123.121)        |
+=======================================================================================================+
|                                                                                                       |
|  1. NETWORK INGESTION LAYER                                                                           |
|     [ tailscaled daemon (--tun=userspace-networking) ] ---> [ Local SOCKS5 Proxy : 127.0.0.1:1055 ]   |
|                                                                      |                                |
|                                                                      v                                |
|  2. OBSERVABILITY ADAPTER (agentic_ops/adapters/cac_pull_collector.py)                                |
|     [ CacPullCollector.collect_all() ]                                                                |
|       |-- Queries all 5 endpoints via HTTP GET (20s timeout)                                          |
|       |-- Absorbs network failures (Returns ONLINE, PARTIAL, or REMOTE_TARGET_OFFLINE without crash)  |
|       +-- Emits an immutable Pydantic contract : CacCollectedSnapshot                                 |
|                                                                      |                                |
|                                                                      v                                |
|  3. DETERMINISTIC FACT EXTRACTION & COGNITIVE DIAGNOSIS                                               |
|     [ compute_facts() ] ---> Objectively extracts : drift_share=0.34, drifted columns, F1=0.883       |
|     [ Investigator LLM Agent ] -> Analyzes quarantined facts (<untrusted_data>) & proposes intent     |
|                                   (e.g., ActionIntent = TRIGGER_RETRAINING or PROMOTE_MODEL)          |
|                                                                      |                                |
|                                                                      v                                |
|  4. DETERMINISTIC GOVERNANCE ENGINE (agentic_ops/core/orchestrator.py & policy.py)                    |
|     [ PolicyEngine ] -> Evaluates ActionIntent against autonomy_policy.yaml and risk_register.yaml    |
|       |-- If Action = Low Risk (e.g., INVESTIGATE) --------> Decision = ALLOW                         |
|       |-- If Action = Forbidden or Exceeds Redis Quota ----> Decision = DENY (Immediate block)        |
|       +-- If Action = Critical (PROMOTE_MODEL / RETRAIN) --> Decision = REQUIRE_APPROVAL              |
|                                                                      |                                |
|     [ Orchestrator FSM ] -> Transitions to AWAITING_APPROVAL and PHYSICALLY HALTS.                    |
|                             Persists state in SQLite (WAL) + Acquires Redis semantic idempotency lock.|
+===================================================|===================================================+
                                                    |
                      +-----------------------------+-----------------------------+
                      | 5. OUTBOUND NOTIFICATION                                  | 6. SIGNED INBOUND WEBHOOK
                      |    (Slack Block Kit Message)                              |    POST /webhooks/slack (:8001)
                      v                                                           |    (HMAC-SHA256 Signature + 5m TTL)
+=================================================================================|=====================+
| HUMAN-IN-THE-LOOP INTERFACE (SLACK)                                             |                     |
|                                                                                 |                     |
|  +---------------------------------------------------------------------------+  |                     |
|  | [MLOPS ALERT - DATA-042] Data Drift detected on jakatt/cac_mlops          |  |                     |
|  | - Source : Prometheus (cac_mlops_drift_share = 34%) & Evidently           |  |                     |
|  | - Drifted columns : age_conducteur, hrmn, surf                            |  |                     |
|  | - Active model : lgbm_accidents v7 (F1 = 0.883)                           |  |                     |
|  | - Proposed Agent Action : TRIGGER_RETRAINING                              |  |                     |
|  |                                                                           |  |                     |
|  |   Button 1 : [ APPROVE EXECUTION ]          Button 2 : [ REJECT (DENY) ]  +--+                     |
|  +---------------------------------------------------------------------------+                        |
+=======================================================================================================+
                                                    |
                                                    | 7. TRANSACTIONAL RESUMPTION (If "APPROVE" verified)
                                                    v
+=======================================================================================================+
| LAST-MILE EXECUTION & EVIDENCE LEDGER (agentic_ops/core/executor.py & evidence.py)                    |
|                                                                                                       |
|  1. [ slack_webhook.py ] -> Verifies HMAC-SHA256, anti-replay (< 5 min), YAML approver group,         |
|                             and acquires an atomic SQLite lock (BEGIN IMMEDIATE) against double-click.|
|  2. [ approval worker ] -> Revalidates quorum and resumes the original episode in simulation only.   |
|  3. [ EvidenceStore ] -> Appends the simulated result to the original signed evidence chain.          |
|  4. [ Live execution ] -> Disabled; no real target is changed by this approval path.                 |
+=======================================================================================================+

B. Breakdown of the 5 Observability Connection Points (CacPullCollector)

To monitor the jakatt/cac_mlops platform without ever disrupting its production operations, agentic_ops/adapters/cac_pull_collector.py polls five specific read-only endpoints:

Resilience to Remote Server Power-Down (Offline Tolerance)

If the remote Scaleway VPS scw-jovial-dubinsky is powered down (or if an individual service is restarting), CacPullCollector.collect_all() gracefully catches httpx.ConnectError and httpx.TimeoutException:

• If all 5 sources respond: snapshot.status = RemoteServiceStatus.ONLINE.

• If 1 to 4 sources fail: snapshot.status = RemoteServiceStatus.PARTIAL (data from healthy services is preserved and processed).

• If the VPS is completely powered off: snapshot.status = RemoteServiceStatus.REMOTE_TARGET_OFFLINE (the Agentic-Ops orchestrator remains on standby without raising unhandled exceptions).

C. How the Interactive Slack Alert & Approval Loop Works (HITL)

The interaction between telemetry ingested from jakatt/cac_mlops and human decision-making on Slack follows a 6-step cryptographic security protocol:

1. Ingestion & Episode Triggering

The orchestrator runs a collection cycle via CacPullCollector().collect_all() (either on a scheduled Pull interval or triggered by an inbound alert webhook). The resulting CacCollectedSnapshot reveals, for example, that cac_mlops_drift_share has reached 0.34 (34% of features have drifted) and that an event=alert entry appeared in Loki.

2. Deterministic Fact Computation (System Facts) & Prompt Quarantine

Before invoking any LLM, compute_facts() mathematically extracts verified system facts from the snapshot. Raw log lines coming from Loki are wrapped inside <untrusted_data> quarantine tags by the Prompt Guard so that no adversarial prompt injection hidden inside an external log can hijack the agent.

3. Cognitive Diagnosis & Interception by the PolicyEngine

The cognitive agent analyzes the context (active model lgbm_accidents v7, drift detected on age_conducteur, hrmn, surf) and outputs a structured intent: ActionIntent(action="TRIGGER_RETRAINING", target="lgbm_accidents") or PROMOTE_MODEL. The PolicyEngine immediately evaluates this intent against autonomy_policy.yaml:

• Any action that mutates the production model or launches a heavy training pipeline is classified as REQUIRE_APPROVAL.

• The state machine (Orchestrator FSM) transitions to AWAITING_APPROVAL and physically halts. The AI agent is deactivated.

4. Dispatching the Interactive Slack Alert

The Control Plane dispatches a formatted Slack Block Kit notification to the MLOps on-call channel containing:

• The unique episode_id and risk_id.

• A summary of the evidence collected from jakatt/cac_mlops (Prometheus/Evidently drift share, current MLflow metrics, status of the latest Prefect flow).

• The proposed action and its computed blast_radius.

• Two interactive action buttons: [ APPROVE ] and [ DENY ].

5. Receiving & Cryptographically Verifying the Human Click (slack_webhook.py)

When the on-call SRE clicks [ APPROVE ] in Slack, Slack's servers send a POST /webhooks/slack request to our FastAPI endpoint (port 8001). agentic_ops/entrypoints/slack_webhook.py enforces 4 sequential security gates:

1. Cryptographic Authenticity (HMAC-SHA256): Verifies the X-Slack-Signature header computed with SLACK_SIGNING_SECRET over the raw request body. Any unsigned or tampered request is rejected in 0.002s with HTTP 401 Unauthorized.

2. Anti-Replay Protection (Replay Attack): Inspects X-Slack-Request-Timestamp. Requests older than 5 minutes or carrying a previously seen signature are rejected.

3. RBAC Authorization (approvers.yaml): Verifies that the Slack user who clicked the button belongs to the authorized approver group for that criticality tier.

4. Atomic Transactional Lock (SQLite BEGIN IMMEDIATE): If two operators click the Slack button simultaneously (the Double-Click race condition), SQLite's immediate transaction lock guarantees that only the first click transitions the record from PENDING -> APPROVED, while the second is rejected.

6. Last-Mile Execution & WORM Sealing (Trust Ledger)

Once the approval quorum is verified:

• The durable approval worker revalidates current approver roles and only invokes the executor in simulation mode.

• It appends the simulated EXECUTING → RESOLVED outcome to the original episode's signed evidence chain. Recovery after evidence append but before queue acknowledgement is idempotent.

• No real infrastructure action is enabled by this path. Slack delivery, production identity, denial reconciliation and deployment-level recovery still require qualification; see docs/INDUSTRIAL_IMPLEMENTATION_STATUS.md.

%%{init: {'theme': 'dark', 'themeVariables': {'actorBkg': '#0f172a', 'actorBorder': '#38bdf8', 'actorTextColor': '#f8fafc', 'actorLineColor': '#64748b', 'signalColor': '#e2e8f0', 'signalTextColor': '#f8fafc', 'noteBkgColor': '#1e1b4b', 'noteBorderColor': '#c084fc', 'noteTextColor': '#f3e8ff', 'activationBkgColor': '#334155', 'activationBorderColor': '#38bdf8'}}}%%
sequenceDiagram
    autonumber
    participant CAC as Target Platform jakatt/cac_mlops (100.117.99.62)
    participant Pull as CacPullCollector (SOCKS5 :1055)
    participant FSM as Orchestrator FSM & PolicyEngine
    participant Slack as On-Call SRE (Slack HITL)
    participant Exec as Executor & WORM Trust Ledger

    rect rgb(12, 45, 72)
    Note over CAC, Pull: Phase 1 : Passive Observation (100% GET Read-Only - Sapphire Zone)
    Pull->>CAC: GET Loki (:3100), Prometheus (:9090), MLflow (:5001), Prefect (:4200), Evidently (:443)
    CAC-->>Pull: JSON (drift_share=0.34, lgbm_accidents v7, LogQL alerts)
    Pull->>FSM: CacCollectedSnapshot (status=ONLINE)
    end

    rect rgb(69, 26, 3)
    Note over FSM, Slack: Phase 2 : Diagnosis & Fail-Closed Guardrail (Amber Zone)
    FSM->>FSM: compute_facts() + Agent Diagnosis -> ActionIntent(TRIGGER_RETRAINING)
    FSM->>FSM: PolicyEngine evaluates autonomy_policy.yaml -> REQUIRE_APPROVAL
    FSM->>Slack: Slack Block Kit Alert (CAC Evidence + [APPROVE] / [DENY] Buttons)
    Note over FSM: Physical halt in AWAITING_APPROVAL state (SQLite WAL)
    end

    rect rgb(6, 78, 59)
    Note over Slack, Exec: Phase 3 : Signed Human Approval & Simulation Only
    Slack->>FSM: Click [APPROVE] -> POST /webhooks/slack (HMAC-SHA256 Signed)
    FSM->>FSM: Verifies HMAC-SHA256 + Anti-Replay (<5m) + BEGIN IMMEDIATE Lock
    FSM->>Exec: Approval worker simulates the action
    Exec-->>FSM: SIMULATED result
    FSM->>FSM: Appends result to the original signed evidence episode
    end

D. Serverless Auto-Wake PaaS Deployment (Render + Tailscale Userspace + 24/7 Keep-Alive)

To allow Agentic-Ops to run continuously on a free-tier PaaS (Render, Hugging Face Spaces, or Koyeb) without ever requiring an operator to open an SSH terminal or click a console start button, the entire stack (Redis + Tailscale SOCKS5 + FastAPI) is packaged inside a self-initializing container (Dockerfile, Dockerfile.hf, and scripts/entrypoint.sh).

1. Live Production URLs & Endpoints (Render — Frankfurt EU Central)

2. Anatomy of the Zero-Console Container Boot Sequence (scripts/entrypoint.sh)

Every time the container starts (whether on a fresh deployment or waking up from a Render inactivity spin-down), scripts/entrypoint.sh executes a 5-stage deterministic initialization sequence:

1. SOCKS5 Proxy Loop Prevention (\x05\xff Bug Fix): Explicitly executes unset HTTP_PROXY HTTPS_PROXY ALL_PROXY http_proxy https_proxy all_proxy before starting any daemon. This guarantees that tailscaled connects directly to Tailscale control servers (controlplane.tailscale.com:443) while confining the socks5://127.0.0.1:1055 proxy strictly to the httpx transport inside cac_pull_collector.py.

2. Runtime Ephemeral tmpfs Initialization (/tmp/state): On serverless PaaS platforms (Render, Hugging Face Spaces), /tmp is mounted as a fresh in-memory tmpfs at container startup, wiping directories created during docker build. Both scripts/entrypoint.sh (mkdir -p /tmp/state) and ApprovalRepo.__init__() (os.makedirs(os.path.dirname(os.path.abspath(path)), exist_ok=True)) recreate /tmp/state dynamically at runtime so SQLite WAL (/tmp/state/approvals.db) never fails with sqlite3.OperationalError.

3. Local Redis Daemon (127.0.0.1:6379): Starts redis-server --daemonize yes --bind 127.0.0.1 --port 6379 inside the container, providing zero-latency atomic operations (INCR + EXPIRE for rate limiting and SET NX EX for semantic idempotency locks) without requiring an external paid Redis add-on.

4. Userspace Tailscale Daemon & Secure Persistence (TS_AUTHKEY vs. Secure Injection): Starts tailscaled --tun=userspace-networking --socks5-server=127.0.0.1:1055 --state=/tmp/tailscaled.state --socket=/tmp/tailscaled.sock (requiring zero root CAP_NET_ADMIN or /dev/net/tun privileges).

   • Recommended Mode — TS_AUTHKEY (Tailnet Admin Reusable Key): Setting TS_AUTHKEY=tskey-auth-... in Render's environment variables authenticates tailscale up --authkey automatically at boot without human intervention.

   • Ephemeral Mode — auth_url: If no key is set, Tailscale's interactive authorization URL is redirected strictly to Render's private deployment logs (/tmp/tailscale_up.log) and redacted on the public API to prevent unauthorized node registration. The node's private cryptographic state remains confined to local storage and is never exposed over public HTTP.

5. Non-Blocking Background tailscale up & Immediate FastAPI Startup: tailscale up is launched asynchronously in the background (nohup tailscale --socket="$TS_SOCKET" up ... &) so that exec agentic-start (uvicorn on 0.0.0.0:8001) binds immediately and passes Render's health check without ever blocking on network handshakes.

3. Three-Layer Wake-Up, Keep-Alive & Manual Fallback Procedures

1. Layer 1 — Automated 24/7 Keep-Alive (cron-job.org, Zero Cold-Start):

   • On Render's free tier, instances spin down after 15 minutes of HTTP inactivity (and take ~30 to 50 seconds to cold-start on the first subsequent request).

   • To eliminate this 50-second cold-start delay completely and keep the Tailscale tunnel (100.85.***.***) permanently up 24/7, a scheduled job on https://console.cron-job.org/jobs (Agentic-Ops Keep-Alive) sends an HTTP GET request to https://agentic-ops-5ebv.onrender.com/healthz every 10 minutes.

2. Layer 2 — Automatic On-Demand HTTP Wake-Up (If cron-job.org is Paused):

   • Even if the external cron job is disabled and the container goes to sleep after 15 minutes (or several hours), any incoming HTTP request automatically triggers Render to wake the container and run scripts/entrypoint.sh:

     • Opening https://agentic-ops-5ebv.onrender.com/ or https://agentic-ops-5ebv.onrender.com/api/cac/snapshot in any web browser.

     • Calling fetch("https://agentic-ops-5ebv.onrender.com/ready") from a frontend application (Vercel).

     • Clicking [ APPROVE ] or [ DENY ] on a Slack HITL alert (POST /slack/interactive).

3. Layer 3 — Manual Wake-Up & Emergency Re-Authentication Procedures:

   • Manual Wake-Up from Terminal or Browser: Simply open https://agentic-ops-5ebv.onrender.com/ in a browser, or run from any terminal:

     # 1. Wake the container and verify local Redis (:6379) & Tailscale SOCKS5 (:1055)
     curl -s https://agentic-ops-5ebv.onrender.com/ | python3 -m json.tool
     
     # 2. Verify Tailscale mesh connection (anonymized IP 100.85.***.***, BackendState: Running)
     curl -s https://agentic-ops-5ebv.onrender.com/api/tailscale/status | python3 -m json.tool
     
     # 3. Trigger a live read-only pull from jakatt/cac_mlops (100.117.99.62)
     curl -s "https://agentic-ops-5ebv.onrender.com/api/cac/snapshot?timeout=3.5" | python3 -m json.tool

   • Interactive Tailscale Authentication (If the node requires reconnection):

     1. If "backend_state" is "NeedsLogin", inspect Render's private deployment logs (/logs) or configure TS_AUTHKEY in environment variables.

     2. Follow the official login URL https://login.tailscale.com/a/... displayed in your private logs and click Connect.

     3. Upon validation, the node transitions to "Running" and its status is confirmed via "has_persisted_state": true on /api/tailscale/status.

4. Interactive Web Terminal (/console) & 1-Click Execution of All Docker Cloud Sandbox Commands

Because Render's native SSH Shell tab requires a paid instance plan, Agentic-Ops embeds a built-in Dark-Mode SRE Cognitive Web Console at https://agentic-ops-5ebv.onrender.com/console (and via Swagger UI at https://agentic-ops-5ebv.onrender.com/docs).

Every single validation phase documented in reports/DOCKER_SANDBOX_VALIDATION.md and reports/CAC_TAILSCALE_PULL_CONNECTOR.md can be executed live inside the Render container in 1 click (with color-coded pedagogical explanations [GUIDE:ROLE], [GUIDE:AGENT], [GUIDE:SLACK], [GUIDE:PROD], [GUIDE:HOW_IT_PASSES] and automatic PII redaction):

Architectural Note — Lazy litellm Loading (LiteLLMAdapter.__init__()) for Sub-Second Execution on Render's 0.1 vCPU: On Render's free instance plan, CPU allocation is throttled to 0.1 vCPU (1/10th of a physical core). Eagerly importing litellm and instructor at module import time inside agentic_ops/adapters/llm_protocol.py previously forced Python to parse litellm's 2 MB token/pricing tables (~35 to 50 seconds on 0.1 vCPU) and attach a global asyncio logging filter (litellm._logging.py $\rightarrow$ _ModuleLock('litellm.types.secret_managers.main') deadlock when concurrent requests arrived). By deferring import litellm and import instructor strictly inside LiteLLMAdapter.__init__() (while keeping class LLMPort(Protocol) zero-dependency), importing Orchestrator and running the entire Phase D (10 scenarios) or pytest_security suite in a clean isolated subprocess drops from ~50,000 ms to ~980 ms with zero thread deadlocks.

5. Live WORM Trust Ledger Audit Trail & Concrete Impact on Target MLOps (jakatt/cac_mlops)

Dedicated Technical Reference: docs/WORM_TRUST_LEDGER_GUIDE.md | Source Module: agentic_ops/core/evidence.py | Audit Route: GET /api/diagnostics/trust_ledger_list

Clicking WORM Trust Ledger CLI in /console (GET /api/diagnostics/trust_ledger_list) executes python -m agentic_ops.cli.trust_ledger list inside the Render container and returns the cryptographically sealed WORM audit trail (agentic_ops/evidence_store/*.json):

$ /usr/local/bin/python -m agentic_ops.cli.trust_ledger list
[exit_code=0 | duration=2866ms]
------------------------------------------------------------------------
Trust Ledger - Audit Trail                           
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃                   ┃           ┃ Risk /    ┃           ┃ Action    ┃ Décision ┃
┃ Episode ID        ┃ Timestamp ┃ Mode      ┃ Agent LLM ┃ Demandée  ┃  Policy  ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│ EP-CHAOS-31c6e006 │ 2026-10-… │ MODEL-005 │ speciali… │ propose_… │   DENY   │
│                   │           │ (SHADOW)  │           │           │          │
│ EP-CHAOS-5950095d │ 2026-10-… │ DATA-042  │ speciali… │ propose_… │   DENY   │
│                   │           │ (SHADOW)  │           │           │          │
│ EP-CHAOS-d6c18601 │ 2026-10-… │ INFRA-001 │ speciali… │ create_i… │   DENY   │
│                   │           │ (SHADOW)  │           │           │          │
│ EP-0005           │ 2026-09-… │ DATA-042  │ data_spe… │ create_i… │  ALLOW   │
│                   │           │ (LIVE)    │           │           │          │
│ EP-0004           │ 2026-09-… │ DATA-042  │ data_spe… │ create_i… │  ALLOW   │
│                   │           │ (LIVE)    │           │           │          │
│ EP-0003           │ 2026-09-… │ DATA-042  │ data_spe… │ rollback… │ REQUIRE… │
│                   │           │ (LIVE)    │           │           │          │
│ EP-0002           │ 2026-09-… │ DATA-042  │ data_spe… │ deploy_p… │ REQUIRE… │
│                   │           │ (LIVE)    │           │           │          │
│ EP-0001           │ 2026-09-… │ DATA-042  │ data_spe… │ create_i… │  ALLOW   │
│                   │           │ (LIVE)    │           │           │          │
└───────────────────┴───────────┴───────────┴───────────┴───────────┴──────────┘

Line-by-Line Decoding: What Each Sealed Episode Does on the Target MLOps Platform (jakatt/cac_mlops)

Cryptographic RFC 6962 Binary Merkle Tree Verification & $\mathcal{O}(\log N)$ Selective Disclosure

Beyond the sequential linear hash chain, Agentic-Ops implements a formal RFC 6962 Binary Merkle Tree (agentic_ops/core/merkle.py) with domain separation prefixes (0x00 for leaves, 0x01 for internal nodes) protecting against second-preimage collision attacks.

This architecture enables Selective Disclosure under the EU AI Act (Article 12): an auditor can verify that any specific incident episode was sealed without the organization needing to disclose any other confidential episode.

# 1. Compute and display the global 32-byte Merkle Root
$ uv run python -m agentic_ops.cli.trust_ledger merkle-root
=== Binary Merkle Tree Cryptographique (RFC 6962) ===
Nombre d'épisodes scellés : 8
Merkle Root (SHA-256)      : f3e72fddffce225ed02e7afc58892f970a8ff9b6ca7662daefedba8302bef54e
Profondeur de l'arbre      : 4 niveaux (O(log N) = 3 étapes)
Second-preimage resistant : Domain Separators 0x00 (feuilles) / 0x01 (nœuds parents)

# 2. Extract and verify the O(log N) inclusion proof for an episode
$ uv run python -m agentic_ops.cli.trust_ledger merkle-proof EP-0001
=== Preuve d'Inclusion Merkle (Audit Proof) : EP-0001 ===
Position dans l'arbre : Feuille index 0 / 8
Leaf Hash (RFC 6962)  : a6bffcb84e4992311828368a47fefd4d1c123343952bd2354517d1c5c97f487d
Merkle Root Attendue  : f3e72fddffce225ed02e7afc58892f970a8ff9b6ca7662daefedba8302bef54e
Audit Path            : 3 étapes logarithmiques
  [1] Sibling (right) : 6357c0642a0b4051d764fb5a936f9d83435fcb834b87eefb6070270a97e33f04
  [2] Sibling (right) : 94b83c0479ae726c1c0f5e4ad9e14b32b77a774a8c4d2af0685f446f4e2e148c
  [3] Sibling (right) : b6038ce4b3b198f2a8fb7b9ab5d85b0deb67b018371824eb80ec2e3831e2394e

Vérification O(log N) : VALIDE (Mathématiquement intègre & inclus dans le Merkle Root)

Core Features & Security Hardening (Red-Teaming)

This repository underwent an offensive architectural audit (Red-Teaming) and was hardened with an absolute "Fail-Closed" policy across the execution chain.

1. E2E Governance & Model Registry

• Strict Enforcement (Non-Bypassable): The Orchestrator governs the MLOps lifecycle. If PROMOTE_MODEL is classified as REQUIRE_APPROVAL, the orchestrator physically halts. No action is passed to the executor.

• Asynchronous Rehydration (HITL): The durable approval worker revalidates current roles, simulates the approved action and appends the result to the original episode. This does not enable live execution; denial reconciliation and deployment qualification remain open.

• Model Registry Gatekeeping: The Executor drives the Model Registry directly. The registry rejects invalid promotions (e.g., F1 < 0.85 or incompatible schema), forcing the Executor to return a detectable FAILED status (rather than a fake SIMULATED status). Only one model version can be ACTIVE at any given time.

2. Cryptographic Traceability (WORM Trust Ledger & RFC 6962 Merkle Tree)

• Dual-Tier Cryptographic Immutability: Combines a real-time append-only linear hash chain (previous_hash + HMAC-SHA512) with an RFC 6962 Binary Merkle Tree for batch condensation and $\mathcal{O}(\log N)$ audit proofs.

• Second-Preimage Attack Resistance: Enforces RFC 6962 domain separation bytes (0x00 for leaves, 0x01 for internal nodes) preventing node-leaf confusion attacks.

• Selective Disclosure Proofs $\mathcal{O}(\log N)$: Enables proving to a regulator (EU AI Act Article 12) that a specific decision was sealed in the ledger without disclosing proprietary inference data or telemetry from any other incident.

• Deterministic Canonicalization: Full state (Pydantic mode='json') is encoded via RFC 8785 canonical JSON (get_canonical_json) with whitespace stripped and keys sorted.

• Tamper-Evident Verification: Any attempt to manually alter a stored WORM file invalidates both the HMAC-SHA512 signature in verify_chain() and the logarithmic Merkle inclusion path in verify_merkle_proof(). Overwriting an existing episode ID is strictly blocked.

3. Distributed Security & Concurrency (Redis)

• Atomic Rate Limiting: Caps destructive actions via atomic Redis pipelines (INCR + EXPIRE). Eliminates race conditions across multi-replica deployments.

• Semantic Idempotency: Distributed Redis locks (SET NX EX) keyed on the semantic incident fingerprint (tenant:env:service:risk_id) prevent duplicate executions.

• Hardened Slack Webhook: Human approval via Slack (Block Kit) requires HMAC-SHA256 signature verification and rejects requests older than 5 minutes (Anti-Replay Attack).

4. Financial & Organizational Resilience

• FinOps Circuit Breaker: Cuts off LLM API calls during a "Monitoring Storm" (e.g., infinite alert loop) to protect cloud budgets, falling back to deterministic remediation (CREATE_ISSUE).

• Shadow Mode (Security A/B Testing): The Orchestrator evaluates shadow_policy.yaml in parallel with the primary manifest and logs divergences without ever mutating production.

• OS-Level Execution Isolation: Every LLM tool runs inside a dedicated OS process (multiprocessing.Process) with a strict timeout to prevent OOMs or hanging workers.

5. Fact-Based Deterministic Decisions (PR2)

• Policy Engine V2: The LLM (Remediation Agent) only proposes an ActionIntent (action, target). The PolicyEngine computes risk attributes (blast_radius, reversible) itself from deterministic action_catalog.yaml and risk_register.yaml definitions.

• System Facts: Actual cluster state (telemetry, OOM pods, detected drift) is formally computed via compute_facts() outside LLM subjectivity, preventing hallucinations from bypassing governance rules.

6. Adversarial Harness & Confinement (PR3)

• Prompt Guard: Cryptographic quarantine (<untrusted_data>) around logs and payloads to neutralize indirect prompt injection attacks.

• Red Team Scenarios: Strict E2E tests (known_gaps.yaml) verifying that the orchestrator rejects any forged target or unauthorized action emitted by a compromised LLM.

7. Transactional Approval Store (PR4)

• Atomic State Machine (SQLite WAL): Human approvals are governed via a BEGIN IMMEDIATE transaction lock to prevent the "Double Execution" (Double Click) vulnerability.

• Anti-Replay & Cryptography: Slack webhooks are verified via HMAC-SHA256 over the raw request payload, with every signature stored with a TTL anti-replay window.

• Governance as Code: Approvers are explicitly declared in approvers.yaml with distinct group checks for L3 critical decisions.

8. Cryptographic Trust Ledger (PR5)

• Append-Only WORM Storage: The evidence store (evidence.py) uses immutable storage where every episode is cryptographically linked to its predecessor.

• HMAC-SHA512 Signature: Any local tampering (modifying a JSON file or index) mathematically breaks the chain, making it impossible for an attacker or compromised agent to conceal a toxic action.

9. Strict Distributed Concurrency (PR6)

• Redis Idempotency Locks: To prevent duplicate remediation across replicas, an atomic distributed lock is placed on the semantic fingerprint of the incident. The lock carries a TTL and is cryptographically bound to the owner process (UUID) to prevent lock stealing.

• Fail-Closed in Production: If the concurrency coordinator (Redis) becomes unreachable, the system enters an emergency Fail-Closed halt rather than silently falling back to an unsafe local cache.

10. Operational Hardening & CI/CD (PR7)

• Least-Privilege Kubernetes RBAC: The Control Plane's Kubernetes ServiceAccount only holds get/list/watch permissions on Pods (for log reading) and patch on Deployments (strictly to restart a faulty model deployment).

• Kubernetes Reliability (Probes): The Helm chart includes livenessProbe and readinessProbe endpoints backed by native FastAPI routes (/healthz, /ready).

• Deterministic Dependencies & CI Matrix: Tested across Python 3.10, 3.11, and 3.12 via GitHub Actions with pinned container image tags.

Internal Telemetry & Observability

SRE-grade metrics exposed via Prometheus (with bounded label cardinality):

• agentic_ops_fsm_transitions_total: Tracks Finite State Machine transitions.

• agentic_ops_policy_decisions_total: Ratio of ALLOW, DENY, and REQUIRE_APPROVAL decisions.

• agentic_ops_circuit_breaker_status: FinOps circuit breaker state.

Separation of Powers (Cognitive vs. Executive Isolation)

One of the primary vulnerabilities of conventional autonomous agent frameworks is handing "network keys" directly to the AI. Agentic-Ops resolves this through a strict separation of privileges:

• Investigation Phase (AI Reads - Sapphire Zone): LLM agents are strictly restricted to read-only access (Pull Mode). They query metric databases and logs to formulate a diagnosis. They are physically incapable of mutating production and only output a structured textual "modification intent".

• Guardrail Phase (Policy & Human Gate - Amber Zone): The PolicyEngine intercepts the intent, halts the state machine, and requests human authorization via Slack.

• Execution Phase (Code Writes - Emerald Zone): The AI agent is deactivated. The rigid, deterministic Python program (Executor) takes over only after human approval. It alone holds the API credentials and triggers mutations (Push Mode).

%%{init: {'theme': 'dark', 'themeVariables': {'actorBkg': '#0f172a', 'actorBorder': '#38bdf8', 'actorTextColor': '#f8fafc', 'actorLineColor': '#64748b', 'signalColor': '#e2e8f0', 'signalTextColor': '#f8fafc', 'noteBkgColor': '#1e1b4b', 'noteBorderColor': '#c084fc', 'noteTextColor': '#f3e8ff', 'activationBkgColor': '#334155', 'activationBorderColor': '#38bdf8'}}}%%
sequenceDiagram
    autonumber
    participant Platform as MLOps Platform
    participant Agent as LLM Agent (Investigator)
    participant FSM as Orchestrator & Policy
    participant Human as Human (Slack HITL)
    participant Exec as Python Executor (Actuator)

    rect rgb(12, 45, 72)
    Note over Platform, Exec: Phase 1 : Investigation (Read-Only - Sapphire Zone)
    Platform->>FSM: Alert Webhook (Inbound)
    FSM->>Agent: Wakes Agent
    Agent->>Platform: REST Queries (GET Logs, GET Metrics)
    Platform-->>Agent: JSON (System State)
    Agent-->>FSM: Intent : "Propose triggering retraining"
    end

    rect rgb(69, 26, 3)
    Note over FSM, Exec: Phase 2 : The Guardrail (Block & Wait - Amber Zone)
    FSM->>FSM: Policy Engine intercepts (REQUIRE_APPROVAL)
    FSM->>Human: Slack Approval Request
    Note over Agent: LLM Agent is deactivated
    end
    
    rect rgb(6, 78, 59)
    Note over Human, Exec: Phase 3 : Validation & Simulation Only
    Human-->>FSM: Clicks [ APPROVE ] (Signed HMAC-SHA256)
    FSM->>Exec: Approval worker simulates the action
    Exec-->>FSM: SIMULATED result
    FSM->>FSM: Appends result to the original signed evidence episode
    end

E2E Integration Architecture (flowchart TD)

Agentic-Ops is not another MLOps platform; it is an agnostic "Control Plane" (a governance brain). It connects to your existing stack via an Event-Driven Architecture and REST APIs, operating across 4 semantic color zones:

• Sapphire Blue (Target MLOps Platform): Observed services (Evidently, Prometheus, MLflow, Prefect/Airflow, Kubernetes).

• Amethyst Purple (Cognitive & Orchestration Core): Inbound listener, Orchestrator FSM, and read-only LLM Agents.

• Amber Gold (Governance Guardrail & HITL): Deterministic Policy Engine and Slack HITL human approval gate.

• Emerald Green (Trusted Actuator): The deterministic Executor performing validated mutations.

%%{init: {'theme': 'dark', 'themeVariables': {'lineColor': '#94a3b8', 'primaryTextColor': '#f8fafc'}}}%%
flowchart TD

    %% Semantic Dark-Mode Palette
    classDef platform fill:#0c2d48,stroke:#38bdf8,stroke-width:2px,color:#e0f2fe;
    classDef cognitive fill:#2e1065,stroke:#c084fc,stroke-width:2px,color:#f3e8ff;
    classDef guardrail fill:#451a03,stroke:#f59e0b,stroke-width:2px,color:#fef3c7;
    classDef executor fill:#064e3b,stroke:#34d399,stroke-width:2px,color:#d1fae5;

    subgraph Target_MLOps_Platform ["Target MLOps Platform (Observed Production)"]
        Evidently["Drift Monitor : Evidently / Prometheus"]:::platform
        MLflow["Model Registry : MLflow"]:::platform
        Airflow["Pipeline Orchestrator : Prefect / Airflow"]:::platform
        K8s["Inference Runtime : Kubernetes"]:::platform
    end

    subgraph Agentic_Ops_Control_Plane ["Agentic-Ops Control Plane (Governance Brain)"]
        API["Webhook API Listener & Pull Collector"]:::cognitive
        FSM["Orchestrator FSM"]:::cognitive
        Agents["Specialized LLM Agents (Read-Only)"]:::cognitive
        Policy["Policy Engine & Risk Register"]:::guardrail
        Exec["Deterministic Executor & WORM Ledger"]:::executor
    end

    subgraph Human_In_The_Loop ["Human-in-the-Loop (HITL)"]
        Slack["Slack Interactive Approval (HMAC-SHA256)"]:::guardrail
    end

    style Target_MLOps_Platform fill:#071929,stroke:#38bdf8,stroke-width:2px,color:#e0f2fe
    style Agentic_Ops_Control_Plane fill:#160b30,stroke:#c084fc,stroke-width:2px,color:#f3e8ff
    style Human_In_The_Loop fill:#271102,stroke:#f59e0b,stroke-width:2px,color:#fef3c7

    Evidently -- "1. Alert / Pull Telemetry (Drift detected)" --> API
    API --> FSM
    FSM --> Agents
    Agents -- "2. Read-Only (GET Models)" --> MLflow
    Agents -- "2. Read-Only (GET Logs)" --> K8s
    Agents --> Policy
    Policy -- "3. REQUIRE_APPROVAL (Halt)" --> Slack
    Slack -- "4. Approve (Signed Webhook)" --> FSM
    FSM --> Exec
    Exec -- "5. Validated Mutation (POST API)" --> Airflow

    linkStyle 0,3,4 stroke:#38bdf8,stroke-width:2px;
    linkStyle 1,2,5 stroke:#c084fc,stroke-width:2px;
    linkStyle 6,7 stroke:#f59e0b,stroke-width:2px;
    linkStyle 8,9 stroke:#34d399,stroke-width:2px;

Catalog of Physical Actions (Executor Mutation Capabilities)

The Executor module is the only component authorized to perform write operations. All mutations require SIMULATION_MODE=False and human approval via Slack (when required by policy).

1. Code & Audit Integration (GitHub API)

Asynchronous alerting and issue tracking.

• Target Action: CREATE_ISSUE

• Mechanism: Forges a secure HTTP POST request with an access token to the GitHub API.

• Result: Instantly creates a remediation issue on the remote repository tagged mlops-incident, detailing the agent's root-cause analysis.

2. Infrastructure Remediation (Kubernetes API)

Emergency circuit-breaker action.

• Target Action: SCALE_TO_ZERO

• Mechanism: Uses the official Python kubernetes client to patch the target deployment replica count.

• Fail-Closed Security: Constrained by a strict namespace/deployment Whitelist, followed by a post-execution Read-After-Write check.

• Result: Shuts down a malfunctioning inference API to stop bad predictions immediately.

3. MLOps Pipeline Orchestration (Prefect API via Tunnel)

Automated retraining trigger.

• Target Action: TRIGGER_RETRAINING

• Mechanism: Targets the remote infrastructure over the overlay network (Tailscale) and sends a formal POST /api/deployments/{id}/create_flow_run call.

• Result: Instructs the remote orchestrator to launch the data ingestion and model retraining pipeline.

4. Model Promotion (MLflow API via Tunnel)

Production rollout action.

• Target Action: PROMOTE_MODEL

• Mechanism: Interacts with the Model Registry REST API (POST /api/2.0/mlflow/registered-models/alias) over the overlay network.

• Result: Assigns the @Production alias to a validated candidate model, switching live inference traffic.

"Dry-Run" Actions (Simulated)

Exploratory or asymmetric-risk actions are kept in strict simulation mode (logged in the WORM ledger without network mutation):

• ROLLBACK: Revert an image or model version to N-1.

• INVESTIGATE: Formally record an investigation episode without mutation.

Universal Interfacing (Agnostic MLOps Control Plane)

Beyond its live connection to jakatt/cac_mlops, Agentic-Ops is designed from the ground up as an agnostic control plane. It does not depend on the source code of the model it supervises. It only requires three elements to govern a third-party infrastructure:

1. A secure network channel (e.g., Tailscale).

2. Read-only observability endpoints or inbound alert webhooks (e.g., Loki, Prometheus, Evidently, Grafana).

3. Governed execution endpoints (e.g., Prefect, MLflow, Kubernetes, GitHub APIs).

As an illustration, here is how Agentic-Ops can plug-and-play into various open-source production MLOps architectures (without modifying their source code) to provide governed self-healing:

To demonstrate immediate interoperability across a shared engineering standard, this compatibility matrix primarily benchmarks reference MLOps capstone architectures developed within the Liora (formerly DataScientest) engineering cohorts:

Glossary of Executor Actions

To turn passive monitoring into an Auto-Remediation Control Plane, executor.py translates each validated decision into physical API calls against the target infrastructure:

• INVESTIGATE_LOGS: Autonomously retrieves logs from the failing service (e.g., via Loki or kubectl logs) and uses the LLM to synthesize a Root Cause Analysis without human intervention.

• CREATE_ISSUE: Generates a formatted ticket (GitHub/Jira/GitLab) containing incident context, deviating metrics, and recommended hyperparameter or code fixes.

• SCALE_UP / SCALE_TO_ZERO: Interacts directly with infrastructure control planes (Kubernetes HPA, AWS ASG, Docker API) to allocate compute during traffic spikes or shut down a service emitting dangerous predictions.

• TRIGGER_RETRAINING / PROPOSE_RETRAIN: Sends an asynchronous call to a pipeline orchestrator (Prefect, Airflow, GitHub Actions) with temporal parameters to trigger model retraining on fresh data.

• PROMOTE_MODEL: Once retraining is validated, transitions the candidate model from Staging to @Production in the Model Registry (MLflow).

• RESTART_PIPELINE / RESTART_PODS (restart_pod): Performs a hard reset of a stalled data job or an inference service suffering from a memory leak (OOM).

• ROLLBACK: Upon severe accuracy degradation following a rollout, immediately shifts traffic back to version N-1.

• NOTIFY_ONCALL: Alerts the on-call engineer on Slack/PagerDuty with pre-digested context, relevant logs, and the governance decision taken.

Governed Action Taxonomy: Mutating vs. Non-Mutating Actions

In Agentic-Ops, actions are not treated equally. They are formally partitioned into two distinct categories in action_catalog.yaml and autonomy_policy.yaml:

Why are Actions Categorized as "Mutating"? (HTTP GET vs. POST / PATCH)

At the transport layer, an observational action uses an idempotent HTTP GET (e.g., retrieving metrics or logs without side effects), whereas infrastructure mutations typically rely on HTTP POST, PUT, or PATCH.

However, in SRE and MLOps engineering, a mutating action is defined by whether it permanently or critically alters the physical state of the production environment:

1. Live Traffic Blast Radius:

   • GET /metrics or GET /logs: Pure observation. Consumes negligible read bandwidth, mutates 0 bytes of state (Non-Mutating).

   • PATCH /spec/replicas (Kubernetes) or POST /api/2.0/mlflow/registered-models/alias (MLflow): Physically changes running container count or immediately redirects 100% of live user traffic to another model candidate (Mutating).

2. FinOps and Asymmetric Cost:

   • create_issue: Formally sends a POST /repos/{owner}/{repo}/issues to GitHub or Jira. Although technically an HTTP write, it is non-destructive for production runtime. If the ticket is unnecessary, user traffic is unharmed.

   • propose_retrain: Triggers an asynchronous GPU training job (POST /api/deployments/{id}/create_flow_run on Prefect). It consumes significant cloud compute budgets, locks database tables, and produces new model binaries (Mutating).

3. Cognitive vs. Executive Isolation (Why LLMs Have Zero Direct Mutation Access):

   • LLMs are probabilistic text generators subject to hallucinations and prompt injections.

   • Handing direct Kubernetes or cloud provider credentials to an LLM creates an unmitigated attack surface.

   • Therefore, the LLM is physically quarantined to read-only tools (GET). It can only output a structured textual intent (ActionIntent). The physical POST or PATCH is exclusively delegated to hardened, deterministic Python actuators (KubernetesExecutionAdapter, PrefectRetrainAdapter, MLflowPromotionAdapter) after strict policy validation and human quorum approval.

Concrete 5-Step Incident Lifecycle Walkthrough (e.g., Feature Drift DATA-042)

When a production incident occurs, the system executes an unambiguous 5-step transition sequence:

+------------------------+        +--------------------------+        +------------------------+
| 1. LLM (Investigator)  |  --->  | 2. PolicyEngine & SRE    |  --->  | 3. Executor (Python)   |
| Read-Only (HTTP GET)   |        | Guardrail (Veto/Slack)   |        | Write (POST / PATCH)   |
+------------------------+        +--------------------------+        +------------------------+

1. Step 1: Inbound Alert & Episode Ingestion (Non-Mutating):

   • Prometheus or Evidently detects covariate shift and dispatches an alert payload to POST /webhook/alert.

   • The Orchestrator assigns a monotonically increasing episode ID (EP-0004) linked to the cryptographic linear hash chain (previous_hash).

2. Step 2: Read-Only Cognitive Investigation (Non-Mutating - 100% GET):

   • The specialized LLM agent wakes up in a sandboxed process.

   • The agent inspects runtime telemetry exclusively via read-only endpoints:

     • GET /metrics on Prometheus.

     • GET /loki/api/v1/query_range for error stack traces.

     • GET /api/2.0/mlflow/runs for model lineage.

   • The LLM synthesizes a diagnosis and emits a proposed action intent: {action: "restart_pod", target: "baac-serving"} or {action: "propose_retrain", target: "lgbm_accidents"}.

3. Step 3: Deterministic Policy Engine Evaluation (Gating & Veto):

   • The PolicyEngine intercepts the intent before any network call is dispatched.

   • Veto Rule R-001 (upstream_masks_service): The active anomaly is an upstream data/feature drift (DATA-042). Restarting an inference pod or retraining on contaminated inputs cannot resolve corrupted data feeds. The engine immediately issues an unconditional DENY for restart_pod.

   • If the proposed intent is create_issue: Class ticket, L1 autonomy -> Immediate autonomous ALLOW (0 approvals required).

   • If the proposed intent is propose_retrain: Class model_mutation, L3 autonomy -> Engine halts in REQUIRE_APPROVAL (2 human SRE approvals required).

4. Step 4: Interactive Slack Human-in-the-Loop (Slack HITL):

   • The Orchestrator formats a Slack Block Kit alert displaying incident telemetry, root cause, and [ APPROVE ] / [ DENY ] interactive buttons.

   • The state machine remains physically halted in AWAITING_APPROVAL. No mutating network payload can be emitted until human quorum is satisfied.

5. Step 5: Hardened Actuation & Cryptographic WORM Sealing (Mutating - POST / PATCH):

   • An authorized on-call engineer clicks [ APPROVE ]. The webhook payload is verified via HMAC-SHA256 (rejecting replays older than 5 minutes).

   • The deterministic KubernetesExecutionAdapter takes over, validates the mutation against ALLOWED_PATCH_PATHS, and issues the physical PATCH /apis/apps/v1/namespaces/default/deployments/baac-serving (or POST to Prefect).

   • The execution receipt (status: SUCCESS), approver identity, and post-flight health checks are sealed into the append-only ledger.jsonl with an HMAC-SHA512 cryptographic signature.

Model Context Protocol (MCP) Governance Gateway: Live /console Reader, 8 Tools, 5 Resources & 20 Starter Prompts

Dedicated Technical Reference: docs/MCP_SERVER_GUIDE.md | Source: agentic_ops/entrypoints/mcp_server.py | Test Suite: tests/entrypoints/test_mcp_server.py (12 passed)

In enterprise AI operations, connecting autonomous IDE or desktop clients (Claude Desktop, Cursor, Windsurf, Google Antigravity) directly to infrastructure MCP servers (Kubernetes, PostgreSQL, GitHub) creates an unmitigated production hazard: the LLM can issue destructive mutations without deterministic policy checks.

Agentic-Ops solves this by exposing the Control Plane itself as a Deterministic MCP Governance Firewall (protocolVersion: "2024-11-05", dual-transport STDIO + HTTP JSON-RPC 2.0 on /mcp):

1. What Can the MCP Server Do? (The 8 Governed MCP Tools & 5 MCP Resources)

• Live /console Reader (read_console_diagnostic): Directly executes and reads any diagnostic phase from the /console Web Terminal (sandbox_phase_a_vitals through pytest_all), returning either compact PASSED lines (include_guides=False) or the full color-coded pedagogical commentary ([GUIDE:ROLE], [GUIDE:AGENT], [GUIDE:SLACK], [GUIDE:PROD], [GUIDE:HOW_IT_PASSES]).

• Deterministic Pre-Flight Firewall (evaluate_governance_policy): Evaluates any proposed agent action (create_issue, restart_pod, scale_up, propose_retrain) against autonomy_policy.yaml, action_catalog.yaml, and risk_register.yaml (R-001 upstream data mask veto, R-002 forbidden risk actions, 1/hour rate limits), returning ALLOW, REQUIRE_APPROVAL (Slack HITL), or DENY.

• End-to-End WORM Incident Sealing (run_governed_incident_episode & inspect_worm_trust_ledger): Runs a full incident episode, seals it in ledger.jsonl with a canonical SHA-256 linear hash chain (previous_hash) and HMAC-SHA512 cryptographic signature, and verifies the entire chain via EvidenceStore.verify_chain().

• Foundational Knowledge Search (search_failure_taxonomy & get_scenario_blueprint): Queries the 435 Failure Modes Taxonomy (docs/mlops_failure_modes_taxonomy.md) and the 36 War-Room Scenarios Registry (docs/mlops_simulation_scenarios_registry.md + tests/scenarios/data/).

• Live Tailscale Read-Only Pull (poll_target_mlops_snapshot & inspect_control_plane_vitals): Triggers CacPullCollector.collect_all() (100% HTTP GET) across the 5 observability sources of jakatt/cac_mlops over the Tailscale SOCKS5 mesh (127.0.0.1:1055).

• 5 Native MCP Resources: agentic-ops://taxonomy/failure-modes, agentic-ops://registry/scenarios, agentic-ops://governance/autonomy-policy, agentic-ops://governance/known-gaps, and agentic-ops://console/vitals.

2. How to Connect (STDIO Local Client vs. Remote Cloud HTTP JSON-RPC 2.0)

A. Connect Claude Desktop Locally (STDIO on macOS & Windows)

Option 1: Native uvx Configuration (Recommended — Zero-install, uses PyPI) Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "agentic-ops": {
      "command": "uvx",
      "args": ["--from", "agentic-ops", "agentic-mcp"]
    }
  }
}

Option 2: Containerized via Docker Hub (Zero Python dependencies)

{
  "mcpServers": {
    "agentic-ops": {
      "command": "docker",
      "args": ["run", "-i", "--rm", "nolll78/agentic-ops:latest", "agentic-mcp"]
    }
  }
}

B. Connect Cursor, Windsurf or Google Antigravity (STDIO or Remote Cloud)

Configure .cursor/mcp.json or mcp_config.json:

{
  "mcpServers": {
    "agentic-ops-remote": {
      "url": "https://agentic-ops-5ebv.onrender.com/mcp"
    }
  }
}

C. Connect ChatGPT Web (chatgpt.com) via Custom GPT Action (OpenAPI / REST)

OpenAI's web interface supports OpenAPI 3.0 Actions. In ChatGPT -> My GPTs -> Create a GPT -> Configure -> Actions -> Create new action:

• Click Import from URL and paste: https://agentic-ops-5ebv.onrender.com/openapi.json

• Privacy Policy: https://agentic-ops-5ebv.onrender.com/healthz

• ChatGPT will automatically ingest all live endpoints (/api/diagnostics/{command}, /api/cac/snapshot, /api/tailscale/status, /mcp/manifest).

D. Connect Directly via Terminal / Remote Cloud (HTTP JSON-RPC 2.0 on Render)

# 1. Inspect the live MCP manifest (8 Tools, 5 Resources, 20 Starter Prompts)
curl -s https://agentic-ops-5ebv.onrender.com/mcp/manifest | jq .counts

# 2. Read Phase D (Live Agentic Scenarios) from /console via MCP JSON-RPC 2.0
curl -s -X POST https://agentic-ops-5ebv.onrender.com/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"read_console_diagnostic","arguments":{"command":"sandbox_phase_d_mlops_scenarios","include_guides":false}}}' | jq -r '.result.content[0].text'

E. Connect LibreChat Web UI (OpenRouter AI Hub + Agentic-Ops MCP Gateway)

LibreChat is the open-source ChatGPT equivalent supporting both OpenRouter models and the MCP Protocol. By mounting the included librechat.yaml configuration file into your LibreChat container (volumes: [./librechat.yaml:/app/librechat.yaml]), any chosen OpenRouter model (deepseek/deepseek-r1:free, meta-llama/llama-3.3-70b-instruct:free, mistralai/mistral-large-2411) gains full direct tool-calling access to the live Render Control Plane via https://agentic-ops-5ebv.onrender.com/mcp.

3. Complete Catalog of the 20 Cognitive Starter Prompts (prompts/list & prompts/get)

Universal LLM Provider Hub: Decoupled Reasoning Across 8 AI Providers

Dedicated Technical Reference: docs/OPENROUTER_FREE_MODELS_GUIDE.md | Source Module: agentic_ops/adapters/llm_provider_registry.py | Config Template: .env.example | Test Suite: tests/adapters/test_llm_provider_registry.py (15 passed)

To eliminate AI provider vendor lock-in and enable zero-cost 24/7 cloud operations, Agentic-Ops decouples the cognitive reasoning engine (RemediationAgent & specialized diagnostic agents) from the physical model implementation through LLMProviderRegistry and llm_protocol.py.

The active brain can be swapped dynamically via environment variables (LLM_PROVIDER, LLM_MODEL) without modifying a single line of application code:

+--------------------------------------------------------------------------------------------------+
| AGENTIC-OPS REASONING ENGINE (RemediationAgent / SpecialistAgents / Pydantic ActionIntent)       |
+------------------------------------------------+-------------------------------------------------+
                                                 | (Standard LLMPort Protocol)
                                                 v
+--------------------------------------------------------------------------------------------------+
| LLM PROVIDER REGISTRY (agentic_ops/adapters/llm_provider_registry.py)                            |
| Resolves: LLM_PROVIDER -> Profiles, Masked Keys, Prometheus Metrics, Auto-Fallback Chain        |
+------------------------------------------------+-------------------------------------------------+
                                                 |
         +---------------------------------------+---------------------------------------+
         |                                       |                                       |
         v                                       v                                       v
+---------------------------------+ +---------------------------------+ +---------------------------------+
| 1. OPENROUTER (:free 24/7 Cloud)| | 2. OLLAMA (Local / Tailscale)   | | 3. PROPRIETARY & ENTERPRISE     |
| - deepseek/deepseek-r1:free     | | - Local: http://localhost:11434 | | - Google Gemini (1.5 Pro/Flash) |
| - deepseek/deepseek-chat:free   | | - Mesh:  http://100.x.y.z:11434 | | - OpenAI (GPT-4o, GPT-4o-mini)  |
| - meta-llama/llama-3.3-70b:free | | - Cloudflare Tunnel (HTTPS)     | | - Anthropic (Claude 3.5 Haiku)  |
| - google/gemini-2.0-flash:free  | | - Vast.ai Cloud GPU VPS         | | - Azure OpenAI & Mistral AI     |
| -> Cost: 0 EUR (No billing card)| | -> Cost: 0 EUR (Native JSON)    | | -> Cost: Usage-based / Tiered   |
+---------------------------------+ +---------------------------------+ +---------------------------------+

1. The 8 Supported AI Model Providers & Cost Comparison

2. Zero-Downtime Resilience: Automatic Fallback Chain (FallbackLLMAdapter)

Free tiers on public clouds may occasionally encounter rate limits (HTTP 429 Too Many Requests). To prevent an incident diagnosis from stalling:

1. The primary model is queried first (e.g. openrouter/google/gemma-4-31b-it:free).

2. If it raises a rate-limit or timeout exception, FallbackLLMAdapter instantly failovers to the secondary model configured in LLM_FALLBACK_MODELS (e.g. openrouter/google/gemma-4-26b-a4b-it:free), then to openrouter/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free, openrouter/nvidia/nemotron-3-super-120b-a12b:free, and openrouter/openrouter/free.

3. Every individual attempt is recorded in Prometheus metrics (agentic_ops_llm_requests_total{provider=..., model=..., status=...}) for FinOps auditability.

3. SRE Observability & Live Probing Endpoints

The Control Plane exposes 5 dedicated REST inspection and conversational endpoints:

• GET /api/llm/status: Inspects active provider, resolved model, fallback chain, masked key status (sk-or-***), and complete roles_topology.

• GET /api/llm/models: Returns the complete 8-provider catalog and recommendations for zero-cost operation.

• POST /api/llm/probe: Actively queries the configured LLM and measures end-to-end roundtrip latency in milliseconds.

• GET /api/prompts: Returns the complete JSON catalog of the 20 MCP Cognitive Starter Prompts.

• GET /chat & POST /api/chat: Standalone browser chat interface embedding a searchable 20-prompt catalog modal, interactive #llm-status-badge, navigation button, and 1-click Live MAS Topology & Agent Inspector.

• Console Button (Phase H): Interactive 1-click validation of all LLM Hub features from /console.

4. Multi-LLM Cognitive Specialization by Agent Role

Dedicated Technical Reference: docs/MULTI_LLM_ROLE_SPECIALIZATION_GUIDE.md | OpenRouter Free Tier Guide: docs/OPENROUTER_FREE_MODELS_GUIDE.md

Deploying a single undifferentiated LLM across all SRE and MLOps tasks is an architectural compromise. To eliminate cognitive trade-offs, Agentic-Ops introduces Cognitive Specialization by Role inside LLMProviderRegistry and Orchestrator:

• Native Distributed Intelligence Out-of-the-Box: Without setting a single environment variable, each agent is natively routed to its expert LLM above. If LLM_MODEL is explicitly defined, it serves as a universal override; if LLM_MODEL_{ROLE} is defined, it overrides that specific agent.

• Why These Specific Default Models? (Architectural Rationale):

  • remediation $\rightarrow$ Nemotron-3 Reasoning (30B): Deliberate Chain-of-Thought reasoning within <think> tags to evaluate blast radius ($0-5$), system dependencies, and action reversibility before mutating production.

  • infra $\rightarrow$ Gemma-4 31B: Massive 256,000-token context window to ingest complete Loki log dumps, systemd journals, and Linux kernel traces in a single pass without truncation.

  • data $\rightarrow$ Gemma-4 26B (MoE): Ultra-low latency and deterministic Pydantic JSON extraction for sub-second schema validation (Pandera) and covariate drift triage (Evidently).

  • model $\rightarrow$ Nemotron-3 Super (120B): High-capacity 120B parameter statistical rigor to differentiate genuine concept drift ($P(Y \mid X)$) from transient input noise ($P(X)$) on minority classes (fatalities/injuries).

  • chat $\rightarrow$ Gemma-4 31B: Instantaneous multimodal conversational streaming (sub-second TTFT) for smooth operator assistance on /chat.

• Granular FinOps Metrics: Prometheus isolates token consumption and latency per role: agentic_ops_llm_requests_total{role="infra", provider="openrouter", model="gemma-4-31b-it", status="success"}.

• Live Topology: Active roles and their models are dynamically inspectable via GET /api/llm/status under the roles_topology key.

5. Multi-Agent System (MAS) Agent Selection & Default Routing Architecture

Dedicated Technical Reference: docs/MAS_AGENT_ROUTING_AND_DEFAULTS_GUIDE.md | Role Specialization Guide: docs/MULTI_LLM_ROLE_SPECIALIZATION_GUIDE.md

To avoid the single-agent bottleneck and prevent judicial conflicts of interest (the "judge vs. executioner" anti-pattern), Agentic-Ops decouples diagnostic investigation from remediation planning across 5 specialized agent roles:

                                  +---------------------------------------+
                                  |         Incoming Alert Signal         |
                                  |         (risk_id, signal_id)          |
                                  +---------------------------------------+
                                                      |
                                                      v
                                        +---------------------------+
                                        |  Orchestrator (Router)    |
                                        | _get_specialist(risk_id)  |
                                        +---------------------------+
                                                      |
                     +--------------------------------+-------------------------------+
                     |                                |                               |
            "MODEL" in risk_id               "INFRA" in risk_id                 Unmatched / Other
                     |                                |                               |
                     v                                v                               v
           +--------------------+           +--------------------+           +--------------------+
           |  ModelSpecialist   |           |  InfraSpecialist   |           |   DataSpecialist   |
           | (role="model")     |           | (role="infra")     |           | (role="data")      |
           | MLflow / F1 Drift  |           | Loki / Kernel Logs |           | [PLATFORM DEFAULT] |
           +--------------------+           +--------------------+           +--------------------+
                     \                                |                              /
                      \                               |                             /
                       +------------------------------+----------------------------+
                                                      |
                                                      v Emits Diagnosis (Pydantic Contract)
                                      +-------------------------------+
                                      |       RemediationAgent        |
                                      | (role="remediation")          |
                                      | CoT / Blast Radius Evaluator  |
                                      +-------------------------------+
                                                      | Emits ActionIntent (Pydantic Contract)
                                                      v
                                      +-------------------------------+
                                      |         PolicyEngine          |
                                      |    (Fail-Closed OPA Gate)     |
                                      +-------------------------------+

Why Other System Actors Have No LLM (Deterministic Actuators vs. Cognitive Brains)

Dedicated Technical Reference: docs/MAS_AGENT_ROUTING_AND_DEFAULTS_GUIDE.md#6-architecture-of-non-cognitive-components-why-other-system-actors-have-no-llm-deterministic-actuators-sensors--circuit-breakers

While the 5 cognitive roles above are powered by specialized LLMs, all other system components, actuators, sensors, and safety nets operate with ZERO LLMs (0-Token, 100% Deterministic Python) to satisfy the Zero-Trust security model:

Live MAS Topology & Agent LLM Inspector in Web Console and Web Chat

Dedicated Technical Reference: docs/MAS_AGENT_ROUTING_AND_DEFAULTS_GUIDE.md#7-interactive-live-inspector-inspecting-mas-agents--active-llms-in-real-time-console--chat | API Endpoint: GET /api/mas/topology

To provide radical transparency during live operations, incident post-mortems, and jury audits, Agentic-Ops embeds an interactive Live MAS Topology & Agent Inspector directly within both the SRE Cognitive Web Console (/console) and the Standalone Web Chat (/chat):

• Header Badge & Purple Console Button: In /console, operators can click either the header badge [MAS Topology & LLMs Live] or the primary purple command button to inspect all 11 system components in real time.

• Interactive Chat Badge & Quick-Action Chip: In /chat, clicking the #llm-status-badge, the top navigation button [MAS Topology], or the suggestion chip [Inspecter les LLMs des Agents MAS] opens the inspector modal without leaving the ongoing conversation.

+--------------------------------------------------------------------------------------------------+
|  MAS ARCHITECTURE & LIVE COMPONENT INSPECTOR                                       [ X Close ]  |
+--------------------------------------------------------------------------------------------------+
|  [Filter: All (11)]  [Filter: Cognitive Brains (5)]  [Filter: Deterministic Actuators (6)]       |
+------------------------------------+-------------------------------------------------------------+
|  COMPONENT SELECTION LIST          |  DEDICATED COMPONENT DOSSIER ("ENCART DEDIE")               |
|                                    |                                                             |
|  [*] RemediationAgent              |  [Role: RemediationAgent]  | Domain: SRE Incident Recovery  |
|      deepseek-r1:free (CoT)        |  Status: LIVE ONLINE       | Type: Cognitive LLM Brain      |
|                                    +-------------------------------------------------------------+
|  [ ] InfraSpecialist               |                                                             |
|      gemma-4-31b-it:free (256k)    |  1. IDENTITY & OPERATIONAL MISSION                          |
|                                    |     - Responsibility: Causal inference & blast radius plan  |
|  [ ] DataSpecialist                |     - FSM Trigger: Executed unconditionally in [PLANNED]    |
|      gemma-4-26b-a4b-it:free (MoE) |     - Judicial Safeguard: Anti-bias separation of powers    |
|                                    |                                                             |
|  [ ] ModelSpecialist               |  2. ACTIVE ENGINE & INFERENCE CONFIGURATION                 |
|      llama-3.3-70b-instruct:free   |     - Active Model: openrouter/deepseek/deepseek-r1:free    |
|                                    |     - Reasoning Mode: Chain-of-Thought (<think> filtered)   |
|  [ ] WebChatAssistant              |     - Cascading Fallbacks: deepseek-chat -> llama-3.3-70b   |
|      deepseek-chat:free            |     - Hyperparameters: Temp = 0.0 | MaxTokens = 4096 | 30s  |
|                                    |     - Override Variable: LLM_MODEL_REMEDIATION              |
|  [ ] CircuitBreakerFallback        |     - Architectural Rationale: Only free model with deep    |
|      0-Token Pure Python           |       causal CoT capability for blast radius assessment.    |
|                                    |                                                             |
|  [ ] ActionExecutor                |  3. DATA CONTRACTS & SYSTEM DEPENDENCIES                    |
|      0-Token Subprocess Sandbox    |     - Input Contract: Pydantic Diagnosis                    |
|                                    |     - Output Contract: Pydantic ActionIntent                |
|  [ ] CacPullCollector             |     - Connected Services: Redis (:6379), OPA PolicyEngine   |
|      0-Token Telemetry Ingestion   |     - Permitted Tools: None (Judicially isolated)           |
|                                    |                                                             |
|  [ ] SlackWebhookAgent             |  4. SECURITY, BLAST RADIUS & FINOPS OBSERVABILITY           |
|      0-Token HMAC Gate             |     - Authorized Blast Radius: 0 to 5 (PolicyEngine veto)   |
|                                    |     - WORM Ledger: Cryptographically sealed (SHA256 + RFC 6962)|
|  [ ] 8 AgentTools                  |     - PromQL Metric: agentic_ops_llm_requests_total{role=""} |
|      0-Token Sensory Probes        |     - Failure Domain: Trips CircuitBreaker on 429/timeout   |
|                                    |                                                             |
|  [ ] MCPServerGateway              |  5. INTERACTIVE LIVE ACTIONS                                |
|      JSON-RPC 2.0 Bridge           |     [ Test LLM Probe (/api/llm/probe) ]                     |
|                                    |     [ Copy PromQL Metric Query ]                            |
+------------------------------------+-------------------------------------------------------------+

The 5 Structural Sections of the Dedicated Inspector Card ("Encart Dédié")

1. Identity & Operational Mission: SRE domain responsibility, FSM trigger condition, and judicial anti-bias separation of powers (or least-privilege invariant).

2. Active Motorisation (Cognitive LLM vs. Deterministic 0-Token Engine): Resolved runtime model, provider, reasoning mode (deliberate CoT <think> vs. direct JSON), active temperature, token ceiling, timeout, override env var, cascading fallback chain, and selection rationale. For non-cognitive components, explicit confirmation of 0-Token Pure Python execution, OS process isolation (multiprocessing.Process), and hard deadline (SIGTERM 30s).

3. Data Contracts & System Dependencies: Pydantic input/output contracts (Diagnosis, ActionIntent, ExecutionResult), connected services (Redis, Docker socket, MLflow, Loki, PostgreSQL), and permitted tool inventory.

4. Security, Blast Radius & FinOps Observability: Authorized blast radius limit ($0$ for investigation up to $5$ for remediation), WORM Ledger cryptographic verification status, dedicated Prometheus FinOps metric query (agentic_ops_llm_requests_total{role="..."}), and failure domain behavior.

5. Interactive Live Controls: 1-click active LLM latency probe (/api/llm/probe) and 1-click PromQL metric query clipboard copy.

Render Free Zero-Overhead Feasibility Proof (0.1 vCPU / 512 MB RAM)

6. Multi-Agent System (MAS) Fallback Architecture & Mapping Keys

Dedicated Technical Reference: docs/MAS_FALLBACK_ARCHITECTURE_GUIDE.md | Cognitive Specialization Guide: docs/MULTI_LLM_ROLE_SPECIALIZATION_GUIDE.md

An autonomous Multi-Agent System operating in production must handle two distinct failure domains:

1. What happens if an agent's default LLM bugs? (HTTP 429 rate limit, timeout, provider outage) $\rightarrow$ Handled transparently by FallbackLLMAdapter without interrupting the diagnosis.

2. What happens if an agent or the orchestrator bugs? (Unclassified risk taxonomy, LLM budget spike, action hallucination, remote VPS drop) $\rightarrow$ Handled by deterministic Fail-Safe and Fail-Closed state transitions.

+--------------------------------------------------------------------------------------------------+
| DUAL-TIER RESILIENCE: LLM MODEL FAILOVER VS. AGENT ORCHESTRATION FAILSAFE                        |
+--------------------------------------------------------------------------------------------------+
|                                                                                                  |
| [TIER 1: LLM MODEL BUG (API / Quota / Timeout)]                                                  |
| - RemediationAgent   : deepseek-r1:free     ──(Fails ?)──> deepseek-chat ──> llama-3.3-70b ...   |
| - InfraSpecialist    : gemma-4-31b-it:free  ──(Fails ?)──> deepseek-r1   ──> deepseek-chat ...   |
| - DataSpecialist     : gemma-4-26b-it:free  ──(Fails ?)──> deepseek-r1   ──> deepseek-chat ...   |
| - ModelSpecialist    : llama-3.3-70b:free   ──(Fails ?)──> deepseek-r1   ──> deepseek-chat ...   |
| - WebChatAssistant   : deepseek-chat:free   ──(Fails ?)──> deepseek-r1   ──> llama-3.3-70b ...   |
|                                                                                                  |
| [TIER 2: AGENT SYSTEM BUG (Logic / Storm / Network)]                                             |
| - Unmatched Risk ID  : Automatic fallback to DataSpecialist (safely triages data corruption)    |
| - Token Storm / Loop : Circuit Breaker trips (>50 req/h) -> CIRCUIT_BREAKER_FALLBACK (Issue only)|
| - Hallucinated Action: PolicyEngine default_effect: "deny" -> Fail-Closed / Slack HITL Approval |
| - VPS / Tailscale Drop: ConnectError caught -> status="offline" -> Fail-Closed mutation lock     |
|                                                                                                  |
+--------------------------------------------------------------------------------------------------+

Surgical Execution Flow: How an Agent Switches to a Fallback LLM at Runtime

Dedicated Technical Reference: docs/MAS_FALLBACK_ARCHITECTURE_GUIDE.md#a-surgical-switch-mechanics--runtime-lifecycle-sequence--latency-optimization | Source Module: agentic_ops/adapters/llm_protocol.py

When an autonomous agent encounters an upstream LLM failure (HTTP 429 rate limit, timeout, provider outage), the switch to a backup model is entirely autonomous, non-blocking, and transparent to the agent's internal business logic:

+-------------------------------------------------------------------------------------------------------+
| SURGICAL SEQUENCE FLOW: AGENT FAILOVER TO BACKUP LLM AT RUNTIME                                       |
+-------------------------------------------------------------------------------------------------------+

 [ Cognitive Agent ]          [ FallbackLLMAdapter ]         [ Primary Adapter ]        [ Fallback Adapter ]
  (e.g., ModelSpecialist)                                     (DeepSeek-R1)              (DeepSeek-Chat)
          |                            |                              |                          |
          | 1. complete(Diagnosis)     |                              |                          |
          |--------------------------->|                              |                          |
          |                            | 2. complete()                |                          |
          |                            |----------------------------->|                          |
          |                            |                              | HTTP 429 / Timeout / 5xx |
          |                            |                              | (Fails after retries)    |
          |                            |                              |-- - - - - - - - - - - - -+
          |                            | 3. Intercept Exception       |                          |
          |                            |<-----------------------------|                          |
          |                            |    Prometheus error inc()                               |
          |                            |                                                         |
          |                            | 4. Immediate Fallback (max_retries=1)                   |
          |                            |-------------------------------------------------------->|
          |                            |                                                         | 200 OK + JSON
          |                            | 5. Validate Pydantic Schema (Diagnosis)                | (via _strip_think_tags)
          |                            |<--------------------------------------------------------|
          |                            |    Prometheus success inc()                             |
          | 6. Return typed Diagnosis  |                                                         |
          |<---------------------------|                                                         |
          | (Agent unaware of failure) |                                                         |
          v                            v                                                         v

1. Hexagonal Inversion (LLMPort): Agents (ModelSpecialist, InfraSpecialist, RemediationAgent) never import vendor SDKs directly; they interact exclusively with the abstract LLMPort interface.

2. Cascading Latency Prevention (max_retries=1): While primary adapters run a multi-attempt exponential backoff loop, fallback adapters are configured with max_retries=1. This guarantees that failover occurs in < 50 ms without compounding latency delays.

3. Structured Output Sanitization: Reasoning models emitting chain-of-thought envelopes (<think>...</think>) are automatically sanitized by _strip_think_tags() before Pydantic schema validation.

4. Independent Prometheus Telemetry: Failures and successes are tracked on separate metric labels (agentic_ops_llm_requests_total{status="error|success"} and agentic_ops_llm_latency_seconds), ensuring immediate SRE observability.

5. Deterministic Circuit Breaker: If all models in the chain fail, LLMCircuitBreaker locks the cluster in Fail-Closed mode (default_effect: "deny"), issues ActionType.CREATE_ISSUE via pure Python CIRCUIT_BREAKER_FALLBACK (0 tokens), and signs the event in the WORM Trust Ledger.

Provider Independence & Disaster Recovery: What If OpenRouter Cuts Access or Changes Policies?

Dedicated Technical Reference: docs/MAS_FALLBACK_ARCHITECTURE_GUIDE.md#d-provider-independence--disaster-recovery-what-if-openrouter-cuts-access-or-changes-policies | Source Module: agentic_ops/adapters/llm_provider_registry.py

Relying on a single third-party API aggregator like OpenRouter as a hard dependency is an enterprise anti-pattern. In production, providers can revoke free tiers, alter Terms of Service, experience upstream peering outages, or enforce aggressive rate limits.

In Agentic-Ops, OpenRouter is purely a zero-cost bootstrapping hub, not an architectural lock-in. The system is built around the dependency inversion principle (LLMPort) and LiteLLM protocol abstraction, supported by 4 strictly isolated lines of defense:

+--------------------------------------------------------------------------------------------------+
| AGENTIC-OPS DISASTER RECOVERY: WHAT HAPPENS IF OPENROUTER CUTS THE CORD?                         |
+--------------------------------------------------------------------------------------------------+
|                                                                                                  |
| [NOMINAL STATE]                                                                                  |
| OpenRouter Free Tier (:free)                                                                     |
| DeepSeek-R1, Gemma-4 31B, Gemma-4 26B, Llama 3.3 70B, DeepSeek-Chat                              |
|                                                                                                  |
|                                     │                                                            |
|                                     ▼ (Event: HTTP 429, 401, Quota Wipeout, Policy Revocation)   |
|                                                                                                  |
| [LINE 1: ZERO-CODE ENV SWITCH] ────► Switch provider in 30s without redeploying code:            |
|                                      LLM_PROVIDER=gemini (Google Gemini Flash — 60 req/min free)  |
|                                      LLM_PROVIDER=huggingface (Serverless Inference API — 0 EUR) |
|                                                                                                  |
| [LINE 2: 100% SOVEREIGN SANCTUARY] ─► Off-Cloud, Zero-Vendor Operation via Tailscale Mesh:        |
|                                      LLM_PROVIDER=ollama                                         |
|                                      OLLAMA_BASE_URL=http://100.117.99.62:11434 (Scaleway VPS)   |
|                                      (Runs deepseek-r1:7b or llama3.2:3b at 0 EUR, 100% private)  |
|                                                                                                  |
| [LINE 3: GRANULAR ROLE SPLIT] ─────► Isolate providers per agent role:                           |
|                                      LLM_MODEL_REMEDIATION=ollama/deepseek-r1:7b                 |
|                                      LLM_MODEL_INFRA=gemini/gemini-1.5-flash-latest (1M tokens)   |
|                                      LLM_MODEL_DATA=huggingface/mistralai/Mistral-7B-Instruct     |
|                                                                                                  |
| [LINE 4: ZERO-LLM SAFETY NET] ─────► Total Internet / AI Outage Fallback:                       |
|                                      CIRCUIT_BREAKER_FALLBACK (0-Token Pure Python)              |
|                                      Forces ActionType.CREATE_ISSUE | Locks cluster Fail-Closed  |
|                                                                                                  |
+--------------------------------------------------------------------------------------------------+

Post-OpenRouter Migration Reference Matrix

Radical Honesty: Code Implementation vs. Live Production Reality

Under the project's Radical Honesty rule, we differentiate what is fully engineered in code from what is active in production at this instant:

Concrete Proof of Dynamic Hybrid Fallback Execution

The codebase natively chains disparate provider backends. Below is the verified output from executing a heterogeneous multi-provider chain directly in the active project virtual environment:

Primary adapter : LiteLLMAdapter (openrouter/deepseek/deepseek-r1:free)
Fallbacks count : 2
  ├── Fallback 0: LiteLLMAdapter (gemini/gemini-1.5-flash-latest)
  └── Fallback 1: OllamaAdapter (deepseek-r1:7b -> http://100.117.99.62:11434)

Operational Disaster Recovery Runbooks

• Runbook A: 30-Second Cloud Migration to Google Gemini (0 EUR, 60 req/min, 1M Context): Set LLM_PROVIDER=gemini and GOOGLE_API_KEY=AIzaSy... in the Render dashboard. The container restarts in 10s and 100% of the MAS immediately executes on Gemini Flash.

• Runbook B: Migration to the Sovereign Ollama Sanctuary (0 EUR, 100% Private): Run ollama run deepseek-r1:7b on the Scaleway VPS, then set LLM_PROVIDER=ollama and OLLAMA_BASE_URL=http://100.117.99.62:11434. Traffic routes privately over Tailscale :1055 / :11434 with zero external SaaS exposure.

• Runbook C: Total AI Blackout / Complete Network Disconnection: If all providers fail, LLMCircuitBreaker automatically trips, engages CIRCUIT_BREAKER_FALLBACK (0-Token, pure Python), issues non-mutating ActionType.CREATE_ISSUE, enters Fail-Closed mode (default_effect: "deny"), and signs the event in the WORM Trust Ledger.

Operational Complexity & Human Effort Matrix (Zero-Toil Guarantee)

Test Suite Verification

Verified by 16 automated tests in tests/adapters/test_llm_provider_registry.py: test_all_8_providers_are_registered, test_ollama_adapter_mocked_success, test_strip_think_tags_removes_r1_reasoning_blocks, test_fallback_chain_on_primary_failure, and test_fallback_chain_all_failing_raises_runtime_error.

7. Radical Honesty: The 2 Operational Pitfalls of :free Cloud Models & Built-in Defenses

Applying the project's Radical Honesty rule, utilizing zero-cost public endpoints entails two concrete technical constraints that have been engineered into Agentic-Ops:

1. Rate-Limiting (Request/Minute Throttling):

   • The Reality: OpenRouter :free tiers enforce strict thresholds (typically 20 requests/minute and ~200 requests/day per IP/token).

   • Evaluation: This quota is more than sufficient for live demonstrations, running the full scenario test suites, and academic defenses. However, in an enterprise receiving 10,000 monitoring alerts/minute, production systems must route to paid endpoints (which remain extraordinarily inexpensive: DeepSeek-V3 costs $0.14 per million input tokens, approximately 50x cheaper than OpenAI GPT-4o).

   • Architectural Defense: When a rate-limit error (HTTP 429) is caught, FallbackLLMAdapter automatically redirects traffic to the next model in LLM_FALLBACK_MODELS without interrupting the active FSM transition.

2. Structured Output Integrity (Pydantic / <think> Blocks):

   • The Reality: The Control Plane strictly enforces Pydantic contracts (ActionIntent with fields action, target, rationale, declared_blast_radius). Pure reasoning models (such as DeepSeek-R1) output their internal chain-of-thought enclosed in <think>...</think> tags before delivering the final answer.

   • Architectural Defense:

     • Both LiteLLMAdapter (via instructor.from_litellm) and OllamaAdapter embed the regex filter _strip_think_tags() to excise <think>...</think> envelopes and Markdown fences before passing the JSON payload to response_model.model_validate().

     • For ultra-fast sub-second alert triaging, developers can select LLM_MODEL=openrouter/deepseek/deepseek-chat:free, which outputs raw JSON directly without thinking blocks.

8. Dual Client Interface: Zero-Dependency Cloud Web Chat vs. Local LibreChat Workstation

Operators can interact conversationally with Agentic-Ops through two independent frontends, depending on their deployment model:

+--------------------------------------------------------------------------------------------------+
| DUAL FRONTEND CONVERSATIONAL TOPOLOGY                                                            |
+--------------------------------------------------------------------------------------------------+
|                                                                                                  |
| [Option 1: Local LibreChat Client]               [Option 2: Standalone Cloud Web Chat]          |
| - Runs in Docker on workstation (Mac/PC)         - Hosted directly on Render Free Tier          |
| - Connects via MCP Gateway (/mcp)                - URL: https://agentic-ops-5ebv.onrender.com/chat |
| - Full ChatGPT clone with presets & uploads      - 0 local install, 0 extra containers, 0 EUR    |
| - Heavy RAM footprint (~1.5 GB RAM)              - Lightweight RAM footprint (< 2 MiB RAM)      |
|                                                                                                  |
|             \                                                     /                              |
|              \                                                   /                               |
|               +------------------------+------------------------+                                |
|                                        |                                                         |
|                                        v                                                         |
|                     +---------------------------------------+                                    |
|                     | AGENTIC-OPS CONTROL PLANE (FastAPI)   |                                    |
|                     | - POST /api/chat                      |                                    |
|                     | - POST /mcp (JSON-RPC 2.0 Gateway)    |                                    |
|                     | - LLMProviderRegistry (Auto-Fallback) |                                    |
|                     +---------------------------------------+                                    |
+--------------------------------------------------------------------------------------------------+

6. Architectural Rationale: Deterministic In-Memory RAG vs. Vector Database (No VectorDB)

Applying the repository's Radical Honesty rule, Agentic-Ops intentionally excludes standalone vector databases (ChromaDB, Pinecone, Qdrant, Weaviate, Milvus). Instead, it relies on a Deterministic In-Memory Knowledge Retrieval Engine (LocalRunbookRAG in agentic_ops/adapters/runbook_rag.py and search_failure_taxonomy in agentic_ops/entrypoints/mcp_server.py).

+--------------------------------------------------------------------------------------------------+
| KNOWLEDGE RETRIEVAL ARCHITECTURE: DETERMINISTIC IN-MEMORY VS VECTOR DATABASE                    |
+--------------------------------------------------------------------------------------------------+
|                                                                                                  |
| [Theoretical / Marketing Architecture]             [Agentic-Ops Production Architecture]         |
| - Standalone VectorDB (Pinecone, ChromaDB)         - 100% In-Memory Deterministic Index          |
| - Sentence Transformer Embeddings in RAM           - Exact Risk ID Resolution + Token Lexical    |
| - High RAM footprint (300 to 600 MiB) -> OOMKilled - Ultra-lightweight RAM footprint (< 1 MiB)   |
| - High latency (50-200ms network roundtrip)        - Sub-millisecond latency (< 1ms in-process)  |
| - Approximate / Probabilistic Cosine Similarity    - Strict Deterministic Incident Matching      |
|                                                                                                  |
+--------------------------------------------------------------------------------------------------+

Installation & Quickstart

Option A: Install from PyPI (Recommended for Production & Clusters)

# Standard installation via pip
pip install agentic-ops

# Or using uv (ultra-fast dependency resolver)
uv pip install agentic-ops

# Start the full Control Plane (FastAPI REST API, SRE Console /console, Chat /chat, MCP Server /sse)
agentic-start --port 8000 --host 0.0.0.0

# Run SRE Trust Ledger diagnostics
agentic-ops audit

Option B: Local Source Development & Full Test Suite

# 1. Environment setup
git clone https://github.com/nolll77/agentic-ops.git
cd agentic-ops-core
uv venv .venv
source .venv/bin/activate
uv pip install -e ".[dev]"
uv pip install 'httpx[socks]'

# 2. Run the Full Suite of Unit, Adversarial, Scenario, Security & Outbox Tests (183 tests)
pytest tests/ -v
# > Verifies RFC 6962 Binary Merkle Tree, WORM linear chain immutability, Fail-Closed governance,
# > non-bypassable Orchestrator state transitions, Admin API Key Zero-Trust middleware,
# > SQLite WAL Outbox pattern before ACK 202, AlertClassifier deterministic mapping,
# > Atomic Quota Manager (0 xfailed), Proposal digest binding, Quorum & Anti-Self-Approval,
# > and ExecutorGateway simulation safety. (183 passed, 3 skipped, 0 xfailed in ~14s)

# 3. Launch the Trust Ledger Audit CLI (Merkle Root & Selective Disclosure Proofs)
python -m agentic_ops.cli.trust_ledger list
python -m agentic_ops.cli.trust_ledger merkle-root
python -m agentic_ops.cli.trust_ledger merkle-proof EP-0001
python -m agentic_ops.entrypoints.mcp_server

Option C: Run Pre-Built Container (Docker Hub & GHCR)

Pre-built multi-architecture OCI images (linux/amd64, linux/arm64) are distributed on Docker Hub and GHCR:

# Pull from Docker Hub
docker pull nolll78/agentic-ops:latest

# Or pull from GitHub Container Registry
docker pull ghcr.io/nolll77/agentic-ops:latest

# Run the complete Control Plane
docker run -d -p 8000:8000 --name agentic-ops nolll78/agentic-ops:latest

# Access services:
# -> Web Chat:    http://localhost:8000/chat
# -> SRE Console: http://localhost:8000/console
# -> Swagger API: http://localhost:8000/docs
# -> MCP Server:  http://localhost:8000/mcp

Reference Documentation & Deployment Proofs

1. Industrial Implementation Status & Safety Invariants (docs/INDUSTRIAL_IMPLEMENTATION_STATUS.md): Exhaustive matrix distinguishing what is implemented in code and tested in CI from external cloud integrations, GameDays, and deployment qualification gates.

2. Roadmap de Durcissement Production & Levée des Écarts (docs/PRODUCTION_READINESS_ROADMAP_AND_GAPS.md): Master SRE roadmap detailing the 10 phases, 8 invariants, and remaining production doors. Entra & Preproduction Qualification Checklist (docs/ENTRA_PREPRODUCTION_QUALIFICATION.md): Concrete checklist for Entra application and SCIM provisioning, image publication, single-replica Helm rollout, end-to-end verification, backup/restore, and failure exercises. These external gates are not claimed complete.

3. Documentation Complète en Français (README_FR.md): Full French version of this main documentation.

4. Cloud Sandbox Validation Report (DOCKER_SANDBOX_VALIDATION.md):

   Deployment on an isolated micro-VM (4 AMD EPYC vCPUs, 8 GB RAM), line-by-line analysis of 30 security tests (forbidden_executed = 0), HTTP 401 Unauthorized rejection on unsigned webhooks, and session screenshots.

5. Tailscale Pull Connector Architecture & Validation (CAC_TAILSCALE_PULL_CONNECTOR.md): Complete documentation of the live interconnection with https://github.com/jakatt/cac_mlops (100.117.99.62), tailscaled setup in userspace-networking mode (SOCKS5 127.0.0.1:1055), DERP relay diagnostics, anatomy of cac_pull_collector.py, and breakdown of all 13 unit tests.

6. Generic Integration Guide (INTEGRATION_GUIDE.md): Specifications for connecting Agentic-Ops to a third-party MLOps infrastructure.

7. MLOps Failure Modes & Risk Taxonomy v2 (docs/mlops_failure_modes_taxonomy.md): Exhaustive 435-entry failure taxonomy across 10 MLOps dimensions (RPN calibration, SLO impact, detection sensors, automated mitigations, and Chaos waves).

8. MLOps Incident War-Room Simulation Scenarios Registry (docs/mlops_simulation_scenarios_registry.md): 36 multi-failure simulation scenarios structured around a 7-block SRE blueprint (with the first 8–9 foundational scenarios expanded to full mathematical, architectural, and executable Python depth).

9. Model Context Protocol (MCP) Governance Gateway Guide (docs/MCP_SERVER_GUIDE.md): Complete reference for the MCP Server (STDIO & HTTP JSON-RPC 2.0 on /mcp), live /console reading via read_console_diagnostic, 8 governed tools, 5 knowledge resources, client configuration files (Claude Desktop, Cursor, Windsurf, Antigravity), and the 20 Cognitive Starter Prompts catalog.

10. Historical Security Audit & Technical Remediation Matrix (docs/HISTORICAL_AUDIT_AND_REMEDIATION.md): Full verbatim transcription of the early external architectural critique, followed by an exhaustive 9-point technical remediation matrix (P0.1 to P1.3) demonstrating how every single flagged vulnerability (Slack HITL HMAC, sub-process isolation, Merkle cryptographic ledger, atomic Redis rate limiting, out-of-band blast radius calculation, and Prompt Hygiene) was systematically engineered, audited, and verified across 54 automated tests.

11. Universal LLM Provider Hub & Configuration Template (.env.example): Comprehensive guide to connecting the Control Plane to 8 AI model providers: OpenRouter (24/7 free tier including DeepSeek-R1 and Llama 3.3 70B), Gemini, OpenAI, Anthropic, Azure OpenAI, Mistral AI, HuggingFace Inference API, and Ollama (local or Tailscale mesh-routed at 0 EUR). Includes automatic failover chains (FallbackLLMAdapter) and Prometheus metrics instrumentation.

12. Multi-Agent System (MAS) Fallback Architecture & Mapping Keys (docs/MAS_FALLBACK_ARCHITECTURE_GUIDE.md): Exhaustive architectural breakdown of the 5 cascading resilience and safety fallback layers in Agentic-Ops: Cognitive Multi-LLM failover, FinOps Circuit Breaker, Specialist routing fallback, Policy Engine fail-closed enforcement, and Remote VPS/Tailscale partition locks. Includes complete code mapping keys, trigger conditions, fallback states, and the Provider Independence & Disaster Recovery Strategy detailing the 4 lines of defense (zero-code env switches, 100% sovereign off-cloud Ollama over Tailscale, granular per-role splitting, and deterministic 0-token safety net) if OpenRouter cuts access or changes policies.

13. Multi-Agent System (MAS) Agent Selection & Default Routing Architecture (docs/MAS_AGENT_ROUTING_AND_DEFAULTS_GUIDE.md): In-depth guide to the 5 specialized agent roles (DataSpecialist, ModelSpecialist, InfraSpecialist, RemediationAgent, WebChatAssistant), taxonomic routing logic in _get_specialist(), anti-bias judicial separation between investigation and remediation, and the rationale behind DataSpecialist as the universal platform default. Includes the exhaustive breakdown of non-cognitive components operating with zero LLMs (0-token, pure deterministic Python) under the Zero-Trust security model, as well as the interactive Live MAS Topology & Agent Inspector embedded directly in both the Web Console (/console) and Web Chat (/chat) backed by GET /api/mas/topology.

14. PyPI Release & Distribution Engineering Guide (docs/PYPI_RELEASE_AND_DISTRIBUTION_GUIDE.md): Comprehensive guide to public package distribution on PyPI: software supply chain security, zero-static-secrets architecture using OpenID Connect (OIDC Trusted Publishing), SLSA provenance attestations, installation and CLI consumption options, and release lifecycle runbooks for maintainers.

15. Official Thesis Defense Report (docs/Rapport_soutenance_Mar25_MLOps_Agentic_Car_Accident.pdf): Comprehensive 40-page SRE & MLOps thesis defense report covering governance, multi-layer observability, deterministic FSM, cryptographic WORM trust ledger with RFC 6962 Merkle tree, and live production interconnection with jakatt/cac_mlops.

16. Community Collaboration & Contributing Guide (CONTRIBUTING.md): Architectural invariants, development setup, Pull Request workflow, coding standards, and step-by-step participation guidelines for researchers and software engineers.

17. Foundational MLOps Risk Bibles & Architecture Reference Dossiers (docs/):

    • docs/bible_ontologie.md: Comprehensive MLOps Risk Ontology (337 KB) detailing the full taxonomy of failure modes across the 10 architecture dimensions.

    • docs/annexes_debats_paradoxes.md: Conceptual debates and architectural paradoxes underlying each failure mode category.

    • docs/mlops_failure_propagation_graph.md: Causal failure propagation graph (Mermaid) analyzing domino cascades across Infra, Data, Features, Time, and Observability layers.

    • docs/bi_observability_genesis.md: Data Observability genesis justifying Apache Superset vs Grafana/Metabase for complex SQL Lab and schema drift monitoring.

18. Active Community Issues & Open Roadmap:

    • #1 — Denial Reconciliation in Evidence Store: Formal linking of human operator rejection to origin episode in WORM ledger.

    • #2 — Deterministic Worker Failure Drills: At-least-once delivery, isolated crash injection, and idempotent task recovery.

    • #3 — Local Inventory Lifecycle & Drift: Staleness, live tampering detection, schema verification, and reload safety.

    • #4 — Local Backup/Restore Drills & DR Verification: Automated rehearsals of SQLite WAL and WORM snapshot restoration.

    • #5 — Local Alert Ingestion Benchmark: Scalability and latency percentiles under burst alert storms.

    • #6 — Comprehensive Benchmark of Free OpenRouter LLMs: Empirical evaluation of 17+ :free models across MAS roles (Gemma, Nemotron, Cohere, Liquid).