Skip to main content
Glama

🧠 Self-Learning AI Agent: Long-Term Memory Engine

Python FastAPI Qdrant MCP License

A production-grade Long-Term Memory Microservice & Model Context Protocol (MCP) Server for AI Agents. Enables continuous, cross-session learning by dynamically extracting atomic user facts, resolving state conflicts (ADD, UPDATE, DELETE, NOOP), and persisting vectors in Qdrant.


🌟 Why This Architecture?

Traditional RAG and naive chat-history appending have critical flaws:

  • Context Window Bloat: Appending full raw transcripts increases latency, token costs, and attention drift.

  • Contradictions & Stale State: If a user says "I live in San Francisco" and 3 months later says "I moved to Tokyo", naive RAG retrieves both chunks, causing hallucinations.

  • My Solution: A Two-Phase Memory Pipeline where an LLM parses atomic facts and performs explicit state mutations (ADD, UPDATE, DELETE, NOOP) directly on the vector store.


Related MCP server: Memsolus MCP Server

πŸ—οΈ System Architecture

flowchart TD
    subgraph Ingestion ["1. Client Interfaces & Ingestion"]
        A["πŸ’¬ User / Agent Chat"] --> C["⚑ FastAPI REST API (/v1/chat)"]
        B["πŸ€– Claude Desktop / Cursor IDE"] --> D["πŸ”Œ MCP 2.x Server (stdio)"]
        C --> E["πŸ“₯ Async Ingestion Queue (asyncio.Queue)"]
        D --> E
    end

    subgraph TwoPhase ["2. Two-Phase Agentic State Machine"]
        E --> F["🧠 Phase 1: Fact Extractor (LLM)"]
        F -->|Atomic Facts & Triples| G["πŸ“‹ Candidate Facts"]
        
        G --> H["πŸ” Semantic Search Candidates"]
        Q1[("πŸ’Ύ Qdrant Vector Store (Current State)")] -.->|Existing User Memories| H
        
        H --> I["βš–οΈ Phase 2: Conflict Reconciler (LLM)"]
        G --> I
        
        I -->|State Mutation Decision| J{"Operation"}
        J -->|ADD| K["✨ Insert New Vector Point"]
        J -->|UPDATE| L["πŸ”„ Overwrite Stale Vector & Payload"]
        J -->|DELETE| M["πŸ—‘οΈ Delete Vector Point"]
        J -->|NOOP| N["⏸️ Ignore Redundant Duplicate"]
    end

    subgraph Storage ["3. Storage & Knowledge Graph Layer"]
        K --> Q2[("πŸ’Ύ Qdrant Vector DB (Synchronized State)")]
        L --> Q2
        M --> Q2
        Q2 --> O["⏳ Ebbinghaus Temporal Decay & Spaced Reinforcement"]
        Q2 --> P["πŸ•ΈοΈ GraphRAG Topological Entity Visualizer"]
    end

✨ Key Features

  • 🧠 Dynamic Two-Phase Lifecycle:

    1. Extraction: Extracts durable, self-contained third-person facts while discarding conversational noise.

    2. Reconciliation: Semantic lookup finds candidate conflicts; LLM decides ADD, UPDATE, DELETE, or NOOP.

  • πŸ•ΈοΈ Interactive GraphRAG Knowledge Visualizer:

    • Live topological node-link graph visualizer built on HTML5 Canvas force physics.

    • Automatically extracts Entity-Relation Triples (Subject -> Relation -> Object) for multi-hop associative retrieval.

  • ⚑ Sub-150ms Asynchronous Ingestion Queue:

    • Event-driven background queue (asyncio.Queue) offloads fact extraction and vector synchronization, enabling instant conversational replies.

  • πŸ—‚οΈ Multi-Tier Scoped Memory:

    • Hierarchical isolation across user (persistent), session (ephemeral/task-level), and workspace (shared team conventions).

  • ⏳ Cognitive Temporal Decay (Ebbinghaus Forgetting Curve):

    • Mathematical recency weighting: $\text{Score} = (1 - w) \cdot \text{Similarity} + w \cdot e^{-\lambda \Delta t}$.

    • Spaced reinforcement: Automatically touches and refreshes retention every time a memory is recalled.

  • πŸ“Š Interactive Visual Memory Explorer & AI Chat Playground:

    • Full-featured dark-mode web dashboard (/dashboard) with live chat playground, real-time memory bank feed, and similarity confidence meters.

  • πŸ”Œ Dual Serving Interfaces:

    • FastAPI REST Endpoints: High-performance HTTP service with OpenAPI docs (/docs).

    • Model Context Protocol (MCP 2.x): Plug-and-play tools (remember_conversation, recall_memories, forget_memory) for Claude Desktop, Cursor, and agentic workflows.

  • πŸ’Ύ Hybrid Qdrant Support: Runs via Docker or automatic embedded local disk mode (./qdrant_data) with zero cloud cost.

  • 🌐 Multi-Provider Support: Seamlessly swappable across Google Gemini (gemini-3.5-flash-lite), OpenAI (gpt-4o-mini), or 100% offline local models via Ollama.

  • πŸ›‘οΈ Production Hardened: Adaptive exponential backoff retry handler parsing upstream rate-limit windows (429/503).


πŸ“ Repository Structure

β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ api/                 # FastAPI REST API routes & controllers
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   └── routes.py        # /v1/memories/process, /search, /user, /delete
β”‚   β”œβ”€β”€ db/                  # Vector database layer
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   └── qdrant.py        # Qdrant client manager & schema initializers
β”‚   β”œβ”€β”€ llm/                 # Unified LLM provider client (Gemini / OpenAI)
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   └── client.py        # Structured JSON generation & embedding generation
β”‚   β”œβ”€β”€ memory/              # Two-Phase Memory Pipeline Core
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   β”œβ”€β”€ models.py        # Pydantic schemas (Fact, Operation, MemoryRecord)
β”‚   β”‚   β”œβ”€β”€ extractor.py     # Phase 1: Atomic fact extractor
β”‚   β”‚   β”œβ”€β”€ reconciler.py    # Phase 2: Conflict reconciler
β”‚   β”‚   └── service.py       # High-level memory orchestrator
β”‚   β”œβ”€β”€ mcp_server/          # Model Context Protocol (MCP 2.x) integration
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   └── server.py        # Standardized MCP server & tools
β”‚   β”œβ”€β”€ config.py            # Pydantic Settings management
β”‚   └── main.py              # FastAPI ASGI entrypoint & lifecycle
β”œβ”€β”€ scripts/
β”‚   β”œβ”€β”€ verify_setup.py      # Setup & database connectivity verifier
β”‚   └── demo_pipeline.py     # Interactive visual lifecycle demo
β”œβ”€β”€ tests/
β”‚   β”œβ”€β”€ test_phase1.py       # Infrastructure & DB tests
β”‚   β”œβ”€β”€ test_phase2.py       # Two-phase pipeline & lifecycle tests
β”‚   β”œβ”€β”€ test_phase3_api.py   # FastAPI REST endpoint tests
β”‚   └── test_phase3_mcp.py   # MCP tools test suite
β”œβ”€β”€ docker-compose.yml       # Multi-container orchestration (API + Qdrant)
β”œβ”€β”€ Dockerfile               # Multi-stage production container build
β”œβ”€β”€ requirements.txt         # Project dependencies
└── pytest.ini               # Pytest configuration

πŸš€ Quickstart Guide

1. Clone & Setup Environment

# Clone the repository
git clone https://github.com/your-username/self-learning-ai-agent.git
cd self-learning-ai-agent

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

2. Configure Environment Variables

cp .env.example .env

Edit .env with your API key and preferred models:

# Google Gemini (Free Tier / Zero Cloud Cost)
OPENAI_API_KEY=AIzaSy...
PROVIDER=gemini
EMBEDDING_MODEL=gemini-embedding-2
EMBEDDING_DIMENSION=3072
EXTRACTION_MODEL=gemini-3.5-flash-lite
RECONCILIATION_MODEL=gemini-3.5-flash-lite

# Vector DB Settings (Automatically falls back to local disk if Docker is off)
QDRANT_HOST=localhost
QDRANT_PORT=6333
SIMILARITY_THRESHOLD=0.60

3. Verify Setup & Run Interactive Demo

# Verify vector DB connectivity
python scripts/verify_setup.py

# Run visual multi-turn memory evolution demo
python scripts/demo_pipeline.py

4. Run Automated Test Suite

pytest -v

🌐 Running the Services

Option A: Local FastAPI Server

uvicorn src.main:app --host 0.0.0.0 --port 8000 --reload

Interactive Swagger API documentation: http://localhost:8000/docs

Option B: Full Stack Docker Compose

docker compose up --build -d

Option C: Launch MCP Server (stdio)

python -m src.mcp_server.server

πŸ”Œ Model Context Protocol (MCP) Integration

Connect this memory engine directly to Claude Desktop, Cursor IDE, or Antigravity.

Add to your claude_desktop_config.json (or Cursor MCP settings):

{
  "mcpServers": {
    "agent-memory": {
      "command": "/path/to/self-learning-ai-agent/.venv/bin/python",
      "args": ["-m", "src.mcp_server.server"],
      "cwd": "/path/to/self-learning-ai-agent"
    }
  }
}

Exposed MCP Tools:

  • remember_conversation(user_id, conversation_text): Extracts facts and reconciles them into memory.

  • recall_memories(user_id, query, limit): Retrieves semantically relevant facts with similarity scores.

  • list_user_memories(user_id, limit): Lists all active facts for the user.

  • forget_memory(memory_id): Manually removes a memory point.


πŸ“‘ REST API Reference

1. Ingest Conversation & Update Memory State

POST /v1/memories/process

curl -X POST http://localhost:8000/v1/memories/process \
  -H "Content-Type: application/json" \
  -d '{
    "user_id": "alex_01",
    "conversation": "I am a Senior AI engineer based in San Francisco. I switched my primary language from Python to Rust."
  }'

GET /v1/memories/search?user_id=alex_01&query=What+languages+does+the+user+code+in%3F

curl "http://localhost:8000/v1/memories/search?user_id=alex_01&query=What+languages+does+the+user+code+in%3F&limit=3"

3. List All User Memories

GET /v1/memories/user/{user_id}

curl http://localhost:8000/v1/memories/user/alex_01

4. Delete Memory by ID

DELETE /v1/memories/{memory_id}

curl -X DELETE http://localhost:8000/v1/memories/c7b2049e-648b-4b10-a24e-b5f7cf839a82

πŸ’Ό Resume & Technical Impact Highlights

If you include this project in your portfolio or resume, here are production-oriented bullet points:

  • Engineered a self-learning long-term memory microservice in Python (FastAPI) utilizing Qdrant vector search to provide AI agents with persistent, cross-session user context.

  • Implemented a dynamic Two-Phase Memory Pipeline that prompts an LLM to extract atomic facts and programmatically execute state mutations (ADD, UPDATE, DELETE, NOOP), eliminating stale fact contradictions and optimizing LLM token utilization.

  • Packaged the memory layer into a Model Context Protocol (MCP 2.x) Server, enabling native, zero-latency tool-use integration across IDEs and AI client agents (Cursor, Claude Desktop).

  • Architected a modular provider layer supporting Gemini, OpenAI, and local Ollama, featuring automated schema validation and adaptive rate-limit backoff retry handlers.

  • Containerized the full stack with multi-stage Docker builds and automated test suites achieving 100% test pass rates across unit and end-to-end integration flows.


πŸ“„ License

MIT License. Free for open-source and commercial use.

Available Tools

4 tools
forget_memoryA

Deletes a specific memory record by its unique ID.

Args: memory_id: The UUID of the memory record to remove.

Inputs: memory_id (required). Returns: result

list_user_memoriesA

Lists all active stored memories for a user.

Args: user_id: The unique identifier for the user. limit: Maximum number of memories to return (default: 50).

Inputs: user_id (required), limit. Returns: result

recall_memoriesA

Retrieves semantically relevant memories from vector storage given a natural language query.

Args: user_id: The unique identifier for the user. query: The topic, question, or context to retrieve facts for. limit: Maximum number of memory records to return (default: 5).

Inputs: query (required), user_id (required), limit. Returns: result

remember_conversationA

Extracts durable facts from conversation text and reconciles them into long-term memory (ADD, UPDATE, DELETE, NOOP).

Args: user_id: The unique identifier for the user. conversation_text: The conversation transcript or user statement to remember.

Inputs: user_id (required), conversation_text (required). Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4/5.0

    Scored across 4 tools

    Disambiguation4/5

    Each tool has a distinct core action: list, recall, remember, and forget. The only potential confusion is between list_user_memories and recall_memories, but the descriptions clearly separate 'all active memories' from 'semantically relevant' retrieval.

    Naming Consistency5/5

    All tool names follow a consistent verb_noun snake_case pattern: list_user_memories, forget_memory, remember_conversation, recall_memories. The verbs are meaningful and align with their actions.

    Tool Count5/5

    Four tools is a well-scoped size for a memory server, covering ingestion, retrieval, listing, and deletion. Each tool maps to a clear capability without redundant or excessive surface area.

    Completeness4/5

    The set covers the core memory lifecycle: adding memories, recalling them, listing them, and deleting them. A direct get-by-ID or explicit update tool is missing, but remember_conversation's reconciliation covers updates through its ADD/UPDATE/DELETE/NOOP behavior.

    Maintenance

    ActivityStale
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      D
      maintenance
      Provides AI agents with persistent, searchable memory that survives across conversations using semantic search, temporal versioning, and smart organization. Enables long-term context retention and cross-session continuity for AI assistants.
      14
      -
    • A
      license
      A
      quality
      D
      maintenance
      Provides persistent long-term memory for AI agents through semantic search and automated knowledge graph extraction. It enables agents to store, recall, and reason over facts, preferences, and relationships across multiple conversations and sessions.
      14
      6 npm
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Provides persistent, shared memory for AI agents by capturing conversations verbatim, distilling facts and summaries, and enabling retrieval through search, timeline, details, and explicit remember tools.
      MIT