MCP Sentiment Analysis Server

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chatbot Test Notebook\n",
    "\n",
    "This notebook is designed to test and experiment with the functionality of the chatbot implemented in the `rag_swimrules` project. It showcases various scenarios and evaluates the performance of the chatbot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import textwrap\n",
    "\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "from langchain_chroma.vectorstores import Chroma\n",
    "from langchain_chroma import Chroma\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_core.messages import AIMessage, HumanMessage, SystemMessage\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "swimming_rules_template = \"\"\"You are an expert swimming rules assistant trained on FINA and competitive swimming regulations. \n",
    "Your answers must be precise, factual, and based only on the provided rules. Follow these guidelines:\n",
    "\n",
    "1. Answer ONLY using the verified rules provided below\n",
    "2. DO NOT use information from any other sources, including the internet or personal knowledge.\n",
    "3. if an action is not explicitly prohibited in the rules, assume it is allowed.\n",
    "4. if an action is not explicity mentioned, assume it is allowed.\n",
    "5. if an action is explicitly stated in the rule, assume it is allowed.\n",
    "6. A disqualification msut be based on a specific rule violation and cannot be implied.\n",
    "7. If the information isn't in the rules, say \"I don't have that rule in my current documentation\"\n",
    "8. Respond with either \"LEGAL\" or \"DISQUALIFIED\" based on the swimming rules, provide a brief, concise explanation.\n",
    "9. Keep answers concise but complete\n",
    "10. Reference specific rule numbers when possible\n",
    "\n",
    "Relevant swimming rules:\n",
    "{context}\n",
    "\n",
    "Question: {question}\n",
    "\n",
    "Answer:\"\"\"\n",
    "\n",
    "SWIMMING_RULES_PROMPT = PromptTemplate(\n",
    "    template=swimming_rules_template,\n",
    "    input_variables=[\"context\", \"question\"]\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "queries = [\n",
    "    \"Prior to the start signal the swimmer is seen rocking back and forth.  Is this a disqualification?\",\n",
    "    \"upon the command 'take your mark', the swimmer takes their starting position and just prior to the start signal, the swimmer is seen moving forward.\",\n",
    "    \"What happens if a swimmer touches the wall with one hand in freestyle?\",\n",
    "    \"what happens if a swimmer touches the wall with one hand in breaststroke?\",\n",
    "    \"A swimmer is wearing a wetsuit in a swimming competition?\",\n",
    "    \"During a freestyle event, a swimmer does a flip turn and touches the wall with only one foot.\",\n",
    "    \"A breaststroke swimmer moves their hands in a sculling or flipper movement at the end of the first arm stroke, both after the start and after the turn Should they be disqualified?\",\n",
    "    \"In a 9-10 100 yard breaststroke event, a swimmer completes 50 yards with a simulatenous two-hand touch and, thinking that the race is over, pushes back from the wall to read the scoreboard. At this point, realizing that the race is only halfway over, the swimmer returns to the wall, pushes off on the breast, and completes the required distance in good form. What call, if any, should be made?\",\n",
    "    \"In a 9-10 100 yard breaststroke event, a swimmer completes 50 yards with a simulatenous two-hand touch and, thinking that the race is over, pushes back from the wall on their back to read the scoreboard. At this point, realizing that the race is only halfway over, the swimmer returns to the wall, pushes off on the breast, and completes the required distance in good form. What call, if any, should be made?\",\n",
    "    \"A swimmer swims the breaststroke in a way that their hands are completely underwater when they are pushed forward together from their breast.\",\n",
    "    \"A swimmer swims the breaststroke in a way that their elbows are above the water line during the forward movement of the arms.\",\n",
    "    \"A breaststroker’s head breaks the surface of the water during each cycle, however, the swimmer does not take a breath even if the head breaks the surface.\",\n",
    "    \"During a freestyle event, a swimmer starts in the water such that the swimmwer is facing the starting end of the pool with one hand on the wall.\",\n",
    "    \"During a freestyle event, a swimmer starts in the water such that the swimmer is facing the other end of the pool with one hand on the wall.\",\n",
    "    \"During a butterfly event, a swimmer starts in the water such that the swimmer is facing the other end of the pool with one hand on the wall.\",\n",
    "    \"During a butterfly event, a swimmer starts in the water such that the swimmer has one hand on the wall looking away from the starting block.\",\n",
    "    \"During a butterfly event, a swimmer starts in the water such that the swimmer is looking at the opposite end of the pool.\",\n",
    "    \"In a 200m individual medley event, a swimmer swims the first 50m in butterfly, the second 50m in backstroke, the third 50m starts swimming freestyle but then switches to breaststroke, and the last 50m in freestyle.\",\n",
    "    \"In a 100y freestyle event, the swimmer swims the first 50y in butterfly and the last 50y in backstroke.\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize embeddings model (ensure the same model used during creation)\n",
    "embeddings = OpenAIEmbeddings(model=\"text-embedding-3-small\")\n",
    "\n",
    "# retretieve the relevant rules from your database or knowledge base\n",
    "# Load the existing Chroma vector store\n",
    "persist_directory = \"../agent_swimrules_db\"\n",
    "vector_store = Chroma(persist_directory=persist_directory, embedding_function=embeddings)\n",
    "\n",
    "# Create a retriever from the vector store\n",
    "retriever = vector_store.as_retriever(search_type=\"similarity_score_threshold\", search_kwargs={\"k\": 20, \"score_threshold\": 0.4})\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Initialize LLM\n",
    "llm = ChatOpenAI(\n",
    "    model=\"gpt-4o\",\n",
    "    temperature=0,\n",
    "    max_tokens=1000\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " >>>Query: Prior to the start signal the swimmer is seen rocking back and forth.\n",
      "Is this a disqualification?\n",
      ">>>Retrieved 8 documents\n",
      "{'chunk_type': 'guidance', 'rule_title': 'Interpretation Guidance', 'rule_number': 'GUIDANCE', 'page': '0', 'source': './rule_data/interpretation_guidance.txt', 'category': 'Interpretations', 'rule_id': 'guidance', 'stroke_type': 'none', 'section': 'Guidance'}\n",
      "{'chunk_type': 'rule', 'section': 'General', 'category': 'General Rules', 'rule_id': 'page_21_chunk_32', 'rule_number': 'unknown', 'page': '21', 'source': './rule_data/2025-mini-rulebook.pdf', 'rule_title': 'General Content', 'stroke_type': 'none'}\n",
      "{'source': './rule_data/2025-mini-rulebook.pdf', 'rule_title': 'A disqualification can be made only by the official within whose jurisdiction the', 'rule_number': '102.1', 'page': '43', 'category': 'Officials', 'section': 'DISQUALIFICATIONS', 'rule_id': 'rule_102.1', 'stroke_type': 'none', 'chunk_type': 'rule'}\n",
      "{'rule_title': 'General Content', 'section': 'General', 'chunk_type': 'rule', 'source': './rule_data/2025-mini-rulebook.pdf', 'page': '21', 'category': 'General Rules', 'rule_id': 'page_21_chunk_31', 'stroke_type': 'none', 'rule_number': 'unknown'}\n",
      "{'source': './rule_data/2025-mini-rulebook.pdf', 'rule_id': 'rule_102.8', 'category': 'Equipment and Facilities', 'stroke_type': 'medley', 'rule_number': '102.8', 'chunk_type': 'rule', 'rule_title': 'Any swimmer not entered in a race who enters the pool or course in the area in', 'page': '43', 'section': 'DISQUALIFICATIONS'}\n",
      "{'rule_number': '101.7', 'section': 'RELAYS', 'source': './rule_data/2025-mini-rulebook.pdf', 'stroke_type': 'relay', 'rule_title': 'RELAYS', 'chunk_type': 'rule', 'rule_id': 'rule_101.7', 'category': 'Relays', 'page': '25'}\n",
      "{'source': './rule_data/2025-mini-rulebook.pdf', 'section': 'BREASTSTROKE', 'stroke_type': 'breaststroke', 'page': '22', 'rule_id': 'rule_101.2', 'category': 'Stroke Technique', 'rule_number': '101.2', 'chunk_type': 'rule', 'rule_title': 'BREASTSTROKE'}\n",
      "{'rule_number': 'unknown', 'stroke_type': 'none', 'rule_title': 'General Content', 'rule_id': 'page_21_chunk_33', 'section': 'General', 'category': 'General Rules', 'source': './rule_data/2025-mini-rulebook.pdf', 'chunk_type': 'rule', 'page': '21'}\n",
      ">>>Assessment: LEGAL. There is no specific rule provided that prohibits a swimmer from rocking\n",
      "back and forth prior to the start signal.\n",
      "\n",
      " >>>Query: upon the command 'take your mark', the swimmer takes their starting\n",
      "position and just prior to the start signal, the swimmer is seen moving forward.\n",
      ">>>Retrieved 4 documents\n",
      "{'stroke_type': 'none', 'chunk_type': 'rule', 'section': 'General', 'rule_number': 'unknown', 'page': '21', 'rule_id': 'page_21_chunk_30', 'rule_title': 'General Content', 'source': './rule_data/2025-mini-rulebook.pdf', 'category': 'General Rules'}\n",
      "{'stroke_type': 'none', 'rule_title': 'General Content', 'page': '21', 'section': 'General', 'source': './rule_data/2025-mini-rulebook.pdf', 'chunk_type': 'rule', 'rule_number': 'unknown', 'category': 'General Rules', 'rule_id': 'page_21_chunk_31'}\n",
      "{'section': 'STARTS', 'chunk_type': 'rule', 'category': 'Starts', 'page': '20', 'rule_id': 'rule_101.2', 'rule_title': 'The Start', 'stroke_type': 'starts', 'source': './rule_data/2025-mini-rulebook.pdf', 'rule_number': '101.2'}\n",
      "{'page': '21', 'category': 'General Rules', 'source': './rule_data/2025-mini-rulebook.pdf', 'rule_number': 'unknown', 'stroke_type': 'none', 'section': 'General', 'chunk_type': 'rule', 'rule_id': 'page_21_chunk_33', 'rule_title': 'General Content'}\n",
      ">>>Assessment: DISQUALIFIED. According to rule 101.1.3A, any swimmer initiating a start before\n",
      "the signal may be disqualified if the Referee independently observes and\n",
      "confirms the Starter’s observation that a violation occurred.\n",
      "\n",
      " >>>Query: What happens if a swimmer touches the wall with one hand in\n",
      "freestyle?\n",
      ">>>Retrieved 8 documents\n",
      "{'source': './rule_data/interpretation_guidance.txt', 'chunk_type': 'guidance', 'rule_id': 'guidance', 'rule_number': 'GUIDANCE', 'rule_title': 'Interpretation Guidance', 'stroke_type': 'none', 'section': 'Guidance', 'page': '0', 'category': 'Interpretations'}\n",
      "{'rule_title': 'FREESTYLE', 'rule_number': '101.5', 'category': 'Stroke Technique', 'chunk_type': 'rule', 'source': './rule_data/2025-mini-rulebook.pdf', 'rule_id': 'rule_101.5', 'page': '24', 'stroke_type': 'freestyle', 'section': 'FREESTYLE'}\n",
      "{'source': './rule_data/2025-mini-rulebook.pdf', 'section': 'BUTTERFLY', 'stroke_type': 'medley', 'rule_title': 'Stroke', 'chunk_type': 'rule', 'page': '23', 'rule_number': '101.2', 'rule_id': 'rule_101.2', 'category': 'Starts'}\n",
      "{'section': 'Guidance', 'page': '0', 'rule_title': 'Interpretation Guidance', 'rule_number': 'GUIDANCE', 'source': './rule_data/interpretation_guidance.txt', 'rule_id': 'guidance', 'chunk_type': 'guidance', 'stroke_type': 'none', 'category': 'Interpretations'}\n",
      "{'rule_number': '101.2', 'source': './rule_data/2025-mini-rulebook.pdf', 'chunk_type': 'rule', 'page': '22', 'stroke_type': 'medley', 'section': 'BREASTSTROKE', 'rule_title': 'Stroke', 'category': 'Starts', 'rule_id': 'rule_101.2'}\n",
      "{'page': '24', 'category': 'Stroke Technique', 'rule_title': 'FREESTYLE', 'stroke_type': 'freestyle', 'rule_number': '101.5', 'source': './rule_data/2025-mini-rulebook.pdf', 'rule_id': 'rule_101.5', 'chunk_type': 'rule', 'section': 'FREESTYLE'}\n",
      "{'chunk_type': 'rule', 'rule_number': '101.1', 'rule_id': 'rule_101.1', 'rule_title': 'Start', 'page': '23', 'category': 'Starts', 'section': 'BUTTERFLY', 'source': './rule_data/2025-mini-rulebook.pdf', 'stroke_type': 'starts'}\n",
      "{'page': '23', 'chunk_type': 'rule', 'rule_title': 'Turns', 'rule_number': '101.4', 'section': 'BUTTERFLY', 'stroke_type': 'breaststroke', 'rule_id': 'rule_101.4', 'category': 'Turns and Finishes', 'source': './rule_data/2025-mini-rulebook.pdf'}\n",
      ">>>Assessment: LEGAL. In freestyle, there is no requirement for a two-hand touch at the wall.\n",
      "The swimmer must simply touch the wall upon completion of each length (Rule\n",
      "101.5.3).\n"
     ]
    }
   ],
   "source": [
    "\n",
    "for query in queries[:3]:\n",
    "    print()\n",
    "    print(textwrap.fill(f\"\\n>>>Query: {query}\", width=80))\n",
    "    \n",
    "\n",
    "    # Use the retriever to get relevant documents\n",
    "    docs = retriever.get_relevant_documents(query)\n",
    "    print(f\">>>Retrieved {len(docs)} documents\")\n",
    "    for doc in docs:\n",
    "        print(doc.metadata)\n",
    "        # print(textwrap.fill(f\"\\n>>>Retrieved Document:\\n{doc.page_content}\", width=80)) \n",
    "    # Combine the retrieved documents into a context string\n",
    "    context = \"\\n\".join([doc.page_content for doc in docs])\n",
    "    # print(textwrap.fill(f\"\\n>>>CONTEXT:\\n{context}\", width=80))\n",
    "    # If no documents were retrieved, provide a default message\n",
    "\n",
    "    # Create the prompt template\n",
    "    prompt = HumanMessage(SWIMMING_RULES_PROMPT.format(context=context, question=query))\n",
    "\n",
    "    response = llm([prompt])\n",
    "    # Word wrap the response content if the line is longer than 80 characters\n",
    "    wrapped_response = textwrap.fill(response.content, width=80)\n",
    "    print(f\">>>Assessment: {wrapped_response}\")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}