@arizeai/phoenix-mcp

{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "<center>\n", " <p style=\"text-align:center\">\n", " <img alt=\"phoenix logo\" src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/phoenix-logo-light.svg\" width=\"200\"/>\n", " <br>\n", " <a href=\"https://arize.com/docs/phoenix/\">Docs</a>\n", " |\n", " <a href=\"https://github.com/Arize-ai/phoenix\">GitHub</a>\n", " |\n", " <a href=\"https://arize-ai.slack.com/join/shared_invite/zt-2w57bhem8-hq24MB6u7yE_ZF_ilOYSBw#/shared-invite/email\">Community</a>\n", " </p>\n", "</center>\n", "<h1 align=\"center\">Evaluate RAG with LLM Evals</h1>\n", "\n", "In this tutorial we will look into building a RAG pipeline and evaluating it with Phoenix Evals.\n", "\n", "It has the the following sections:\n", "\n", "1. Understanding Retrieval Augmented Generation (RAG).\n", "2. Building RAG (with the help of a framework such as LlamaIndex).\n", "3. Evaluating RAG with Phoenix Evals." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Retrieval Augmented Generation (RAG)\n", "\n", "LLMs are trained on vast datasets, but these will not include your specific data (things like company knowledge bases and documentation). Retrieval-Augmented Generation (RAG) addresses this by dynamically incorporating your data as context during the generation process. This is done not by altering the training data of the LLMs but by allowing the model to access and utilize your data in real-time to provide more tailored and contextually relevant responses.\n", "\n", "In RAG, your data is loaded and prepared for queries. This process is called indexing. User queries act on this index, which filters your data down to the most relevant context. This context and your query then are sent to the LLM along with a prompt, and the LLM provides a response.\n", "\n", "RAG is a critical component for building applications such a chatbots or agents and you will want to know RAG techniques on how to get data into your application.\n", "\n", "<img src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/images/RAG_Pipeline.png\">" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stages within RAG\n", "\n", "There are five key stages within RAG, which will in turn be a part of any larger RAG application.\n", "\n", "- **Loading**: This refers to getting your data from where it lives - whether it's text files, PDFs, another website, a database or an API - into your pipeline.\n", "- **Indexing**: This means creating a data structure that allows for querying the data. For LLMs this nearly always means creating vector embeddings, numerical representations of the meaning of your data, as well as numerous other metadata strategies to make it easy to accurately find contextually relevant data.\n", "- **Storing**: Once your data is indexed, you will want to store your index, along with any other metadata, to avoid the need to re-index it.\n", "\n", "- **Querying**: For any given indexing strategy there are many ways you can utilize LLMs and data structures to query, including sub-queries, multi-step queries, and hybrid strategies. \n", "- **Evaluation**: A critical step in any pipeline is checking how effective it is relative to other strategies, or when you make changes. Evaluation provides objective measures on how accurate, faithful, and fast your responses to queries are.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Build a RAG system \n", "\n", "Now that we have understood the stages of RAG, let's build a pipeline. We will use [LlamaIndex](https://www.llamaindex.ai/) for RAG and [Phoenix Evals](https://arize.com/docs/phoenix/llm-evals/llm-evals) for evaluation.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install -qq \"arize-phoenix[evals,llama-index]\" \"arize-phoenix-client\" \"llama-index-llms-openai\" \"openai>=1\" gcsfs nest_asyncio 'httpx<0.28'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For this tutorial we will be using OpenAI for creating synthetic data as well as for evaluation. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "from getpass import getpass\n", "\n", "if not (openai_api_key := os.getenv(\"OPENAI_API_KEY\")):\n", " openai_api_key = getpass(\"🔑 Enter your OpenAI API key: \")\n", "os.environ[\"OPENAI_API_KEY\"] = openai_api_key" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# The nest_asyncio module enables the nesting of asynchronous functions within an already running async loop.\n", "# This is necessary because Jupyter notebooks inherently operate in an asynchronous loop.\n", "# By applying nest_asyncio, we can run additional async functions within this existing loop without conflicts.\n", "import nest_asyncio\n", "\n", "nest_asyncio.apply()\n", "\n", "import pandas as pd\n", "from llama_index.core import SimpleDirectoryReader, VectorStoreIndex\n", "from llama_index.core.node_parser import SimpleNodeParser\n", "from llama_index.llms.openai import OpenAI\n", "\n", "import phoenix as px\n", "from phoenix.client import Client\n", "from phoenix.client.types.spans import SpanQuery # pyright: ignore[reportUnusedImport]\n", "\n", "client = Client()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "During this tutorial, we will capture all the data we need to evaluate our RAG pipeline using Phoenix Tracing. To enable this, simply start the phoenix application and instrument LlamaIndex." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "px.launch_app().view()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Enable Phoenix tracing via `LlamaIndexInstrumentor`. Phoenix uses OpenInference traces - an open-source standard for capturing and storing LLM application traces that enables LLM applications to seamlessly integrate with LLM observability solutions such as Phoenix." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from openinference.instrumentation.llama_index import LlamaIndexInstrumentor\n", "\n", "from phoenix.otel import register\n", "\n", "tracer_provider = register(endpoint=\"http://127.0.0.1:6006/v1/traces\")\n", "LlamaIndexInstrumentor().instrument(skip_dep_check=True, tracer_provider=tracer_provider)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load Data and Build an Index" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's use an [essay by Paul Graham](https://www.paulgraham.com/worked.html) to build our RAG pipeline." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import tempfile\n", "from urllib.request import urlretrieve\n", "\n", "with tempfile.NamedTemporaryFile() as tf:\n", " urlretrieve(\n", " \"https://raw.githubusercontent.com/Arize-ai/phoenix-assets/main/data/paul_graham/paul_graham_essay.txt\",\n", " tf.name,\n", " )\n", " documents = SimpleDirectoryReader(input_files=[tf.name]).load_data()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Define an LLM\n", "llm = OpenAI(model=\"gpt-4o\")\n", "\n", "# Build index with a chunk_size of 512\n", "node_parser = SimpleNodeParser.from_defaults(chunk_size=512)\n", "nodes = node_parser.get_nodes_from_documents(documents)\n", "vector_index = VectorStoreIndex(nodes)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Build a QueryEngine and start querying." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "query_engine = vector_index.as_query_engine()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "response_vector = query_engine.query(\"What did the author do growing up?\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check the response that you get from the query." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "response_vector.response" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default LlamaIndex retrieves two similar nodes/ chunks. You can modify that in `vector_index.as_query_engine(similarity_top_k=k)`.\n", "\n", "Let's check the text in each of these retrieved nodes." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# First retrieved node\n", "response_vector.source_nodes[0].get_text()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Second retrieved node\n", "response_vector.source_nodes[1].get_text()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Remember that we are using Phoenix Tracing to capture all the data we need to evaluate our RAG pipeline. You can view the traces in the phoenix application." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"phoenix URL\", px.active_session().url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can access the traces by directly pulling the spans from the phoenix session." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "spans_df = client.spans.get_spans_dataframe()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "spans_df[[\"name\", \"span_kind\", \"attributes.input.value\", \"attributes.retrieval.documents\"]].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that the traces have captured the documents that were retrieved by the query engine. This is nice because it means we can introspect the documents without having to keep track of them ourselves." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "spans_with_docs_df = spans_df[spans_df[\"attributes.retrieval.documents\"].notnull()]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "spans_with_docs_df[[\"attributes.input.value\", \"attributes.retrieval.documents\"]].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have built a RAG pipeline and also have instrumented it using Phoenix Tracing. We now need to evaluate it's performance. We can assess our RAG system/query engine using Phoenix's LLM Evals. Let's examine how to leverage these tools to quantify the quality of our retrieval-augmented generation system." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Evaluation\n", "\n", "Evaluation should serve as the primary metric for assessing your RAG application. It determines whether the pipeline will produce accurate responses based on the data sources and range of queries.\n", "\n", "While it's beneficial to examine individual queries and responses, this approach is impractical as the volume of edge-cases and failures increases. Instead, it's more effective to establish a suite of metrics and automated evaluations. These tools can provide insights into overall system performance and can identify specific areas that may require scrutiny.\n", "\n", "In a RAG system, evaluation focuses on two critical aspects:\n", "\n", "- **Retrieval Evaluation**: To assess the accuracy and relevance of the documents that were retrieved\n", "- **Response Evaluation**: Measure the appropriateness of the response generated by the system when the context was provided." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Generate Question Context Pairs\n", "\n", "For the evaluation of a RAG system, it's essential to have queries that can fetch the correct context and subsequently generate an appropriate response.\n", "\n", "For this tutorial, let's use Phoenix's `llm_generate` to help us create the question-context pairs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, let's create a dataframe of all the document chunks that we have indexed." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Let's construct a dataframe of just the documents that are in our index\n", "document_chunks_df = pd.DataFrame({\"text\": [node.get_text() for node in nodes]})\n", "document_chunks_df = document_chunks_df.sample(10, random_state=42)\n", "document_chunks_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have the document chunks, let's prompt an LLM to generate us 3 questions per chunk. Note that you could manually solicit questions from your team or customers, but this is a quick and easy way to generate a large number of questions." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "generate_questions_template = \"\"\"\\\n", "Context information is below.\n", "\n", "---------------------\n", "{text}\n", "---------------------\n", "\n", "Given the context information and not prior knowledge.\n", "generate only questions based on the below query.\n", "\n", "You are a Teacher/ Professor. Your task is to setup \\\n", "3 questions for an upcoming \\\n", "quiz/examination. The questions should be diverse in nature \\\n", "across the document. Restrict the questions to the \\\n", "context information provided.\"\n", "\n", "Output the questions in JSON format with the keys question_1, question_2, question_3.\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import json\n", "\n", "from phoenix.evals import OpenAIModel, llm_generate\n", "\n", "\n", "def output_parser(response: str, index: int):\n", " try:\n", " return json.loads(response)\n", " except json.JSONDecodeError as e:\n", " return {\"__error__\": str(e)}\n", "\n", "\n", "questions_df = llm_generate(\n", " dataframe=document_chunks_df,\n", " template=generate_questions_template,\n", " model=OpenAIModel(model=\"gpt-3.5-turbo\"),\n", " output_parser=output_parser,\n", " concurrency=20,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "questions_df.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Construct a dataframe of the questions and the document chunks\n", "questions_with_document_chunk_df = pd.concat([questions_df, document_chunks_df], axis=1)\n", "questions_with_document_chunk_df = questions_with_document_chunk_df.melt(\n", " id_vars=[\"text\"], value_name=\"question\"\n", ").drop(\"variable\", axis=1)\n", "# If the above step was interrupted, there might be questions missing. Let's run this to clean up the dataframe.\n", "questions_with_document_chunk_df = questions_with_document_chunk_df[\n", " questions_with_document_chunk_df[\"question\"].notnull()\n", "]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The LLM has generated three questions per chunk. Let's take a quick look." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "questions_with_document_chunk_df.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Retrieval Evaluation\n", "\n", "We are now prepared to perform our retrieval evaluations. We will execute the queries we generated in the previous step and verify whether or not that the correct context is retrieved." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# loop over the questions and generate the answers\n", "for _, row in questions_with_document_chunk_df.iterrows():\n", " question = row[\"question\"]\n", " response_vector = query_engine.query(question)\n", " print(f\"Question: {question}\\nAnswer: {response_vector.response}\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have executed the queries, we can start validating whether or not the RAG system was able to retrieve the correct context. Let's extract all the retrieved documents from the traces logged to phoenix. (For an in-depth explanation of how to export trace data from the phoenix runtime, consult the [docs](https://arize.com/docs/phoenix/how-to/extract-data-from-spans))." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from openinference.semconv.trace import DocumentAttributes, SpanAttributes\n", "\n", "retrieved_documents_df = client.spans.get_spans_dataframe(\n", " query=(\n", " SpanQuery()\n", " .where(\"span_kind == 'RETRIEVER'\")\n", " .select(\"trace_id\", SpanAttributes.INPUT_VALUE)\n", " .explode(\n", " SpanAttributes.RETRIEVAL_DOCUMENTS,\n", " reference=DocumentAttributes.DOCUMENT_CONTENT,\n", " document_score=DocumentAttributes.DOCUMENT_SCORE,\n", " )\n", " )\n", ")\n", "retrieved_documents_df.rename(columns={\"input.value\": \"input\"}, inplace=True)\n", "retrieved_documents_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's now use Phoenix's LLM Evals to evaluate the relevance of the retrieved documents with regards to the query. Note, we've turned on `explanations` which prompts the LLM to explain it's reasoning. This can be useful for debugging and for figuring out potential corrective actions." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from phoenix.evals import (\n", " RelevanceEvaluator,\n", " run_evals,\n", ")\n", "\n", "relevance_evaluator = RelevanceEvaluator(OpenAIModel(model=\"gpt-4o\"))\n", "\n", "retrieved_documents_relevance_df = run_evals(\n", " evaluators=[relevance_evaluator],\n", " dataframe=retrieved_documents_df,\n", " provide_explanation=True,\n", " concurrency=20,\n", ")[0]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "retrieved_documents_relevance_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now combine the documents with the relevance evaluations to compute retrieval metrics. These metrics will help us understand how well the RAG system is performing." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "documents_with_relevance_df = pd.concat(\n", " [retrieved_documents_df, retrieved_documents_relevance_df.add_prefix(\"eval_\")], axis=1\n", ")\n", "documents_with_relevance_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's compute Normalized Discounted Cumulative Gain [NCDG](https://en.wikipedia.org/wiki/Discounted_cumulative_gain) at 2 for all our retrieval steps. In information retrieval, this metric is often used to measure effectiveness of search engine algorithms and related applications." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "from sklearn.metrics import ndcg_score\n", "\n", "\n", "def _compute_ndcg(df: pd.DataFrame, k: int):\n", " \"\"\"Compute NDCG@k in the presence of missing values\"\"\"\n", " n = max(2, len(df))\n", " eval_scores = np.zeros(n)\n", " doc_scores = np.zeros(n)\n", " eval_scores[: len(df)] = df.eval_score\n", " doc_scores[: len(df)] = df.document_score\n", " try:\n", " return ndcg_score([eval_scores], [doc_scores], k=k)\n", " except ValueError:\n", " return np.nan\n", "\n", "\n", "ndcg_at_2 = pd.DataFrame(\n", " {\"score\": documents_with_relevance_df.groupby(\"context.span_id\").apply(_compute_ndcg, k=2)}\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ndcg_at_2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's also compute precision at 2 for all our retrieval steps." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "precision_at_2 = pd.DataFrame(\n", " {\n", " \"score\": documents_with_relevance_df.groupby(\"context.span_id\").apply(\n", " lambda x: x.eval_score[:2].sum(skipna=False) / 2\n", " )\n", " }\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "precision_at_2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Lastly, let's compute whether or not a correct document was retrieved at all for each query (e.g. a hit)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hit = pd.DataFrame(\n", " {\n", " \"hit\": documents_with_relevance_df.groupby(\"context.span_id\").apply(\n", " lambda x: x.eval_score[:2].sum(skipna=False) > 0\n", " )\n", " }\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's now view the results in a combined dataframe." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "retrievals_df = client.spans.get_spans_dataframe(\n", " query=(\n", " SpanQuery()\n", " .where(\"span_kind == 'RETRIEVER' and input.value is not None\")\n", " .select(\"input.value\")\n", " )\n", ")\n", "rag_evaluation_dataframe = pd.concat(\n", " [\n", " retrievals_df[\"input.value\"],\n", " ndcg_at_2.add_prefix(\"ncdg@2_\"),\n", " precision_at_2.add_prefix(\"precision@2_\"),\n", " hit,\n", " ],\n", " axis=1,\n", ")\n", "rag_evaluation_dataframe" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Observations\n", "\n", "Let's now take our results and aggregate them to get a sense of how well our RAG system is performing." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Aggregate the scores across the retrievals\n", "results = rag_evaluation_dataframe.mean(numeric_only=True)\n", "results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see from the above numbers, our RAG system is not perfect, there are times when it fails to retrieve the correct context within the first two documents. At other times the correct context is included in the top 2 results but non-relevant information is also included in the context. This is an indication that we need to improve our retrieval strategy. One possible solution could be to increase the number of documents retrieved and then use a more sophisticated ranking strategy (such as a reranker) to select the correct context." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have now evaluated our RAG system's retrieval performance. Let's send these evaluations to Phoenix for visualization. By sending the evaluations to Phoenix, you will be able to view the evaluations alongside the traces that were captured earlier." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from phoenix.client.__generated__ import v1\n", "\n", "ndcg_rows = ndcg_at_2.reset_index()[[\"context.span_id\", \"score\"]].dropna()\n", "prec_rows = precision_at_2.reset_index()[[\"context.span_id\", \"score\"]].dropna()\n", "\n", "ndcg_annotations: list[v1.SpanAnnotationData] = [\n", " v1.SpanAnnotationData(\n", " name=\"ndcg@2\",\n", " span_id=str(row[\"context.span_id\"]),\n", " annotator_kind=\"CODE\",\n", " result={\"score\": float(row[\"score\"])},\n", " )\n", " for _, row in ndcg_rows.iterrows()\n", "]\n", "\n", "precision_annotations: list[v1.SpanAnnotationData] = [\n", " v1.SpanAnnotationData(\n", " name=\"precision@2\",\n", " span_id=str(row[\"context.span_id\"]),\n", " annotator_kind=\"CODE\",\n", " result={\"score\": float(row[\"score\"])},\n", " )\n", " for _, row in prec_rows.iterrows()\n", "]\n", "\n", "client.spans.log_span_annotations(span_annotations=ndcg_annotations, sync=False)\n", "client.spans.log_span_annotations(span_annotations=precision_annotations, sync=False)\n", "\n", "# Document-level relevance annotations are omitted until document annotation logging is supported\n", "# in the client." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Response Evaluation\n", "\n", "The retrieval evaluations demonstrates that our RAG system is not perfect. However, it's possible that the LLM is able to generate the correct response even when the context is incorrect. Let's evaluate the responses generated by the LLM." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from phoenix.client.types.spans import SpanQuery as _SpanQuery\n", "\n", "qa_df = client.spans.get_spans_dataframe(\n", " query=(\n", " _SpanQuery()\n", " .select(\"span_id\", \"input.value\", \"output.value\")\n", " .where(\"parent_id is None\")\n", " .with_index(\"trace_id\")\n", " )\n", ")\n", "\n", "docs_df = client.spans.get_spans_dataframe(\n", " query=(\n", " _SpanQuery()\n", " .where(\"span_kind == 'RETRIEVER'\")\n", " .concat(\"retrieval.documents\", reference=\"document.content\")\n", " .with_index(\"trace_id\")\n", " )\n", ")\n", "\n", "if qa_df is None or qa_df.empty:\n", " print(\"No spans found.\")\n", " qa_with_reference_df = qa_df\n", "elif docs_df is None or docs_df.empty:\n", " print(\"No retrieval documents found.\")\n", " qa_with_reference_df = None\n", "else:\n", " ref = docs_df[[\"reference\"]]\n", " qa_with_reference_df = pd.concat([qa_df, ref], axis=1, join=\"inner\").set_index(\n", " \"context.span_id\"\n", " )\n", "\n", "qa_with_reference_df.rename(\n", " columns={\"input.value\": \"input\", \"output.value\": \"output\"}, inplace=True\n", ")\n", "qa_with_reference_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have a dataset of the question, context, and response (input, reference, and output), we now can measure how well the LLM is responding to the queries. For details on the QA correctness evaluation, see the [LLM Evals documentation](https://arize.com/docs/phoenix/llm-evals/running-pre-tested-evals/q-and-a-on-retrieved-data)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from phoenix.evals import (\n", " HallucinationEvaluator,\n", " OpenAIModel,\n", " QAEvaluator,\n", " run_evals,\n", ")\n", "\n", "qa_evaluator = QAEvaluator(OpenAIModel(model=\"gpt-4-turbo-preview\"))\n", "hallucination_evaluator = HallucinationEvaluator(OpenAIModel(model=\"gpt-4-turbo-preview\"))\n", "\n", "qa_correctness_eval_df, hallucination_eval_df = run_evals(\n", " evaluators=[qa_evaluator, hallucination_evaluator],\n", " dataframe=qa_with_reference_df,\n", " provide_explanation=True,\n", " concurrency=20,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "qa_correctness_eval_df.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hallucination_eval_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Observations\n", "\n", "Let's now take our results and aggregate them to get a sense of how well the LLM is answering the questions given the context." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "qa_correctness_eval_df.mean(numeric_only=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hallucination_eval_df.mean(numeric_only=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our QA Correctness score of `0.91` and a Hallucinations score `0.05` signifies that the generated answers are correct ~91% of the time and that the responses contain hallucinations 5% of the time - there is room for improvement. This could be due to the retrieval strategy or the LLM itself. We will need to investigate further to determine the root cause." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since we have evaluated our RAG system's QA performance and Hallucinations performance, let's send these evaluations to Phoenix for visualization." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from phoenix.client.__generated__ import v1\n", "\n", "# Expect dataframes indexed by context.span_id\n", "qac_rows = qa_correctness_eval_df.reset_index()[\n", " [\"context.span_id\", \"score\", \"label\", \"explanation\"]\n", "].dropna(how=\"all\", subset=[\"score\", \"label\", \"explanation\"])\n", "hall_rows = hallucination_eval_df.reset_index()[\n", " [\"context.span_id\", \"score\", \"label\", \"explanation\"]\n", "].dropna(how=\"all\", subset=[\"score\", \"label\", \"explanation\"])\n", "\n", "# Construct annotations in the same style as retrieval metrics\n", "qa_annotations: list[v1.SpanAnnotationData] = [\n", " v1.SpanAnnotationData(\n", " name=\"Q&A Correctness\",\n", " span_id=str(row[\"context.span_id\"]),\n", " annotator_kind=\"LLM\",\n", " result={\n", " **({\"score\": float(row[\"score\"])} if pd.notna(row[\"score\"]) else {}),\n", " **({\"label\": str(row[\"label\"])} if pd.notna(row[\"label\"]) else {}),\n", " **({\"explanation\": str(row[\"explanation\"])} if pd.notna(row[\"explanation\"]) else {}),\n", " },\n", " )\n", " for _, row in qac_rows.iterrows()\n", "]\n", "\n", "hall_annotations: list[v1.SpanAnnotationData] = [\n", " v1.SpanAnnotationData(\n", " name=\"Hallucination\",\n", " span_id=str(row[\"context.span_id\"]),\n", " annotator_kind=\"LLM\",\n", " result={\n", " **({\"score\": float(row[\"score\"])} if pd.notna(row[\"score\"]) else {}),\n", " **({\"label\": str(row[\"label\"])} if pd.notna(row[\"label\"]) else {}),\n", " **({\"explanation\": str(row[\"explanation\"])} if pd.notna(row[\"explanation\"]) else {}),\n", " },\n", " )\n", " for _, row in hall_rows.iterrows()\n", "]\n", "\n", "client.spans.log_span_annotations(span_annotations=qa_annotations, sync=False)\n", "client.spans.log_span_annotations(span_annotations=hall_annotations, sync=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now have sent all our evaluations to Phoenix. Let's go to the Phoenix application and view the results! Since we've sent all the evals to Phoenix, we can analyze the results together to make a determination on whether or not poor retrieval or irrelevant context has an effect on the LLM's ability to generate the correct response." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"phoenix URL\", px.active_session().url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion\n", "\n", "We have explored how to build and evaluate a RAG pipeline using LlamaIndex and Phoenix, with a specific focus on evaluating the retrieval system and generated responses within the pipelines. \n", "\n", "Phoenix offers a variety of other evaluations that can be used to assess the performance of your LLM Application. For more details, see the [LLM Evals](https://arize.com/docs/phoenix/llm-evals/llm-evals) documentation." ] } ], "metadata": { "kernelspec": { "display_name": "dev", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 2 }