@arizeai/phoenix-mcp

{ "cells": [ { "cell_type": "markdown", "id": "8a1f0d77-0261-469e-ac52-5fd1e82de4d6", "metadata": { "jp-MarkdownHeadingCollapsed": true }, "source": [ "# Overview\n", "\n", "Observability for all model types (LLM, NLP, CV, Tabular)\n", "\n", "Phoenix Inferences allows you to observe the performance of your model through visualizing all the model’s inferences in one interactive UMAP view.\n", "\n", "This powerful visualization can be leveraged during EDA to understand model drift, find low performing clusters, uncover retrieval issues, and export data for retraining / fine tuning." ] }, { "cell_type": "markdown", "id": "59a31a79-d75a-4217-8d5b-1061b94d6268", "metadata": {}, "source": [ "# Quickstart\n", "\n", "The following Quickstart can be executed in a Jupyter notebook or Google Colab.\n", "\n", "We will begin by logging just a training set. Then proceed to add a production set for comparison." ] }, { "cell_type": "markdown", "id": "204a1126-8ef2-4f64-ba39-0502f6061c12", "metadata": {}, "source": [ "## Step 1: Install and load dependencies\n", "\n", "Use `pip` or `conda` to install `arize-phoenix`." ] }, { "cell_type": "code", "execution_count": null, "id": "1b7ba3d4", "metadata": {}, "outputs": [], "source": [ "%pip install -Uq 'arize-phoenix[embeddings]'" ] }, { "cell_type": "markdown", "id": "38aa109d-337e-47ac-81f0-8416741abfb6", "metadata": {}, "source": [ "## Step 2: Prepare Model Data\n", "\n", "Phoenix visualizes data taken from pandas dataframe, where each row of the dataframe compasses all the information about each inference (including feature values, prediction, metadata, etc.)\n", "\n", "For this Quickstart, we will show an example of visualizing the inferences from a computer vision model. See example notebooks for all model types [here](https://arize.com/docs/phoenix/notebooks).\n", "\n", "Let’s begin by working with the training set for this model." ] }, { "cell_type": "markdown", "id": "5256d4bf-7ba6-41c8-9d43-ea4084fbb68e", "metadata": {}, "source": [ "### Download the dataset and load it into a Pandas dataframe." ] }, { "cell_type": "code", "execution_count": null, "id": "b78779d7", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "import phoenix as px\n", "\n", "train_df = pd.read_parquet(\n", " \"http://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/cv/human-actions/human_actions_training.parquet\"\n", ")" ] }, { "cell_type": "markdown", "id": "249ae962-918d-407b-a506-1b460e47afd7", "metadata": {}, "source": [ "### Preview the dataframe (optional)\n", "\n", "Note that each row contains all the data specific to this CV model for each inference." ] }, { "cell_type": "code", "execution_count": null, "id": "83a12c72-3a60-4399-b704-b6cdbcb57c33", "metadata": {}, "outputs": [], "source": [ "train_df.head()" ] }, { "cell_type": "markdown", "id": "ddaf9f03-4fdf-425f-8f14-fb05d50131f1", "metadata": {}, "source": [ "## Step 3: Define a Schema\n", "\n", "Before we can log these inferences, we need to define a Schema object to describe them.\n", "\n", "The Schema object informs Phoenix of the fields that the columns of the dataframe should map to.\n", "\n", "Here we define a Schema to describe our particular CV training set:" ] }, { "cell_type": "code", "execution_count": null, "id": "76b04e52", "metadata": {}, "outputs": [], "source": [ "train_schema = px.Schema(\n", " timestamp_column_name=\"prediction_ts\",\n", " prediction_label_column_name=\"predicted_action\",\n", " actual_label_column_name=\"actual_action\",\n", " embedding_feature_column_names={\n", " \"image_embedding\": px.EmbeddingColumnNames(\n", " vector_column_name=\"image_vector\",\n", " link_to_data_column_name=\"url\",\n", " ),\n", " },\n", ")" ] }, { "cell_type": "markdown", "id": "7d3b229c-c34c-4ec5-bcfa-ae8fefc9aee2", "metadata": {}, "source": [ "***Important:*** The fields used in a Schema will vary depending on the model type that you are working with.\n", "\n", "For examples on how Schema are defined for other model types (NLP, tabular, LLM-based applications), see example notebooks under [https://arize.com/docs/phoenix/notebooks#embedding-analysis](Embedding Analysis) and [https://arize.com/docs/phoenix/notebooks#structured-data-analysis](Structured Data Analysis)." ] }, { "cell_type": "markdown", "id": "2542e3f4", "metadata": {}, "source": [ "## Step 4: Wrap into Inference Object\n", "\n", "Wrap your `train_df` and schema `train_schema` into a Phoenix `inferences` object:" ] }, { "cell_type": "code", "execution_count": null, "id": "ec7cbaf4", "metadata": {}, "outputs": [], "source": [ "train_ds = px.Inferences(dataframe=train_df, schema=train_schema, name=\"training\")" ] }, { "cell_type": "markdown", "id": "525803af", "metadata": {}, "source": [ "## Step 5: Launch Phoenix!\n", "\n", "We are now ready to launch Phoenix with our Inferences!\n", "\n", "Here, we are passing `train_ds` as the `primary` inferences, as we are only visualizing one inference set (see Step 6 for adding additional inference sets)." ] }, { "cell_type": "code", "execution_count": null, "id": "6c4e42bb", "metadata": {}, "outputs": [], "source": [ "session = px.launch_app(primary=train_ds)" ] }, { "cell_type": "markdown", "id": "de0f2a9b-dc3c-432c-9cca-be18d3176a31", "metadata": {}, "source": [ "Running this will fire up a Phoenix visualization. Follow in the instructions in the output to view Phoenix in a browser, or in-line in your notebook. \n", "Optional - try the following exercises to familiarize yourself more with Phoenix:\n", "\n", "\n", "**You are now ready to observe the training set of your model!**" ] }, { "cell_type": "markdown", "id": "d16e6019-ae50-4814-961b-ed45d1fe48dd", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "b1c4a3da-9d7b-4577-ae64-4d846a84a0c5", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "b8a0711a-54c9-4b13-8583-0e0d22d1295c", "metadata": {}, "source": [ "# Optional actions and activities" ] }, { "cell_type": "markdown", "id": "8de7a986-ac2c-49f8-9f4d-1f36d01faf8c", "metadata": {}, "source": [ "## Checkpoint A - Exercises to familiarize yourself more with Phoenix:\n", "\n", "- [ ] Click on `image_embedding` under the Embeddings section to enter the UMAP projector view\n", "- [ ] Select a point where the model accuracy is <0.78, and see the embedding visualization below update to include only points from this selected timeframe\n", "- [ ] Select the cluster with the lowest accuracy; from the list of automatic clusters generated by Phoenix\n", " - Note that Phoenix automatically generates clusters for you on your data using a clustering algorithm called HDBSCAN (more information: [https://arize.com/docs/phoenix/concepts/embeddings-analysis#clusters](https://arize.com/docs/phoenix/concepts/embeddings-analysis#clusters)\n", "- [ ] Change the colorization of your plot - e.g. select Color By ‘correctness’, and ‘dimension'\n", "- [ ] Describe in words an insight you've gathered from this visualization\n", "\n", "*Discuss your answers in our [https://arize-ai.slack.com/join/shared_invite/zt-2w57bhem8-hq24MB6u7yE_ZF_ilOYSBw#/shared-invite/email](community)!*" ] }, { "cell_type": "markdown", "id": "196b6e18-80ca-402f-a385-e798ed25d2f1", "metadata": {}, "source": [ "## Add comparison data\n", "\n", "In order to visualize drift, conduct A/B model comparisons, or in the case of an information retrieval use case, compare inferences against a [corpus](https://arize.com/docs/phoenix/~/changes/v6Zhm276x8LlKmwqElIA/how-to/define-your-schema/corpus-data), you will need to add a comparison dataset to your visualization.\n", "\n", "We will continue on with our CV model example above, and add a set of production data from our model to our visualization.\n", "\n", "This will allow us to analyze drift and conduct A/B comparisons of our production data against our training set." ] }, { "cell_type": "markdown", "id": "ebe1f414-55f3-4a20-b772-6c5e95cf4fd6", "metadata": {}, "source": [ "### Prepare production inferences" ] }, { "cell_type": "code", "execution_count": null, "id": "23300c39-299b-4a58-978b-a0042b265d9b", "metadata": {}, "outputs": [], "source": [ "prod_df = pd.read_parquet(\n", " \"http://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/cv/human-actions/human_actions_training.parquet\"\n", ")" ] }, { "cell_type": "markdown", "id": "66e2e229-a472-4971-880f-19953611ad76", "metadata": {}, "source": [ "#### View Data (Optional)" ] }, { "cell_type": "code", "execution_count": null, "id": "c34faac3-fc3c-4a6d-a5e4-a8fe884511b9", "metadata": {}, "outputs": [], "source": [ "prod_df.head()" ] }, { "cell_type": "markdown", "id": "062202f0-ec5c-4707-baad-eb6412e8328d", "metadata": {}, "source": [ "### Define model schema\n", "\n", "Note that this schema differs slightly from our `train_schema` above, as our `prod_df` does not have a ground truth column!" ] }, { "cell_type": "code", "execution_count": null, "id": "207ced60-da1d-4cd7-9175-37e0e13347fc", "metadata": {}, "outputs": [], "source": [ "prod_schema = px.Schema(\n", " timestamp_column_name=\"prediction_ts\",\n", " prediction_label_column_name=\"predicted_action\",\n", " embedding_feature_column_names={\n", " \"image_embedding\": px.EmbeddingColumnNames(\n", " vector_column_name=\"image_vector\",\n", " link_to_data_column_name=\"url\",\n", " ),\n", " },\n", ")" ] }, { "cell_type": "markdown", "id": "6891e7af-8e14-458e-ac40-061de06cc69d", "metadata": {}, "source": [ "***When do I need a different schema?***\n", "\n", "In general, if both sets of inferences you are visualizing have identical schemas, you can reuse the Schema object.\n", "\n", "However, there are often differences between the schema of a primary and reference dataset. For example:\n", "- Your production set does not include any ground truth, but your training set does.\n", "- Your primary dataset is the set of prompt-responses in an LLM application, and your reference is your corpus.\n", "- Your production data has differing timestamps between all inferences, but your training set does not have a timestamp column.\n", "\n", "Read more about comparison dataset Schemas here: [How many schemas do I need?](https://arize.com/docs/phoenix/~/changes/v6Zhm276x8LlKmwqElIA/quickstart/phoenix-inferences/inferences)" ] }, { "cell_type": "markdown", "id": "06472099-60bb-45e4-9bc7-f845bcedd5b5", "metadata": {}, "source": [ "### Wrap into Inferences object" ] }, { "cell_type": "code", "execution_count": null, "id": "e6e52bc5", "metadata": {}, "outputs": [], "source": [ "prod_ds = px.Inferences(dataframe=prod_df, schema=prod_schema, name=\"production\")" ] }, { "cell_type": "markdown", "id": "5f3861be-5ec8-414c-a5a9-2e5eb09607ba", "metadata": {}, "source": [ "### Launch Phoenix with both Inferences!\n", "\n", "This time, we will include both `train_ds` and `prod_ds` when calling `launch_app`." ] }, { "cell_type": "code", "execution_count": null, "id": "97697c34-4121-466e-9dcb-d11532dbed4f", "metadata": {}, "outputs": [], "source": [ "session = px.launch_app(primary=prod_ds, reference=train_ds)" ] }, { "cell_type": "markdown", "id": "e3b6cbb5-6785-43f9-9eb6-5c9335f7b5f8", "metadata": {}, "source": [ "***What data should I set as `reference` and as `primary`?***\n", "\n", "Select the inferences that you want to use as the referential baseline as your `reference`, and the dataset you'd like to actively evaluate as your `primary`.\n", "\n", "In this case, training is our referential baseline, for which we want to gauge the behavior (e.g. evaluate drift) of our production data against." ] }, { "cell_type": "markdown", "id": "706d445e-d084-4b42-898e-8c6729981dd8", "metadata": {}, "source": [ "Once again, enter your Phoenix app with the new link generated by your session." ] }, { "cell_type": "markdown", "id": "bbaaab26-4200-4577-ab06-835d05d5283c", "metadata": {}, "source": [ "You are now ready to conduct comparative Root Cause Analysis!\n", "\n", "" ] }, { "cell_type": "markdown", "id": "5cc6a724-0616-4699-a5b3-8f36595de780", "metadata": {}, "source": [ "### Checkpoint B - try the following exercises to familiarize yourself more with Phoenix:\n", "\n", "- [ ] Click into `image_embedding` under the Embeddings listing to enter the UMAP projector \n", "- [ ] Select a point on the time series where there is high drift (hint: as given by Euclidean Distance), and see the datapoints from the time selection being rendered below \n", "- [ ] While colorizing the data by 'Dataset', select the datapoints with the lasso tool where there exists only production data (hint: this is a set of data that has emerged in prod, and is a cause for the increase in drift!) \n", "- [ ] Export the selected cluster from Phoenix \n", "- [ ] Describe in words the process you went through to understand increased drift in your production data \n", "\n", "Discuss your answers in our [community](https://arize-ai.slack.com/join/shared_invite/zt-2w57bhem8-hq24MB6u7yE_ZF_ilOYSBw#/shared-invite/email)!" ] }, { "cell_type": "markdown", "id": "a6273382-307c-4b7f-976b-6191183c4dfd", "metadata": {}, "source": [ "## Export data\n", "\n", "Once you have identified datapoints of interest, you can export this data directly from the Phoenix app for further analysis, or to incorporate these into downstream model retraining and finetuning flows.\n", "\n", "See more on exporting data [here](https://arize.com/docs/phoenix/inferences/how-to-inferences/export-your-data)." ] }, { "cell_type": "markdown", "id": "ba966787-5d60-45ac-9bdd-73a1b1d48650", "metadata": {}, "source": [ "## Enable production observability with Arize\n", "\n", "Once your model is ready for production, you can add Arize to enable production-grade observability. Phoenix works in conjunction with Arize to enable end-to-end model development and observability.\n", "\n", "With Arize, you will additionally benefit from:\n", "- Being able to publish and observe your models in real-time as inferences are being served, and/or via direct connectors from your table/storage solution\n", "- Scalable compute to handle billions of predictions\n", "- Ability to set up monitors & alerts\n", "- Production-grade observability\n", "- Integration with Phoenix for model iteration to observability\n", "- Enterprise-grade RBAC and SSO\n", "- Experiment with infinite permutations of model versions and filters\n", "\n", "Create your [free account](https://arize.com/join) and see the full suite of [Arize](https://docs.arize.com/arize/) features." ] }, { "cell_type": "markdown", "id": "6b7012ca-483a-4d51-ac24-ff4014f153bf", "metadata": {}, "source": [ "# Where to go from here?\n", "\n", "Read more about Embeddings Analysis [here](https://arize.com/docs/phoenix/inferences/use-cases-inferences/embeddings-analysis)" ] }, { "cell_type": "markdown", "id": "3ad6b2ea-690f-436b-a02c-1c2eb0d38eec", "metadata": {}, "source": [ "# Questions\n", "\n", "Join the [Phoenix Slack community](https://join.slack.com/t/arize-ai/shared_invite/zt-1ppbtg5dd-1CYmQO4dWF4zvXFiONTjMg) to ask questions, share findings, provide feedback, and connect with other developers." ] }, { "cell_type": "code", "execution_count": null, "id": "da74ce24-b2da-435f-978c-2024b9378886", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }