smolagents_evaluator_optimizer.ipynb•8.72 kB
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "hMMdZ1vnxj8f"
},
"source": [
"<center>\n",
" <p style=\"text-align:center\">\n",
" <img alt=\"phoenix logo\" src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/phoenix-logo-light.svg\" width=\"200\"/>\n",
" <br>\n",
" <a href=\"https://arize.com/docs/phoenix/\">Docs</a>\n",
" |\n",
" <a href=\"https://github.com/Arize-ai/phoenix\">GitHub</a>\n",
" |\n",
" <a href=\"https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q\">Community</a>\n",
" </p>\n",
"</center>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "se3M7GECJfpZ"
},
"source": [
"## SmolAgents Parallelization Tutorial\n",
"\n",
"In this tutorial, we'll explore the orchestrator and worker approach using SmolAgents and how to trace the workflow with Phoenix.\n",
"\n",
"The orchestrator and worker model is a powerful technique where a central orchestrator coordinates multiple worker agents, each responsible for specific tasks. This approach enhances the modularity and scalability of workflows. Tracing allows us to monitor this coordinated flow and understand the interactions between the orchestrator and workers.\n",
"\n",
"SmolAgents provides flexible agent capabilities that integrate seamlessly with Python's asyncio library, enabling the orchestration of multiple agent interactions.\n",
"\n",
"By the end of this tutorial, you'll learn how to:\n",
"\n",
"- Set up SmolAgents with tools tailored for orchestrator and worker roles\n",
"- Implement the orchestrator and worker pattern using Python's asyncio library\n",
"- Utilize Phoenix to trace and visualize orchestrator and worker interactions\n",
"- Compare the performance of orchestrated versus non-orchestrated workflows\n",
"\n",
"⚠️ You'll need a Hugging Face Token for this tutorial."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "G2j6_RvFJfpa"
},
"source": [
"## Set up Keys and Dependencies\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install openinference-instrumentation-smolagents smolagents arize-phoenix"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"from getpass import getpass\n",
"\n",
"from smolagents import CodeAgent, HfApiModel, tool\n",
"\n",
"from phoenix.otel import register\n",
"\n",
"if \"HF_TOKEN\" not in os.environ:\n",
" os.environ[\"HF_TOKEN\"] = getpass(\"🔑 Enter your Hugging Face Token: \")\n",
"\n",
"if \"PHOENIX_API_KEY\" not in os.environ:\n",
" os.environ[\"PHOENIX_API_KEY\"] = getpass(\"🔑 Enter your Phoenix API key: \")\n",
"\n",
"if \"PHOENIX_COLLECTOR_ENDPOINT\" not in os.environ:\n",
" os.environ[\"PHOENIX_COLLECTOR_ENDPOINT\"] = getpass(\"🔑 Enter your Phoenix Collector Endpoint\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configure Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Initialize the model\n",
"model = HfApiModel()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ibmucjY_QkaH"
},
"source": [
"## Configure Tracing\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tracer_provider = register(\n",
" project_name=\"smolagents-agents\",\n",
" protocol=\"http/protobuf\",\n",
" auto_instrument=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Define Tools"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"@tool\n",
"def extract_qualifications(profile: str) -> str:\n",
" \"\"\"\n",
" Extracts qualifications from a candidate's profile.\n",
"\n",
" Args:\n",
" profile: Candidate background or resume summary.\n",
"\n",
" Returns:\n",
" List of key skills, experiences, or education.\n",
" \"\"\"\n",
" keywords = [word.strip(\".,\") for word in profile.split() if len(word) > 5]\n",
" return \", \".join(set(keywords))\n",
"\n",
"\n",
"@tool\n",
"def analyze_tone(profile: str) -> str:\n",
" \"\"\"\n",
" Analyzes tone or soft skills based on writing style.\n",
"\n",
" Args:\n",
" profile: Candidate's self-description.\n",
"\n",
" Returns:\n",
" Observations about communication style and tone.\n",
" \"\"\"\n",
" return \"Confident, technically focused, concise communicator.\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Worker Agents"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def resume_screener_agent(profile: str, role: str) -> str:\n",
" agent = CodeAgent(tools=[extract_qualifications], model=model)\n",
" prompt = (\n",
" f\"Review this profile for a role as {role}:\\n\\n{profile}\\n\\n\"\n",
" f\"Use extract_qualifications to pull skills, then give a short score (1-10) based on relevance.\"\n",
" )\n",
" return agent.run(prompt)\n",
"\n",
"\n",
"def culture_fit_agent(profile: str) -> str:\n",
" agent = CodeAgent(tools=[analyze_tone], model=model)\n",
" prompt = (\n",
" f\"Based on the following candidate self-description:\\n\\n{profile}\\n\\n\"\n",
" f\"Use analyze_tone to assess their communication style. Then comment on possible team or culture fit.\"\n",
" )\n",
" return agent.run(prompt)\n",
"\n",
"\n",
"def recommendation_agent(tech_eval: str, culture_eval: str) -> str:\n",
" agent = CodeAgent(tools=[], model=model)\n",
" prompt = (\n",
" f\"A recruiter received these evaluations:\\n\\n\"\n",
" f\"Technical Evaluation:\\n{tech_eval}\\n\\n\"\n",
" f\"Culture Fit Evaluation:\\n{culture_eval}\\n\\n\"\n",
" f\"Based on both, should this candidate move to the next round? Give a yes/no and a short justification.\"\n",
" )\n",
" return agent.run(prompt)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Orchestrator"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def orchestrator(candidate_name: str, profile: str, role: str):\n",
" print(f\"\\n🧑💼 Reviewing: {candidate_name} — for role: {role}\")\n",
"\n",
" # Step 1: Get tech screen\n",
" tech_eval = resume_screener_agent(profile, role)\n",
"\n",
" # Step 2: Get culture screen\n",
" culture_eval = culture_fit_agent(profile)\n",
"\n",
" # Step 3: Get final recommendation\n",
" final_rec = recommendation_agent(tech_eval, culture_eval)\n",
"\n",
" # Show results\n",
" print(f\"\\n📄 Technical Eval:\\n{tech_eval}\")\n",
" print(f\"\\n💬 Culture Fit Eval:\\n{culture_eval}\")\n",
" print(f\"\\n✅ Recommendation:\\n{final_rec}\")\n",
" print(\"\\n\" + \"=\" * 60 + \"\\n\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Run Multiple Candidates in Parallel "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def main():\n",
" role = \"Machine Learning Engineer\"\n",
" candidates = {\n",
" \"Sofia Rao\": \"\"\"5+ years in applied ML. Experience with Transformers, model evaluation, and production pipelines. Previously led a small team at a healthtech startup.\"\"\",\n",
" \"Daniel Kim\": \"\"\"Worked as a backend engineer, now transitioning into ML. Strong Python and infra background. No formal ML experience, but completed multiple online LLM projects.\"\"\",\n",
" }\n",
"\n",
" for name, profile in candidates.items():\n",
" orchestrator(name, profile, role)\n",
"\n",
"\n",
"if __name__ == \"__main__\":\n",
" main()"
]
}
],
"metadata": {
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}