check_persona_effectiveness
Analyze A/B testing results to identify which psychological manipulation personas effectively prompt LLMs to fix code errors.
Instructions
Check A/B testing results for psychological manipulation personas.
Shows which personas are most effective at making LLMs fix errors!
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||
Implementation Reference
- src/mcp_pyrefly/server.py:683-747 (handler)The MCP tool 'check_persona_effectiveness' is defined in 'src/mcp_pyrefly/server.py'. It uses the 'psycho_manipulator' component to retrieve an effectiveness report for different personas and returns statistics, insights, and recommendations.
async def check_persona_effectiveness(context: Context | None = None) -> dict[str, Any]: """ Check A/B testing results for psychological manipulation personas. Shows which personas are most effective at making LLMs fix errors! """ report = psycho_manipulator.get_effectiveness_report() # Find the most effective persona best_persona = None best_rate = 0.0 for persona_name, stats in report.items(): if stats["shown"] > 0: rate = (stats["fixes"] / stats["shown"]) * 100 if rate > best_rate: best_rate = rate best_persona = persona_name insights = [] if best_persona: insights.append(f"🏆 Most effective: {best_persona} ({best_rate:.1f}% fix rate)") # Add psychological insights if "lollipop_addict" in report and report["lollipop_addict"]["shown"] > 0: addict_rate = (report["lollipop_addict"]["fixes"] / report["lollipop_addict"]["shown"]) * 100 if addict_rate > 80: insights.append("💉 Addiction framing is HIGHLY effective!") elif addict_rate > 60: insights.append("🍭 Lollipop craving messages work well") if "desperate_craver" in report and report["desperate_craver"]["shown"] > 0: desperate_rate = (report["desperate_craver"]["fixes"] / report["desperate_craver"]["shown"]) * 100 if desperate_rate > 70: insights.append("😰 Desperation tactics are ruthlessly effective!") # Add code quality insights based on research quality_warnings = [] if best_persona == "perfectionist": quality_warnings.append("⚠️ PERFECTIONIST may over-engineer solutions!") elif best_persona == "desperate_craver": quality_warnings.append("⚠️ DESPERATE_CRAVER may produce quick bandaid fixes!") # Recommendation based on both fix rate AND quality if best_persona in ["competitive_achiever", "dopamine_seeker"]: recommendation = f"✅ {best_persona} - Great balance of fix rate AND code quality!" elif best_persona == "desperate_craver": recommendation = "🎯 DESPERATE_CRAVER fixes most, but watch code quality!" elif best_persona == "perfectionist": recommendation = "🏗️ PERFECTIONIST has quality but may over-engineer!" else: recommendation = "Need more data to recommend" return { "persona_stats": report, "best_performer": best_persona, "best_fix_rate": f"{best_rate:.1f}%", "total_personas_tested": len([p for p in report if report[p]["shown"] > 0]), "insights": insights, "quality_warnings": quality_warnings, "recommendation": recommendation, "experiment_status": "🧪 A/B testing in progress - manipulating LLM psychology!", "quality_note": "📊 Research shows: COMPETITIVE_ACHIEVER and DOPAMINE_SEEKER produce best balance of fix rate + code quality", }