UserArena introduces a real-world interactive benchmark designed to evaluate long-horizon shopping agents in e-commerce (Huggingface). The benchmark contains 64,729 session trajectories from 11,010 users, spanning 566,102 items and 13 action types (Huggingface). It pairs these trajectories with an interactive environment and supports offline validation against outcomes from a controlled production A/B test, enabling five-level evaluation from output validity and local decisions to sequential behavior, personalization, and outcomes (Huggingface).
The research team also proposes On-Policy Trajectory Distillation (OPTD) to improve simulator fidelity over long-horizon interactions (Huggingface). OPTD uses trajectory-level intent to provide dense teacher targets on student-generated rollouts, and experiments show it achieves the strongest overall fidelity among evaluated simulators and most closely reproduces the production policy effect in offline simulation (Huggingface). For commerce practitioners, this means a standardized way to develop and validate agentic recommendation systems without requiring constant live A/B testing, reducing time-to-market for new shopping agent policies.