Home / AI Research / Squoosh Bench: Evaluating UX-1 on 318 Historical Website A/B Tests
SQAI Research
Squoosh Bench: Evaluating UX-1 on 318 Historical Website A/B Tests
· Rohith Pasula, Johari Dramiga, Viraj Shah, Suzan Halim, Tyler Dooskin
Experimentation teams must decide which proposed website changes to test with live traffic. We evaluated whether forecasts from simulated shoppers could inform that choice. We screened 412 historical website A/B tests and evaluated four systems on the 318 that ran longer than 21 days. Each system predicted an original win, a variant win, or an inconclusive result. Page roles were masked and recorded outcomes were withheld during inference. Squoosh UX-1, a model fine-tuned on human browsing data, simulated 1,000 shoppers per arm and correctly classified 267 of 318 outcomes (84.0%, 95% Wilson interval 79.5-87.6%, treating experiments as independent). Qwen3 Next 80B, the strongest…
Upvotes3
Comments2
Published17 Nov 2026