User adaptivity can reverse LLM rankings between offline and live tests
Offline difference-in-means tracks a shared prompt distribution, while live A/B tests track prompts users form after seeing each model.
2 stories, newest first · one card per story
Offline difference-in-means tracks a shared prompt distribution, while live A/B tests track prompts users form after seeing each model.
In a churn-reactivation A/B test it raised conversion 1% and cut promotion cost 6% versus Segment-EMA.
Today 2.6K readers · 2.9K views · 145 reads · all time 206K views