Optimizely’s Stats Accelerator: Understanding Its Real World Use Case

By Atticus Li

Lead, Conversion Rate Optimization & UX Fortune 150 Company

As a professional responsible for Conversion Rate Optimization and User Experience at a large enterprise, our team has transitioned to Optimizely to execute and analyze experiments. Recently, I realized I had a misunderstanding regarding Optimizely’s Stats Accelerator feature. I previously thought Stats Accelerator could help accelerate any standard A/B test by dynamically reallocating traffic between the control and a single variation. However, after consulting with the Optimizely team, I learned this was not the intended use case.

Here, I want to clarify precisely what Stats Accelerator is designed for and how it differs from other dynamic allocation methods such as Multi-Armed Bandit (MAB) and Optimizely’s Auto Allocate and hope this information would be helpful to other teams.

What is Stats Accelerator, and How Does It Work?

Stats Accelerator is specifically built for experiments with multiple variations compared against a control, such as A/B/C tests or other multi-variation setups. It uses Bayesian statistical methods to dynamically allocate traffic, focusing more visitors toward the variations showing the most promise. This helps experiments reach statistical significance faster.

The key point here is:

• Stats Accelerator requires at least two variations plus a control (three arms total) to operate effectively.

This requirement exists because Stats Accelerator relies on having multiple alternatives to compare, enabling it to intelligently shift traffic and determine the best-performing option more efficiently.

Clarifying the Use Case for Standard A/B Tests

I initially assumed Stats Accelerator would be applicable for traditional two-arm (A/B) tests. However, the feature explicitly requires three total arms (one control and two variations) for effective use.

For standard two-arm A/B tests, the recommended method remains Optimizely’s default Sequential Testing—which allows for early peeks at results without compromising statistical accuracy.

How Stats Accelerator Differs from Multi-Armed Bandit and Auto Allocate

Understanding when to use each dynamic allocation method provided by Optimizely is crucial:

• Stats Accelerator: Ideal for multiple variations tested simultaneously against a single control, optimized for quickly learning which variation performs best with statistical significance.
• Multi-Armed Bandit (MAB): Ideal when the main goal is maximizing conversions during the experiment. MAB continuously reallocates traffic toward the highest-performing variant, with less emphasis on traditional statistical significance.
• Auto Allocate: A simpler dynamic allocation method similar to MAB, focusing primarily on conversion optimization rather than statistical depth or multi-variation comparisons.

Practical Recommendations for Experimentation

To effectively leverage Optimizely's features:

• Use Sequential Testing: When running standard A/B tests with just one variation.
• Use Stats Accelerator: When you have multiple meaningful variations and your goal is to quickly and statistically validate the best-performing variant.
• Use Multi-Armed Bandit or Auto Allocate: When optimizing real-time performance (such as conversions) is more critical than detailed learning or precise statistical significance.

Final Thoughts

This clarification helped me and my team better align our experimental designs with appropriate Optimizely features. Stats Accelerator is an excellent tool, but it has specific conditions for optimal use. Recognizing these nuances ensures your team choose the right experimentation approach and effectively utilize your traffic and resources.