BrazilImproving online channels and products: Alternatives to traditional A/B testing
When it comes to exploring how to improve online channels or digital products, the process of validation is essential. While most large companies rapidly turn to A/B testing, for smaller companies, this may not always be possible or the optimal solution. A new e-book from Digital Power explores the various alternative methods and their pros and cons.
Experimentation and optimization are key for any company’s online presence and products, and for achieving a process of continuous improvement cycle. Smooth-flowing websites, user-intuitive digital products or slicker conversion journeys can make a huge difference.
Experimentation is a broad spectrum of inquiry that can include exploratory studies and comparative analyses. It is a systematic approach that tests hypotheses and often attempts to answer specific questions. In the online world, A/B testing is one such experimental form.
Validation is the process of confirming or disproving any hypothesis or expectation. This can be done through a range of approaches like user research, expert evaluation, prototype testing, and A/B testing.
The two steps are logically intertwined – by validating ideas through experimentation, companies can augment the level of their decision-making, as they then test and confirm based on concrete results, instead of assumptions or ‘gut feelings’. Key KPIs for validation include quantitative factors such as larger number of visitors and higher conversion rates for websites, and higher usage and engagement rates for digital products.
Speaking about the value of validation, Mila van der Zwaag, researcher and customer experience specialist at Digital Power, says: “With online and digital product optimization, making assumptions is a bit like shooting in the dark – you might hit the target, but there’s a good chance you’ll miss. That’s why validation is our best friend here. It’s all about challenging our guesses and making sure we’re on the right track.”
Getting the hypothesis right
When companies apply this cycle, it is key to upfront ensure that the experiment’s design is clear, and underpinned by a hypothesis that is supportive and researchable. “Every hypothesis is built upon underlying assumptions. When formulating a hypothesis, carefully consider the specific problem you are attempting to solve and the rationale behind the proposed solution, including why you anticipate it will be effective.”
So, before the validation phase, Mila urges researchers to ensure their hypothesis has sufficient supporting evidence. “Are you truly validating a solution, or is more foundational research needed? If so, revisit your initial exploration phase. Also, part of this assessment is ensuring that you select the right method, as different methods address different questions.”
A/B testing
In terms of models for experimentation and validation, A/B testing stands out as a go-to choice. The basic idea of A/B testing is to directly compare two different options to see which performs better based on a company’s specific goals.
Also known as a ‘bucket test’, A/B testing is popular because it provides a rigorous and data-driven approach to decision-making. It allows for objective assessment of different variations of a feature, helping to identify which ones perform best with real users. This can inform the path to improved user experience, better content engagement, and increased return on investment.
The main drawback of A/B testing is, however, that it requires a more workload-heavy setup (replicating two different versions with two user groups) and needs a relatively high level of traffic for results to be truly reliable.
“Running A/B tests can be incredibly insightful and can help companies make changes that increase engagement and conversion rates. But when traffic to a website is low, the results are typically not very reliable,” notes Mila. “Achieving statistical significance – and the confidence to move ahead unaverred – is then debatable.”
That is because small sample sizes result in higher variability, making it challenging to distinguish true effects from random fluctuations in the experiment’s options. “Moreover, it is hard to determine if the found effect is a fair representation of the real-life effect.”
Validation beyond A/B testing
Luckily, for companies that run low traffic websites, there are numerous other UX research methods for validation, many of which can do the needed trick with less data.
For example, new features can be tested by individual participants who are then surveyed on their experience – a classic and easy-to-apply form of UX testing.
Another potential approach is to ask individual testers which of the two options they prefer in a preference test. Other quick surveys like this can also be conducted with a broader scope – even people recruited in public places.
Usability testing can provide deep insights and immediate feedback. Mila: “Usability testing involves observing real users as they interact with a digital product, aiming to identify usability issues and understand user behavior. Usability testing is often focused on the ease of use and overall user experience, while A/B testing can validate broader hypotheses, including those related to conversion rates or other specific metrics.”
Before-and-after testing is a method that can be easily set up and uses real-life data. “In digital product optimisation, a before-and-after test involves comparing the performance of a product feature or design when fully live against an earlier period when the previous version was active,” says Mila. “This test is easier to deploy compared to an A/B test.”
There are many other methods that can be tried – each with its own pros and cons. Take for example ‘guerrilla testing’. This involves quickly gathering feedback by approaching people in public places (like coffee shops) to evaluate a product or design concept on the spot. This method is fast, affordable, and can provide insight from a wide variety of different users. But drawbacks include that it is less controlled and often only really provides surface-level insights.
Choosing the right method
Any method that a company chooses will have its own set of benefits and drawbacks. An important first step in selecting the right method is recognizing the challenges posed by limited traffic and going beyond A/B testing, Mila notes.
“By emphasizing a structured workflow for continuous improvement and risk mitigation, we encourage a culture of data-driven decision-making. Integrating diverse research methods – both quantitative and qualitative, behavioral and attitudinal – enables a comprehensive evaluation of online websites and products.”
