When Not to Use A/B Testing

[sumaiyakhatun26]

Let's take a closer look.

The product is at the start and has little traffic. In such a situation, you risk not getting results in the required time. For example, when a landing page with startup services is being tested or a new product is being launched, there is not much traffic yet. You will have to wait a long time to get a statistically significant result from the experiment. During this time, the hypothesis being tested may become irrelevant.

Special calculators will help you calculate how justified it is to conduct an A/B test for australia consumer email list a product. You can determine the sample size with Evan Miller's calculator , and the test duration with the VWO calculator .

A product for B2B, B2G or premium segment. These areas have a high value of each client. During an A/B test, part of the audience may see a "raw" version of the solution and refuse to cooperate. This can happen if you show users an inconvenient interface or unclear tariffs.

In the B2B and B2G product market, there is a possibility that customers communicate with each other. Part of the audience may learn that the “chosen ones” are offered special conditions that are not available to them.