Imagine changing an ad creative and launching it to a new audience. Leads increase. Was the result caused by the creative, the audience or their combination? The answer determines what to do next with your advertising and budget.

A/B testing compares two variants under comparable conditions. To understand the effect of a specific variable, change it separately.

1. Choose what to test

🎨 Creatives with the same audience. Compare a photo and a video advertising the same service. Keep the audience settings, offer, landing page and campaign objective the same; change only the creative. Use the advertising platform’s A/B testing mode if available. In a standard campaign, the algorithm may give one ad more impressions, making results harder to compare.

👥 Audiences with the same creative. To find out whether qualified leads cost less from a broad or interest-based audience, use the same ad, offer and landing page. Change the targeting settings.

💰 Offers. Compare a 10% discount with free delivery for the same audience. Keep the ad format consistent and change the customer offer. When assessing results, consider how much the business earns per order under each offer.

🌐 Landing pages. Send visitors from one ad to two page versions: one form has five fields, the other three. Compare the proportion of visitors who submit a qualified lead.

To test both creatives and audiences, run two separate tests. Compare creatives with one audience first. Then take one creative and compare audiences. A creative that works well with the first audience may perform differently with another.

2. Choose the primary metric in advance

Start with the decision you need to make. If you want to know which ad should receive more budget to generate qualified enquiries, the primary metric could be cost per qualified lead.

Suppose each ad costs 1 million Uzbek soums:

  • Variant A: 200 clicks and 10 qualified leads. Cost per qualified lead: 100,000 soums.
  • Variant B: 150 clicks and 20 qualified leads. Cost per qualified lead: 50,000 soums.

A leads on clicks; B leads on cost per qualified lead. If that cost was chosen as the primary metric, use it to assess the result. Before reallocating budget, check whether there is enough data and how leads from each variant progress to sales.

Before launch, agree with the sales team on what counts as a qualified lead, where the data comes from and how long to allow for qualification. Apply the same rules to both variants.

3. Decide how much data you need

Suppose 3 out of 100 website visitors currently submit a lead. You want to test whether a shorter form increases conversion from 3% to 4%. With equal traffic, this comparison requires approximately 5,300 visitors per version, at a 5% significance level and 80% power. This is a planning estimate for comparing two proportions; the calculation depends on the method and experiment conditions.

Planning requires two figures: the current result and the smallest improvement that matters to the business. The smaller the difference you want to detect, the more visitors you need.

Run variants at the same time. Before starting, set the required sample size, minimum test duration and time needed to process the final leads. If one variant is ahead on day three, continue according to the plan.

4. Check whether you can make a decision

First confirm that you reached the planned sample size and counted leads by the same rules for both variants. Then ask two questions.

How confidently can you see a difference? For example, conversion rose from 3% to 4%. Look at the confidence interval: a range indicating how precisely the difference was estimated. If it includes both an increase and a decrease, there is not yet enough evidence to confidently choose a winner.

Is the difference large enough for the business? Even if the improvement is supported, estimate how many additional qualified leads it could produce per month, how many may become sales and whether that justifies implementation costs.

📋 Marketing experiment card

Before launch

  • Business question and decision. What do we want to learn, and what decision will we make? ___
  • Hypothesis. If we change ___, we expect ___ because ___.
  • Test variants. A — current version ___ / B — new version ___. Which variable are we testing? ___
  • Launch conditions. Channel and audience ___ / delivery settings and measurement rules shared by A and B ___.
  • Primary metric. What we measure ___ / calculation ___ / data source ___ / measurement period ___.
  • Success criterion. Current result ___ / minimum improvement worth implementing ___ / acceptable thresholds for other important metrics ___.
  • Sample and timing. Data required per variant ___ / minimum test duration ___ / when final leads or sales will be included ___.

After completion

  • Data check. Did we reach the planned sample? Were A and B measured consistently? Were there disruptions or changes during the test? ___
  • Result. A ___ / B ___ / difference ___ / precision of the estimate and whether the business threshold was met ___.
  • Decision and next step. Implement B, keep A or plan another test. Decision and reason ___ / owner ___ / deadline ___.