Most "A/B tests" run by small advertisers are noise. The winner picked at day 7 isn't the real winner — it just happened to be ahead at the moment you looked. Real A/B testing requires sample sizes most accounts don't have, which is why we'll cover both the right way and the pragmatic way.
What you can actually test
- Headlines — typically 30-100% impact on CTR.
- Descriptions — smaller impact (10-30%).
- Images / videos — huge impact on social ads.
- CTAs — "Learn more" vs "Get started" vs "Free trial" — meaningful.
- Landing pages — biggest lever, hardest to test fairly.
Don't test multiple variables at once unless you have massive volume. You won't know which change caused the lift.
The math: how much volume do you need?
For a CTR test moving from 3% to 4% (a 33% relative lift), with 95% confidence, you need ~2,300 impressions per variant. That's reachable for most accounts.
For a conversion-rate test moving from 4% to 5% (a 25% relative lift), with the same confidence, you need ~2,300 clicks per variant. That's much harder. At a 3% CTR, you'd need 75,000 impressions per variant — tens of thousands of dollars of spend.
The right way: ad rotation
Modern Google Ads and Meta both auto-test when you give them multiple ads in an ad group. Just upload 5 headlines + 4 descriptions and the platform serves them in combinations. After enough volume, the platform converges on what works.
Set ad rotation to "Optimize" (default in Google Ads) — not "Rotate evenly," which is for manual A/B and almost never the right pick for a small account.
The pragmatic way: the 5-second test
Show your two ad variants to 5-10 strangers (or use a service like UsabilityHub) for 5 seconds each. Ask them which makes more sense. If 8 out of 10 prefer one, ship it. You don't need 95% statistical confidence to prefer the obviously-better option.
Tests that lie to you
- Looking at a test before significance. "We've been running 3 days, A is winning 12% to 8%." Sample is too small; the variance can swing 20% just from random visitor behavior. Pick a duration in advance and don't peek.
- Day-of-week effects. Test ran Monday-Wednesday. Weekend behavior is different. Always run tests in full weeks.
- Seasonality. Started a test the week before Black Friday. Conversion rates spiked because of the holiday, not because of your variant.
- Different audiences. Variant A served to mobile-heavy traffic, variant B to desktop. Not the same test.
Landing page tests
Use a real testing tool — Google Optimize is gone, but Optimizely, VWO, or Unbounce work. Don't roll your own with random redirects; the moment a returning visitor hits the wrong variant, your test is contaminated.
Hold the test for at least 2 weeks AND at least the volume your math says you need — whichever is longer.
What to test, in order
- The big claim in the H1.
- The first proof element above the fold.
- The CTA button text.
- The form length (how many fields).
- The page hero (image vs video vs static).
Don't test font sizes, button colors, or other tiny variations until the big stuff is settled. The lifts there are real but rounded down to noise on most accounts.