Question 1

What should I A/B test first in cold email?

Accepted Answer

Start with subject lines — they have the highest impact since they determine whether your email gets opened at all. Once you have a winning subject line, test opening lines, then CTAs. Test one variable at a time to get clean results.

Question 2

How many emails do I need for a valid A/B test?

Accepted Answer

It depends on the metric you're measuring. For open rates (subject lines), you typically need 200-500 per variant. For reply rates, you may need 500-2,000+ per variant because reply rates are lower and require larger samples to detect meaningful differences.

Question 3

What is statistical significance?

Accepted Answer

Statistical significance means the difference between your variants is unlikely to be due to random chance. A 95% confidence level (the industry standard) means there's only a 5% probability the result is a fluke. Don't call a winner too early — wait for significance.

Question 4

Can I test more than two variants?

Accepted Answer

Yes, but it increases the sample size needed. For cold email, stick to two variants (A/B) rather than multivariate tests. Your list sizes are typically limited, and two-variant tests give you clean, actionable results faster.

Question 5

How long should I wait before declaring a winner?

Accepted Answer

For subject line tests (open rate), wait at least 48 hours. For reply rate tests, wait 5-7 days to account for delayed responses and follow-ups. Never declare a winner after just a few hours — you'll get skewed results from timezone effects.

Cold Email A/B Test Planner

Frequently Asked Questions

A/B Test Automatically with ColdRelay