What to Actually A/B Test in Klaviyo (and What's a Waste of Time)

I've watched a brand spend two weeks deciding whether the button should say "Shop Now" or "Shop the Sale." They picked a winner off 11 sales and felt like they'd run a real test. They hadn't.

A/B testing is one of the most misused features in Klaviyo. Brands test the wrong things, on too few people, and then read the results like tea leaves. Here's what's worth testing and what isn't.

The thing that breaks most tests

Before you test anything, one number decides whether the test will mean anything: how many people see each version.

An A/B test compares two versions and tells you which won. But if version A gets six sales and version B gets nine, the gap is too small to trust. Run it again next week and the result could easily flip.

This is where most small brands get stuck, and Klaviyo is upfront about the math. According to Klaviyo's help docs, a flow A/B test isn't called statistically significant until at least 500 people have received each version. For a campaign, Klaviyo flags the test as not significant when the win probability is still under 75% after 1,800 people per version have seen it.

And those thresholds are for opens and clicks. Testing for actual purchases needs far more. In an example on Klaviyo's own blog, detecting a 30% lift on a 2% conversion rate takes roughly 9,800 people per version to call it reliably.

Read that again. If your whole active list is a few thousand people, a trustworthy conversion test is out of reach. The math needs more volume than the list has. On a smaller list, test big, obvious changes, judge them on opens and clicks instead of purchases, and accept that some tests won't give a clean answer.

What's worth testing

When you do test, go after the things that change what people do.

The offer or angle. This is the highest-value test there is. Free shipping versus a percentage off. A bundle versus a single product. Leading with the problem versus leading with the result. These change whether the email actually sells, so the differences are big enough to read.

The subject line. Still worth testing, with one catch. Klaviyo notes that Apple Mail inflates open rates, so don't crown the winner on opens alone. Pick a metric that actually matters, like clicks or placed orders, and judge the test on that.

Send time and day. Worth testing at the campaign level, especially if you've never checked. Tuesday morning versus Thursday evening can shift results more than any wording change.

The hero, or the first thing they see. The lead image and opening line do most of the work in an email. Testing that whole top section is worth far more than testing one line buried in the middle.

Flow timing. In an automated Klaviyo flow, testing the delay between emails, an hour after abandon versus four hours, can matter more than the copy inside them.

What's a waste of time

Button color and tiny wording. "Shop Now" versus "Shop Today." Teal button versus a slightly darker teal. These almost never produce a gap big enough to measure, and even if one looks like it won, you can't trust it.

Testing five things at once. If version B has a new subject line, a new image, a new offer, and a new button, and it wins, you've learned nothing. You don't know which change did it. Test one thing at a time, or you're just guessing with extra steps.

Anything on a list too small to reach a real result. If you only have a few hundred people who actually open and buy, most tests will come back inconclusive. Spend that energy on the offer and the flows instead.

Declaring a winner too early. A test that ran for two hours, or got crowned after a dozen sales, hasn't told you anything yet. Give it enough time and enough volume, or don't call it a test.

How Klaviyo actually runs the test

Klaviyo has A/B testing built in, in two places.

For campaigns, you can test subject lines or full content variations. You set how big the test group is, pick the metric that decides the winner, and choose how long Klaviyo waits before it sends the winning version to everyone else. Klaviyo recommends a test size proportional to your list, often around 20% to each version, then sends the winner to the rest. The setting people miss is the winning metric. Klaviyo lets you pick opens, clicks, or placed orders. Pick the one closest to revenue you can.

For flows, you can A/B test an individual message inside the flow. That's how you test timing, a subject line, or the offer in an automated email without touching the rest of the flow.

In both cases, the tool only reflects what you feed it. Feed it one meaningful change, tested cleanly, and it will tell you something true.

Before you run a test

• Are you testing something that changes what people do, not how the email looks?

• Are you testing one thing at a time?

• Is your list big enough to reach a result you can trust?

• Did you pick a winning metric tied to revenue, not just opens?

• Are you giving it enough time and enough sales before calling it?

• If the test comes back inconclusive, will you accept that instead of forcing a winner?

A handful of good tests a year, aimed at things that actually change what people do, will teach you more than a steady stream of button-color experiments. Start by testing what matters, and give it the volume to mean something.

If you're not sure what's worth testing on your list, or whether your list is even big enough to test the way you're trying to, book a free call and we'll look at it together. No pressure, no pitch.

Email is a system, not a send.

— Alex

Need help implementing these strategies in your email marketing program?

Frequently asked questions

What should I A/B test in Klaviyo first?

Start with your offer or angle, because that's the change most likely to move sales. Test something like free shipping against a percentage discount, or one core message against another. Save subject-line and timing tests for after you've tested the things that change whether people buy.

Why do my Klaviyo A/B tests come back inconclusive?

Usually because not enough people received each version. Klaviyo won't call a flow test significant until at least 500 people have received each variation, and it needs more than that for campaigns. If your list is small or the change is minor, the test often can't reach a clear result. On a small list, test bigger, more obvious changes, or accept that some tests won't give a clean answer.

Should I trust an A/B test win based on open rate?

Not on its own. Apple Mail inflates open rates, so a subject line that "wins" on opens may not win on clicks or sales. When you set up the test in Klaviyo, choose a winning metric closer to revenue, like clicks or placed orders.

How long should a Klaviyo A/B test run?

Long enough to collect data you can trust, which depends on your send volume and how many sales each version produces. For campaigns, give it enough time that the winning metric reflects real purchase behavior, not the first hour of opens. A test crowned after a handful of sales hasn't told you anything yet.