Three Simple Mistakes You May Be Making with Your Email Split Tests

July 9th, 2012

Split tests are windows into guest behavior patterns. It’s true. Every time you split test an email, you are giving yourself a chance to see how they act or react based on various marketing principles, uses of specific words/images, or the time/place that they get your email.

Even with easy-to-use tools, it’s easy to make mistakes. Here are 3 things you’ll want to remember or avoid the next time you run a split test – which might as well be the next time you send an email, right?

1) Send to a Large Enough Group
You’ve probably heard the term “statistical significance” before. It’s a way of knowing how confident you can be in your results. CopyBlogger once shared a simple rule of thumb for this:

  • If there are 100 impressions in your sample, you need to see a 20% difference between variations to be sure that they actually mean something.
  • If there are 1,000 impressions, you need a 6.3% difference.
  • If there are 10,000 impressions, you need a 2% difference.
  • If there are 100,000 impressions, you need a 0.063% difference.

If you don’t have a large group, don’t let that dissuade you from doing a split test. Even small samples can provide a good direction to build off of for future campaigns and tests.

2) Isolate Your Variables
Another common mistake is trying to learn too much from one test. The key is to have only one thing be different between the two versions. If you are testing the use of “discount” or “special offer” in your subject line, you can’t also send at different times of day to different demographic groups.

Same goes for the email copy. You can only identify which image works better than another at encouraging clicks if the rest of the email remains unchanged. Ironically, when you try to learn more by tweaking multiple elements, you don’t learn anything at all because you don’t know what contributed to the difference in results.

3) Record What You Learn
Before sending your next split test, pick out a way to record what you learn. Then, as you plan future campaigns, you can look over what you have learned, use the more effective choice, and build on lessons from the past rather than relearn them.

The real key to split tests is to actually do them. Every email sent without a B-version is a lost opportunity to learn something new.

Share on:

Have us speak at your next event.

Insight-filled presentations backed by billions of hospitality-specific data points.