Analyzing A/B Testing Email Marketing Data

Welcome to the wild world of A/B testing emails! If you are responsible for email marketing, you know that every subject line, word choice, and link placement are variables that affect your email performance. Not to mention, factors such as send time, day of week, month, etc.

The crazy thing is, the performance is all measured, right down to the last click.

If you’re like most marketers, you’re probably spending dozens of hours testing different variations of the same email to:

1. Identify factors that are most effective at driving conversions and engagement.
2. Discover the type of content that resonates with your target audience.
3. Learn which version is highly likely to avoid the spam filter.

But there’s just one problem with A/B testing your emails. You generate a lot of data, especially if you have a large email list.

Your marketing automation tool isn't likely the best tool for data exploration and analysis. Therefore, you are going to want to export the data, open the CSV online, and use a friendly spreadsheet format for analysis.

What is the best way to Analyze it? Well, let’s have a look.

A/B Testing Email Marketing Examples

I pitched this idea to my close friend Will Andrews, who’s in charge of Product and Marketing at Gigasheet. He collaborated with me to build a dummy A/B email testing dataset – where we together came up with:

• 1,000 random emails
• We added an email variation column group - where we’ve set the entries between 1 and 5 for five different email variations. We randomly assigned email variation numbers to 1,000 emails in our list.
• We randomly assigned “Yes” or “No” entries to the following column groups:
• Opened (Yes or No) – To understand which version of the email helped us achieve the highest open rate.
• Clicked-Through (Yes or No) – To understand which version of the email helped us achieve the highest click-through rate.
• Responded (Yes or No) – To understand which version of the email our recipients replied to the most.
• Booked a Meeting (Yes or No) - To understand which version of the email helped book the most meetings.
• Unsubscribed (Yes or No) – To discover which version of the email helped us get the lowest unsubscription rate.  

Here’s a sneak peek at our dataset:

Now, let’s analyze it!

Using Google Sheets to Analyze My A/B Test Email Campaign Data

I work as a freelance content partner for Gigasheet. But that doesn’t mean I hate Google Sheets or Microsoft Excel.

In fact, I love them!

To conduct my analysis, I loaded my spreadsheet to Google Sheets, which you may find here.

If you’re like me, you already know how complex Google Sheets is. From giving you the ability to play with spreadsheets and perform calculations to advanced features like conditional formatting, data validation, data filtering, and sorting, data visualization, and more, Google Sheets is incredibly complex.

Every time I had to apply a filter or formula – I had to look it up via “Help.”

While you’ll soon get the hang of the platform (know all about its features), it takes time. Google Sheets has a steep learning curve.

First, I wanted to sort my data by email variations. I’ll admit – it was simple – all I had to do was create a filter and sort the data in the email variation column by “Sort A -> Z.” You can also hide or unhide column groups in Google Sheets easily.

But then came the hard part.

I wanted to find out – which version of my email had the highest open and click-through rate. I thought of applying a filter – but if I went that route, I’d have to count open and click-through rates for every email variation – which is just too much work.

Well – you can always apply grouping – but when I tried doing it in Google Sheets – I found out that it was one of the hardest things on the planet. Using Google Sheets to group your data is like getting a toddler to eat their veggies – it’s HARD.

On top of this, did you know that – Microsoft Excel and Google Sheets have their own limitations? At present, Google Sheets can only process 10 million cells. Excel’s row and column limits stand at 1,048,576 and 16,384 respectively.

Also, if you try to access a large spreadsheet file using Microsoft Excel or CSV, it may result in your PC crashing or browser/software freezing. I have been known to cause Excel to crash!

That’s the reason we encourage people to use Gigasheet.

Using Gigasheet to Analyze Email A/B Testing Campaign Data

As a freelance writer, I keep my clients limited. And the reason I work with Gigasheet is that I love the platform. No matter how large your spreadsheet file is or whether it has billions of rows, Gigasheet operates at cloud scale – so neither your browser will hang, nor you’ll find it tough to play around.

And the best part is – Gigasheet has so many advanced features like data filtering, sorting, grouping, validation, and more – which will help you get your hands on the exact information you need. Insights are easy with Gigasheet!

Let’s try it out.

Open Rate A/B Testing Analysis

Let’s perform the same operation that we were trying to do above. I want to find out – which version of my email has the highest open rate. I loaded my spreadsheet to Gigasheet – and added two layers of column grouping:

Here are the results:

Here’s the story data told me:

• I sent the first variation of the email to 209 email addresses, of which 113 opened my email whereas 96 didn’t. My open rate was 54.06%
• I rolled out the second variation of the email to 199 email addresses, of which 110 opened my email whereas 89 didn’t. My open rate was 55.27%.
• I sent the third variation of the email to 193 email addresses, of which 101 opened my email whereas 92 didn’t. My open rate was 52.33%.
• I sent the fourth variation of the email to 205 email addresses, of which 94 opened my email whereas 111 didn’t. My open rate was 45.85%
• I sent the fifth variation of the email to 194 email addresses, of which 103 opened my email whereas 91 didn’t. My open rate was 53.09%.

In terms of open rate, the second variation won – with an open rate of 55.27%. Whereas, the fourth variation had the lowest open rate, i.e. 45.85%. This helped me understand that the second variation has the best subject line.

Click-through Rate A/B Testing Analysis

Similarly, I looked at the click-through rate data by applying the grouping:

Here are the results:

You’ll find that there are blank cells in my file – that’s because if an email was not opened, then actions like click-through, responding back, booking a meeting and unsubscribing were not taken. So, I have kept those cells blank.

Upon analyzing the data, I found out that –

• Variation 1’s CTR is 29.18%.
• Variation 2’s CTR is 28.64%.
• Variation 3 has a CTR of 27.46%.
• Variation 4’s CTR is 24.39%.
• Variation 5’s CTR is equal to 24.22%.

Here, Variation 1 is the winner with a CTR of 29.18%. Whereas, Variation 5 has the lowest CTR of 24.22%. Variation 4 had the lowest open rate; however, variation 5 has the lowest CTR. So maybe Variation 5 doesn’t have an up-to-the-mark email copy.

Similarly, you can add or modify your grouping to find out information like which variation –

• Had the lowest unsubscription rate
• Helped you book the most meetings.
• And more!

Unsubscribe Rate A/B Testing Analysis

In today’s world, we generate a lot of data. It’s manually impossible to process and make meaning out of it – that’s exactly where Gigasheet saves the day.

Next, I wanted to identify the emails that unsubscribed – so I applied this filter:

I found out that 132 people unsubscribed. Find it at the bottom of the screenshot below.

Use Gigasheet to Analyze Your Email Marketing Data!

Like I said before, the amount of A/B test data you generate depends on the size of your email list and the number of people you choose to roll out emails to, if you run normal email marketing campaigns, you’ll be generating a lot of data. And you’ll need a platform to make meaning out of it.

In my opinion, there’s no other platform better than Gigasheet when it comes to processing and analyzing large data.

I’m already in love. Are you? Try out Gigasheet today for free!

‍