Cold Email Subject Line Split Test

Cold email has been a hot topic for both digital marketing firms looking for more leads and affiliate marketers looking for link building opportunities. But, for those with more experience in SEO and SEM, email marketing is a whole new beast to explore. It's such a big segment of digital marketing that there are entire courses devoted to it. 

So, I figured the best way to do some split testing with cold email is to break up the whole process into very small sections. For this test, the main concept is to test a few different subject lines and see which one has the best open rate. 

Expectations & Limitations

This is just a simple case study. It's only testing the subject lines of initial cold emails.

I also only targeted one specific niche: family therapy. Other niches may yield slightly different results.  

There is also a fairly small sample size, but it should be enough to get some useful takeaways. 

Design

This cold email outreach followed a very simple design. Here are the key elements:

  • This was pure cold email outreach
  • Three different subject lines were tested
  • Each subject line was sent to 100 different recipients
  • The same "from" email was used for all the outbound emails
  • a G-suite branded email was used (not a generic @gmail)
  • All 300 emails used the same body to ensure there were not deliverability differences

As you can see, it was a very vanilla and straight froward design. As far as the subject lines tested, they were chosen based on offering the recipient a slightly different idea of what the email might be about. 

Here are the three subjects that were used and why.

Subject Line 1 - Quick Question

I wanted a very generic subject line. It was short and did not really give the recipient an idea of what the email was about. It did however suggest I was asking them for something instead of me trying to jam some unsolicited advertising in front of them.

Subject Line 2 - Help More People

The second subject line was more revealing in that it targeted the recipients' businesses. Since I contacted family therapists, helping people is something that is of interest to them.

Subject Line 3 - Help More People?

The last subject line tested was a combination of the first two. It was targeted and not generic, but it included a question mark to make it a little softer in tone than Subject Line 2.

Assumptions

Before looking into the results, we can make some assumptions about what to expect. 

My assumptions were:

  • The generic subject line would perform the worst
  • There would be little difference between Subject Line 2 and Subject Line 3, meaning that adding a question mark was not going to make any measurable difference

These are both pretty safe and simple assumptions considering the differences in the subject lines.

Results

Enough words, let's look at the stats.

Quick Question

quick question results

The only fields that are important for this test are "Sent", "Opened", and "Bounced". 

As we can see, of the 100 sent emails, 4 never made it to a live inbox. We still managed to get 96 emails through, and they had a combined open rate of 56%

That's not too shabby. But, how does the generic subject line match up to the other two?

Help More People

help more people stats

There were a few more bounces here than with the first 100 emails sent. Those extra bounces brings the total delivered emails down from 100 to 93. 

That's still a decent amount of emails hitting inboxes though. But, the open rate here was considerable lower at 48%.

That's almost a double digit drop off from the generic subject line.

There's still one more to check out.

Help More People?

help more people question stats

These results look extremely similar to the generic subject line. They both had 4 bounces, and the total opened percentage is very close. 

For this we got a 55% open rate. That's not bad at all for a cold email and basically only one open less than the generic subject.

Takeaways 

These was a very small but targeted case study, so we don't have tons to look at here. But, we did get some interesting and in my opinion unexpected results. 

Let's see if my initial assumptions were accurate - 

Assumption 1: The generic subject line would have the worst open rate of the three tested

  • Incorrect

Wow. I could not have been more wrong. Not only was it not the worst of the three, but it was the winner. 

I have a feeling by keeping the contents of the email vague, but by making the subject line more about asking for something instead of pushing something, it tickled the curiosity of the recipients. 

Assumption 2: There would be little difference between "Help More People" and "Help More People?"

  • Incorrect

Who would have known that adding something as simple as a question mark would make such a big difference on open rates? But, from this test we had almost a double digit difference - 55% vs. 48%.

I feel even with the small sample size, that is a pretty significant difference. 

In Summary

Overall, this is a fairly small scale case study. However, that doesn't mean we can't take away some useful information. 

First of all, it's clear that subject lines can have an effect on open rates. Most people know this already, or at least believe this to be true. Having some stats to back it up though is always nice. 

The bigger takeaway though is that the exact wording of your subject line might be less important than the general tone. 

It's pretty clear that "Quick Question" and "Help More People?" are very different in their wording. But, they share a tone that piques curiosity unlike "Help More People", which is just basically a statement.

That being said, I'm still shocked at just how wrong as I was in my assumptions. I guess the biggest takeaway of all is that no matter what your assumptions are in any aspect of your marketing work, unless you test them you'll never know if you are correct or not.