A.I. (ChatGPT) vs. Human Generated Subject Line Experiment – Who Wins?

Apr 17, 2023 | Blog, Sales and Marketing

What Is Data Visualization and Why Is It Crucial to Mitigate Risk

ChatGPT has gained incredibly quick popularity amongst marketing and sales pioneers looking for the next big thing to help reduce time in creating content, answering questions, and saving time.  As the technology continues to evolve and branch into new applications, we thought we would try an experiment between human generated subject lines vs. A.I generated subject lines in relation to cold B2B email.

Synopsis

In this experiment we looked at what would be some of the more popular human generated cold email subject lines that sales teams use to reach out and try to garnish a response off the first touch of an email.  We looked at the popular marketing content provider Hubspot as to what some of the more popular human generated subject lines would be and found 10 that we could A/B test against.  We sent 500 emails a day utilizing verified email data from Lead411.  

We did the same thing with ChatGPT generated subject lines asking ChatGPT “What are the 10 best cold email subject lines for B2B sales for a data provider platform?”  Utilizing these subject lines we used A/B testing and sent 500 emails a day utilizing verified email data from Lead411.

The titles of the email addresses we utilized were sales directors, SDRs, Business Development, and VP of Sales in the United States that worked in companies that were growing and expanding (based on our Lead Scoring Data).  We extracted contacts that had email addresses that were either verified or unverified and we sent these emails during the months of late March and early April of 2023.  We are looking at the open rates only.

*Note for this experiment, we used the email sending provider PersistIQ sending 5000 emails in both campaigns, and had a 7% bounce rate during the ChatGPT campaign and a 5% bounce rate during the Human Generated campaign. The email we used was an internal email – lastname@l4mail.com

 

Here are the results:

ChatGPT 

  1. Unlock the potential of your sales prospecting with Lead411 – 43%
  2. Take the first step to success with Lead411 – Schedule a Product Tour Now – 39%
  3. Get Ready for a game-changer: Lead411 is here to help optimize your sales process – 39%
  4. Get ahead of the competition with Lead411 – Let’s Chat! – 38%
  5. Ready to see sales results? Lead411 can help you achieve them. – 38%
  6. Say Goodbye to cold sales leads – Lead411 can help you today! 38%
  7. Revolutionize your lead prospecting with Lead411 – Are you interested in learning more – 37%
  8. Experience the power of Lead411 – sign up for a product tour today! – 36%
  9. Want to sell more in 2023? Lead411 can get you there faster. – 35%
  10. Are you tired of cold sales leads? Lead411 can help! – 33%

 

Human Generated 

  1. Sales at (Recipient Company Name) – 31%
  2. Maybe you can help? – 29%
  3. Leads for (Recipient Company Name) – 29%
  4. New Quarter, New Quota – 29%
  5. Question about Lead Gen? 28%
  6. Have you tried Lead411? – 28%
  7. Meeting Request – 27%
  8. Idea for Sales – 27%
  9. Hi (Recipient First Name), do you need more prospects? – 27%
  10. This is a sales email – 25%

 

Evaluation

Looking at these results at-a-glance, you can see that ChatGPT had significantly higher open rates than the human generated subject lines.  There are a few variables that should be mentioned before coming to a definitive conclusion.  

First, the chatGPT emails were sent at the end of Q1, so the demand for a new lead generation provider could be heightened during the end of a quarter, as budgetary allocations, and time-sensitive deadlines can contribute to a sense of urgency for a new or additional data provider like Lead411.

Second, all of the subject lines that ChatGPT suggested did have Lead411 in the subject line, so recipients automatically knew it was from a reputable company and the email was safe to open.

Lastly, the keep it short-and-sweet methodology from Hubspot back-fired for the human generated emails, as the length of each subject line was 65 characters for ChatGPT and only 23 characters for the human generated emails.

 

Conclusion

ChatGPT was the clear winner for subject line generation.  This experience aimed to keep all variables the same but there were some factors to consider including timing, length of subject lines, and content (mentioning company name in the email for reputability purposes) that were different among each campaign.  Another underlying variable is the email provider used for such experiment, the email used to send (and the reputation of that email) meaning, not listed on blacklists or spam checkers. 

As A.I. continues to become more and more suitable for B2B practices, this experiment proves that there is value to ask ChatGPT to generate results for use with cold emails subject lines. There was a noticeable difference averaging a 37.6% open rate for ChatGPT compared to a 28% open rate for the Human Generated Subject Lines. However, the next intended step for B2B sales is to get responses from these emails and not just simply an opened email.  Future experiments will look at ChatGPT generated Email Templates vs. Human Generated Email Templates.

 

Recent Posts