How Motley Fool Doubled Email Registrations with Multivariable Tests (Includes Before-and-After)

CHALLENGE

"Our whole email marketing machine starts with an email address," says The Motley Fool's New Member Experience Director, Greg Martz.

Like many content sites, The Fool has a mixed revenue stream. First they use email as a tool to keep millions of registered users coming back frequently to read free content against which the team sell ads to sponsors. Then, The Fool tries to up-sell as many happy readers as possible to premium memberships to get paid members-only content on the site.

To grow the list and keep up with churn -- unsubscribes plus the estimated 30% of email addresses that go bad annually -- the email team have to sign up thousands of new names every single day.

Along with co-registration, they used two standard tactics:

Tactic #1. A pop-up appearing on the home page at the start of a new visitor's visit, and then again after the visitor had browsed for a while.

Tactic #2. A landing page for paid search marketing campaigns, on Google and Overture under search terms such as "Motley Fool" and "Retirement", offering free article access in return for email address.

The team's been testing campaigns to convince visitors to hand over email addresses since the 1990s... so their control creative was pretty good to start with.

For example, the home page pop-up routinely got a 2.99% conversion rate and a typical Overture campaign landing page did even better at 10.7%. However, the team was convinced they could do even better....

CAMPAIGN

Everyone had ideas about what they wanted to test, from colors to copy. Running a/b tests for each element would be extremely time consuming. So, Martz decided to try out multivariable testing for the site's pop-ups instead.

Multivariable testing (which we've written about in several Case Studies in the past - link to library below) allows you to test an enormous array of changes all at once. You can have several almost completely different looking pages or pop-ups running against each other and learn which of the many different elements on each creative actually hurt or helped results.

Among other factors, the team tested:

- Describing email subscriptions a "membership" versus just a "sign-up."

- Longish copy (roughly 100 words) describing benefits versus much shorter copy basically saying "sign up, it's free."

- A $29 new member bonus report that each email sign up would get for free versus no freebie.

- Offering four checkboxes for topic preferences such as daily news and investment strategies, versus no choices at all.

- Varying copy on the submit button from "Become a member" to "Click Here -- It's Free!"

Once the winning pop-up was determined, the creative team applied those learnings to other sign-up pages, including the search marketing landing pages (naturally).

RESULTS

"I saw a 50% change (for the better) on the home page pop-up completion rate," says Martz.

Here's the exact data:
Before After
Initial home pop 2.99% 4.50%
Later in visit pop 3.**% 5.80%
Google landing pages 8.3% 17.9%
Overture landing pages 10.7% 22.7%

The value of these additional names to the company in terms of repeat visits and email lifetime, plus average percent converting to a paid membership was slightly lower than in the past. (Which makes perfect sense, when you push a free offer harder, you get more iffy prospects.) However, Martz is quick to note that higher volume more than made up for a slight decline in quality.

What worked? Extreme simplicity.

Shorter copy, simpler layout, simpler offer. People would rather just signup for sign access than become a "member" and get bonuses.... In fact Martz suspected The Fool's past offers were just a case of overselling.

Example - the winning button says "Click here -- It's Free".

Inspired by these results, the team began testing anew this month. Their latest test is to make the home page pop-up even simpler, but now it's an overlay instead of a pop. So far Martz says results are "incredible."

You can see samples of the winning and losing creative at the link below.


Useful links related to this article

Creative samples - before and after screenshots - for these tests:
http://www.marketingsherpa.com/motley/study.html

Optimost - the multivariable testing service The Motley Fool uses for tests:
http://www.optimost.com

The Motley Fool
http://www.fool.com