The Scientific Method for Conversion Rate Optimization Testing

There are two angles to approach A/B and Multivariate testing.

There’s the common “Opportunity-Out Method,” which starts with identifying the biggest area of impact before building out hypothesis, but I prefer to start with a bit more rigor.

Call me an academic, but the best approach to conversion rate optimization testing is the scientific method.

That’s right:

Everything you need to know about #CRO, You Learned in Grade School Click To Tweet

Here, we’ll quickly cover applying the scientific method to your conversion rate optimization tests.

  • Identify the Problem/Purpose
  • Observation/Background Research
  • Form a Hypothesis
  • Design an Experiment
  • Analyze Results
  • Conclude (and Continue!)

At each step, I’ll share an introduction, and the continuation of an ongoing example to help see this method in action.

Identify the Problem/Purpose

What key performance indicator (KPI) do you want to tick up? What stage of the customer journey would you like to improve?

Start with a clear purpose to focus your early research and observations.

  • Email Example: “I want to improve my email open rate.”
  • Web Example: “I want to improve my website’s conversion rate.”

Observation/Background Research

Review the data you have surrounding your question, such as analytics of current or past approaches, blog posts from similar tests in other companies or even scouring academia in Google Scholar.

Allow yourself time to build an understanding of your question with your own and borrowed information.

Most importantly, don’t be afraid to challenge long-standing status-quo – in fact, that’s what you’ve come to do, isn’t it?

  • Email Example: Review historical email performance. Scan some blog posts on the topic.
  • Web Example: Review previous tests and long-term benchmarks, making note of ‘accidental tests’ such as specific offers or product launches.

Form a Hypothesis

Using your background research, make an informed hypothesis. (Literally, an educated guess.) A good hypothesis will be specific enough to guide your test, and includes what you’re testing and your projected outcome.

For early testing, be sure your hypothesis covers elements you can accurately measure such as analytics information (conversions and element clicks), as well as email metrics (open and click-through-rate).

Avoid ‘softer’ metrics such as ‘comfort,’ ‘trust’ or ‘awareness’ unless you have the time, traffic volume, tools and ginormous budget required to test these accurately.

If you find yourself using a psychometric measure in your hypothesis, pair it with hard measures. ‘Trust’ and ‘comfort,’ may show as improved conversion rate or website engagement in time on site and page depth.

  • Email Example: Highlighting an offer as the subject line in our newsletter will improve email open rate.
  • Web Example: Including an ‘infographic’ at the top of our homepage will improve understanding, reducing bounce rate and improving conversion rate.
It’s called A/B testing, not A/A-ish testing. #CRO Click To Tweet

Design a Conversion Rate Optimization Experiment

Whether you’re using MailChimp subject line tests, or Analytics 360, it’s up to you to build your experiment – the ‘B.’

Remember, your ‘A’ should always be your status quo – the current approach. This ensures relevancy in your results and lays the groundwork for further optimization.

Think clinically. If you send your placebo medication group to watch Netflix and the test medication group to the gym, how will you know if your results were the medication or the gym?

To ensure accuracy in your test, maintain as much similarity between versions as possible, adding only the variable of your test. Note in the sample below with subject lines that each include the same subject, adding only the ‘offer’ test at the lead of the test.

Finally, fight the urge to get caught up in minutia. Following on one of our examples, there are countless types of ‘offers,’ but choose one to start, and know that follow-up testing can go for ‘which offer’ if offers do, indeed, improve conversion rate.

Build the most representative version of your test you can, and run it against your control.

  • Email Example:
    • Control Subject Line (Status Quo) (A) – “Your Weekly Newsletter from Acme Corp.”
    • Test Subject Line (B) – “Save this month in Your Weekly Newsletter from Acme Corp.
  • Web Example: Using Visual Website Optimizer or similar, test inclusion of infographic.

Analyze Results

In the lab, this is riveting and highly time-consuming.

In marketing, this step pretty much meets you automatically.

Thankfully, your testing platform includes all the measurement needed to watch your test in action and analyze the results. MailChimp sends you an email when it’s done, and VWO will let you watch the results in near-real-time while the margin of error slowly slims down to a solid business decision.

Review your results carefully, and apply victory honestly.

The only failed #CRO test is the one with 0% lift. Click To Tweet

Conclude (and Continue!)

Did your hypothesis hold, or is the status quo your top performer? Treat yourself to follow-up testing to push your CRO efforts further!

  • Can you apply your learning against other areas of the customer journey?
  • Can you be more specific in your hypothesis to uncover more? Example: What kind of offer improves your open rate?
  • Can you modify your new approach for further improvement? A video in-place of your infographic? An expiration on your offer?

Follow-up testing can yield valuable extra percentage points, but rarely yields the lift of the initial test as you continue to dive deeper into ‘micro’ details. Keeping detailed notes while testing will allow you to return later once you’ve capitalized on as many ‘macro’ tests as you can.

Share and Swipe

I hope this framework helps you and your team build more impactful tests for your brand. Pass this along to help the internet be a better-converting place for all!

The Scientific Method for #CRO Testing Click To Tweet It’s called A/B testing, not A/A-ish testing. #CRO Click To Tweet The only failed #CRO test is the one with 0% lift. Click To Tweet