What Should You Be A/B Testing?

How do you come up with A/B testing ideas for your site?

It can be a challenge, especially if you are new to split testing.

We’re going to provide a list of ideas that you can consider as a starting point.

It’s important to remember that the tests you run should be based on the data you gather about your customers.

That’s the point of doing user tests and conducting surveys after all.

This list provides a starting point and something you can compare against your data, but you definitely want to come up with tests based on the data you’ve already gathered.

It’s also a good idea to keep a list of test ideas so you can keep track of all the ideas you come up with. Whenever you come up with a new hypothesis for testing, keep track of it there. You can then go down the list and run test after test in order to optimize your site.

Here is an example of a spreadsheet you can create to list your test ideas and prioritize them.

Now on to my list of 20 A/B test ideas to help you figure out what you should be testing.

Test Idea 1: Headlines

New headlines are easy to test and provide a good opportunity for an initial test.

For starters, there are a lot of poorly written headlines. Copywriters and business owners frequently get caught up in writing something clever that misses the point and doesn’t convince visitors to continue reading. Nor do many headlines convey a strong value proposition to customers. So headlines are a good place to begin your testing. They’re an important part of any page and one of the first pieces of copy visitors will read on your site. You’ll also get immediate feedback on your headline variations and learn right away which version performs better than the others.

In a case study from Visual Website Optimizer, for example, Movexa improved their conversions 89.97% by adding one word to their headline. The control was “Natural Joint Relief” and the new headline was “Natural Joint Relief Supplement.” Isn’t it amazing that one word can make such a big difference?

So how can you improve your headlines?
& what should you be testing?

The first thing to test is a headline that’s more clear. Businesses often use clever and creative headlines that end up being confusing. If this is your case, one of the first things you can test is a headline that clearly explains who you are and what you do, instead of a clever headline that attempts to make you seem cool.

The next thing you can consider are three typical headline variations: questions, benefits, and loss aversion.

• With a question headline, you create a headline that asks the visitor a question that will pique their interest. On Crazy Egg, for example, you could ask a question like “Looking for an Easy Way To Optimize Your Website?” This headline poses a question that most website owners would be interested in having answered
• The next headline formula you can consider is mentioning a benefit for using a product or service. You can mention that the product will help them lose weight, save time, make more money, etc. A benefit headline for Crazy Egg could be something like “Optimize Your Site with Heatmaps That Show Exactly Where Your Customers Click.” It’s clear, straight forward, and provides a benefit for using the service.
• The third headline formula you can consider is a loss aversion headline. With a loss aversion headline, you mention what people will miss out on by not using a product or service, and quite often, people are more worried about missing out on something than gaining a benefit. A loss aversion headline for Crazy Egg could be something like “Find Out Where Your Site Is Leaking Revenue with Crazy Egg Heatmaps.”

So which one of these headlines is better? Option 1, 2, or 3?

Here’s quick look at all three:

1. Looking for an Easy Way to Optimize Your Website?
2. Optimize Your Site with Heatmaps That Show Exactly Where Your Customers Click
3. Find Out Where Your Site Is Leaking Revenue with Crazy Egg Heatmaps

At this point you may have a preference for one headline over the others, but really, there’s no way to know without running a test. You first need to enter all three variations into Visual Website Optimizer or Optimizely, and then see which one increases conversions the most. The good news is that based on the results mentioned above with Movexa, a better headline can increase conversions up to 89.97% and possibly more!

Test Idea 2: Call-To-Action Buttons

The next test you can consider are call to action buttons. You can test different color variations, new CTA copy, and different button sizes and shapes to see if a change leads to higher conversions. CTA buttons are another piece of low hanging fruit that is easy to test.

As has already been mentioned, Switch Video increased conversions 221% by changing one word in their CTA copy. When they changed “Get a Free Consultation” to “Get a Free Quote,” qualified leads generated from their homepage increased 221%. That’s an awesome win from only changing one single word.

So how can you improve your call-to-action buttons?

First, consider new copy you can test. Are you using generic copy borrowed from another website like “Buy Now” or “Get a Free Consultation,” or are you using copy that’s specific for your product? Take a moment to consider what might make sense for your business.

You should also consider what it is that customers want to do at this point in the process. Do they want to begin a free trial, are they interested in buying now, or are they just looking to learn more? Do your best to match the CTA copy to your product and what visitors are looking for at that point in the buying process.

Next, consider the color of your CTA button. Is there a high contrast with the rest of your site, or does the button blend in? If it blends in, you need to consider an accent color that causes the CTA button to stand out. If your site uses a lot of blue, you may want to consider green for your CTAs, etc. You should also consider whether red, green, orange, or some other color leads to higher conversions. Yes, color seems insignificant, but tests have shown it can make a difference.

KISSmetrics uses a green CTA button that contrasts with their primary color which is blue.

Something else to consider is the location of the CTA button. Most sites make sure to put their CTA above the fold, but that doesn’t always lead to the highest conversion rate. For example, I found that including the CTA above the fold on my landing page decreased conversions by 17%. In most cases it’s a good idea to include the CTA above the fold, but this test showed that’s not always the case.

Placing the CTA above the fold on a landing page at QuickSprout.com decreased conversions by 17%.

When it comes to CTA buttons, you can test color, size, shape, location, and copy to see what works best on your website.

Test Idea 3: Long Vs. Short Copy

One of the biggest debates in copywriting and marketing circles is about whether long or short copy sells better. The right answer is that it depends on the product or service. Some products require longer copy to overcome customer objections and to explain how the product works. In other cases, short copy that gets to the point is better because less selling is needed and more copy just gets in the way. It really comes down to the product or service you’re selling.

But if you don’t test, you’ll never know. You may think that shorter copy works best for your product when in fact longer copy would increase conversions. There are some ways to get an idea if longer or shorter copy is better, but you’ll never know for sure without running a test. However, here are some cues you can consider:

• 1. Longer copy is generally needed for selling high-dollar products. A $4,000 purchase requires more consideration than a $25 one, so bigger purchases usually require longer copy.
• 2. Complicated products also need more copy. If your product is difficult to understand, then you’ll need to do more to educate your customers about how it works. Consider writing more copy if your product or service is something your customers have trouble understanding.
• 3. Your copy should be long enough to answer all of your customers’ objections. Through surveys you should find out what objections your customers have about making a purchase. Are they concerned about price or quality? Is anything else preventing them from making a purchase such as not understanding how the product works? Once you identify these objections, you should make sure your copy addresses each and every one of them.

Those three tips give you a starting point for knowing whether or not you need longer copy, and Crazy Egg provides a great example of this.

While working on the Crazy Egg website, Conversion Rate Experts proposed that a longer page would increase conversions. They believed that additional copy that addressed more of the objections they found through surveys would get more people to sign up. To test this, they created a page that was 20 times longer than the original. That might sound crazy, but the new version outperformed the control by 30%.

This flies in the face of what a lot of people consider common knowledge that web visitors don’t read anymore. Some experts like to say that people don’t read online so you shouldn’t write too much copy when that’s not actually true as this example shows.

Those who have copywriting and CRO experience know that quite often longer copy generates more sales. If someone’s interested in a product or service, they’re willing to read to learn more about it. If they’re not interested, they won’t read, but that’s not the kind of visitor you’re worried about anyway. If they are interested, they’ll read everything they can to become well informed before making a purchase.

So long versus short copy is another important initial test you can consider.

Test Idea 4: Buy Now Vs. Free Trial

If you haven’t tested a free trial yet, you should give it strong consideration. Numerous tests have confirmed that offering a free trial increases conversion rates, which makes sense when you think about it. You give customers the option to try your product for free, and they in turn have 15 to 30 days to decide whether or not they want to use your product for the long term by buying. It’s a win for customers, and it’s also a win for you when conversion rates go up, even though some people will inevitably cancel their free trial and not sign up.

Get Response experimented with adding a free trial button to their homepage and found that it increased conversions by 158.6%. At that rate they can afford for some customers to cancel the free trial and still come out ahead.

By adding a Free Trial button next to a Buy Now option, Get Response increased conversions 158.6%.

Test Idea 5: Credit Card Vs. No Credit Card

Another test you can consider is requiring a credit card versus not requiring a credit card. Quite often companies ask for a credit card up front before offering a free trial, but this obviously lowers conversions. Requiring a credit card creates a hurdle that visitors have to decide whether or not they want to jump over. They may be interested in your product, but they may be worried about getting charged if they don’t cancel in time.

Totango recently tested offering a free trial without requiring a credit card and found it increased free trials by 500% and paid subscriptions by 50%. That’s a big win and makes offering a free trial without requiring a credit card something worth testing.

Totanga increased free trials 500% and paid accounts 50% by not requiring a credit card up front.

Test Idea 6: Trust Symbols

It’s easy to overlook trust symbols, but they help to improve sales online because lots of people are skeptical about handing over their credit card information. They may like your product, but they could be unsure whether or not they can trust you with their credit card number. That’s where trust symbols come in handy.

Trust symbols show that your business or checkout process is verified by a third party, and they give people a reason to trust you. Barcodes UK, for example, tested adding a Trust Verified symbol to their website and increased sale conversions by 28% over the course of an 8-week trial. The test including 9,666 visitors and found the version with the trust seal converted at 2.33% while the one without it converted at 1.82% with a 96% likelihood of the results not happening by chance. That’s an impressive jump in conversions by only making one change to the website.

Barcodes UK increased conversions 28% with the version on the right that added a Trust Verified symbol.

Test Idea 7: Live Chat

Live chat is another feature you can consider testing on your site. With live chat, you add a plug-in like Olark or Snap Engage to your site, and then use it to talk to your visitors. It’s a great way to learn more about the objections visitors have when they’re on your site, and it’s also a great way to overcome those objections right on the spot.

For example, let’s say you sell cowboy boots online, and you’re not quite sure why sales aren’t as high as you’d like them to be so you decide to add live chat to your site. Within just a few days, you’re shocked with what you learn. You find out that five out of ten people are the most concerned about the return policy. They like the boots and the prices but aren’t sure how difficult it is to return boots if they don’t fit right.

In this case you can answer customers objections right away which will increase sales. You can explain your return policy to the 50% of customers that are concerned with that problem, and in turn, those customers are more likely to make a purchase. That’s one benefit from using live chat.

The other benefit is that you’ll learn how you can improve your site. You can create a spreadsheet and keep track of the questions that customers ask on your site. Then you’ll have a good idea of what objections they have and what’s preventing them from making a purchase. In the case of return shipping, now that you know it’s a problem, you can figure out how to change your site so your return shipping policy is more prominent. Once the change is made, you can run a test to see if it improves conversions or not. This is a great way to use live chat to optimize your site.

So what kind of results can you expect? It’s impossible to know for sure since every site is different, but Ez Texting, a Visual Website Optimizer customer, found that live chat increased sign ups by 31%. They added Olark live chat on the sign up page so customers could ask questions if they had any. This is what Josh Malin from Ez Texting had to say about the test for the case study on VisualWebsiteOptimizer.com.

I know that testing can contradict deeply held beliefs but it is helpful every now and then to go through a test that confirms this. It makes it that much easier in the future to test other long-held beliefs. And of course it’s always great to have solid data to explain the inclusion of some counterintuitive element on our front end sales site to everyone else at the company.

Josh Mali, Ez Texting

This quote provides a great example of why A/B testing is so valuable. Not only will you scientifically improve your site, but you’ll also gather data that will make decision making easier within your organization. Instead of trusting your team’s opinions, you’ll make decisions based on votes visitors cast on your site.

EZ Texting increased conversions 31% by adding live chat in the bottom right corner of their sign-up page.

Test Idea 8: Helping Vs. Selling

Most companies take a hard sell approach to selling online. They put their best foot forward and lean into the most persuasive pitch they can muster. In many cases this works, but in some it doesn’t.

The problem with a hard sell approach is that customers are on guard for anything that seems too salesy. Consumers are more educated than ever before and have their guard up against overly salesy pitches. Yes, in a lot of cases they still work, but they’re not always the best approach as proven by the experience of ActiveNetwork.

ActiveNetwork decided to take a more supportive, helpful approach with new e-mail copy they were testing. They included lines like, “We believe our product sells itself, so we’re just here to provide you with whatever assistance you need in getting your event up and running.” Did it make a difference? Yes, yes it did.

The new supportive, helpful copy increased leads by 349%. This goes to show that sometimes being salesy will turn your customers off. Instead of trying to be ultra-persuasive, you should be friendly and helpful to get a conversation and a conversion opportunity started.

ActiveNetwork found that e-mail copy with a more helpful tone increase leads generated by 349%.

Test Idea 9: Removing Form Fields

It’s well known in conversion rate optimization circles that extra form fields decrease conversions. You never know by how much, but even a single extra field is likely to decrease conversions by a certain percentage.

This means you want to use as few fields as possible. It also means you should only ask for necessary information. If you don’t need to ask for an e-mail address or credit card verification code, for example, then you shouldn’t.

A test I ran on Quick Sprout that removed one form field proves this point. The original form asked for First Name, Email Address, Website URL, and Revenue, and the new form deleted the Revenue field. Removing that single field on an already short form increased conversions by 26% which goes to show than even one field can make a big difference.

This doesn’t mean that less fields will always increase conversions. Sometimes people are expecting more fields and will be confused when they don’t find what they’re looking for. There’s a chance that an extra field or two will actually increase conversions, or at the very least generate more qualified leads, but you better test to find out. You don’t want to have four fields on your form when having three will increase conversions by 26%.

Test Idea 10: Create An Explainer Video

Explainer videos are short animated videos that websites use to explain what they do online. They’re usually 60 to 90 seconds long and help to explain complicated products or to emotionally connect with visitors through images, audio, and text. They’re quite awesome, and they’ve also been shown to increase conversions.

Crazy Egg, for example, increased conversions 64% on their homepage by adding an explainer video. They took their time to write a really awesome script, and the result was a video that resonated with visitors and convinced them to sign up for a free trial. That’s one positive case study.

In another example, Rypple, a Switch Video client, found that adding an explainer video to their homepage increased conversions by 20%. They also found that it lowered customer acquisition cost and led to customers being more engaged and ready to use the product. Those are all great benefits from creating an explainer video and putting it on your homepage or other pages of your site.

Rypple increased conversions 20% by adding an explainer video to their homepage.

Test Idea 11: Free Trial Vs. Money-Back Guarantee

You may think that a free trial is the same as a money-back guarantee because they seem like the same thing, but your customers may disagree.

I ran a test on Quick Sprout to see if one converted better than the other. I removed all of the money-back guarantee badges and replaced them with 30-day free-trial badges. The result? Conversions went up 116%. This goes to show that small changes can make a big difference and that customers perceive free trials and money-back guarantees differently.

Test Idea 12: Free Trial Length

What do you think is better: a 14-day or a 30-day free trial? If you’re like most people, you probably think 30 days is better since it gives customers more time to try out your product. That’s the same thing I thought until before testing a 14-day trial versus a 30-day trial on KISSmetrics.com.

The same number of people signed up for a free trial with both versions so that ended up being a wash. What was interesting, though, is that the number of people who used the product went up by 102% with the 14-day trial which means that the shorter trial period convinced people to try out the free trial sooner and ensured they didn’t forget about it over the longer trial period. The extra usage also boosted revenue as more people found out how valuable KISSmetrics is.

Test Idea 13: Adding A Dollar Value To Free Offers

Have you ever seen a late night commercial that used this line: “Order now and get a free carrot peeler worth $19.99 completely free. That’s right, order today and get a free bonus carrot peeler valued at $19.99.” That’s what it means to add a dollar value to a free offer.

By providing a dollar value, you quantify the benefit that someone gets by receiving a product, service, or eBook for free. A free product worth $19.99 is much more exciting than a free product worth an unknown amount. This tactic can be used for free downloads, bonus products, etc.

On Quick Sprout, for example, I tested adding the dollar value of the course I give away for free when people subscribe. By stating that the course is valued at $300, I was able to increase the e-mail opt-in rate by 22%. If you’re currently giving anything away for free, you may want to consider adding the dollar value to see if it increases conversions.

Test Idea 14: Show A Live Version Of Your Product Instead Of Screenshots

Lots of people use screenshots on their homepage to show how their product works because it’s a great way to give visitors a behind the scenes look at what they’ll get by signing up for your product. But not as many people show a live version instead of screenshots.

With a live version, you actually demonstrate the product before people are required to make a purchase or sign up for a free trial. Three companies that show a live version are Optimizely, Olark, and Qualaroo.

Instead of trying to explain what they do in detail on their homepage, Optimizely invites you to test out their software by typing in your URL. It’s a great way to show off how easy their product is to use and to give visitors a behind the scenes look at how Optimizely looks.

Optimizely’s nifty little “try it out” box makes it easy for visitors to try the product even before they sign up for a free trial.

Olark draws attention to the product demonstration on their homepage with a pop-up that says “Hey, We’re Olark. We Built This.” It’s an effective way to draw attention to show off the product they built on their homepage.

Qualaroo also demonstrates their product on the homepage by using a pop-up survey to ask visitors a question. It serves as an example and also a way to learn more about their visitors. It also helped to boost conversions the conversion rate by 38%.

Demonstrating their product on the homepage boosted conversions for Qualaroo by 38%.

Test Idea 15: Create A Two-Step Checkout Process

Something else you can try is a two-step checkout process. In many cases, limiting the number of steps and pages people need to pass through is a good idea, but in some cases, adding an extra step or two can increase conversions.

At Crazy Egg, Conversion Rate Experts suggested a three-page checkout process. First, you select your plan; next, you create your account; and finally, you enter your payment information. It had the same number of fields as the previous two-step process, but they were broken up into three separate pages. The result was a 10% increase in conversions.

Test Idea 16: Offer A Time-Based Bonus

Bonuses are a great way to increase conversions. With a bonus, you offer something extra in addition to your initial offer, such as an extra video interview or a free guide. A knife retailer, for example, could offer a free sharpener as a bonus for all purchases over $50. The bonus costs a minimal amount and entices more customers to make a purchase. Bonuses alone are a great way to increase conversions, but there’s something else you can do.

That something else is offering a time-based bonus. At Quick Sprout, I increased conversions 11% by offering a free video course and a free plug-in with the Quick Sprout Traffic System. But then I tried something else. Michael Williams suggested offering an additional bonus where the first 50 or 100 people to sign up got something extra special. The result? Conversions went up another 47% with the time-based bonus.

Good copywriters and conversion experts know that one of the best ways to generate more sales is to create a sense of urgency, and that’s exactly what time-based bonuses do. They entice interested customers to buy right away so they don’t miss out on the special, limited time offer. If this makes sense for your business model, you should give it a try to see if you can increase conversions 47% or more.

Test Idea 17: Testimonials

Testimonials are another CRO testing staple. They work well because they allow you to say things on your site that you couldn’t say otherwise. It looks bad for you to brag about your business, but it’s ok if someone else does it. They also work because customers trust what other people say about you much more than they trust what you say. You’re obviously biased. Happy customers provide a much more objective evaluation of your business.

Here are some tips to keep in mind about testimonials:

1. They need to seem realistic. If the testimonial seems too far fetched or doesn’t seem realistic, customers won’t trust it as much as another testimonial.
2. It should say something specific about your business. Testimonials that just say, “Business X is amazing. I love them so much!” isn’t as good as, “Business X knows more about social media than any other company we’ve worked with. They helped us double our social media following in three short months!” The more specific the testimonial, the better.
3. Use pictures and video to make testimonials even more believable. Text testimonials aren’t the only tool you have at your disposal. Pictures of the testimonial giver can be used to make it even more realistic, and videos can be used to provide an enriched testimonial that fully captures your customer’s satisfaction.
4. Placement matters too. Not only do you need to use testimonials that mention something specific about your business, but you also need to use testimonials in the right places. For example, a well-placed testimonial on the check-out page of the Quick Sprout Traffic System increased conversions by 6.38%.

Conversion Rate Experts uses video testimonials to provide more in-depth testimonials and case studies on their website.

Test Idea 18: Copywriting

This is an obvious testing opportunity that we haven’t talked about yet. Quite often, small or big changes to copy can have a big impact on conversions.

For example, lots of companies ask people to sign up for e-mail newsletters without providing a reason to do so. They usually say something like, “Sign up for free updates,” which is great, but everyone provides free updates so you’re not doing anything special. Instead, you could provides reasons or benefits for signing up.

Michael Aargaard ran a test about this on his site ContentVerve.com. The control used “Get fresh updates from ContentVerve.com” to get people to sign up. The challenger used “Get fresh updates” as the headline and added three bullet-point benefits for signing up. Here’s the copy he added:

• Case studies & test results
• How-to videos and articles
• Podcasts w/thought leaders

The result? 83.75% more people signed up with the new version compared to the control with a 97% statistical confidence level. This test shows that adding or changing copy can have an impact on conversions.

When it comes to copy, you can test adding, removing, editing, or changing copy, which includes headlines, sub-headlines, bullet points, product descriptions, etc.

Test Idea 19: Navigation Copy

Something else you can consider testing is the copy in your navigation menu because the words you use do make a difference.

For example, maybe you use “Contact Us” as a menu item when really what you want people to do is hire you. In this case you could test changing the copy to “Hire Us” to see if the click-throughs and form submissions create more qualified leads.

An example of this can be found in Dan Siroker and Pete Koomen’s book A/B Testing. In the book they discuss a test run by Formstack, an easy-to-use online form builder. Prior to testing, Formstack used “Why Use Us” as the first menu item in their main navigation.

Then, after giving it some thought, they postulated that “How It Works” may be better copy to use in that position on the site.

However, not wanting to make a decision without testing, they decided to run an A/B test. The result was nearly 50% more traffic to the page and an 8% increase in free trial sign ups. These results quelled the internal debate about which version was better and allowed Formstack to move on and conduct additional tests.

Formstack changed the copy in their first menu item from “Why Use Us” to “How It Works” and increased page traffic by 50% and free trial sign ups by 8%.

Test Idea 20: Carousel Vs. Static Image

There are many tests you can conduct for the media on your homepage, including video vs. image, people vs. product images, etc. One important test you should consider running is a carousel vs. a static image.

Why? Because for some odd reason, business after business defaults to using a carousel on their homepage without considering whether or not it’s the most effective way to present their offer. Maybe it’s because it seems high tech, or maybe web designers just fall in love with them. Either way, many UX and conversion professionals question the efficacy of sliders, and tests have been run that show they’re not as effective as static content.

In two such tests, Blair Keen, an Optimization Manager at Adobe, found that removing a slider in one test increased sales by 23% and removing it in another had no effect on conversions.

In this test, the challenger increased conversions 23% by removing the slider from the homepage.

So what do we learn from these tests? First, we learn that sometimes, if not many times, a static image will outperform a slider when it comes to conversions. Second, we also learn that there are times when a slider and a static image will perform at the same level which means it’s up to the business owner which option they’d like to choose. Either way, it’s recommended you test your slider to make sure you’re not sabotaging sales with a distracting slider that frustrates users.

Conclusion

• Over time your going to come up with a lot of testing ideas. The best way to keep track of these ideas is to create a spreadsheet where you can list all of your ideas and prioritize them based on your best guess of how beneficial a change will be.
• Two easy tests to run are headlines and call-to-action buttons. With headlines, you can try a benefit, loss aversion, or question headline to see if one works better than the other.
• Long versus short copy is another important test you can run. If people are interested in your product, there’s a good chance they’ll want to read more about it, so don’t convince yourself short copy is better before you give long copy a try.
• Free trials, not asking for credit card information before a free trial, and trust symbols are great opportunities for conversion testing.
• Live chat is another test you can run to see if helping people in real time can increase conversions.
• Speaking of helping people, sometimes helpful copy works better than super persuasive, hard-sell copy.
• Reducing form fields is another great way to increase conversions if you have any superfluous fields that can be removed.
• If you haven’t tried an explainer video yet, you may want to give it a go. Crazy Egg increased conversions 64% on their homepage with an explainer video, and Rypple increased conversions by 20%.
• A free trial versus a money-back guarantee and the length of a free trial are two more things you can consider testing.
• You can also try adding a dollar value to free offers as a way to entice customers to make a purchase or download a free eBook.
• Showing an example of your product instead of screenshots is another opportunity for testing.
• You can also try a two or three step checkout process to see if the same number of fields broken up into multiple steps will increase conversions.
• Here’s a great list of ecommerce testing ideas.