Contentful vs Adobe Experience Manager: On A/B Testing (Round 2)

A/B testing is one of the most powerful weapons in a marketer’s arsenal. Or in fact, in anyone’s arsenal.

But it is also probably the most under-utilized weapon. It’s the shotgun that sits in everyone’s closet but they almost never take it out. And often they take it out only when it’s too late.

First, before jumping in to the question of the day — Adobe Experience Manager vs Contentful, in regards to their A/B testing abilities — let’s do a refresher: what is it, and why is it so under-utilized?

A/B testing and its sister multivariate testing is the ability to show different versions of a page to different people–and then measure which one ends up being better for your goals. So, for example, you can show half the people a blue-themed site and the other half a pink-themed site and then see which one results in more sales for for you. 

But it gets endlessly complex: what if you have different user flows on each you want to test, like fully different experiences? What if you want to make the changes consistent across multiple visits? Or across multiple platforms? What if you want to only test it on a small portion of the visitors? What if you want to test something sensitive like the price, where you need to make sure that users never see a lower price than higher price moments later because they’ll flip out? And so forth.

Just think through these sorts of experiments and you quickly realize that by doing change X you can increase your sales by 2%, then by doing change Y you can increase it by another 3% and so forth… and suddenly, you’ve doubled your sales!

But why is it so under-used then? For two reasons. First, lots of marketers are just uncreative: you need creative brainpower to map out and implement A/B testing strategies in a serious and rigorous way.

But there’s a second reason: most platforms have traditionally made it really complex – for the precise reasons I listed above (and many more). Imagine, say, wanting to do a complex test and thinking about building out these different user flows and then navigating them in an A/B test structure on the back-end… I almost want to have a heart attack thinking about it.

Well! How do our two favorite content platforms, Contentful and Adobe Experience Manager, deal with this question? The answer is, in opposite ways.

For Adobe Experience Manager, the functionality is there–and there is a bunch of complex back-end work to make it happen. Just reading their documentation about how to do it, and after the first sentence you think, “Okay, I need the software architect to come and plan this tiny little test I want to do! So much work just to see if our visitors might prefer a pink color!”

It is hard to do–but powerful, once you do it. As the instructions point out, you can integrate with external tools to manage it like Omniture, and you have enterprise-grade A/B testing reading to go, so you can deal with every multivariate edge case out there.

Contentful takes a different approach. (Google Docs just auto-completed that sentence correctly after I wrote “tak” so my writing is getting predictable. Alas, alas!). Contentful prides itself on its integrated-in, easy-to-use A/B testing. 

Here’s an easy example: on the Contentful content creation and editing screens… they have A/B testing buttons right there. Just click, edit, go–and voila! You have a test, there, done, live! (If only all of life were so easy as… Google Docs, are you going to auto-complete that sentence for me, too?)

So, there are three noteworthy observations on that fact.

One is that this is consistent with the general AEM vs Contentful broader difference: Contentful’s first and foremost focus on the ease of use. Yes, just knowing that, this difference shouldn’t be surprising.

Second is that, of course the easy button approach can’t account for the more complex cases. You want to have a weird complex series of use cases in particular you want to do things like make this one screen here, multiple screens there, doing something totally different – yeah, you will need to work with the API and need some developers for that. 

The third is a more subtle difference that Contentful doesn’t emphasize enough but I think they should, and I would if I led their marketing: how they’re a system that’s great for helping you learn how to get conversions. It’s a content system designed more for helping you track and achieve goals like more sales or more engagement. Rather than have to struggle to figure out how to do this test to improve your numbers – it’s right there. You can’t miss it.

And that, ladies and gentlemen (another Google Docs auto-complete!), is where the real, hidden power of Contentful lies: not just easy to use testing, but a system designed for conversion optimization. While the content creators may not fully appreciate this, the marketers will – and it’s their responsibility to get the content creators excited about actually using it. Because no matter how easy to use a feature is, if it is unused, it collects dust and it is all for nought, just like the nihilist’s view of life. (Google Docs, you didn’t see that one coming!)