What I learned at SXSW Interactive
This is part 2 of a 5-part series about my time at the SXSW Interactive Conference 2017. I’ve consolidated information around five trends and some of the most interesting things I learned about each of them. (Originally sent as a series of emails to a smaller group of associates, RPA thought this was so great that we decided to archive them on our website.)
Today’s trend is testing, and how tech-oriented brands are using ongoing A/B testing to drive product innovation.
Can’t resist including the attached comic that (RPA Associate Strategic Planning Director) Rich Bina sent when we were working on our last testing project.
A/B testing can sometimes feel almost…manipulative. You are, after all, doing free, live focus groups while sitting back and watching where all the sheep run. Testing, and specifically A/B testing, came up over and over again in various sessions and panels at SXSW. Brands have heard the Kodak and Blockbuster examples quoted too many times to be content sticking with business as usual. But how can product, advertising or websites be constantly changed to keep up with the ways our audience is changing? A/B testing! Netflix is a brand that is championing this methodology and it is the main driver for their product development. They value disproven hypotheses as highly as proven ones. With that mentality, every test provides value and a foundation for the next test.
I thought this example made a pretty strong case for the importance of testing and observing what people do versus what they say:
Netflix wanted to increase customer sign-ups. In a survey of potential customers, 50% indicated content was what they wanted to know before signing up. So a pretty simple hypothesis – showing the content people will get when they sign up will increase acquisitions. In prototype testing focus groups Netflix ran, non-members overwhelmingly said they would be more likely to sign up when shown the prototype version where they could browse titles and see the content they would get. Implement that prototype, right? No! In actual A/B testing, the sign-up page without the titles was the best at getting acquisitions. People would get too focused on whether specific titles were or weren’t on Netflix. It marginalized the product to just be whether a few movies could be found or not. In the end, Netflix said they found a way to be somewhere in between business and customer goals.
Some other A/B testing insights from across SXSW Interactive-
Back tomorrow for III.