This is part 2 of a 5-part series about my time at the SXSW Interactive Conference 2017. I’ve consolidated information around five trends and some of the most interesting things I learned about each of them. (Originally sent as a series of emails to a smaller group of associates, RPA thought this was so great that we decided to archive them on our website.)

Today’s trend is testing, and how tech-oriented brands are using ongoing A/B testing to drive product innovation.

Can’t resist including the attached comic that (RPA Associate Strategic Planning Director) Rich Bina sent when we were working on our last testing project.


A/B testing can sometimes feel almost…manipulative. You are, after all, doing free, live focus groups while sitting back and watching where all the sheep run. Testing, and specifically A/B testing, came up over and over again in various sessions and panels at SXSW. Brands have heard the Kodak and Blockbuster examples quoted too many times to be content sticking with business as usual. But how can product, advertising or websites be constantly changed to keep up with the ways our audience is changing? A/B testing! Netflix is a brand that is championing this methodology and it is the main driver for their product development. They value disproven hypotheses as highly as proven ones. With that mentality, every test provides value and a foundation for the next test.

I thought this example made a pretty strong case for the importance of testing and observing what people do versus what they say:

Netflix wanted to increase customer sign-ups. In a survey of potential customers, 50% indicated content was what they wanted to know before signing up. So a pretty simple hypothesis – showing the content people will get when they sign up will increase acquisitions. In prototype testing focus groups Netflix ran, non-members overwhelmingly said they would be more likely to sign up when shown the prototype version where they could browse titles and see the content they would get. Implement that prototype, right? No! In actual A/B testing, the sign-up page without the titles was the best at getting acquisitions. People would get too focused on whether specific titles were or weren’t on Netflix. It marginalized the product to just be whether a few movies could be found or not. In the end, Netflix said they found a way to be somewhere in between business and customer goals.

Some other A/B testing insights from across SXSW Interactive-

  • Develop a hypothesis, run your experiment, get the result - if false, that's great because you benefit from saved time and investment you could have made seeing that idea fully through to implementation
  • Don't take beliefs for granted when setting up a test. Tests built on assumptions can quickly fall apart
  • Netflix uses an evergreen A/B testing approach. It’s unlikely you are seeing the exact same experience that your neighbor is. They also layer in surveys, research sessions/focus groups, and evaluate trends in their user data. But everything is A/B tested before going live
  • Be reasonable with the potential impact a test could have. Google’s 41 shades of blue test netted a +$200M/yr improvement in revenue - but their entire business model is getting people to click links (
  • In testing, your metric needs to be the compass for everyone involved
  • Design to the extremes- for the testing variations, the more extreme the better. Our intuition is often wrong, so it’s important to explore the boundaries to better understand them. Our instinct may be to be explore variation in the middle of the axis, but you can learn way more from being in the corners
  • Be ok with some ambiguity with what drove the results. Can retest after to parse out what moved the needle. Don’t get bogged down with variations initially – think more conceptually about your variations
  • Don’t discount the importance real data (content fidelity) will have on your test variations. Whenever possible, integrate live APIs into prototypes. Also allows for a better understanding of technical limitations earlier in the process

Back tomorrow for III.