Executives have a unique role to play in encouraging the practice of experimentation within their companies. But arbitrary deadlines and a narrow focus can often hinder the process.
According to Hazjier Pourkhalkhali, Global Head of Strategy at Optimizely, there’s two key thing execs need to focus on to encourage experimentation: being customer-centric and creating an environment where their employees can deliver their best work.
“For experimentation what people really need at the end of the day isn’t so much resources as much as they need the time to come up with hypotheses, the traffic to get to the right degree of statistical validity and the freedom to make these kinds of big changes,” Pourkhalkhali said.
“On those fronts we have often seen executives come in with arbitrary deadlines that really hurt statistical validity and being overly excited about one particular route of research that hurts their teams’ abilities to be creative and find the best possible answer.”
Speaking to Which-50 between sessions at the company’s Modernize event in Sydney last week, Pourkhalkhali said it’s important business leaders accept the first quarter of any experiment isn’t about ROI.
“There’s nothing inherently wrong with a quarterly mentality, but what most executives need to realise is that no initiative needs to deliver ROI in the same quarter as it is started. For some reason when it comes to testing they believe from the second you start to the moment of ROI there can be no more than a month of time,” Pourkhalkhali said.
“The first quarter isn’t about ROI – it’s about building a scalable process, it’s about being able to drive insight and learning, it’s about setting the foundation for a lot of opportunities for ROI down the line. Being experimental is not something you get done in a single quarter.”
Embrace a multitude of directions
The other common mistake the companies make is getting overly excited about one particular path early in the process, Pourkhalkhali said.
“Someone in the c-suite will say ‘we want to build this new check-out flow’… and people have to prove that this route that has been per-determined for them is the one that works. But that’s not experimental, they are not letting the data guide them to the right answers.”
“By overly advocating for a particular routes, forcing people to narrow their focus early on, they’ve hurt the very craft of testing.”
The advice from Pourkhalkhali is not to limit a test to just two outcomes — for example one group of customers sees your existing app as is while to other sees a version with the new feature — but to take on a “portfolio of risk.”
For example if you segment the audience into five groups you can test the existing app versus the app with a new feature in different designs as well as a “shoot for the moon idea.”
Pourkhalkhali provided a real life example of multivariate testing from Booking.com. The hotel site wanted to re-write all their hotel descriptions to see if that was going to encourage more bookings.
“As you can imagine that’s quite an ordeal you are about to commit to and running that as a experiment doesn’t make sense because you’ve now re-written 50,000 hotel descriptions. That’s a lot of effort and labour gone to waste if it loses. So how do you test that upfront?”
What Booking.com did, Pourkhalkhali says, was test the assumption that people read the hotel descriptions. The experiment tested three versions of the site: one with the hotel descriptions as the were, one version where the hotel description is removed and one where the hotel version was replaced with random information about the city that was completely unrelated to the hotel.
“What they found was when they deleted the hotel descriptions conversion rates dropped, people reacted negatively, they felt less faith in the recommendation.”
“When they replaced the hotel description with gibberish they didn’t respond at all: what this said to them was, people don’t actually read our hotel descriptions, they just like knowing that someone did the work.”
Pourkhalkhali compared this outcome to the ingredient label on the back of a can of Coke, you don’t read everyone of the ingredients, but you like knowing the FDA has taken the time to review the contents.
The lesson being that companies need to determine how to test their assumptions before building every component of their grand vision.