Brian Christian reports in the May 2012 edition of Wired that (now-classic) A/B testing of web site variations has virtually replaced HiPPO decision-making in tech circles, at least for web design decisions. (HiPPO is “highest-paid person’s opinion.”)
“Over the past decade, the power of A/B testing has become an open secret of high-stakes web development. It’s now the standard (but seldom advertised) means through which Silicon Valley improves its online products. Using A/B, new ideas can be essentially focus-group tested in real time: Without being told, a fraction of users are diverted to a slightly different version of a given web page and their behavior compared against the mass of users on the standard site. If the new version proves superior—gaining more clicks, longer visits, more purchases—it will displace the original; if the new version is inferior, it’s quietly phased out without most users ever seeing it. A/B allows seemingly subjective questions of design—color, layout, image selection, text—to become incontrovertible matters of data-driven social science.”
A/B testing applications are not limited to web design; we’ve used the methodology successfully to achieve a deeper understanding of customers’ purchase triggers and behavior, especially for B2B transactions.
Process recommendation > Manufacturers who sell items at retail can use the same technique, over-labeling their primary consumer package with a different label version, changing POP, hangtag or display materials, or on-pack promotional offers. Sales volume then tells the story whether to change to the new label, POP or promotion. Data tells the story.
When you approach a big decision, amass whatever relevant data is available to counteract the HiPPO forces in your own decision process. If there is no data, then a market test might well be the appropriate next step in addition to starting a running box score on HiPPO decisions. If the batting average on HiPPO decisions doesn’t beat “Panda” Sandoval’s, include a PowerPoint slide with the HiPPO box score for the next decision round.
(“7 Ways to Spot the Next Big Thing p. 152 may be an even more valuable read.)