Updated 7 a.m. 6/3/2014 with clarifications and a statement from Optimizely.
Optimizely, a popular service that helps web site owners conduct tests to improve usability, is leaking information about those tests.
“It’s really useful if you’re looking at competitors’ stuff, and seeing what they’re running,” McLaughlin told VentureBeat.
McLaughlin built a site, whatyatesting.com, to show off the vulnerability. For instance, you can see which Optimizely tests Starbucks is conducting, or which tests Healthcare.gov has done. Other sites McLaughlin has scoped out include payroll-processor ADP, freelance marketplace oDesk, domain registrar GoDaddy, and news site CNN.
Optimizely simplifies the process of doing A/B tests, in which a site randomly delivers one of two variations to each visitor, then collects data about which variation visitors click on more. A/B tests can be handy for deciding which color to make a “buy” button, how large of a font to use, what header image to use, and so on.
But A/B tests can also be used to try out new products on a subset of a site’s audience — or to try out new pricing schemes. If those tests reached a wider public — or a site’s competitors — the leaks could be potentially damaging.
Code education company General Assembly, for instance, appears to be testing a price change from $29 to $49 for some of its online classes. And Alexa.com is testing new products, but only with a subset of the worldwide audience, McLaughlin said.
In a response, Optimizely stated:
We recognize that some customers may prefer that their experiment and variation names not be visible in source code, even if it makes integrations with third-party tools a little bit harder. To address this, we will soon release an option for customers to mask Optimizely experiment and variation names in source code.
McLaughlin said he’d brought this vulnerability to Optimizely’s attention before but had received no response. Eventually that prompted him to bring the story to the public, via VentureBeat.
Optimizely said that it had no record of McLaughlin contacting the company.