Optimizely vulnerability lets you see what other sites are testing

Updated 7 a.m. 6/3/2014 with clarifications and a statement from Optimizely.

Optimizely, a popular service that helps web site owners conduct tests to improve usability, is leaking information about those tests.

The leak, which is embedded in the JavaScript code used to implement Optimizely on each site, doesn’t appear to endanger the sites or their fundamental security. But it does enable anyone, with very little effort, to see exactly what tests an Optimizely customer is conducting.

That’s because the details of each test are encoded directly in the JavaScript. In fact, according to John McLaughlin, a marketing consultant who discovered the leak, the code contains details on all the Optimizely tests a site has done in the past as well as current tests. (Update: The code does not contain archived tests, an Optimizely representative stated.)

“It’s really useful if you’re looking at competitors’ stuff, and seeing what they’re running,” McLaughlin told VentureBeat.

Update: Optimizely says that the leak is not unintentional — but that rather, it is a feature of all JavaScript-based testing tools. However, the company said it will also soon release a version of its tool that enables customers to hide the details of their tests.

McLaughlin built a site, whatyatesting.com, to show off the vulnerability. For instance, you can see which Optimizely tests Starbucks is conducting, or which tests Healthcare.gov has done. Other sites McLaughlin has scoped out include payroll-processor ADP, freelance marketplace oDesk, domain registrar GoDaddy, and news site CNN.

Optimizely simplifies the process of doing A/B tests, in which a site randomly delivers one of two variations to each visitor, then collects data about which variation visitors click on more. A/B tests can be handy for deciding which color to make a “buy” button, how large of a font to use, what header image to use, and so on.

But A/B tests can also be used to try out new products on a subset of a site’s audience — or to try out new pricing schemes. If those tests reached a wider public — or a site’s competitors — the leaks could be potentially damaging.

Code education company General Assembly, for instance, appears to be testing a price change from $29 to $49 for some of its online classes. And Alexa.com is testing new products, but only with a subset of the worldwide audience, McLaughlin said.

In a response, Optimizely stated:

To clarify, this is not a security vulnerability. All JavaScript experimentation platforms, including Optimizely, have information that is visible in source code. To make integrations with third party tools (such as analytics platforms) simpler for our customers, we originally chose to include Optimizely experiment and variation names in the source code.

We recognize that some customers may prefer that their experiment and variation names not be visible in source code, even if it makes integrations with third-party tools a little bit harder. To address this, we will soon release an option for customers to mask Optimizely experiment and variation names in source code.

McLaughlin said he’d brought this vulnerability to Optimizely’s attention before but had received no response. Eventually that prompted him to bring the story to the public, via VentureBeat.

Optimizely said that it had no record of McLaughlin contacting the company.