Pinterest is bringing full automation to Shop the Look, a feature that helps users buy products from companies that work with Pinterest, so you can, for example, buy a pair of jeans you see in a picture.
Previously, Shop the Look, which made its debut in 2017, was powered by artificial intelligence to find objects in Pinned photos that resemble products in stock from vendors, but included a human in the loop for curation. The automated Shop the Look will begin with the home decor category.
Full automation for Shop the Look also moves the company closer to its goal to make the world pinnable.
Altogether, Pinterest users have made more than 175 billion Pins. Full automation for Shop the Look has led to a 22.5x growth in Shop the Look coverage across Pins and products, and in early testing has led to a 7 percent lift in engagement with Shop the Look, a company spokesperson told VentureBeat in an email.
The process necessary to automate Shop the Look was explained in a Medium post by Pinterest engineer Kunlong Gu. It started with a dataset of 270,000 scene-product matching pairs and the Google Product Taxonomy (GPT).
“Initially we strictly followed GPT, but the model didn’t perform well in some coarse-grained categories (i.e. beddings, tables). We found that products in these coarse-grained categories are of vastly different shapes and functions,” Gu wrote. “For example, the category of ‘bedding’ include[s] bed canopies, bed sheets, pillows. We then manually cleaned up the existing datasets (fine-grained labeling) and improved the model significantly.”
Each scene is then processed using a recurrent convolutional neural network to parse each image into individual objects.
Automation frees up Pinterest staff from the need to curate Shop the Look and will inform future computer vision and ecommerce efforts at the company.
“In the long term, the scene images are great resources to learn the relationship between objects, i.e. what objects complement each other or go well together in a certain style. We hope to leverage this rich data of object occurrence and build a sophisticated object graph for every object in the world, making Pinterest a personalized stylist for home, fashion and more,” Gu said.
Other computer vision-powered products from Pinterest, a company whose content is made almost entirely of images, include Lens visual search and Lens Your Look. As of February 2018, Pinterest powered 600 million visual searches a month.
Pinterest has accrued a variety of artificial intelligence talent in recent years, especially from Google, including head of engineering Li Fan, who has since moved on to Lime, and visual search and computer vision lead Chuck Rosenberg, who previously acted as head of computer vision research at Google.