How Pinterest, Google and Amazon Are Improving Visual Search

Google, Snap, Inc. and Amazon revealed new visual search tools this week, highlighting the upcoming change in how consumers find products and information.

Companies relying on search have long envisioned search engines that launched queries with images rather than text. Recent improvements in computer-vision technology have finally made this dream a reality, and the biggest technology companies, including Pinterest, Google, Amazon, Microsoft and eBay, now have visual search tools. As they attract users and improve accuracy and relevancy, these tools will continue to post strong growth in visual search volume.

eMarketer's latest report, "Visual Search 2018: New Tools from Pinterest, eBay, Google and Amazon Increase Accuracy, Utility" looks at what is possible now using images as a starting point for search.

Pinterest, for example, has an ambitious visual search strategy, which has produced tools such as Shop the Look, which lets users select blue dots within a pin to find similar products, and Lens, a real-time visual search tool embedded in the camera of the Pinterest App (and elsewhere). Pinterest’s bet on visual search has paid dividends, with searches on Lens increasing 140% between February 2017-2018.

Pinterest rolled out its first visual search tools in 2015, which actually followed early tools by Google and Amazon. Over the past two years, both companies quietly continued to develop their image recognition technology and have released major new visual search tools.

Google notably released Google Lens in 2017. This general visual search engine can analyze and return results for images ranging from products to landmarks to dog breeds. With an update in May 2018, it’s now embedded in the camera, Google Assistant and Google Photos apps of many Android smartphones. The update also introduced real-time results within the camera, style match that suggests similar products, and smart text selection, which gives users the ability to save and even translate text from images or the real world. This week, Google announced plans to release Lens within Google Images in fall 2018, giving users a chance to identify specific elements within an image to search for additional relevant images or product pages.

Amazon offers its visual search technology to other companies through its Rekognition platform. The Rekognition application programming interface (API) has a suite of image recognition tools, including object identification that can enable visual search. In fact, the API helps Pinterest interpret text within images, and Amazon’s visual search technology also powers Snapchat’s visual search tool.

Snapchat’s tool links directly to an Amazon product detail page and is specifically positioned as a product search tool. In the early version of the visual search function Snap, Inc. is rolling out in the US in late September 2018, Snapchat users are able to press and hold on the camera screen while it’s pointed at an object, and an Amazon product card will pop up on the screen. With augmented reality and the camera already big components of the platform, visual search is a natural extension for Snapchat to keep up with Google's and Pinterest's latest offerings.

Bing and eBay also recently released visual search tools joining dozens of smaller visual search providers, many of which are already selling tools embedded in retailer sites.

The wave of new tools has increased awareness of visual search among smartphone users. In a July 2017 survey by the National Retail Federation, 27% of US internet users were aware of visual search technology. A more recent survey by RichRelevance found that more than half of US internet users (52.3%) would like to be shown similar products after taking a photo via a retailer’s mobile app.

Greater competition and awareness is leading to growing consumer uptake, which in turn increases the accuracy of visual search tools. It's likely this virtuous cycle has already pushed monthly visual searches above 1 billion queries. Although this pales next to the hundreds of billions (at a minimum) monthly text searches generate, growth will likely continue to be strong as competition among providers heats up and accuracy and utility improve.

In the latest episode of "Behind the Numbers," eMarketer analysts Yory Wurmser and Andrew Lipsman explain what visual search is and how it works.