Facebook has revamped Marketplace, a platform where users can buy and sell stuff, with a bunch of AI-powered features that, it claims, can automatically recognize and label objects in pictures as well as transform 2D images into a 3D view.
At the heart of what it calls its “universal computer vision system” is a model called GrokNet. Trained on millions of real images posted by sellers on Marketplace, GrokNet spits out descriptions of objects in pictures, and can cope with nearly everything from cars to home decoration.
For example, it can categorize gear and identify its properties, such as its color, material, style, and the like. Given an image of, say, a hat, it is able to label it as a “beige corduroy watchcap” or a “cream cableknit fedora,” or whatever is appropriate.
GrokNet is a big model that uses 83 loss functions to identify items it saw and learned from in seven image datasets. “We discovered that each data set has an inherent level of difficulty,” a team of Facebook research scientists and engineers explained this week.
“Easier tasks don’t need much supervision and can be given a smaller weight — this insight helped us improve accuracy on multiple tasks simultaneously. For difficult tasks, we need to apply a very large weight in the loss function and apply many images for each batch when training.
“For simpler data sets, we can use a small weight and a small number of images per batch. For this unified model, we allocated most of the batch size to the more challenging data sets and only one or two images per batch to the simpler ones. This allowed us to scale up and ensure that all 83 loss functions work well at the same time.”
The model – written using PyTorch – was trained on 96 GPUs, we’re told. Now that it’s in production, it runs on servers built using the Facebook-championed Open Compute Project blueprints, with the PyTorch runtime used for inference.
Its predictions can be used to enhance Facebook Marketplace’s search and recommendation systems. Buyers interested in a particular item can describe what they're looking for, and the system will spit out matching products from their automatically AI-generated descriptions. This is perhaps a little more trustworthy than relying on seller-written captions.
GrokNet is able to label multiple objects in a photo by segmenting an image into parts and placing an outline around each detected and identified item. A tool known as Instance Mask Projection is used to predict the overall shape of something, like a scarf around someone’s neck, so that its pixels can be analyzed to extract details like its color or pattern.
At the moment, the model‘s outputs are used to improve search on Marketplace. Facebook’s ultimate ambition, however, is to use its universal computer vision system across its apps, whether its Instagram or Marketplace, so it can automatically detect and identify objects in snaps, allowing people to – for example – select things that interest them in a photo, pull up the AI-generated description, and place online orders for the stuff.
Taking 2D footage and turning it into a 3D model
Another feature is called Rotating View. Essentially, it turns a set of 2D images into a 3D view, which is particularly handy for viewing things like furniture at different angles.
Facebook's mega-chatbot has 'a persona, discusses nearly any topic, shows empathy.' Perfect for CEO version 2READ MORE
“If, for example, sellers want to list an 18x60 chestnut wood rustic coffee table on Marketplace, they can capture a short video clip. Potential buyers can then spin and move the table up to 360 degrees to see whether the color, size, style, and condition would work well for their space,” Facebook's eggheads said.
In effect, an algorithm maps features detected in the video clip's 2D stills to a 3D view of the object that can be smoothly rotated all around. Rotating View is only available for Marketplace on iOS, for initial testing, at the moment.
“Our long-term vision is to build an all-in-one AI lifestyle assistant that can accurately search and rank billions of products, while personalizing to individual tastes. That same system would make online shopping just as social as shopping with friends in real life," the team said.
"Going one step further, it would advance visual search to make your real-world environment shoppable. If you see something you like (clothing, furniture, electronics, etc.), you could snap a photo of it and the system would find that exact item, as well as several similar ones to purchase right then and there.”
Facebook is also extending its social media platform to support more e-commerce options. It rolled out Shops, a new way for businesses to advertise their products in an online catalogue.
Customers can interact with sellers through Whatsapp, Messenger, or Instagram Direct. Shops is available now. There’s also a version for Instagram too known as Instagram Shop, which is expected to roll out later this year in the Summer. ®