Oh no, you're thinking, yet another cookie pop-up. Well, sorry, it's the law. We measure how many people read us, and ensure you see relevant ads, by storing cookies on your device. If you're cool with that, hit “Accept all Cookies”. For more info and to customize your settings, hit “Customize Settings”.

Review and manage your consent

Here's an overview of our use of cookies, similar technologies and how to manage them. You can also change your choices at any time, by hitting the “Your Consent Options” link on the site's footer.

Manage Cookie Preferences
  • These cookies are strictly necessary so that you can navigate the site as normal and use all features. Without these cookies we cannot provide you with the service that you expect.

  • These cookies are used to make advertising messages more relevant to you. They perform functions like preventing the same ad from continuously reappearing, ensuring that ads are properly displayed for advertisers, and in some cases selecting advertisements that are based on your interests.

  • These cookies collect information in aggregate form to help us understand how our websites are being used. They allow us to count visits and traffic sources so that we can measure and improve the performance of our sites. If people say no to these cookies, we do not know how many people have visited and we cannot monitor performance.

See also our Cookie policy and Privacy policy.

This article is more than 1 year old

It's, it's, a red-and-blue striped golfing umbrella... Facebook teaches its online tat bazaar to auto-identify stuff for sale

You wanted robo-butlers, you got robo-object-recognizers

Facebook has revamped Marketplace, a platform where users can buy and sell stuff, with a bunch of AI-powered features that, it claims, can automatically recognize and label objects in pictures as well as transform 2D images into a 3D view.

At the heart of what it calls its “universal computer vision system” is a model called GrokNet. Trained on millions of real images posted by sellers on Marketplace, GrokNet spits out descriptions of objects in pictures, and can cope with nearly everything from cars to home decoration.

For example, it can categorize gear and identify its properties, such as its color, material, style, and the like. Given an image of, say, a hat, it is able to label it as a “beige corduroy watchcap” or a “cream cableknit fedora,” or whatever is appropriate.

GrokNet is a big model that uses 83 loss functions to identify items it saw and learned from in seven image datasets. “We discovered that each data set has an inherent level of difficulty,” a team of Facebook research scientists and engineers explained this week.

“Easier tasks don’t need much supervision and can be given a smaller weight — this insight helped us improve accuracy on multiple tasks simultaneously. For difficult tasks, we need to apply a very large weight in the loss function and apply many images for each batch when training.

“For simpler data sets, we can use a small weight and a small number of images per batch. For this unified model, we allocated most of the batch size to the more challenging data sets and only one or two images per batch to the simpler ones. This allowed us to scale up and ensure that all 83 loss functions work well at the same time.”

The model – written using PyTorch – was trained on 96 GPUs, we’re told. Now that it’s in production, it runs on servers built using the Facebook-championed Open Compute Project blueprints, with the PyTorch runtime used for inference.

Its predictions can be used to enhance Facebook Marketplace’s search and recommendation systems. Buyers interested in a particular item can describe what they're looking for, and the system will spit out matching products from their automatically AI-generated descriptions. This is perhaps a little more trustworthy than relying on seller-written captions.

GrokNet is able to label multiple objects in a photo by segmenting an image into parts and placing an outline around each detected and identified item. A tool known as Instance Mask Projection is used to predict the overall shape of something, like a scarf around someone’s neck, so that its pixels can be analyzed to extract details like its color or pattern.

At the moment, the model‘s outputs are used to improve search on Marketplace. Facebook’s ultimate ambition, however, is to use its universal computer vision system across its apps, whether its Instagram or Marketplace, so it can automatically detect and identify objects in snaps, allowing people to – for example – select things that interest them in a photo, pull up the AI-generated description, and place online orders for the stuff.

Taking 2D footage and turning it into a 3D model

Another feature is called Rotating View. Essentially, it turns a set of 2D images into a 3D view, which is particularly handy for viewing things like furniture at different angles.

Mark Zuckerberg

Facebook's mega-chatbot has 'a persona, discusses nearly any topic, shows empathy.' Perfect for CEO version 2

READ MORE

“If, for example, sellers want to list an 18x60 chestnut wood rustic coffee table on Marketplace, they can capture a short video clip. Potential buyers can then spin and move the table up to 360 degrees to see whether the color, size, style, and condition would work well for their space,” Facebook's eggheads said.

In effect, an algorithm maps features detected in the video clip's 2D stills to a 3D view of the object that can be smoothly rotated all around. Rotating View is only available for Marketplace on iOS, for initial testing, at the moment.

“Our long-term vision is to build an all-in-one AI lifestyle assistant that can accurately search and rank billions of products, while personalizing to individual tastes. That same system would make online shopping just as social as shopping with friends in real life," the team said.

"Going one step further, it would advance visual search to make your real-world environment shoppable. If you see something you like (clothing, furniture, electronics, etc.), you could snap a photo of it and the system would find that exact item, as well as several similar ones to purchase right then and there.”

Facebook is also extending its social media platform to support more e-commerce options. It rolled out Shops, a new way for businesses to advertise their products in an online catalogue.

Customers can interact with sellers through Whatsapp, Messenger, or Instagram Direct. Shops is available now. There’s also a version for Instagram too known as Instagram Shop, which is expected to roll out later this year in the Summer. ®

 

Similar topics

TIP US OFF

Send us news


Other stories you might like