Google focuses Lens on combined image and text searching

Want that thing, but in a different color, and don't know the name? Multisearch has your back


Google's latest feature is making its Lens visual search tool mingle with text for image searches for those difficult-to-describe vague queries.

That may not sound like a big deal, but what Google calls Multisearch "a feat of machine learning" could change the way people perform some of the most common searches and take a bite out those moments when you know the rough idea of what you're looking for but lack a suitable way to define it in text or image searches alone.

Retailers will salivate over the vague search option, for one, as highlighted by Belinda Zeng, Google Search product manager, who described several examples of how Multisearch could work: "How many times have you tried to find the perfect piece of clothing, a tutorial to recreate nail art or even instructions on how to take care of a plant someone gifted you – but you didn't have all the words to describe what you were looking for?"

Multisearch would address all of those situations: a photo of a dress could be queried with the name of a preferred color, photos of a finished set of nails could be searched with the phrase "how to," and unknown plants can be Googled with "care instructions." 

An example of Google's Multisearch tool

An example of Google's Multisearch tool

In each of those cases, Google Search is addressing multiple unknowns: what the object in the image is, how the text request relates to the image, and how to combine the two into what the user wants.

In short, what you need is an expert on those particular subjects with the skill to infer, based on what it's presented with, the best course of action.

Say hi to MUM

Multisearch is possible thanks to Google's Multitask Unified Model, or MUM. MUM is an improvement on BERT, Google's previous AI language model released in 2019. BERT improved Google search results by 10 percent for English speakers, Google said. MUM, it added, is 1,000 times more powerful than BERT.

Google's Pandu Nayak described the difference between BERT and MUM using the planning of a hiking trip as an example: ask an expert how to prepare for a hike and they'll take all sorts of things into account. 

"Today, Google could help you with this, but it would take many thoughtfully considered searches – you'd have to search for the elevation of each mountain, the average temperature in the fall, difficulty of the hiking trails, the right gear to use, and more," Nayak said. 

BERT is what you'd use to make multiple searches (Nayak said eight on average for a query of that level of complication), but MUM is designed to respond like the aforementioned professional. "MUM could understand you're comparing two mountains, so elevation and trail information may be relevant. It could also understand that, in the context of hiking, to 'prepare' could include things like fitness training as well as finding the right gear," Nayak said. 

As with all things search, Google isn't providing specifics about its algorithms, so the inner workings of MUM and its Multisearch capabilities are likely to remain a mystery. Google said that Multisearch is available in beta through its iOS and Android apps in the US only, though as of writing it doesn't appear to be available in the iOS version of the app.

Google said MUM-powered Multisearch currently works best for shopping-related searches (e.g. drapes with pattern X, a dress in a different color, etc.). Other searches will work, but Google's not vouching for their effectiveness. We've asked Google if it can explain the technology behind Multisearch and future applications in more detail.

In calendar 2021, 'Google Search and other' brought in $43.3 billion in revenue, up from $31.9 billion in the prior twelve months. The entire company turned over $75.3 billion for the year. ®


Other stories you might like

  • End of the road for biz living off free G Suite legacy edition
    Firms accustomed to freebies miffed that web giant's largess doesn't last

    After offering free G Suite apps for more than a decade, Google next week plans to discontinue its legacy service – which hasn't been offered to new customers since 2012 – and force business users to transition to a paid subscription for the service's successor, Google Workspace.

    "For businesses, the G Suite legacy free edition will no longer be available after June 27, 2022," Google explains in its support document. "Your account will be automatically transitioned to a paid Google Workspace subscription where we continue to deliver new capabilities to help businesses transform the way they work."

    Small business owners who have relied on the G Suite legacy free edition aren't thrilled that they will have to pay for Workspace or migrate to a rival like Microsoft, which happens to be actively encouraging defectors. As noted by The New York Times on Monday, the approaching deadline has elicited complaints from small firms that bet on Google's cloud productivity apps in the 2006-2012 period and have enjoyed the lack of billing since then.

    Continue reading
  • Google battles bots, puts Workspace admins on alert
    No security alert fatigue here

    Google has added API security tools and Workspace (formerly G-Suite) admin alerts about potentially risky configuration changes such as super admin passwords resets.

    The API capabilities – aptly named "Advanced API Security" – are built on top of Apigee, the API management platform that the web giant bought for $625 million six years ago.

    As API data makes up an increasing amount of internet traffic – Cloudflare says more than 50 percent of all of the traffic it processes is API based, and it's growing twice as fast as traditional web traffic – API security becomes more important to enterprises. Malicious actors can use API calls to bypass network security measures and connect directly to backend systems or launch DDoS attacks.

    Continue reading
  • FTC urged to probe Apple, Google for enabling ‘intense system of surveillance’
    Ad tracking poses a privacy and security risk in post-Roe America, lawmakers warn

    Democrat lawmakers want the FTC to investigate Apple and Google's online ad trackers, which they say amount to unfair and deceptive business practices and pose a privacy and security risk to people using the tech giants' mobile devices.

    US Senators Ron Wyden (D-OR), Elizabeth Warren (D-MA), and Cory Booker (D-NJ) and House Representative Sara Jacobs (D-CA) requested on Friday that the watchdog launch a probe into Apple and Google, hours before the US Supreme Court overturned Roe v. Wade, clearing the way for individual states to ban access to abortions. 

    In the days leading up to the court's action, some of these same lawmakers had also introduced data privacy bills, including a proposal that would make it illegal for data brokers to sell sensitive location and health information of individuals' medical treatment.

    Continue reading

Biting the hand that feeds IT © 1998–2022