Would you Looker that: Google bakes in BigQuery caching layer to boost BI interactivity

'BigQuery is pretty fast anyway so you'd really need to value the extra speed'


Google has introduced a caching layer in Big Query - its cloud data warehouse - designed to speed up responses to users as they explore and experiment with data.

Under the moniker BI Engine, the data layer is designed to bridge the gap between BI tools that are built for interactivity like Tableau, which extracts data from the data warehouse, and an in-memory database for greater interactivity and Google's Looker, which queries data directly.

“The cloud databases... like BigQuery weren't built to support interactive BI, they were built for scale event analytics and scale-type questions,” said Colin Zima, chief analytics officer with Looker, the cloud BI biz Google bought for $2.6bn in February 2020.

The point of Looker, Zima said, was to allow more interactive BI on data in the database rather than creating a second repository. “We're probably the BI tool that works the best with the database, essentially in building out a data model that writes SQL to the database, and the database passes back SQL,” he said. Looker supports 55 flavours of SQL, he added.

The BI Engines caching layer within BigQuery is designed to overcome compromises in performance in terms of interactivity. Other BI tools including Tableau can take advantage of BI Engine through the BigQuery API, Zima said.

“It's a passive cache, which you allocate resource to: it intelligently attempts to figure out the types of things that you're going [to do] that need interactivity. It passively builds itself in the database so it can be queried interactively at much lower latency,” Zima said.

Solar panel company Sunrun has been using the tool. The Google Cloud Architect on its BI team, Kiran Manne, claimed he'd seen a 40 per cent performance increase across more than 1,000 interactive users.

However, Harvinder Atwal, chief data science officer at Big Query user MoneySupermarket told The Register that a real advantage would depend on how critical speed is to the use case.

"BigQuery is pretty fast anyway so you'd really need to value the extra speed if you're already a user. It's going to be more useful for organisations with separate ETL and marts for their BI. Putting the data into BigQuery and using BI Engine plus a BI tool would provide a much bigger benefit," he said.

He also cautioned users to consider their pockets before turning on the tool. "If you're not on BigQuery flat-rate pricing you have to reserve on-demand RAM for BI Engine, which could get pricey," he said.

Philip Howard, research director at analyst firm Bloor Research said the approach to caching was not "exactly original" but could be useful. "Anything that improves performance and reduces latency is potentially useful," he said. ®

Similar topics

Broader topics


Other stories you might like

  • Google has more reasons why it doesn't like antitrust law that affects Google
    It'll ruin Gmail, claims web ads giant

    Google has a fresh list of reasons why it opposes tech antitrust legislation making its way through Congress but, like others who've expressed discontent, the ad giant's complaints leave out mention of portions of the proposed law that address said gripes.

    The law bill in question is S.2992, the Senate version of the American Innovation and Choice Online Act (AICOA), which is closer than ever to getting votes in the House and Senate, which could see it advanced to President Biden's desk.

    AICOA prohibits tech companies above a certain size from favoring their own products and services over their competitors. It applies to businesses considered "critical trading partners," meaning the company controls access to a platform through which business users reach their customers. Google, Apple, Amazon, and Meta in one way or another seemingly fall under the scope of this US legislation. 

    Continue reading
  • Makers of ad blockers and browser privacy extensions fear the end is near
    Overhaul of Chrome add-ons set for January, Google says it's for all our own good

    Special report Seven months from now, assuming all goes as planned, Google Chrome will drop support for its legacy extension platform, known as Manifest v2 (Mv2). This is significant if you use a browser extension to, for instance, filter out certain kinds of content and safeguard your privacy.

    Google's Chrome Web Store is supposed to stop accepting Mv2 extension submissions sometime this month. As of January 2023, Chrome will stop running extensions created using Mv2, with limited exceptions for enterprise versions of Chrome operating under corporate policy. And by June 2023, even enterprise versions of Chrome will prevent Mv2 extensions from running.

    The anticipated result will be fewer extensions and less innovation, according to several extension developers.

    Continue reading
  • I was fired for blowing the whistle on cult's status in Google unit, says contractor
    The internet giant, a doomsday religious sect, and a lawsuit in Silicon Valley

    A former Google video producer has sued the internet giant alleging he was unfairly fired for blowing the whistle on a religious sect that had all but taken over his business unit. 

    The lawsuit demands a jury trial and financial restitution for "religious discrimination, wrongful termination, retaliation and related causes of action." It alleges Peter Lubbers, director of the Google Developer Studio (GDS) film group in which 34-year-old plaintiff Kevin Lloyd worked, is not only a member of The Fellowship of Friends, the exec was influential in growing the studio into a team that, in essence, funneled money back to the fellowship.

    In his complaint [PDF], filed in a California Superior Court in Silicon Valley, Lloyd lays down a case that he was fired for expressing concerns over the fellowship's influence at Google, specifically in the GDS. When these concerns were reported to a manager, Lloyd was told to drop the issue or risk losing his job, it is claimed. 

    Continue reading
  • End of the road for biz living off free G Suite legacy edition
    Firms accustomed to freebies miffed that web giant's largess doesn't last

    After offering free G Suite apps for more than a decade, Google next week plans to discontinue its legacy service – which hasn't been offered to new customers since 2012 – and force business users to transition to a paid subscription for the service's successor, Google Workspace.

    "For businesses, the G Suite legacy free edition will no longer be available after June 27, 2022," Google explains in its support document. "Your account will be automatically transitioned to a paid Google Workspace subscription where we continue to deliver new capabilities to help businesses transform the way they work."

    Small business owners who have relied on the G Suite legacy free edition aren't thrilled that they will have to pay for Workspace or migrate to a rival like Microsoft, which happens to be actively encouraging defectors. As noted by The New York Times on Monday, the approaching deadline has elicited complaints from small firms that bet on Google's cloud productivity apps in the 2006-2012 period and have enjoyed the lack of billing since then.

    Continue reading
  • UK competition watchdog seeks to make mobile browsers, cloud gaming and payments more competitive
    Investigation could help end WebKit monoculture on iOS devices

    The United Kingdom's Competition and Markets Authority (CMA) on Friday said it intends to launch an investigation of Apple's and Google's market power with respect to mobile browsers and cloud gaming, and to take enforcement action against Google for its app store payment practices.

    "When it comes to how people use mobile phones, Apple and Google hold all the cards," said Andrea Coscelli, Chief Executive of the CMA, in a statement. "As good as many of their services and products are, their strong grip on mobile ecosystems allows them to shut out competitors, holding back the British tech sector and limiting choice."

    The decision to open a formal investigation follows the CMA's year-long study of the mobile ecosystem. The competition watchdog's findings have been published in a report that concludes Apple and Google have a duopoly that limits competition.

    Continue reading
  • Google recasts Anthos with hitch to AWS Outposts
    If at first you don't succeed, change names and try again

    Google Cloud's Anthos on-prem platform is getting a new home under the search giant’s recently announced Google Distributed Cloud (GDC) portfolio, where it will live on as a software-based competitor to AWS Outposts and Microsoft Azure Stack.

    Introduced last fall, GDC enables customers to deploy managed servers and software in private datacenters and at communication service provider or on the edge.

    Its latest update sees Google reposition Anthos on-prem, introduced back in 2020, as the bring-your-own-server edition of GDC. Using the service, customers can extend Google Cloud-style management and services to applications running on-prem.

    Continue reading

Biting the hand that feeds IT © 1998–2022