BigQuery analytics goes multi-cloud: Use our service without moving your data, says Google – but there is no magic

Query data in one cloud and export the results to another

1 Reg comments Got Tips?

Google's virtual Cloud Next event has kicked off with the introduction of BigQuery Omni, letting customers extend the tentacles of the analytics service to probe data across Google Cloud, AWS and Microsoft Azure.

The service is currently in private beta for AWS and "very soon BigQuery Omni is going to be available for Azure," said Google's Data Analytics GM Debanjan Saha.

BigQuery Omni lets users use Google’s business intelligence and big data analytics service across data in multiple clouds.

BigQuery Omni lets users use Google’s business intelligence and big data analytics service across data in multiple clouds

The thinking behind BigQuery Omni is that most enterprises, whether by accident or design, find themselves using more than one public cloud. This is true of more than 80 per cent of organisations, said Saha. "When you are in multiple clouds the data is siloed, and if you want to run analytics on that siloed data, you have to move data from one cloud to another which is cumbersome and expensive," he added.

Note that data is moved temporarily within AWS from your data storage to the BigQuery clusters running on Anthos to execute queries

BigQuery Omni lets you use Google's analytics service on the cloud where the data sits. Features of BigQuery include building and running ML (machine learning) models, using an in-memory analysis service, and linking to location data for geospatial analysis.

The way this works is that Google uses Anthos – an application platform based on Kubernetes – to run its BigQuery service on other clouds. Customers will not have a dedicated BigQuery service on AWS or Azure, but will use Google's managed BigQuery as a shared resource. This enables use of SQL queries and other BigQuery features on these third-party clouds.

In the case of AWS, Saha said: "Behind the scenes, BigQuery's query engine is running on our Anthos clusters within the BigQuery managed service. BigQuery gets the data from data storage within your account once you've authorized permissions via your other public clouds' IAM roles.

"Note that data is moved temporarily within AWS from your data storage to the BigQuery clusters running on Anthos to execute queries."

Google does not envisage that users will attempt to query data stored in multiple clouds in one operation. The latency issues and data transfer required for this would make it inefficient. The idea is that users can query data in one cloud and export the results to another, and then perform further analytics against the combined data.

With BigQuery Omni, the control plane runs on Google Cloud Platform but queries are carried out on the cloud where the data resides. If you need to query across clouds you should export the results of one query to the cloud where you want to query the joined data.

The control plane runs on Google Cloud Platform, but queries are carried out on the cloud where the data resides. If you need to query across clouds you should export the results of one query to the cloud where you want to query the joined data

An example would be where a customer wants to combine advertising data from Google Analytics with logs from an ecommerce platform running on AWS. Users can run a query on AWS, export the results to Google Cloud Platform, and then "build a dashboard that allows you to visualise your audience behaviour and purchases alongside your advertising spend," said Saha.

There is no magic and if the result sets were large, such an operation would still require some export of data from one cloud to another. That said, BigQuery Omni does mean that users have a single analytics system to learn, all controlled from a single user interface or API on Google Cloud.

Procter and Gamble (P&G) is one BigQuery customer mentioned by Google, with the claim that P&G will "transform its business by shifting from mass market to ...mass personalization."

It added: "By continuing to expand the implementation of Google Cloud's BigQuery data analytics technology, P&G can combine siloed customer data to enrich the shopping journey for existing customers, while also attracting new consumers to its product."

Those seeking to push out an "enriched shopping journey" will surely be delighted. ®

SUBSCRIBE TO OUR WEEKLY TECH NEWSLETTER


Biting the hand that feeds IT © 1998–2020