Why cloud databases are taking over the world
It’s a question of scale
Sponsored The cloud has a habit of breathing new life into venerable and slightly dull mature markets, making them exciting again. We've seen it with everything from enterprise application software through to hosted storage, but perhaps the most impressive turnaround has been in databases. Two decades ago, relational databases running in your own data center or a colocation center were standard, and that hadn't changed in years. Then, the cloud brought sexy back to databases.
Gartner expects that 75 percent of all databases will be deployed or migrated to cloud platforms by next year, dwarfing the five percent that it says will be repatriated to on-premises systems. Tony Baer, founder and principal of technology advisory firm dbInsight, says that the cloud quickly grew into a major disruptor because of its ability to scale.
"When you put a database in the cloud and take advantage of cloud native architecture, you can really rethink what you can do with the database," he says. The cloud enabled companies to scale out rather than up, bringing linear scalability within their reach for the first time. That opened up new possibilities at a time when companies were becoming increasingly data hungry.
Gartner's 2020 Cloud Database Magic Quadrant estimated that $17bn (31 percent) of the $55.4bn in worldwide DBMS market revenues came from the cloud. The cloud database market shows no sign of slowing down, with 70 percent of DBMS market growth coming from cloud databases in 2019. It will still be contributing half of the overall market's growth in 2023, Gartner said.
Challengers and visionaries
Gartner's quadrants break down market players into Niche players (those with limited market appeal), Visionaries, Challengers, and Leaders. There are only two companies that Gartner sees as challengers, and they're unlikely bedfellows.
Redis Labs, with its open-source database, has three strong points. First, it covers multiple cloud providers, giving customers options for their public cloud infrastructure. Second, it offers multi-model support. That means you can cover traditional relational models with it, but also others ranging from key-value stores through to things like text and geolocation models, all against a single back-end engine. Finally, it has carved out a strong name for itself in the in-memory database market, making it a go-to for low-latency use cases like real-time fraud detection.
The other company in the challenger space is Snowflake. For a company less than a decade old, it has seen an impressive rise, culminating in a September IPO that saw its market cap burst past IBM's for a while. It's one of a new generation of young database companies that Baer likens to a "datamart in the sky". The company, which started with a focus on cloud data warehouses, is broadening its sights to becoming a data sharing marketplace that lets companies publish their data for each other in read-only format.
Had it not grown so large so quickly, Snowflake might well have landed in the visionary category, reserved for small, nimble companies with a high degree of innovation. Here, you'll find Snowflake rival Databricks, which focuses on coupling large, sprawling unstructured data lakes with smaller, more refined collections of data that it calls 'data lakehouses'. It, too, plans an IPO this year.
While there are two and four players in these two quadrants respectively, it's notable that its Leaders quadrant has eight. Leaders are well-established players with broad use case support, mature products, and support for hybrid and cloud-native deployment models. This is a sign that in spite of the frothy nature of the cloud database market, there are large, well-established companies with deep technology stacks and broad coverage that still have a competitive advantage.
Those in the Leader category are typically low-risk options, but that doesn't mean they aren't innovators. These companies have a solid sense of what's coming next.
Some of these companies come from traditional pre-cloud enterprise backgrounds, embracing the cloud either with solutions designed to run on other peoples' infrastructure, or with their own cloud services. SAP is in the Leader quadrant with its range of operational and analytics offerings, while Oracle has also made the leap to cloud with its Autonomous Database product, hosted in its own infrastructure or on customers' premises.
The mega-vendor's attempt to capture the cloud database market reminds Baer of the uphill run it did over a decade ago as it persuaded people to buy into its Exadata hardware/software combo.
"The core clients did not take to Exadata immediately," he says. "It took a few years for it to prove itself and to say that it actually is feasible to do this database consolidation."
Not all cloud service providers fit into the Leaders category (we're looking at you, Tencent and Alibaba) but the other heavy hitters do.
IBM is in Gartner's Leaders quadrant, but it's close to the bottom of that region in its ability to execute. The company's historical fear that cloud databases would cannibalise its mainframe business held it back, says Baer.
With that said, IBM has performed some impressive analytics work in hybrid cloud environments, demonstrating some leadership among cloud service providers in spanning multi-cloud environments with its Cloud Pak for Data product. This binds tools together to manage the data lifecycle, including data virtualization tools that read from multiple sources, and also features support both for the company's own Db2 Warehouse and MongoDB. It's available on multiple other CSP's platforms.
The top three cloud database providers are, predictably, the top three service providers. Microsoft embraced the cloud database concept early on and has gradually fleshed out its offerings with a multi-pronged approach. Alongside a cloud-native version of its SQL Server database, it also offers Cosmos DB, which supports APIs for the Apache Cassandra database originally created by Facebook, and MongoDB. It also launched a cloud-native managed Cassandra instance in March. It has plentiful analytics capabilities too.
Google has been busy muscling in on the cloud database market over the last few years with products like its Spanner managed relational database, and its NoSQL offerings Datastore, Firestone, and BigTable, which between them offer key-value and document storage. The company also has BigQuery, a data warehouse product that it offers across other cloud platforms in the form of BigQuery Omni. That's a sign of movement in the cloud database space as CSPs begin to offer their home-grown databases on other cloud platforms.
The rise of AWS
Then, there's the granddaddy of cloud service database platforms: Amazon Web Services (AWS). This platform ranked first among the leaders in Gartner's Magic Quadrant, which points to the company's market dominance as a key advantage.
As a Cloud Service Provider that launched several years before its main competitors, AWS got a head start on developing database systems. It rolled out its first database service, Amazon Relational Database Service (RDS), which manages a collection of cloud-based relational systems, in 2009 before launching its scale-out DynamoDB key-value database in 2012. That same year, it also rolled out its Amazon Redshift data warehouse.
Over time, the company fleshed out its database offerings with a variety of home-baked products including the ElastiCache in-memory database (2011), its Aurora RDBMS (2014), and the Neptune graph database (2018). AWS rolled out its Quantum Ledger Database (QLDB), which helps customers build ledger-like applications, in 2018, followed by the DocumentDB NoSQL database, which also supports MongoDB workloads, in 2019.
AWS has kept up the pace of development for its managed database portfolio, launching Timestream, its time series database for workloads including IoT, and its wide column database with Apache Cassandra compatibility, called Keyspaces, last year.
What's next for the cloud database market? Amazon leads the pack in terms of the sheer number of different managed database types on offer. Baer expects to see a shift towards layering more functionality on top of the raw database engines. This shift in momentum will take cloud service providers in new directions, he predicts.
"One is towards filling out these data platforms. The other is knitting together more coherent cloud services and building an integration with other cloud services like AutoML [automated machine learning], or visualization." Automation of data pipelines will also be a focal point, he says, and we're already seeing this with enhancements to projects like AWS Glue, a data integration service which now combines and replicates data from multiple data stores, using a capability called Elastic Views.
That's a reasonable prediction. Two things are certain in the rapidly-evolving cloud database market: companies are going to want more scale as their data acquisition and generation grows, but they're also going to need better ways of governing and managing it. This innovative space is set to be hot for some time to come
Sponsored by Amazon Web Services
Click here for more information on AWS purpose-built Databases.