Boosting throughput for cloud databases

Aiven's bold move to scale up services for Redis users

Sponsored Feature As applications continue to scale up in the cloud, businesses face a daunting problem: how can they deliver the data to power those online services without missing a beat?

Increasingly, organisations are turning to cloud databases as a solution. RDBMS and NoSQL systems operating in the cloud are unshackled from local resource constraints.

Cloud databases can scale, but for real speed, companies are turning to in-memory solutions. These databases, like Redis®, sit in front of long-term data storage to serve queries from large user bases at lightning speed.

Aiven, a company that offers data infrastructure as a cloud-based service, has long provided a cloud-based version of the open source version of the Redis database. However, as its customers' speed and scalability constraints continued to grow, it searched for an alternative with higher throughput capabilities.

In March, Redis made the need for a Redis alternative more urgent by  changing its licensing in a move that saw it exit open source. From version 7.4, Redis Open Source ceases to exist, instead becoming Redis Community Edition. The company abandoned the BSD license that it previously used for the software, replacing it with a dual-license system that it already used for Redis Stack and other commercial products.

These licenses - the Redis Source Available License version 2 (RSALv2) and the Server Side Public License version 1 (SSPLv1) - prevent companies from using the free version of Redis, starting with version 7.4, to build a competitive offering, including Redis-based managed services in the cloud. One caveat is that a vendor could enter into a Partner agreement with Redis for version 7.4 and beyond.

March saw Aiven launch its alternative solution, Aiven for Dragonfly, into general availability.

Every byte, everywhere, all the time

The business world's hunger for more data is insatiable. Analytics and AI applications crave as much information as possible to be more efficient. This demand for data is stimulating enterprise interest for scalable database technology to new levels, focusing the lens on the cloud.

Gartner's global Forecast Analysis for Database Management Systems says that the DBMS market as a whole is set for a 16.8 percent CAGR from 2023-2027 to reach $203.6bn, but it also notes that the lion's share of that growth is coming from the cloud. With a 55.2 percent share of the DBMS market as of 2022, cloud-based database platform as a service (dbPaas) is already surpassing on-premises databases' 44.8 percent market share.

"The demand for more data is just one side of the story," says Larry Heathcote, head of core services product marketing at Aiven. "The other challenge driving enterprise demand for scalable database storage is an increasing volume of users."

"One big trend we've seen is the need to support highly successful applications with millions of users rather than tens of thousands," says Heathcote. "This has created a need for databases that can store more data and operate at faster speeds to meet performance requirements."

Those applications must also offer increasingly low response times as user demands for performance increase. This is especially critical in specialist applications like gaming, where split-second delays can cause extreme frustration and a loss of gamers.

Enter the Dragonfly

Aiven already supports a range of databases for its customers. On the relational side, companies can deploy PostgreSQL, MySQL, and ClickHouse on the service, and on the NoSQL side it offers support for Cassandra and Redis. However, as Aiven noted this increasing demand for scalable performance, it began hunting for another option for Redis customers.

"Redis is one of the most popular NoSQL databases used by developers, but it has its limitations," explains Heathcote. One of these is scalability.

"When queries grow beyond certain volumes, the system begins to hit limits. This is something that Redis solved with its paid Redis Enterprise product, which introduced sharding technology to divide up databases across different machines in a cluster.

Redis Enterprise unlocks more performance for users, but it's costly. Running multiple shards means adding extra machines, all of which need large amounts of memory. It also takes considerable configuration and infrastructure management. Both of these demands increase the cost that a cloud database service provider must pass on to its customers.

Redis's move away from open source also makes it more likely that developers will look for alternatives. The licensing change is a move to protect Redis's intellectual property and to help it sustain a profitable business, following similar decisions by companies like MongoDB and ElasticSearch. However, as Redis admits, the licenses for Redis version 7.4 and beyond are no longer technically open source as defined by the Open Source Initiative. This will make many developers nervous, just as it has done in other cases.

These technical limitations and business changes made it increasingly important for Aiven to look at alternative solutions in the marketi - a search that culminated in it adopting support for Dragonfly.

Dragonfly is a modern in-memory data store that serves as a replacement for Redis, offering performance improvements without the associated costs. It offers hundreds of thousands of operations per second and also handles workloads in the terabyte range. It promises to do all this on a single instance, eliminating the need for multiple machines.

The Dragonfly project, which made it to version 1.0 a year ago, claims to deliver higher performance than Redis when running on a single instance via its optimisation for modern hardware, and already claims thousands of users. Many of those users have undoubtedly been attracted to one of its most appealing features: complete API compatibility with Redis. This enables existing Redis users who are feeling the strain to migrate over to Dragonfly simply and seamlessly.

Higher throughput and lower memory usage

With these benefits in mind, Aiven launched its Aiven for Dragonfly service in early March. The company benchmarked its performance on AWS, using instances with 32 vCPUs and 512Gb of memory. It found a nearly 700 percent higher throughput in requests per second compared to Aiven for Redis, along with a 25 percent lower memory usage.

Through its support for Dragonfly, Aiven is also able to support some features that customers would otherwise have to migrate to Redis Enterprise to access. These include support for JSON documents, which have become the lingua franca of the NoSQL database community. Developers can store, update, and load JSON values directly within Dragonfly just as they can for other Redis data types. It gives them more intuitive access to nested data structures, allowing them to conduct operations without any heavy data parsing.

Another feature that Aiven for Dragonfly brings to Redis developers is vector-based search. This feature, which is again only available in Redis Enterprise, enables the storage, indexing, and querying of vector data which is important for AI applications.

Vectors are useful when handling unstructured data because they encode features of that data into a numerical format. This makes it easier to store and index a representation of that data efficiently, which in turn makes it easier to compare different vectors - and therefore different pieces of data - for similarities.

Developers would need to upgrade to Redis Enterprise to gain vector-based capabilities like semantic search, recommendation-based queries, image and video search, natural language processing, and anomaly detection. Vector-based searches bring those capabilities within a developer's grasp, propelling them beyond the limitations of basic keyword searches without needing expertise in machine learning.

Leveling up workloads

The use cases for Redis and Dragonfly are similar: high-throughput workloads that need low-latency responses. Like Redis, Dragonfly can act as a cache for a main database, providing access to recently-stored information. Aiven provides Redis developers with an easy transition path when scaling things up beyond a certain point.

"The most important limitation with Redis comes when we are increasing the throughput requirements, that is, how many queries per second can you do against a Redis database," says Heathcote. "What we are recommending to our customers is that you can use Aiven for Redis up to 70,000 operations per second. If your requirements go beyond that, and we're talking about hundreds of thousands of operations per second, then Aiven for Dragonfly is the much better alternative."

Because Redis and Dragonfly are API-compatible, no code changes are necessary to migrate between the two. Customers migrating from Redis can simply specify Dragonfly as their target in the Aiven management console, and their data will automatically migrate. They can then test their application against Aiven for Dragonfly and then switch it into production when ready, pointing their production application at the Dragonfly instance.

While Aiven for Dragonfly went into general availability in March, companies have been beta testing the system with what looks like impressive results. One such user, Conrad Electronics, provides an advanced sourcing platform for technical supplies to 21 million global customers.

The German company, which was already using Aiven for Redis, needed a mixture of more speed and scalability to support its customer experience. This prompted it to test Aiven for Dragonfly and saw significant performance improvements. Senior executives reported a seamless migration once the testing phase was complete.

Another customer in the advertising industry that had been using a third-party Redis implementation is in the process of switching to Dragonfly after its own successful beta test. Aiven for Dragonfly reportedly handled an average of 500,000 operations per second for the customer without a glitch.

Managing everything in the cloud

Customers like these have something else in common aside from the need for speed and scalability: they don't want to handle their caching database requirements on their own premises. That would take a significant capital investment in servers packed with expensive memory at a time when DRAM prices are on the rise.

Instead, they turned to Aiven because of its ability to manage performant data infrastructure in the cloud. The company provides the database services and software to handle these scalable in-memory caching requirements, but beyond that it also handles the administrative tasks necessary to set up and run the database. These include backup management, secure configuration, and automatic software upgrades, while the service offers a 99.99 percent uptime guarantee.

Aiven also integrates with third-party services so that customers can extend their data management functionality. These relationships allow developers and administrators to activate a range of integrations from the Aiven console, including Splunk, Snowflake, and Datadog. Bring Your Own Cloud (BYOC) makes it possible to deploy Aiven's managed data services directly to customers' own AWS, Google Cloud, or Microsoft Azure accounts. This provides more control over their data and opportunities to optimize cloud infrastructure costs.

Aiven for Dragonfly is now available, giving customers another option to plan and upgrade their infrastructure as the pressures of success begin to bite. With even higher throughput now available on demand, companies can continue to grow without the headaches of nursing their infrastructure through sharding and multi-instance management.

Sponsored by Aiven.