This article is more than 1 year old
Intel DAOS 2.2 and Red Hat Stratis 3.3 released
New versions of two next-gen Linux storage engines may belie fading interest
Intel's DAOS is a distributed parallel filesystem for supercomputers, while Stratis is Red Hat's answer to ZFS – new versions of each were released today.
DAOS, or Distributed Application Object Storage, is Intel's system for fast supercomputer storage, while Stratis is Red Hat's next-generation filesystem, which combines logical volume management and COW snapshots. Both have uncertain futures.
DAOS (not to be confused with Taos)
DAOS version 2.2 is interoperable with the existing DAOS 2.0 and supports two new network providers: libfabric/tcp
replaces the older libfabric/sockets
as the preferred libfabric transport, and for clusters that use InfiniBand instead, it now supports UCX fabric for Unified Communication X communications. The new release also adds support for Rocky Linux 8 and Alma Linux 8, while removing support for the discontinued CentOS 8.
Which won't mean much if you're not interested in high-performance computing and clustered supercomputers. For some background, The Reg's sister site devoted to storage, Blocks & Files, offered an overview in 2019. As recently as February, another of our siblings, the supercomputer-focused Next Platform was impressed by DAOS's performance.
But the problem is that DAOS was designed for use with persistent memory – specifically, Intel's canceled Optane storage. In 2018, Intel sold off its other cluster filesystem tech, Lustre, to DDN, which you might know as the home of openSolaris derivative Nexenta. Very shortly afterwards, Lustre was removed from the Linux kernel.
As we have covered, other vendors have their own upcoming persistent memory tech, but it's not clear if Intel will continue to sponsor development of DAOS given that it was originally intended as a way to sell more Optane storage.
Stratis (not to be confused with Btrfs or ZFS)
Stratis 3.3 is Red Hat's planned next-generation file system. It combines parts of two existing technologies which are already used in other Red Hat products: the LVM2 logical volume manager, plus the XFS filesystem from IRIX, which SGI donated to Linux in 1999.
The technological and legal situation that Stratis was designed to address definitely still exists. What is not so clear is whether the competitive situation in the market still does.
Back in 2019, Ubuntu was still increasing its support for ZFS, despite legal controversies around using the Sun/Oracle CDDL-licensed filesystem with the GPL2-licensed Linux kernel.
SUSE is heavily invested in the Btrfs filesystem, while it was removed from RHEL some years ago – although Fedora subsequently adopted it.
Canonical and SUSE use these filesystems because both ZFS and Btrfs offer a function which are immensely useful in building more robust, fault-tolerant Linux systems: filesystem snapshots. The snapshot concept is simple, but implementing it is complex.
The idea of snapshots is that they are a very rapid way to capture a complete image of a Unix machine's whole filesystem at one moment in time. The old way to do this was to take a backup: to copy all the files somewhere else, which is very robust but takes a long time. In the 1980s and 1990s, that was acceptable because many computers were not connected to the internet, and software updates were infrequent events, often costing large amounts of money, and therefore could be planned for well in advance.
- Hints about SUSE's 'Adaptable Linux Platform' emerge
- 'Unbreakable' Oracle Linux 9 is a RHEL rebuild with built-in Btrfs support
- Workstation, server, IoT? No worries. Fedora 36 is out – all 13 editions of it
- Devs of bcachefs try to get filesystem into Linux again
However, contemporary DevOps culture reflects the main use of Linux, running inside VMs in server farms: deployed rapidly in large numbers, and deleted when they're no longer needed. Software updates are a very frequent event, even multiple times per day. There isn't time to back up all these VMs, update them, and then test if the update worked.
Snapshots use copy-on-write instead. Before applying a software update, the OS instructs the filesystem to redirect writes to a new area of storage, leaving the old versions of the files in a separate area, which thus functions as an immediate stored instance of the state of the filesystem before the update was applied. Instead of copying all the old files somewhere else, which is slow, you just start to put all the new ones somewhere else, which is immediate. If there's a problem, the machine reboots back to the old snapshot, and the new one is discarded, allowing the state of the whole OS to be rolled back like a database transaction.
Traditional xNix filesystems, such as Linux's ext4 and XFS, can't do this. The only way to perform snapshotting with such a filesystem is by inserting an extra layer of storage management underneath the filesystem, between it and the storage medium, so that the filesystem isn't directly managing physical disks, but virtual storage "volumes". This is what the Linux kernel's Logical Volume Manager does, and so it enables snapshots – but it's quite complicated. LVM is also used for other purposes, too, such as providing RAID functionality and performing full-disk encryption using LUKS.
Btrfs provides some of this functionality, including snapshot support and RAID, within the filesystem – and for added fun and games, it can run on top of LVM or the kernel's built-in RAID support. This means considerable duplication of functionality. It's complicated, and it also doesn't have a good reputation for stability, as we mentioned looking at Fedora's use of it.
An alternative is the still-in-development bcachefs, which delivers snapshots without the complexity of Btrfs and delegates all the fancy RAID stuff to LVM or mdraid
.
ZFS is much simper to configure and administer than all of these because it combines the functions of logical volume management, RAID, encryption, snapshots, and filesystem all in one. What's more, in its OpenZFS form it's open source… but under a license incompatible with that of the Linux kernel.
The Stratis project aimed to replicate the functionality of ZFS, or of Btrfs-plus-LVM, in a single integrated whole built from all-GPL code.
This is a laudable goal, but like DAOS, the conditions that made the company sponsor it have changed. Ubuntu has been refocusing its efforts for some time: the GitHub repository for its zsys
tool, which integrated ZFS into Ubuntu, shows a sad lack of activity, and the experimental option to install on ZFS has disappeared from the latest LTS release.
The Linux industry today is focusing less on fancy filesystems, and more on immutable OSes, with read-only root filesystems and containerized apps. Red Hat's Project Atomic is "sunset", and its CoreOS ContainerLinux is end-of-life. The Fedora Project's flagship effort here is the Kinoite immutable desktop and Fedora CoreOS for servers.
Meanwhile, parent company IBM has swallowed some of Red Hat's storage projects. Commercial companies have to follow the money, and the trend there is towards scaling workloads via containers and VMs, rather than increasing the capabilities of the underlying OS. Where there is innovation in fault tolerance in desktop Linux, it's in terms of ChromeOS with its multiple partitions, as also used in Linux Deepin. ®