Linux has literally lost its Lustre – the filesystem favoured by HPC types has vanished in the first release candidate of version 4.18 of the Linux kernel.
Linus Torvalds’ announcement of the new release lauds the fact it’s shrunk markedly, much of which can be attributed to the removal of Lustre.
“The removal of Lustre may not be all that notable, because it does look like a lot of the development has been happening out of tree, which may be why it never really ended up working as well as people hoped in the staging tree,” Torvalds wrote. “ Greg [ Kroah-Hartman] clearly got pretty frustrated about it, so now it's gone.”
How frustrated? Kroah-Hartman explained Lustre's omission by saying it has "been in the kernel tree for over 5 years now" but "has not really moved forward into the 'this is in shape to get out of staging' despite many half-completed attempts."
"And getting code out of staging is the main goal of that portion of the kernel tree," Kroah-Hartman wrote. "Code should not stagnate and it feels like having this code in staging is only causing the development cycle of the filesystem to take longer than it should."
"There is a whole separate out-of-tree copy of this codebase where the developers work on it, and then random changes are thrown over the wall at staging at some later point in time. This dual-tree development model has never worked, and the state of this codebase is proof of that."
"So, let's just delete the whole mess," he decided, but left the door open for Lustre to be re-admitted once Lustre's developers "take the time they have spend doing those types of housekeeping chores and get the codebase into a much better shape, and it can be submitted for inclusion into the real part of the kernel tree when ready."
Lustre developers appear to be unfussed by the news: our scan of the project’s mailing lists didn’t find any comment.
Linux is not alone in losing interest in Lustre: Intel hired more than a handful of its prominent developers and added support for Hadoop but in April 2017 decided to stop offering its own version of filesystem and instead offered its code to the open-source community.
Hasta la vista Lustre, so long Spectrum Scale: Everyday HPC is hereREAD MORE
When Lustre emerged in the year 2003 it had little competition for creation of large-scale filesystems. Nearly 15 years on and Red Hat offers Gluster, IBM’s Spectrum Scale (aka the GPFS General Parallel File System) and scale-out NFS can all do plenty of what made Lustre useful. HDFS has emerged, too, for big data workloads.
Linux losing Lustre won’t therefore deprive penguinistas of access to scale-out storage clusters. Indeed, even HPC rigs won’t necessarily miss the filesystem, given such operators tend not to need the very latest kernel. And even if they do want the newest kernel, there’s no indication they won’t be able to mount Lustre.
Torvalds’ post on the new release lauds 4.18’s new features, among them support for AMD GPUs, fixes for Spectre V4 – aka Speculative Store Bypass - on Arm CPUs, and drivers to make two-in-one Chromebooks better at switching between desktop and tablet mode.
“We had 1500+ developers attributed as authors in this merge window,” Torvalds concluded, adding that the changes in this release involved “91 maintainers, so just think about that for a while.” ®