The successor to Research Unix was Plan 9 from Bell Labs

A better UNIX than UNIX isn't a UNIX at all

FOSDEM 2024 To move forwards, you have to let go of the past. In the 1990s that meant incompatibility, but it no longer has to.

This article is the third based on The Reg FOSS desk's talk at FOSDEM 2024.

The first part of this short series talked about the problem of software bloat. The second talked about the history of UNIX, and how its development continued at Bell Labs after commercial vendors picked up earlier versions and took them to market.

One of the results of this pattern of development was that Eighth Edition Unix didn't have much industry impact, and little if anything drew significantly upon the Ninth and Tenth Editions. Despite this, work continued, but what followed the Tenth Edition wasn't called "Unix" at all. Instead it was dubbed "Plan 9 from Bell Labs." This time, we want to look at why Plan 9 wasn't called Unix in the first place, and what can be done about that using features of 21st century hardware and software.

The Reg FOSS desk has looked at Plan 9 a few times, including a more recent look at the most active fork, 9Front. Last year, we discussed the significance of 9Front, especially comparing its size with that of modern Linux.

Plan 9 took the concepts that led to Unix, and rethought them for the 1990s. This, like any big change, had consequences, and not all of them were positive.

The good bits

Plan 9 was in some way a second implementation of the core concepts of Unix and C, but reconsidered for a world of networked graphical workstations. It took many of the trendy ideas of late-1980s computing, both of academic theories and of the computer industry of the time, and it reinterpreted them through the jaded eyes of two great gurus, Kenneth Thompson and Dennis Ritchie (and their students) – arguably, design geniuses who saw their previous good ideas misunderstood and misinterpreted.

In Plan 9, networking is front and center. There are good reasons why this wasn't the case with Unix – it was being designed and built at the same time as local area networking was being invented. UNIX Fourth Edition, the first version written in C, was released in 1973 – the same year as the first version of Ethernet.

Plan 9 puts networking right into the heart of the design. While Unix was later used as the most common OS for standalone workstations, Plan 9 was designed for clusters of computers, some being graphical desktops and some shared servers.

Plan 9 takes the idea of "everything is a file" and it makes it real. In Unix, it's really just an empty marketing slogan. Lots of things, such as trees of process IDs, aren't part of the file system. In some versions, they can be seen via bolted-on extras, such as the /proc in-memory pseudo file system – but you won't find a /proc in macOS, for instance.

Under Rio, the current Plan 9 window system (which replaced the older 8½ system), on-screen windows are represented as directories in the file system, and their contents are files. Other Plan 9 machines on the network are visible in your file system – subject to access permissions, of course – and you can see some of their file system right inside yours.

Because everything really is a file, displaying a window on another machine can be as simple as making a directory and populating it with some files. You can start programs on other computers, but display the results on yours – all without any need for X11 or any visible networking at all.

This means all the Unixy stuff about telnet and rsh and ssh and X forwarding and so on just… goes away. It makes X11 look very overcomplicated, and it makes Wayland look like it was invented by Microsoft.

As everything goes through the file system, it eliminates much of one of the biggest complex problems of microkernel designs – inter-process communications. Plan 9 is not a microkernel as such; it's more that in some ways, it makes the defining features of microkernels somewhat irrelevant.

Plan 9 is not just a replacement kernel, or the layers above that. It's also a clustering system, and a network filesystem, and a container management system. In terms of functional blocks, we are talking about replacing Linux, and Ceph or Gluster or whatever, and all hypervisors, and all container systems, and Kubernetes and all that.

As it was conceived as one tool for all this, it is much, much simpler than the gigabytes of complexity layered on top of Linux that makes Linux able to do this stuff.

Even so, Plan 9 is tiny. I talked to a couple of people from the 9front community when I wrote about it in November, and one chap gave me some numbers.

The kernel is 5,119,091 bytes – 5MB or so.

The entire distribution including all sources, documents, local Git repository and binaries is circa 530MB for amd64.

(That's including all the apps, demos and games, including Doom.)

As for complexity: the Plan 9 kernel has 38 system calls.

I've not been able to find a firm number for Linux, but I have some estimates. As of 2016, kernel 4.7 had about 335. By 2017 that was up to 341. I've tried to estimate the number for kernel 6.8 and I think there are 520 across all architectures.

For comparison, Kubernetes alone is about two million lines of code. This is the price of bolting on a cluster management layer, instead of embedding it into your design.

A brief aside about file systems

In case this all sounds a bit too theoretical, one of the problems if everything is in the file system is, of course, deciding where everything is located.

Back in 2011, I wrote about containers and called them a sysadmin's dream come true. That was about the same time that Docker launched.

Containers are a sort of chroot on steroids. All the file paths in a given process start at a new root directory, and since in Unix almost everything is relative to the root directory, changing that isolates that process (and any children) from the rest of the OS.

Nowadays there are several forms of containerization on Linux, but some of them use the Linux kernel's cgroups feature, which came out of Google. This separates the processes inside containers from one another by putting them inside different namespaces.

The problem is that Unix, and therefore Linux, doesn't just have one global namespace because when it was designed these ideas were a bit new and undeveloped. Unix concepts that aren't part of a single namespace include user IDs, and group IDs, and process IDs, and all sorts of things that aren't represented in or expressed through that single global file system.

This is why cgroups namespaces exist – to split those namespaces up as well for more complete isolation. The problem is that it's a later, bolted-on feature so just as anything that works with /proc won't work on macOS – or on OpenBSD – then anything that uses cgroups namespaces won't work on FreeBSD.

Back to the Plan

Namespaces are a core part of the Plan 9 design, and every individual process under Plan 9 has its own namespace. Because far more of the communications between parts of the OS is done via the file system. Thus, every process has its own view of the file system.

So, in effect, every process is in a container… Note, in a project announced in 1990, a full decade before the first release of FreeBSD jails in FreeBSD 4.0.

The bad part: it flopped

If Plan 9 is so gosh-durned clever, why aren't we all using it? Well, some reasons are easy to enumerate.

Firstly, just as Unix itself at the beginning, it was an experimental OS. In their early years, they weren't intended for production use, and Plan 9 was a tool for research into operating system concepts. It was only licensed for commercial use with the Second Edition in 1995

Secondly, back in the 20th century, Plan 9 wasn't open source. But in 2000, the Third Edition was released as FOSS under its own custom licence agreement, and then in 2014 it was re-licensed under the GPL.

Thirdly, Plan 9's very clean, minimalist conceptual design brings penalties. Since there is no "real" underlying view of the "true" file system, that means that there are quite a lot of things Plan 9 simply does not do. There are no links, either symbolic or hard.

Originally, its mv command simply copied files then deleted the originals. This is the price of having the kernel talk to the local file system the same way that it talks to the file systems on other machines – via the 9P protocol.

(Yes, this can be slow, and sounds extremely limiting, but this vulture was some five years into his career when the first Microsoft OS with hard links, Windows NT 3.1, was launched. Until MS-DOS 5 in 1991, DOS didn't have a MOVE command, and no Microsoft OS had anything like symbolic links prior Windows 95's "shortcuts.")

Plan 9 is unfriendly, unforgiving, and hard to use. I am not some frothing Plan 9 advocate. I do not routinely use it myself, although I keep a copy of 9front around in a VM. Over the years, I occasionally get a fresh version and play around with it.

9front has come quite a long way from the AT&T version. A non-guru such as myself can install it and try it, which is more than I could with the original version. But, as the FQA page says:

Plan 9 is not for you

Let's be perfectly honest. Many features that today's "computer experts" consider to be essential to computing (JavaScript, CSS, HTML5, etc.) either did not exist when Plan 9 was abandoned by its creators, or were purposely left out of the operating system. You might find this to be an unacceptable obstacle to adopting Plan 9 into your daily workflow. If you cannot imagine a use for a computer that does not involve a web browser, Plan 9 may not be for you.

Plan 9 has a GUI, but it's very strange. But then it's a very strange OS.

The thing is, though, that when I started playing with Linux – roughly around Slackware 3.0 in 1995, with kernel 1.0 – Linux was extremely basic, very unfriendly, and not very much actual use for anything. Linux has come an extraordinarily long way since then, even if it still has little desktop penetration.

The killer is that Plan 9 is different enough from Unix – any Unix – that it's incompatible with Unix. Its creators did not call it Unix Eleventh Edition for good reasons. It changes fundamental parts of the design in ways that mean that bringing existing source code across and recompiling it won't work.

To pick a trivial example, Plan 9's version of C prohibits nested #include directives. Most non-trivial C programs contain lots of files, and those files all contain lots of #include statements. In Plan 9 C, only the top-level C file in a program can #include other files, and header files are not allowed #include at all. This means a bit more work for the programmer, but it makes compilation much faster.

It was good, but it wasn't good enough

In order to displace an existing, well-established and deeply entrenched product – in this case, conventional Unix – a new successor product has to be significantly better than its forerunner to make it worth the cost of switching, even if that cost is just in time and effort, not money. We don't know of a snappy term for this rule, but there may be one, perhaps akin to Clayton Christensen's revolutionary innovation. To paraphrase an old OS/2 marketing slogan, Plan 9 was "a better Unix than Unix" – but it wasn't better enough.

As another small side-note, Plan 9 wasn't the end of the original Unix line. There were ten editions of Research Unix, then a successor that was so different it wasn't Unix any more. That's Plan 9, and in a way, it is "UNIX 2.0."

There is also a very different successor to Plan 9 itself. It is called Inferno, and it is, in a manner of speaking, "UNIX 3.0." I'm not going to go into any detail about it here today, as this article is already long enough, but it's interesting and well worth investigation.

9front is clever and impressive. It's also weird and inscrutable, but so was Linux around the time of Plan 9's first commercial release … and now look at it. Linux powers ChromeOS and Android, which means that aside from countless invisible servers, several billion people use Linux every day, even though they've never even heard of it.

Even most "Linux distributions" software for the tiny number of people who know what an "operating system" is and care which one they use, are pretty easy now.

But Plan 9 is not Unix, and it can't run Unix programs. You couldn't simply port Firefox or LibreOffice to Plan 9, as has been done to Windows and macOS and all the BSDs.

There are tools to help with stuff like this, but they are a little rudimentary because the sort of people who value Plan 9 either don't want such things, or do things like that on other computers.

For example, there is a Plan 9 Linux emulator, comparable to the FreeBSD linuxulator. Plan 9's is called Linuxemu). It's old enough that it's 32-bit only, but until a few years ago so were Solaris LX zones – then Joyent modernized them for the SmartOS implementation.

There is a Linux compatibility layer for porting source code called APE. There's an X11 server, Equis.

More importantly, 9front has a hypervisor called VMX, which uses the built-in virtualization of Intel and AMD chips. There's even a guide to installing Void Linux in it. A VM can have just one core, but it is there and it does work, although, as the documentation says, VMX can and will crash your kernel.

So here is my idea.

It's 2024. We all have 64-bit machines now, and x86-32 is fading fast. Most distros no longer support it; even one of the last holdouts, Debian, is planning to drop support in the next release.

Modern PCs have lots of memory and lots of storage. These days, in Linux land, there are a number of "microVMs" such as Amazon's Firecracker. Recently that gained support for FreeBSD guests too. There are others, such as Intel Clear Containers, which merged into Kata Containers, and so on.

MicroVMs can start a guest in milliseconds, and they are used for "serverless" cloud computing. Another lie, but let's not get into that.

The idea is tiny VMs that are as quick and as ephemeral as containers. You start them, typically to run just one program, and when that completes, the VM exits.

If it were possible to run a microVM on 9front, you could, in principle, start a Linux app, use it, and quit it again, and on a modern computer not notice the delay from starting the VM.

That also reduces the need for a Linux emulator. This is a complex task, and one that needs constant maintenance to track the moving target of the current Linux kernel. So let it go.

There's an X server. If that were running already, or a dependency, new microVMs could connect to it, and thus run seamlessly inside Rio. I suspect that Equis needs some love and maintenance, and this would be very flaky at first, but the component parts of this tech are already out there.

There's already a model for this. Qubes has been around for over a decade and it is in active development. All user programs run inside dedicated VMs. In the case of Qubes, this is for additional security. Qubes is based on Fedora and the Xen hypervisor, but it's the concept rather than the implementation that is important here.

This need not be focused on graphical apps. So long as the app in a microVM gets its input from the file system and writes its output to the file system as well, this could enable Linux apps to offer some degree of interoperability with native Plan 9 ones, without bloating Plan 9 with all of the complexity of a modern Linux system. The messy stuff is kept neatly boxed up.

There is potential here for incrementally bringing some tools across from Unix to Plan 9 in a way that was impossible when Plan 9 was built. Stage one: isolating apps in VMs. Stage two: bring the most useful smaller, simpler apps across to APE or something like it. Stage three: potentially rewrite the essentials as native Plan 9 apps. (For console apps, a native version of something like ncurses would make life a great deal easier.)

Over time, a move like this would inevitably mean that Plan 9 itself gradually got bigger and more complex … but then there already is a successor to Plan 9 in the form of the Inferno OS. As others have observed before:

We can't just bring Plan 9 namespaces into Linux or any other Unix-like OS. Only a system designed from the start properly, with such coherence, can achieve something so simple yet so powerful and integrate with the whole system.

This is just one option of many. But it's a potential path forward to a cleaner, less bloated future. There is a postscript to this coming soon, as well – the final part of the original FOSDEM talk looked at stripping down Linux into a dedicated VM operating system, with all the bare-metal functionality removed. ®