Fed up with Python setup and packaging? Try a shot of Rye

For those envious of Rust and Cargo

The recent debut of a Python tool called Rye has raised hopes that the programming language's long standing setup and packaging shortcomings may be solvable.

Rye, a personal project developed by Armin Ronacher, an open source developer and creator of the Python web framework Flask, addresses two issues: bootstrapping (setup) and packaging.

Bootstrapping involves getting a Python environment installed and running; packaging involves the distribution, installation, and management of modular software libraries, or packages, in Python applications.

Python has a reputation for being one of the more approachable and easy to learn programming languages. But it's also fairly old at this point, dating back to 1991, so it has accrued a lot of technical debt. Things can go sideways quickly when developer tools behave in unanticipated ways.

The language is managed by the Python community, with the support of the Python Software Foundation. Consequently, a lot of different tools exist for bootstrapping and packaging, and in the absence of a single, correct approach for these tasks, developers often make choices that complicate their lives and cause code to break.

The confusing nature of Python packaging was featured in an XKCD comic on April 30, 2018. As Brett Cannon, a member of the Python Steering Council and software engineering manager at Microsoft, noted a year later, "So this obviously touched a nerve."

Cannon attempted to spread the blame for the situation depicted in the comic on its author Randall Munroe, on the third-party macOS package manager Homebrew, and on Python. And while it's fair to scold developers when they shoot themselves in the foot with a footgun, it still falls to the Python community to reduce the availability of footguns.

Fast forward to 2021 and we find the Python Software Foundation hiring a packaging project manager. The discussion of Python's packaging deficiencies now includes whether a more cohesive, directed approach can work.

The Python community touched on these issues earlier this year and noted that feedback from a user survey indicates the community believes there are too many tools and too much complexity. That's a tough message for an open source community to digest – everyone is welcome to contribute, but please not so much.

Pradyun Gedam, maintainer of the Python tool pip, explored these concerns further in a subsequent blog post in January.

"A class of users expect a packaging tool that provides a cohesive experience (like npm (NodeJS), cargo (Rust), gem (Ruby), pub (Dart), dotnet (C#/.NET), etc) – a single tool that provides a build system, dependency manager, publishing, running project-specific tasks/scripts, etc," he wrote.

"Today, each of these pieces is a separate tool for Python and doesn’t have a strict 1:1 mapping to the 'best practices'/'secure' workflows. This is at odds with the expectations that these users have. This class of users, by and large, want consolidation and a single-tool experience."

Many have tried to improve Python's setup and packaging story over the years and these efforts are ongoing. Cannon himself has created a project called the Python Launcher for Unix, to make it easier to be sure which Python interpreter you're running. Then there are packaging and dependency management tools like Poetry and Pyflow. And developer Gregory Szorc has tackled an adjacent issue with self-contained, portable Python distributions and the PyOxidizer Project.

Ronacher's Rye in fact makes use of Szorc's python-build-standalone project. This simplifies the installation experience by avoiding the problems that arise from having multiple different Python distributions and it obviates the need to compile Python by providing pre-prepared binaries.

Written in Rust, of course

Rye is built in Rust, which has a certain cachet these days, though it only works on macOS and Linux presently. Even so, the project has been well-received in the Python community. Ronacher's demonstration of Rye on YouTube shows that it's possible for Python to have a setup and dependency experience that's more like Rust's Cargo tool.

"Rye wants to be an experiment of improving the Python developer experience to get closer to what Rust developers are already used to," explained Ronacher in an email to The Register. "I have received a lot of positive and unexpected interest in this project.

"I built various iterations of this over the last three years out of personal interest, but I was cautious about publishing anything out of fear of adding yet another tool into the mix. Right now I’m reaching out to other people in the packaging ecosystem to see if there is a way to work together towards something, and I’m trying to get a better public description out of how I wish packaging in the Python ecosystem could work to align people on a vision."

Ronacher is well-aware that efforts to simplify Python's packaging story are not necessarily helped by the addition of yet another in a long line of tools and acknowledges his reluctance to foist software developed for personal use onto the wider community. In a GitHub Issues post he asks whether Rye should exist?

I want to never have to think about Python packaging and project management ever again

"I really have no desire to contribute more to the proliferation of Python packaging," he wrote. "Today the [Python Packaging Authority] on Github is what appears to be dozens of competing, largely independent efforts. In addition to that there is Conda. There are now even companies appearing that are putting layers around Conda as well. It's a pretty big mess.

"I just want it solved, and I want to never have to think about Python packaging and project management ever again."

Some of those responding to Ronacher's musing argue that Rye, or some similar singular solution, should exist. Others disagree. And there, in a nutshell, you have the cat herding that is Python governance.

Simon Willison, an open source developer and maintainer of Datasette, tried Rye recently and believes it can at least move the community toward a more coherent setup and packaging scheme.

"Python packaging is a crowded space, but Rye has some really interesting ideas in it – especially around how Python itself can be installed in a way that reduces the chance of conflicting with system defaults," he told The Register.

"I'm personally a big fan of using prototypes and running code to help illustrate new ideas, so I welcome Rye as a project that can help drive quality conversations." ®