Google's Grumpy code makes Python Go

With a transcompiling tool, YouTube aims to overcome Python's limitations

Google on Wednesday introduced an open-source project called Grumpy to translate Python code into Go programs.

The company's front-end server for YouTube and its YouTube API are mainly written in Python, and run on CPython 2.7, as opposed to other implementations (Jython, PyPy, or IronPython).

The issue for Google is performance. Python in its various forms doesn't excel at concurrency – running multiple, independent threads at the same time – explains YouTube engineer Dylan Trotter in a blog post.

Grumpy is a source code transcompiler and runtime. It compiles Python source code into Go source code that gets compiled into native code rather than bytecode that requires a VM.

The compiled Go code makes calls to the Grumpy runtime, which is a Go library that functions similarly to the Python C API.

Trotter characterizes Grumpy as experimental, but that doesn't mean support for Python 3 is forthcoming as the project matures. In response to a request for Python 3 support, Trotter on GitHub said, "We have a large Python 2.7 codebase so that's what we've been focused on. I definitely would like to support Python 3. It's just a bunch of work." He suggested those interested in Python 3 support should fork the project.

One possible rationale for creating Grumpy may be the planned end-of-life date for Python 2.7 in 2020. Rather than trying to update all its Python code from version 2.7 to 3 – which has been slow to catch on – Google appears to be planning to convert at least some of its Python code to Go over the next few years.

Google didn't immediately respond to a request for comment.

In a post to Hacker News, Trotter offered his thoughts on the matter. "The idea is to continue to write code in Python," he said. "The transcompiled code is not suitable for working with directly. That said, there is the possibility of rewriting bits and pieces in Go (eg, performance-critical stuff) and then call into it from Python. Sort of a hybrid approach."

Three years hence and beyond, that hybrid approach can be expected to include a lot more Go code, provided Google's developers become as proficient with Go as they are with Python.

Go, as Trotter suggests, has performance advantages over Python (though benchmarks can vary significantly, depending upon optimizations and alternative implementations). It also has brand advantages: Go was developed at Google and the company's willingness to eat its own dogfood, so to speak, could convince other organizations that Go is the way to go.

Google doesn't benefit directly from Go adoption, but broader usage will deepen the talent pool of Go developers, who could in time end up working at Google or improving Go libraries that Google uses. There's also the opportunity cost imposed on competitors – the more time developers spend with Go, the less they have to spend with languages backed by Apple, Microsoft, Mozilla, Oracle, or others. ®

Broader topics

Narrower topics

Other stories you might like

  • Verizon: Ransomware sees biggest jump in five years
    We're only here for DBIRs

    The cybersecurity landscape continues to expand and evolve rapidly, fueled in large part by the cat-and-mouse game between miscreants trying to get into corporate IT environments and those hired by enterprises and security vendors to keep them out.

    Despite all that, Verizon's annual security breach report is again showing that there are constants in the field, including that ransomware continues to be a fast-growing threat and that the "human element" still plays a central role in most security breaches, whether it's through social engineering, bad decisions, or similar.

    According to the US carrier's 2022 Data Breach Investigations Report (DBIR) released this week [PDF], ransomware accounted for 25 percent of the observed security incidents that occurred between November 1, 2020, and October 31, 2021, and was present in 70 percent of all malware infections. Ransomware outbreaks increased 13 percent year-over-year, a larger increase than the previous five years combined.

    Continue reading
  • Slack-for-engineers Mattermost on open source and data sovereignty
    Control and access are becoming a hot button for orgs

    Interview "It's our data, it's our intellectual property. Being able to migrate it out those systems is near impossible... It was a real frustration for us."

    These were the words of communication and collaboration platform Mattermost's founder and CTO, Corey Hulen, speaking to The Register about open source, sovereignty and audio bridges.

    "Some of the history of Mattermost is exactly that problem," says Hulen of the issue of closed source software. "We were using proprietary tools – we were not a collaboration platform before, we were a games company before – [and] we were extremely frustrated because we couldn't get our intellectual property out of those systems..."

    Continue reading
  • UK government having hard time complying with its own IR35 tax rules
    This shouldn't come as much of a surprise if you've been reading the headlines at all

    Government departments are guilty of high levels of non-compliance with the UK's off-payroll tax regime, according to a report by MPs.

    Difficulties meeting the IR35 rules, which apply to many IT contractors, in central government reflect poor implementation by Her Majesty's Revenue & Customs (HMRC) and other government bodies, the Public Accounts Committee (PAC) said.

    "Central government is spending hundreds of millions of pounds to cover tax owed for individuals wrongly assessed as self-employed. Government departments and agencies owed, or expected to owe, HMRC £263 million in 2020–21 due to incorrect administration of the rules," the report said.

    Continue reading

Biting the hand that feeds IT © 1998–2022