'We're finding bugs way faster than we can fix them': Google sponsors 2 full-time devs to improve Linux security

Plus: Why the Chocolate Factory only uses code it builds from source

Interview Worried about the security of Linux and open-source code, Google is sponsoring a pair of full-time developers to work on the kernel's security.

The internet giant builds code from its own repositories rather than downloading outside binaries, though given the pace at which code is being added to Linux, this task is non-trivial. Google's open-source security team lead Dan Lorenc spoke to The Register about its approach, and why it will not use pre-built binaries despite their convenience.

But first: the two individuals full-time sponsored by Google are Gustavo Silva, whose work includes eliminating some classes of buffer overflow risks and on kernel self-protection, and Nathan Chancellor, who fixes bugs in the Clang/LLVM compilers and improves compiler warnings.

Both are already working at the Linux Foundation, so what is new? "Gustavo's been working on the Linux kernel at the Linux Foundation for several years now," Lorenc tells us. "We've actually been sponsoring it within the Foundation for a number of years. The main change is that we're trying to talk about it more, to encourage other companies to participate. It's a model that works, we're trying to expand it, find contributors that want to turn this into a full-time thing, and giving them the funding to do that."

It is in the nature of open source that Google's funding benefits other Linux users, and it is also in the company's interests. How important is Linux to Google? "It's absolutely critical. Google started on Linux. We use it everywhere," says Lorenc.

That being the case, why can Google only manage "Gold" membership of the Linux Foundation ($100,000 per annum), whereas others including Microsoft, Intel, Facebook, and Red Hat are "Platinum", which contributes $500,000 annually? "I'm not sure about that stuff. There are dozens of sub-foundations which we are also members of," he adds. Google is ahead of AWS, which is a mere "Silver" member ($20,000 a year).

Lorenc explains some of the steps Google takes to ensure the security of open-source code that it uses internally, including Linux. "One of the things that we try to do for any open source that we use, and something we recommend anybody uses, is being able to build it yourself. It is not always easy or trivial to build, but knowing that you can is half the battle, in case you ever need to.

We require that all open source we use is built by us, from our internal repositories

"We require that all open source we use is built by us, from our internal repositories, just to prove that we can, if we ever need to make a patch, and so that we have better provenance, knowing where it is coming from. They are technically forks, but just a copy of the [public] repo that we keep up to date. We rarely carry patches long-term in any of the projects we work on, it is just a maintenance nightmare, but we can if we need to.

"For the most part we build our own Linux kernels, but that's the Linux model. For Linux it is not strange to be doing it this way, it is strange in a number of other projects where we do it."

The consequence is that Google loses the convenience most Linux users enjoy, downloading a binary image for a Linux distribution and installing what is needed via a package manager.

What are the challenges in keeping Linux secure? "It's way better than many projects," says Lorenc. "They have a well-documented process with tiers of reviewers. The challenge is that the Linux kernel is huge, it's software and all software has bugs, the more software there is the more bugs there are. The other thing is we've gotten very good at finding bugs, static analysis has come a long way, fuzzing has come along way, and we're finding bugs way faster than we can fix them. The next challenge is finding ways to fix them.

"We don't want to stop finding them, but if you're finding bugs that nobody has time to look at, you're not really solving the whole problem."

This then is part of the rationale for getting "more hands on keyboard", but sponsoring two developers who already work on this is not close to a solution.

If you're finding bugs that nobody has time to look at, you're not really solving the whole problem

It is still worthwhile, says Lorenc, as there are not many such people. "We did an announcement last week around the Python interpreter. We added one full-time person working on CPython for the next year. Best estimate is that is a 50 per cent increase in the number of people working on the Python interpreter full-time." The situation with full-time developers working on Linux security may be similar, he says.

With Linus Torvalds regularly commenting on the large amount of new code in kernel releases, how can this be addressed? "We are going to have to get creative," Lorenc tells The Reg. "Our basic approach to security at scale across Google and in the rest of the industry is to try to engineer away entire classes of problems. We do have to fix the bugs we find, but at the same time think about ways to fix entire classes of bugs."

Penguins in Antarctica

Hidden Linux kernel security fixes spotted before release – by using developer chatter as a side channel


Examples would be the buffer overflow problem that has long afflicted software written in system languages like C, used for the Linux kernel. Switch language? "We know we have a whole new set of programming languages now, like Rust and Go and Swift, that operate in completely new ways," he says. "By using these languages you've engineered away entire classes of memory safety bugs."

Microsoft has had similar thoughts with regard to Windows, but change is not easy. Is Google working on using Rust for kernel code, or rewriting code in Rust? "We're thinking about it," Lorenc reveals. "The last update was at the Linux Plumber's conference, there was a session on what it would take. That was run by a Google engineer from the Android team... it's not far enough along to know if it's going to work or not."

Given the scale of the challenge and the large flow of new kernel code, is the security aspect winning? Lorenc says: "I don't think we're losing. We're making a lot of great progress. Each release is getting bigger but we're getting better with each release too."

Not losing, then – but not winning either. ®

Other stories you might like

  • Cheers ransomware hits VMware ESXi systems
    Now we can say extortionware has jumped the shark

    Another ransomware strain is targeting VMware ESXi servers, which have been the focus of extortionists and other miscreants in recent months.

    ESXi, a bare-metal hypervisor used by a broad range of organizations throughout the world, has become the target of such ransomware families as LockBit, Hive, and RansomEXX. The ubiquitous use of the technology, and the size of some companies that use it has made it an efficient way for crooks to infect large numbers of virtualized systems and connected devices and equipment, according to researchers with Trend Micro.

    "ESXi is widely used in enterprise settings for server virtualization," Trend Micro noted in a write-up this week. "It is therefore a popular target for ransomware attacks … Compromising ESXi servers has been a scheme used by some notorious cybercriminal groups because it is a means to swiftly spread the ransomware to many devices."

    Continue reading
  • Twitter founder Dorsey beats hasty retweet from the board
    As shareholders sue the social network amid Elon Musk's takeover scramble

    Twitter has officially entered the post-Dorsey age: its founder and two-time CEO's board term expired Wednesday, marking the first time the social media company hasn't had him around in some capacity.

    Jack Dorsey announced his resignation as Twitter chief exec in November 2021, and passed the baton to Parag Agrawal while remaining on the board. Now that board term has ended, and Dorsey has stepped down as expected. Agrawal has taken Dorsey's board seat; Salesforce co-CEO Bret Taylor has assumed the role of Twitter's board chair. 

    In his resignation announcement, Dorsey – who co-founded and is CEO of Block (formerly Square) – said having founders leading the companies they created can be severely limiting for an organization and can serve as a single point of failure. "I believe it's critical a company can stand on its own, free of its founder's influence or direction," Dorsey said. He didn't respond to a request for further comment today. 

    Continue reading
  • Snowflake stock drops as some top customers cut usage
    You might say its valuation is melting away

    IPO darling Snowflake's share price took a beating in an already bearish market for tech stocks after filing weaker than expected financial guidance amid a slowdown in orders from some of its largest customers.

    For its first quarter of fiscal 2023, ended April 30, Snowflake's revenue grew 85 percent year-on-year to $422.4 million. The company made an operating loss of $188.8 million, albeit down from $205.6 million a year ago.

    Although surpassing revenue expectations, the cloud-based data warehousing business saw its valuation tumble 16 percent in extended trading on Wednesday. Its stock price dived from $133 apiece to $117 in after-hours trading, and today is cruising back at $127. That stumble arrived amid a general tech stock sell-off some observers said was overdue.

    Continue reading
  • Amazon investors nuke proposed ethics overhaul and say yes to $212m CEO pay
    Workplace safety, labor organizing, sustainability and, um, wage 'fairness' all struck down in vote

    Amazon CEO Andy Jassy's first shareholder meeting was a rousing success for Amazon leadership and Jassy's bank account. But for activist investors intent on making Amazon more open and transparent, it was nothing short of a disaster.

    While actual voting results haven't been released yet, Amazon general counsel David Zapolsky told Reuters that stock owners voted down fifteen shareholder resolutions addressing topics including workplace safety, labor organizing, sustainability, and pay fairness. Amazon's board recommended voting no on all of the proposals.

    Jassy and the board scored additional victories in the form of shareholder approval for board appointments, executive compensation and a 20-for-1 stock split. Jassy's executive compensation package, which is tied to Amazon stock price and mostly delivered as stock awards over a multi-year period, was $212 million in 2021. 

    Continue reading
  • Confirmed: Broadcom, VMware agree to $61b merger
    Unless anyone out there can make a better offer. Oh, Elon?

    Broadcom has confirmed it intends to acquire VMware in a deal that looks set to be worth $61 billion, if it goes ahead: the agreement provides for a “go-shop” provision under which the virtualization giant may solicit alternative offers.

    Rumors of the proposed merger emerged earlier this week, amid much speculation, but neither of the companies was prepared to comment on the deal before today, when it was disclosed that the boards of directors of both organizations have unanimously approved the agreement.

    Michael Dell and Silver Lake investors, which own just over half of the outstanding shares in VMware between both, have apparently signed support agreements to vote in favor of the transaction, so long as the VMware board continues to recommend the proposed transaction with chip designer Broadcom.

    Continue reading

Biting the hand that feeds IT © 1998–2022