SmartNICs, IPUs, DPUs de-hyped: Why and how cloud giants are offloading work from server CPUs

Where this technology grew from, and what it offers you

Systems Approach The recent announcements from Intel about Infrastructure Processing Units (IPUs) have prompted us to revisit the topic of how functionality is partitioned in a computing system.

As we noted in an earlier piece, The Accidental SmartNIC, there is at least thirty years’ history of trying to decide how much one should offload from a general purpose CPU to a more specialized NIC, and an equally long tussle between more highly specialized offload engines versus more general-purpose ones.

The IPU represents just the latest entry in a long series of general-purpose offload engines, and we’re now seeing quite a diverse set of options, not just from Intel but from others such as Nvidia and Pensando. These latter firms use the term DPU (data processing unit) but the consensus seems to be that these devices tackle the same class of problems.

There are several interesting things going on here. The first is that there is an emerging consensus that the general purpose x86 (or Arm) server is no longer the best place to run the infrastructure functions of a cloud. By “infrastructure functions” we mean all the things that it takes to run a multi-tenant cloud that are not actually guest workloads: the hypervisor, network virtualization, storage services and so on.

Whereas the server used to be the home of both guest workloads and infrastructure services, these functions are increasingly viewed as “overhead” that is only taking cycles away from guests. One oft-cited paper is Facebook’s Accelerometer study, which measures overhead within Facebook’s data centers as high as 80 per cent, although this may not be generalizable to cloud providers. More plausibly, Google reported [PDF] in 2015:

'Datacenter tax' can comprise nearly 30 per cent of cycles [...], which makes its constituents prime candidates for hardware specialization in future server systems-on-chips.

Amazon Web Services presumably saw the same issue of overheads cutting into the revenue-generating workloads, and started to use specialized hardware for infrastructure services when it acquired Annapurna Labs in 2015, laying the groundwork for its Nitro architecture. The impact was to move almost all infrastructure services out of the servers, leaving them free to run guest workloads and little else.

Once you decide to move a function out of the general-purpose CPU complex into some sort of offload engine, the question is how to retain the appropriate level of flexibility. These offloaded functions are not static, so putting them into fixed-function hardware would be a short-sighted move.

This is why we have seen NICs move in recent years from fixed-function offloads such as TCP segmentation to the more flexible architecture of SmartNICs. So the goal is to build an offload system that is more optimized for the offloaded services than a general-purpose CPU, yet still programmable enough to support innovation and evolution of offloaded services.

The goal is to build an offload system that is more optimized for the offloaded services than a general-purpose CPU, yet still programmable enough to support innovation and evolution of offloaded services

Intel’s IPU family contains several entrants, which take different approaches to delivering that flexibility, including both FPGA- and ASIC-based versions. The Mount Evans ASIC is particularly interesting as it includes both Arm CPU cores and programmable networking hardware (from the Barefoot Networks team) that is P4-programmable.

This is a subject dear to our hearts here at Systems Approach, as the P4 toolchain is central to much of the technology that we wrote about in our SDN book.

Putting a P4-programmable switch in an IPU/DPU makes lots of sense, since the networking functions that are likely to be offloaded include those of a virtual switch. And one thing we learned at Nicira and later in the NSX team at VMware was that if you want to move the vswitch to an offload engine, that engine needs to be fully programmable.

If a NIC is insufficiently general to implement the whole vswitch, you can only move some subset of the vswitch functionality to the offload engine. Even if you could move 90 per cent of the functionality, that remaining 10 per cent that you have to keep doing in the CPU is likely to be a bottleneck.

So a P4-programmable offload engine based on the Protocol Independent Switching Architecture (PISA) provides the required level of flexibility and programmability to make offloading of the whole vswitch possible. Combine this with some other programmable hardware (such as Arm cores) and you can see how the entire set of infrastructure functions, including the hypervisor, storage virtualization, etc., can be offloaded to the IPU.

The virtual switch of a network virtualization system is a candidate for offloading

One way to view the latest generation of DPUs/IPUs is that the efforts of the SDN movement to create more programmable switches has enabled innovation in a new space.

SDN initially promised to drive control plane innovation by decoupling the switching hardware from the software that controlled it. Network virtualization was one of the first applications of SDN to take off, with the separation of control and data planes and highly flexible software switches enabling networks to be created entirely in software (on top of a hardware-based underlay).

PISA and P4 led to a more flexible form of switching hardware and a new way to define the hardware-software interface (improving on the earlier efforts of OpenFlow). All of these threads–control plane innovation, network virtualization, and flexible, programmable switch hardware–are now being brought together in the creation of IPUs and DPUs.

We can also view the development of IPUs/DPUs as a continuation of the trend in which processors are both highly flexible and yet specialized for certain tasks. GPUs and TPUs are really flexible, being used for everything from crypto-mining to machine learning to graphics processing, but are nevertheless quite specialized compared to CPUs. (GPUs were even used for packet processing in an era before we had PISA and P4.)

DPUs and IPUs now seem well established as a new category of highly programmable devices that are optimized for a specific set of tasks that need to be performed in a modern cloud data center. With that greater specialization comes greater efficiency, while flexibility remains high enough to support future innovation. ®

Larry Peterson and Bruce Davie are the authors of Computer Networks: A Systems Approach and the related Systems Approach series of books. All their content is open source and available on GitHub. You can find them on Twitter, their writings on Substack, and past The Register columns here.

Send us news

Lawyers say changes to UK data law will make life harder for international businesses

Concerns raised over government drive to implement distinct post-Brexit policy

Legal experts say UK government plans to create new data protection laws will make more work and add costs for business, while also creating the possibility of challenges to data sharing between the EU and UK.

Last week, the Queen's Speech – in which the British government sets out its legislative plans – said the ruling Conservative party planned to replace the EU's General Data Protection Regulation (GDPR) to ease the burden on business with an approach to data protection that encourages innovation while retaining protection of personal data and privacy.

Continue reading

September 16, 1992, was not a good day to be overly enthusiastic about your job

If I get in early and work hard, everyone will notice, right?

Who, Me? "The early bird trashes the business" is a saying that we've just made up, but could easily apply to the Register reader behind a currency calamity in today's episode of Who, Me?

Our hero, Regomized as "Mike", was working as a "data entry operative" for a tourism company in 1992. The company ran bus tours to the then brand-new EuroDisney, parent company of Disneyland Paris (now the most visited theme park in Europe), which had opened earlier that year.

Mike was an eager beaver, his youthful naivete having convinced him that if he worked extra hard, came in extra early, and kept the in-tray clear, then his efforts would be both noticed and rewarded with promotion and a bump in pay.

Continue reading

(Our) hardware is still key in a multicloud world, Dell ISG chief insists

IT giant may be shifting its focus to software and services, but systems remain the foundation

Analysis At this month's Dell Technologies World show in Las Vegas, all the usual executives were prowling the keynote stages, from CEO Michael Dell to co-COOs Chuck Witten and Jeff Clark, all talking about the future of the company.

Noticeably absent were the big servers or storage systems that for decades had joined them on stage, complete with all the speeds and feeds. Though a PC made an appearance, there was no reveal of big datacenter boxes.

It's a continuing scenario that is likely to play out to various degrees at user events for other established IT hardware vendors, such as when Hewlett Packard Enterprise later next month convenes its Discover show, also in Las Vegas. It's having to adapt to the steady upward trend in multicloud adoption, the ongoing decentralization of IT and the understanding that in today's world, data is king, Hardware is still needed, but the outcomes they deliver are what is most important.

Continue reading

Ad-tech firms grab email addresses from forms before they're even submitted

Researchers find widespread harvesting of info without consent

Tracking, marketing, and analytics firms have been exfiltrating the email addresses of internet users from web forms prior to submission and without user consent, according to security researchers.

Some of these firms are said to have also inadvertently grabbed passwords from these forms.

In a research paper scheduled to appear at the Usenix '22 security conference later this year, authors Asuman Senol (imec-COSIC, KU Leuven), Gunes Acar (Radboud University), Mathias Humbert (University of Lausanne) and Frederik Zuiderveen Borgesius, (Radboud University) describe how they measured data handling in web forms on the top 100,000 websites, as ranked by research site Tranco.

Continue reading

Arm CPU ran on electricity generated by algae for over six months

AA-battery-sized biological photovoltaic cell touted as ideal for IoT applications

Researchers at the University of Cambridge's Department of Biochemistry have run an Arm CPU for six months using algae as a power source.

As explained in a paper titled Powering a microprocessor by photosynthesis, the biochem boffins built an AA-battery-sized device that hosts an algae named Synechocystis that "naturally harvests energy from the sun through photosynthesis."

The boffins found a way to turn that harvested energy into current by using an aluminium anode, and fed it into a board hosting an Arm Cortex M0+ CPU.

Continue reading

China's Kylin Linux targets second RISC-V platform

Is state-approved Ubuntu distro how the Middle Kingdom will replace PCs with home-grown kit?

China's military-derived and government-approved Linux distribution, Ubuntu Kylin, has revealed plans to target a second RISC-V platform.

Ubuntu Kylin is Ubuntu’s official version for China and was developed in partnership with Chinese authorities, including the military.

In March 2022, a version of the OS was released for the HiFive Unmatched board – a SiFive product in the Mini-ITX form factor and packing a five-core Freedom U740 SoC.

Continue reading

Toshiba says it's talking to 10 suitors about possible sale

Hires external advisors to bolster decision making capacity and hints it could consider multiple buyout plans

Ailing Japanese giant Toshiba has revealed it has 10 potential suitors for its possible sale.

A Friday announcement revealed that Toshiba's decision to consider a sale to a private buyer has progressed to the point at which discussions are under way with §0 parties who have expressed an interest in submitting a proposal to buy the company.

Those talks have become sufficiently serious that Toshiba has appointed two sets of advisors – from Mizuho Securities and JP Morgan Securities – to offer financial advice and assist the special committee Toshiba assembled to consider offers.

Continue reading

Demand for PC and smartphone chips drops 'like a rock' says CEO of China’s top chipmaker

Markets outside China are doing better, but at home vendors have huge component stockpiles

Demand for chips needed to make smartphones and PCs has dropped "like a rock" – but mostly in China, according to Zhao Haijun, the CEO of China's largest chipmaker Semiconductor Manufacturing International Corporation (SMIC).

Speaking on the company's Q1 2022 earnings call last Friday, Zhao said smartphone makers currently have five months inventory to hand, so are working through that stockpile before ordering new product. Sales of PCs, consumer electronics and appliances are also in trouble, the CEO said, leaving some markets oversupplied with product for now. But unmet demand remains for silicon used for Wi-Fi 6, power conversion, green energy products, and analog-to-digital conversion.

The CEO's "like a rock" comment came in the Q&A section of the call, after previous scripted remarks mentioned a "destocking phase" among SMIC clients.

Continue reading

Colocation consolidation: Analysts look at what's driving the feeding frenzy

Sometimes a half-sized shipping container at the base of a cell tower is all you need

Analysis Colocation facilities aren't just a place to drop a couple of servers anymore. Many are quickly becoming full-fledged infrastructure-as-a-service providers as they embrace new consumption-based models and place a stronger emphasis on networking and edge connectivity.

But supporting the growing menagerie of value-added services takes a substantial footprint and an even larger customer base, a dynamic that's driven a wave of consolidation throughout the industry, analysts from Forrester Research and Gartner told The Register.

"You can only provide those value-added services if you're big enough," Forrester research director Glenn O'Donnell said.

Continue reading

D-Wave deploys first US-based Advantage quantum system

For those that want to keep their data in the homeland

Quantum computing outfit D-Wave Systems has announced availability of an Advantage quantum computer accessible via the cloud but physically located in the US, a key move for selling quantum services to American customers.

D-Wave reported that the newly deployed system is the first of its Advantage line of quantum computers available via its Leap quantum cloud service that is physically located in the US, rather than operating out of D-Wave’s facilities in British Columbia.

The new system is based at the University of Southern California, as part of the USC-Lockheed Martin Quantum Computing Center hosted at USC’s Information Sciences Institute, a factor that may encourage US organizations interested in evaluating quantum computing that are likely to want the assurance of accessing facilities based in the same country.

Continue reading

Bosses using AI to hire candidates risk discriminating against disabled applicants

US publishes technical guide to help organizations avoid violating Americans with Disabilities Act

The Biden administration and Department of Justice have warned employers using AI software for recruitment purposes to take extra steps to support disabled job applicants or they risk violating the Americans with Disabilities Act (ADA).

Under the ADA, employers must provide adequate accommodations to all qualified disabled job seekers so they can fairly take part in the application process. But the increasing rollout of machine learning algorithms by companies in their hiring processes opens new possibilities that can disadvantage candidates with disabilities. 

The Equal Employment Opportunity Commission (EEOC) and the DoJ published a new document this week, providing technical guidance to ensure companies don't violate ADA when using AI technology for recruitment purposes.

Continue reading