Open Compute Summit Facebook is using Intel's Xeon D processors to build stacks of web servers for the 1.39 billion people who visit the social network every month.
The OpenRack server design is codenamed Yosemite, is pictured above, and is available for anyone to use under the OpenCompute project. The hardware "dramatically increases speed and more efficiently serves Facebook traffic," the website's engineers boast.
Each sled holds four boards, and on each board sits a single Xeon D-1540 processor package with its own RAM and flash storage. That D-1540 part features eight cores (16 threads) running at 2GHz, plus two 10Gb Ethernet ports, PCIe and other IO.
Each processor consumes up to 65W, 90W for the whole server card, and 400W (TDP) for a full sled. A single rack can hold 48 sleds, which adds up to 192 Xeon Ds and 1,536 Broadwell cores. The Yosemite motherboard has a 50Gb/s multi-host network interconnect that hooks the four CPU boards through a single Ethernet port.
The key thing is that this design is easier for Facebook's software engineers to program. Each independent server is essentially a single socket processor with its own RAM, storage and NIC, whereas previous designs are two-socket affairs. The single-socket design gets rid of all the NUMA headaches present in a two-socket system, when writing and tuning multi-threaded code to generate and serve web pages.
"We took a look at our heavy workloads ... and the challenges with balancing NUMA, and came up with Yosemite," Jay Parikh, Facebook's veep of engineering, told The Register at the OpenCompute Summit today in San Jose, California.
"There are four trays, and each tray has a one-socket computer in it. This is a four-in-one design for each sled. Each card has a CPU, memory and SSD, so you can boot it up as a server.
"890 million people visit Facebook on mobile every day. We have to build the infrastructure to support this."
Halving the socket count also cuts power consumption, we're told, and reduces the footprint of each server. Parikh added that Facebook's data centers are "100 per cent" powered by OpenCompute gear, and has saved the biz $2bn in the past three years in terms of energy saved, and so on.
Facebook and Intel have apparently been working on using Xeon Ds for the past couple of years while, in the background, the social network has been toying with the idea of using dense ARM-powered system-on-chips for front-end servers. So far, it seems, Intel is winning out.
"We started experimenting with SoCs about two years ago. At that time, the SoC products on the market were mostly lightweight, focusing on small cores and low power," said Facebook hardware design engineer Harry Li.
"Most of them were less than 30W. Our first approach was to pack up to 36 SoCs into a 2U enclosure, which could become up to 540 SoCs per rack.
"But that solution didn't work well because the single-thread performance was too low, resulting in higher latency for our web platform. Based on that experiment, we set our sights on higher-power processors while maintaining the modular SoC approach."
In other developments
- Apple is joining the OpenCompute project, according to OCP president Frank Frankovsky. "Apple has been involved in this project quietly for a very long time. A lot of people know Apple as a consumer tech company. But they also have excellent infrastructure engineering people," he added in an opening speech to the conference. OpenCompute was founded by Facebook.
- The social network's engineers are opening up their baseboard management controller (BMC) software, aptly calling it OpenBMC. This is the code that runs on a microcontroller on server motherboards, and allows sysadmins to monitor a computer's temperature, control its power, and talk to the kernel via serial-over-LAN. It's useful for managing machines even if the operating system has fallen over. Facebook's software started life during a 24-hour hackathon by the site's engineers.
- Broadcom is open-sourcing a software library called OpenNSL so other code can talk to and control its network ASICs. Now Facebook has released FBOSS Agent that runs on its network switches, and uses OpenNSL to control the underlying Broadcom ASICs in its Wedge top-of-rack switch.
- Facebook has published the blueprints to its Wedge top-of-rack switch. Accton, Broadcom, Cumulus, and Big Switch are building the hardware for others to buy; Accton is ready to ship the gear in the first half of this year. ®