Dell ARMs up for hyperscale servers

But are they dangerous to anyone except HP?


Look out Intel. Here comes another ARM box to the microserver party.

If people didn't want ARM-based servers, Dell wouldn't build them, and so with the launch of the "Copper" ARM server sled for the "Viking" C5000 microserver chassis we know that people want ARM servers. And this is not some experiment that Dell is doing, either, Steve Cumings, executive director of marketing for Dell's Data Center Solutions bespoke server unit, tells El Reg.

"This is not only real, but it is going out the door," says Cumings, with a jab at rival Hewlett-Packard's "Project Moonshot" hyperscale server effort and its "Redstone" many-ARMed server nodes launched last November. Those servers made a big splash, but neither HP nor Calxeda have said much since then about who is using these concept machines – or what the plan is to ramp up the Redstone product line with other ARM chips with more oomph and memory capacity.

Those HP Redstone servers are based on Calxeda's 32-bit ECX-1000 processors, which also debuted last fall, and put four processors on a card, 18 cards in a tray, and 288 nodes in a 3U space – all interconnected by the EnergyCore Fabric Switch embedded on each Calxeda chip. This interconnect can implement a 2D torus, mesh, fat tree, or butterfly tree topology and can scale across 4,096 sockets. This is just the thing for certain hyperscale web and data-munching workloads that need lots of compute cores but not the high-bandwidth interconnect that top-end Ethernet or InfiniBand switches and adapters would offer.

Cumings says that the Copper server sleds have been in development and test for the past year among its hyperscale server customers, who generally are looking for the cheapest clocks and lowest watts to cram more computing per dollar and per watt into their server racks to support various workloads.

The reason why it has taken Dell so long to get a production-grade ARM server to market has little to do with hardware – 32-bit and 40-bit ARM chips that could be used in servers have been around for a while – but rather is related to the fact that the ecosystem of software had to evolve around the machines to give them something useful to do. Even the most hyper of the hyperscale data centers does not want to roll its own Linux and software stacks unless it absolutely has to.

Dell's ARMed PowerEdge C5000 microserver chassis

Dell's ARMed PowerEdge C5000 microserver chassis

Now that Ubuntu 12.04 LTS is out with an ARM variant, there is a commercial-grade Linux and application stack you can run on an ARMv7-compatible chip. And the Fedora Project is also doing tweaks to support ARM chips, allowing those who like RHELish Linux to experiment as well.

The LAMP stack and Hadoop will run on these Linuxes on ARM processors, and the OpenStack cloud controller was demoed running on LARM (what else are we going to call it?) this month, too. The KVM hypervisor is expected to be ready soon for the Cortex-A15 ARM processor and Java also works on the chip, too. Dell is also working to get its "Crowbar" server configuration tool working on the Copper servers before the end of the year, and is working to get a variant of Crowbar that can set up Hadoop clusters out the door, too.

The important thing is not that all of this software is running or will soon be running, but that the software is getting architecture-specific optimizations as people see the hardware and software ecosystem for ARM-based servers coming into being.

"We spend a lot of time on hyperscale, and we have a pretty clear understanding of what customers want to do and where their interests lie," explains Cumings. "We have had this system in the lab for over a year, and we have been trying to figure out what it might be good for."

Initially, it looks like what you might expect: Web front ends and Hadoop data munching are the two big workloads that will deliver better performance per watt and better performance per dollar compared to X86 servers.

Cumings is not about to provide any specifics about how much better or what these things cost, but each ARM server node on the Copper sled only burns a peak of 15 watts. That's pretty low, and as low as Intel is trying to shoot with its Xeon and Atom chips.

Dell has chosen Marvell's Armada XP 78460 variant as the basis of the Copper sled. This is a quad-core system-on-chip (SoC) with four Marvell "Sheeva" PJ4B cores, which are a variant of the ARMv7-MP design that has the 40-bit memory addressing added to it. The cores run at 1.6GHz and have both symmetric multiprocessing and asymmetric multiprocessing enabled. SMP allows the cores to be ganged up into a single image behind main and cache memories, while AMP allows the cores to be carved up into isolated virtual hardware slices.

Block diagram of Marvell's Armada XP 78460 chip

Block diagram of Marvell's Armada XP 78460 chip

As you can see, the heart of the Armada XP 78460 is a system crossbar interconnect that links all of the elements of the SoC together – components that might otherwise be on a server motherboard separated from the CPUs and glued together by a chipset. Each Sheeva CPU has a floating point unit and is linked over a coherency fabric to a 2MB on-die L2 cache. The memory controller supports up to 8GB of DDR3 main memory with ECC scrubbing and running at up to 1.6GHz. There are four PCI-Express 2.0 controllers on the chip, as well as controllers to drive two SATA peripheral ports and four Gigabit Ethernet ports. There is also a 4Gb/sec packet processor that could come in very handy, as well as a security engine for encrypting and decrypting data and a controller to link to three USB 3.0 ports. That is a lot of stuff to cram into a 15 watt thermal envelope – hence the excitement about ARM servers.

Dell Copper ARM server sled

Dell's "Copper" ARM server sled (click to enlarge)

The Copper sled server, which slides into the PowerEdge C5000 chassis from the DCS unit, puts four of these Armada XP 78460 processors on a system board with four memory slots; each processor links to one slot and it has 8GB of memory in it. This board is designed by Dell; it is not clear where it is manufactured. The sled has room for four 3.5-inch SATA drives, one for each ARM server node.

Off to the right of the four processors is another processor and some other gadgetry that implements a Layer 2 Ethernet switch across all of the four nodes on the Copper sled as well as the dozen Copper sleds that can be slid into the C5000 chassis. This network can also span multiple enclosures if customers want to do it. The exact feeds and speeds of this on-board, distributed Ethernet switch are not being divulged at this time, but it was clearly designed to compete against the EnergyCore Fabric Switch embedded on each Calxeda ARM server chip. Cumings says that Dell expects for customers to use normal top-of-rack switches to link multiple Copper sled enclosures together, but they don't have to.

Cumings says that the Copper sleds have been shipping to selected seed customers for some time and that Dell will be standing up racks of the ARM-based servers in its solution centers as well as in the Texas Advanced Computing Center (TACC) at the University of Texas. The latter is a supercomputing center that has the facilities to allow secure remote access from all over the world, and Dell's plan is to let developers sign up for some timeslices on the Copper servers to give their code a whirl before they commit to buying the custom iron from DCS.

A single C5000 chassis can hold 48 ARM processors, for a total of 192 cores. That works out to 2,688 cores in a rack if you fill it top to bottom with C5000s – or 2,496 cores if you leave 3U open for top-of-rack Ethernet switches.

The HP Redstone machines can cram a lot more cores into the rack if you don't put storage on them, and still offer higher density if you put SSDs or 2.5-inch SATA disks into some of the SL6500 enclosure's trays. The issues then become the cost of each machine and the premium, if any, that HP is commanding for the Redstone density and the Calxeda interconnect. As Copper and Redstone are not shipping commercially and do not have official prices, it is hard to tell how it all plays out. ®

Broader topics


Other stories you might like

  • Florida's content-moderation law kept on ice, likely unconstitutional, court says
    So cool you're into free speech because that includes taking down misinformation

    While the US Supreme Court considers an emergency petition to reinstate a preliminary injunction against Texas' social media law HB 20, the US Eleventh Circuit Court of Appeals on Monday partially upheld a similar injunction against Florida's social media law, SB 7072.

    Both Florida and Texas last year passed laws that impose content moderation restrictions, editorial disclosure obligations, and user-data access requirements on large online social networks. The Republican governors of both states justified the laws by claiming that social media sites have been trying to censor conservative voices, an allegation that has not been supported by evidence.

    Multiple studies addressing this issue say right-wing folk aren't being censored. They have found that social media sites try to take down or block misinformation, which researchers say is more common from right-leaning sources.

    Continue reading
  • US-APAC trade deal leaves out Taiwan, military defense not ruled out
    All fun and games until the chip factories are in the crosshairs

    US President Joe Biden has heralded an Indo-Pacific trade deal signed by several nations that do not include Taiwan. At the same time, Biden warned China that America would help defend Taiwan from attack; it is home to a critical slice of the global chip industry, after all. 

    The agreement, known as the Indo-Pacific Economic Framework (IPEF), is still in its infancy, with today's announcement enabling the United States and the other 12 participating countries to begin negotiating "rules of the road that ensure [US businesses] can compete in the Indo-Pacific," the White House said. 

    Along with America, other IPEF signatories are Australia, Brunei, India, Indonesia, Japan, South Korea, Malaysia, New Zealand, the Philippines, Singapore, Thailand and Vietnam. Combined, the White House said, the 13 countries participating in the IPEF make up 40 percent of the global economy. 

    Continue reading
  • 381,000-plus Kubernetes API servers 'exposed to internet'
    Firewall isn't a made-up word from the Hackers movie, people

    A large number of servers running the Kubernetes API have been left exposed to the internet, which is not great: they're potentially vulnerable to abuse.

    Nonprofit security organization The Shadowserver Foundation recently scanned 454,729 systems hosting the popular open-source platform for managing and orchestrating containers, finding that more than 381,645 – or about 84 percent – are accessible via the internet to varying degrees thus providing a cracked door into a corporate network.

    "While this does not mean that these instances are fully open or vulnerable to an attack, it is likely that this level of access was not intended and these instances are an unnecessarily exposed attack surface," Shadowserver's team stressed in a write-up. "They also allow for information leakage on version and build."

    Continue reading
  • A peek into Gigabyte's GPU Arm for AI, HPC shops
    High-performance platform choices are going beyond the ubiquitous x86 standard

    Arm-based servers continue to gain momentum with Gigabyte Technology introducing a system based on Ampere's Altra processors paired with Nvidia A100 GPUs, aimed at demanding workloads such as AI training and high-performance compute (HPC) applications.

    The G492-PD0 runs either an Ampere Altra or Altra Max processor, the latter delivering 128 64-bit cores that are compatible with the Armv8.2 architecture.

    It supports 16 DDR4 DIMM slots, which would be enough space for up to 4TB of memory if all slots were filled with 256GB memory modules. The chassis also has space for no fewer than eight Nvidia A100 GPUs, which would make for a costly but very powerful system for those workloads that benefit from GPU acceleration.

    Continue reading
  • GitLab version 15 goes big on visibility and observability
    GitOps fans can take a spin on the free tier for pull-based deployment

    One-stop DevOps shop GitLab has announced version 15 of its platform, hot on the heels of pull-based GitOps turning up on the platform's free tier.

    Version 15.0 marks the arrival of GitLab's next major iteration and attention this time around has turned to visibility and observability – hardly surprising considering the acquisition of OpsTrace as 2021 drew to a close, as well as workflow automation, security and compliance.

    GitLab puts out monthly releases –  hitting 15.1 on June 22 –  and we spoke to the company's senior director of Product, Kenny Johnston, at the recent Kubecon EU event, about what will be added to version 15 as time goes by. During a chat with the company's senior director of Product, Kenny Johnston, at the recent Kubecon EU event, The Register was told that this was more where dollars were being invested into the product.

    Continue reading
  • To multicloud, or not: Former PayPal head of engineering weighs in
    Not everyone needs it, but those who do need to consider 3 things, says Asim Razzaq

    The push is on to get every enterprise thinking they're missing out on the next big thing if they don't adopt a multicloud strategy.

    That shove in the multicloud direction appears to be working. More than 75 percent of businesses are now using multiple cloud providers, according to Gartner. That includes some big companies, like Boeing, which recently chose to spread its bets across AWS, Google Cloud and Azure as it continues to eliminate old legacy systems. 

    There are plenty of reasons to choose to go with multiple cloud providers, but Asim Razzaq, CEO and founder at cloud cost management company Yotascale, told The Register that choosing whether or not to invest in a multicloud architecture all comes down to three things: How many different compute needs a business has, budget, and the need for redundancy. 

    Continue reading

Biting the hand that feeds IT © 1998–2022