Mellanox FDR InfiniBand pushes PCI-Express 3.0 to the limits

10GE, 40GE can't keep up with IB

ISC 2012 If you want to try to choke a PCI-Express 3.0 peripheral slot, you have to bring a fire hose. And that is precisely what InfiniBand and Ethernet switch and adapter card maker Mellanox Technology has done with a new Connect-IB server adapter.

Mellanox was on hand at the International Super Computing event in Hamburg, Germany this week, showing off its latest 56Gb/sec FDR InfiniBand wares and boasting of the uptake in InfiniBand technology in the Top 500 rankings of supercomputers and its general uptake in database cluster, data analytics, clustered storage arrays, and other segments of the systems racket.

Mellanox is the dominant supplier now that QLogic has sold off its InfiniBand biz to Intel, and it is milking the fact that it has FDR switches and adapters in the field when QLogic is still at 40Gb/sec QDR InfiniBand. (QLogic, if it had not been eaten by Intel, would counter that it can get the same or better performance from its QDR gear than Mellanox delivers with its FDR gear.) These are good days for Mellanox, which ate rival Voltaire to get into the Ethernet racket and which is enjoying the benefits of the rise of high-speed clusters.

At least until Intel comes back at Mellanox in a big way, pursuing all of its own OEM partners with the Xeon-QLogic-Fulcrum-Cray Aries quadruple whammy. Intel did not buy QLogic, Fulcrum Microsystems, and the Cray supercomputer interconnect business to sit on these assets, like some kind of knickknacks sitting on shelf.

Intel is going to try to become a supplier of supercomputing interconnects that do all kinds of things and that hook into its Xeon processors and chipsets tightly and seamlessly, and that will eventually make it very tough for Mellanox.

But not so at ISC this year. As El Reg previously reported, for the first time in the history of the Top 500 rankings of supercomputers, InfiniBand has edged out Ethernet, with 208 machines using InfiniBand and 207 using Ethernet. Drilling down into the data a bit, there were 195 machines that used Gigabit Ethernet switches and adapters to link server nodes together, and another 12 that used 10 Gigabit Ethernet.

There are still 78 machines on the list that use earlier InfiniBand gear, but there are 110 machines using QDR InfiniBand, and 20 machines that use FDR InfiniBand. There are a few hybrid interconnects as well on the list that mix InfiniBand with some other network.

The remainder are a mix of custom interconnects like the Cray "SeaStar" XT and "Gemini" XE routers, the Silicon Graphics NUMAlink, IBM's BlueGene/Q, Fujitsu's "Tofu," and a few others. Gigabit Ethernet is by far the most popular of any single speed or type, of course, but it is dramatic how InfiniBand has really blunted the uptake of 10GE networks at the top end of supercomputer clusters. The idea seems to be that if you are going to spend money on anything faster than Gigabit Ethernet, then you might as well skip 10GE or even 40GE and get the benefits of QDR or FDR InfiniBand.

This is certainly what Mellanox is hoping customers do, and that is why it is bragging about a new server adapter card called Connect-IB that can push two full-speed FDR ports.

The Connect-IB dual-port InfiniBand FDR adapter

The Connect-IB dual-port InfiniBand FDR adapter card (click to enlarge)

This new Connect-IB card, which is sampling now, will be available for both PCI-Express 3.0 and PCI-Express 2.0 slots, and eats an x16 slot. Up until now, network adapter cards have generally been x8 slots, with half as many lanes of traffic and therefore a lot less theoretical and realized bandwidth available to let the network chat up the servers. By moving to servers that support PCI-Express 3.0 slots, you can put two FDR ports on each adapter using an x16 slot and still run them at up to 100Gb/sec aggregate across the two ports.

If your server is using older PCI-Express 2.0 slots – and at this point, that means anything that is not using an Intel Xeon E5-2400, E5-2600, E5-4600, or E3-1200 v2 processor since no other server processor maker is supporting PCI-Express 3.0 yet – then there is an x16 Connect-IB card that has one port that you can try to push all the way up to 56Gb/sec speeds.

These new cards have a single microsecond MPI ping latency and support Remote Direct Memory Access (RDMA), which is one of the core technologies that gives InfiniBand its performance edge over Ethernet and which allows for servers to reach across the network directly into each other's main memory without going through that pesky operating system stack. Mellanox says the new two-port Connect-IB card can push 130 million messages per second – four times that of its competitor. (That presumably means you, QLogic, er, Intel.)

There is also a single and dual-port option on the Connect-IB cards that slide into x8 slots. It is not clear how much data these x8 slots can really push, and until they are tested in the field, Mellanox is probably not even sure.

In theory, an x8 slot running at PCI-Express 3.0 speeds should be able to do 8GB/sec (that's bytes, not bits) of bandwidth in both directions, for a total of 16GB/sec of total bandwidth across that x8 link. This should not saturate the x8 link.

What is certain is that an x8 slot running at PCI-Express 2.0 speeds could not really handle FDR InfiniBand, with only 64Gb/sec of bandwidth (8GB/sec) each way available. That was getting too close to the ceiling.

Now, the ConnectX chips on the Mellanox adapters as well as the SwitchX ASICs at the heart of its switches swing both ways, Ethernet and InfiniBand, so don't jump to the wrong conclusion and think Mellanox doesn't love Ethernet.

The company was peddling its 40GE adapters and switches, which support RDMA over Converged Ethernet (RoCE) and which give many of the benefits of InfiniBand to customers who don't want to build mixed InfiniBand-Ethernet networks. (Or, perhaps more precisely, they want Mellanox to do it inside of the switch and inside of the adapter cards and mask the transformation from the network.) Mellanox says that it is showing up to an 80 per cent application performance boost using its 40GE end-to-end compared to 10GE networks on clusters.

In addition, Mellanox also announced that the latest FDR InfiniBand adapters will also support Nvidia's GPUDirect protocol, which is a kind of RDMA for the Tesla GPU coprocessors that allowed GPUs inside of a single machine to access each other's memory without going through the CPU and OS stack to do it.

With the current Tesla K10 and future Tesla K20 GPU coprocessors, GPUDirect will allow for coprocessors anywhere in a cluster to access the memory of any other coprocessor, fulfilling Nvidia's dream of not really needing the CPU for much at all. This GPUDirect support will be fully enabled in Mellanox FDR adapters. ®

Other stories you might like

  • How ICE became a $2.8b domestic surveillance agency
    Your US tax dollars at work

    The US Immigration and Customs Enforcement (ICE) agency has spent about $2.8 billion over the past 14 years on a massive surveillance "dragnet" that uses big data and facial-recognition technology to secretly spy on most Americans, according to a report from Georgetown Law's Center on Privacy and Technology.

    The research took two years and included "hundreds" of Freedom of Information Act requests, along with reviews of ICE's contracting and procurement records. It details how ICE surveillance spending jumped from about $71 million annually in 2008 to about $388 million per year as of 2021. The network it has purchased with this $2.8 billion means that "ICE now operates as a domestic surveillance agency" and its methods cross "legal and ethical lines," the report concludes.

    ICE did not respond to The Register's request for comment.

    Continue reading
  • Fully automated AI networks less than 5 years away, reckons Juniper CEO
    You robot kids, get off my LAN

    AI will completely automate the network within five years, Juniper CEO Rami Rahim boasted during the company’s Global Summit this week.

    “I truly believe that just as there is this need today for a self-driving automobile, the future is around a self-driving network where humans literally have to do nothing,” he said. “It's probably weird for people to hear the CEO of a networking company say that… but that's exactly what we should be wishing for.”

    Rahim believes AI-driven automation is the latest phase in computer networking’s evolution, which began with the rise of TCP/IP and the internet, was accelerated by faster and more efficient silicon, and then made manageable by advances in software.

    Continue reading
  • Pictured: Sagittarius A*, the supermassive black hole at the center of the Milky Way
    We speak to scientists involved in historic first snap – and no, this isn't the M87*

    Astronomers have captured a clear image of the gigantic supermassive black hole at the center of our galaxy for the first time.

    Sagittarius A*, or Sgr A* for short, is 27,000 light-years from Earth. Scientists knew for a while there was a mysterious object in the constellation of Sagittarius emitting strong radio waves, though it wasn't really discovered until the 1970s. Although astronomers managed to characterize some of the object's properties, experts weren't quite sure what exactly they were looking at.

    Years later, in 2020, the Nobel Prize in physics was awarded to a pair of scientists, who mathematically proved the object must be a supermassive black hole. Now, their work has been experimentally verified in the form of the first-ever snap of Sgr A*, captured by more than 300 researchers working across 80 institutions in the Event Horizon Telescope Collaboration. 

    Continue reading
  • Shopping for malware: $260 gets you a password stealer. $90 for a crypto-miner...
    We take a look at low, low subscription prices – not that we want to give anyone any ideas

    A Tor-hidden website dubbed the Eternity Project is offering a toolkit of malware, including ransomware, worms, and – coming soon – distributed denial-of-service programs, at low prices.

    According to researchers at cyber-intelligence outfit Cyble, the Eternity site's operators also have a channel on Telegram, where they provide videos detailing features and functions of the Windows malware. Once bought, it's up to the buyer how victims' computers are infected; we'll leave that to your imagination.

    The Telegram channel has about 500 subscribers, Team Cyble documented this week. Once someone decides to purchase of one or more of Eternity's malware components, they have the option to customize the final binary executable for whatever crimes they want to commit.

    Continue reading
  • Ukrainian crook jailed in US for selling thousands of stolen login credentials
    Touting info on 6,700 compromised systems will get you four years behind bars

    A Ukrainian man has been sentenced to four years in a US federal prison for selling on a dark-web marketplace stolen login credentials for more than 6,700 compromised servers.

    Glib Oleksandr Ivanov-Tolpintsev, 28, was arrested by Polish authorities in Korczowa, Poland, on October 3, 2020, and extradited to America. He pleaded guilty on February 22, and was sentenced on Thursday in a Florida federal district court. The court also ordered Ivanov-Tolpintsev, of Chernivtsi, Ukraine, to forfeit his ill-gotten gains of $82,648 from the credential theft scheme.

    The prosecution's documents [PDF] detail an unnamed, dark-web marketplace on which usernames and passwords along with personal data, including more than 330,000 dates of birth and social security numbers belonging to US residents, were bought and sold illegally.

    Continue reading

Biting the hand that feeds IT © 1998–2022