HPC

Microsoft claims it has spun up a top-five AI supercomputer for its pals at OpenAI – but won't reveal the full specs

This big beast has more than '285,000 CPU cores, 10,000 GPUs'


Build Microsoft has spun up a custom AI supercomputer in its Azure cloud for OpenAI that ranks within the top five publicly known most powerful supers on Earth, it is claimed.

The behemoth is said to contain “more than 285,000 CPU cores and 10,000 GPUs,” with up to 400Gbps of network connectivity for each GPU server.

If this beast is in the top five, that would mean – judging from the latest list of the world's most powerful publicly known supercomputers – it is somewhere between the 450,000-core 38-PFLOPS Frontera system at the University of Texas, and Uncle Sam's 2.4-million-core 200-PFLOPS Summit.

We have a feeling it's somewhere around fifth place and fourth, which is China's 4.9-million-core 100PFLOPS Tianhe-2A, but we're just guessing.

Microsoft's brag was made during its annual Build conference, a virtual developer-focused event taking place this week.

AI research is computationally intensive, and OpenAI needs copious amounts of compute to its train giant machine-learning models from massive blocks of data. Some of its most ambitious projects include GPT-2, a text-generating system with up to a billion parameters that required 256 Google TPU3 cores to train it on 40GB of text scraped from Reddit; and the OpenAI Five, a bot capable of playing Dota 2, which needed more than 128,000 CPU cores and 256 Nvidia P100 GPUs to school.

Microsoft is researching cryogenic quantum computing to solve the world's problems

Google: We've achieved quantum supremacy! IBM: Nope. And stop using that word, please

READ MORE

“As we’ve learned more and more about what we need and the different limits of all the components that make up a supercomputer, we were really able to say, ‘If we could design our dream system, what would it look like?” said OpenAI CEO Sam Altman. “And then Microsoft was able to build it.”

It’s difficult to know how brawny the cluster really is, however, since the Windows giant declined to disclose any further technical details beyond the above numbers. When pressed, a spokesperson told The Register: “We were not able to provide the system’s benchmark processing speed, only that it would rank among the top five on the TOP500 list of the fastest supercomputers in the world,” adding that it had “no information to share on types [of chips] used at the moment.” OpenAI also said it had “no additional details to share beyond that at this time.”

The custom Azure-hosted cluster allocated exclusively for OpenAI is part Microsoft’s $1bn investment in the San Francisco-based research lab. Last year, it pledged to support OpenAI’s efforts to build “a hardware and software platform within Microsoft Azure which will scale to artificial general intelligence." As part of the deal, OpenAI promised to make MICROS~1 its “preferred partner,” if and when it decides to commercialize any of its machine-learning projects.

Microsoft said its partnership with OpenAI was a “first step toward making the next generation of very large AI models and the infrastructure needed to train them available as a platform for other organizations and developers to build upon.”

The Windows giant is particularly interested in natural language models that it believes will improve search, and generate and summarize text. Earlier this year, it built Microsoft Turing, the world’s largest language model with a whopping 17 billion parameters. The model has been used to boost Microsoft’s Bing, Office, and Dynamics software, the US super-corp said. Now, it’s planning to open source its various Microsoft Turing models along with instructions on how to train it all for specific tasks in its Azure cloud platform.

“The exciting thing about these models is the breadth of things they’re going to enable,” gushed Microsoft Chief Technical Officer Kevin Scott.

“This is about being able to do a hundred exciting things in natural language processing at once and a hundred exciting things in computer vision, and when you start to see combinations of these perceptual domains, you’re going to have new applications that are hard to even imagine right now."

Other flashy AI announcements from Microsoft Build include updated versions of DeepSpeed and the ONNX Runtime. Deepspeed is a Python library that speeds up the process of training large machine-learning models across multiple compute nodes. ONNX Runtime is similar, and it’s more flexible in that it can optimize models for training and inference for a number of machine-learning frameworks. ®

Updated to add

We understand the supercomputer uses AMD second-generation Rome Epyc server processors, and Nvidia V100 GPUs, from sources in the industry.

Broader topics


Other stories you might like

  • North Korea pulled in $400m in cryptocurrency heists last year – report

    Plus: FIFA 22 players lose their identity and Texas gets phony QR codes

    In brief Thieves operating for the North Korean government made off with almost $400m in digicash last year in a concerted attack to steal and launder as much currency as they could.

    A report from blockchain biz Chainalysis found that attackers were going after investment houses and currency exchanges in a bid to purloin funds and send them back to the Glorious Leader's coffers. They then use mixing software to make masses of micropayments to new wallets, before consolidating them all again into a new account and moving the funds.

    Bitcoin used to be a top target but Ether is now the most stolen currency, say the researchers, accounting for 58 per cent of the funds filched. Bitcoin accounted for just 20 per cent, a fall of more than 50 per cent since 2019 - although part of the reason might be that they are now so valuable people are taking more care with them.

    Continue reading
  • Tesla Full Self-Driving videos prompt California's DMV to rethink policy on accidents

    Plus: AI systems can identify different chess players by their moves and more

    In brief California’s Department of Motor Vehicles said it’s “revisiting” its opinion of whether Tesla’s so-called Full Self-Driving feature needs more oversight after a series of videos demonstrate how the technology can be dangerous.

    “Recent software updates, videos showing dangerous use of that technology, open investigations by the National Highway Traffic Safety Administration, and the opinions of other experts in this space,” have made the DMV think twice about Tesla, according to a letter sent to California’s Senator Lena Gonzalez (D-Long Beach), chair of the Senate’s transportation committee, and first reported by the LA Times.

    Tesla isn’t required to report the number of crashes to California’s DMV unlike other self-driving car companies like Waymo or Cruise because it operates at lower levels of autonomy and requires human supervision. But that may change after videos like drivers having to take over to avoid accidentally swerving into pedestrians crossing the road or failing to detect a truck in the middle of the road continue circulating.

    Continue reading
  • Alien life on Super-Earth can survive longer than us due to long-lasting protection from cosmic rays

    Laser experiments show their magnetic fields shielding their surfaces from radiation last longer

    Life on Super-Earths may have more time to develop and evolve, thanks to their long-lasting magnetic fields protecting them against harmful cosmic rays, according to new research published in Science.

    Space is a hazardous environment. Streams of charged particles traveling at the speed of light, ejected from stars and distant galaxies, bombard planets. The intense radiation can strip atmospheres and cause oceans on planetary surfaces to dry up over time, leaving them arid and incapable of supporting habitable life. Cosmic rays, however, are deflected away from Earth, however, since it’s shielded by its magnetic field.

    Now, a team of researchers led by the Lawrence Livermore National Laboratory (LLNL) believe that Super-Earths - planets that are more massive than Earth but less than Neptune - may have magnetic fields too. Their defensive bubbles, in fact, are estimated to stay intact for longer than the one around Earth, meaning life on their surfaces will have more time to develop and survive.

    Continue reading

Biting the hand that feeds IT © 1998–2022