HPC

Yahoo! invites world of boffins into 4,000-node Hadoop cluster

Hello! Yahoo! is! a! tech! company!


Yahoo! has opened up its Hadoop research cluster to computer science boffins at four additional US universities, including Stanford, the University of Washington, the University of Michigan, and Purdue.

The company's M45 cluster — a Hadoop setup spanning 4,000 processors and 1.5-petabyte of disk space inside a data center at Yahoo!'s Sunnyvale headquarters —– was originally launched in 2007, and is now available for Big Data research at eight universities across the country. Other participants include founding member Carnegie Mellon, plus the The University of California at Berkeley, Cornell University, and the University of Massachusetts at Amherst.

Speaking with The Reg, Thomas Kwan, director of research operations for Yahoo! Research, reaffirmed the company's commitment to Hadoop research, saying that the M45 continues not only to drive development of the open source distributed number-crunching platform, but to serve as a means of educating future Yahoo! employees.

Though the M45 is a godsend for Big Data boffins, it's a rather large inconvenience for the mainstream media, which insists on telling the world that Yahoo! is not a tech company. It can be difficult to explain why a non-tech company is funding a 4,000-processor and 1.5-petabyte data cluster that crunches epic amounts of data using an open source distributed-computing platform that now underpins everything from Facebook to Twitter to a portion of Microsoft Bing.

Hadoop — whose development was largely driven by Yahoo! — is based on the GFS and MapReduce distributed-computing platforms developed at Google. In 2004, Mountain View published a pair of papers on these technologies, and soon, Doug Cutting — known previously for developing Lucene, the open-source retrieval library — used them to start an open source shadow project. He called it Hadoop after his son's stuffed elephant.

Today, the M45 drives such projects as the Carnegie Mellon natural language–processing system known as NELL (Never Ending Language Learning System) and the University of Washington's sweeping knowledge database project KnowItAll. The cluster is only available to universities. ®

Similar topics

Broader topics


Other stories you might like

  • Tesla driver charged with vehicular manslaughter after deadly Autopilot crash

    Prosecution seems to be first of its kind in America

    A Tesla driver has seemingly become the first person in the US to be charged with vehicular manslaughter for a deadly crash in which the vehicle's Autopilot mode was engaged.

    According to the cops, the driver exited a highway in his Tesla Model S, ran a red light, and smashed into a Honda Civic at an intersection in Gardena, Los Angeles County, in late 2019. A man and woman in the second car were killed. The Tesla driver and a passenger survived and were taken to hospital.

    Prosecutors in California charged Kevin George Aziz Riad, 27, in October last year though details of the case are only just emerging, according to AP on Tuesday. Riad, a limousine service driver, is facing two counts of vehicular manslaughter, and is free on bail after pleading not guilty.

    Continue reading
  • AMD returns to smartphone graphics with new Samsung chip for your pocket computer

    We're back in black

    AMD's GPU technology is returning to mobile handsets with Samsung's Exynos 2200 system-on-chip, which was announced on Tuesday.

    The Exynos 2200 processor, fabricated using a 4nm process, has Armv9 CPU cores and the oddly named Xclipse GPU, which is an adaptation of AMD's RDNA 2 mainstream GPU architecture.

    AMD was in the handheld GPU market until 2009, when it sold the Imageon GPU and handheld business for $65m to Qualcomm, which turned the tech into the Adreno GPU for its Snapdragon family. AMD's Imageon processors were used in devices from Motorola, Panasonic, Palm and others making Windows Mobile handsets.

    Continue reading
  • Big shock: Guy who fled political violence and became rich in tech now struggles to care about political violence

    'I recognize that I come across as lacking empathy,' billionaire VC admits

    Billionaire tech investor and ex-Facebook senior executive Chamath Palihapitiya was publicly blasted after he said nobody really cares about the reported human rights abuse of Uyghur Muslims in China.

    The blunt comments were made during the latest episode of All-In, a podcast in which Palihapitiya chats to investors and entrepreneurs Jason Calacanis, David Sacks, and David Friedberg about technology.

    The group were debating the Biden administration’s response to what's said to be China's crackdown of Uyghur Muslims when Palihapitiya interrupted and said: “Nobody cares about what’s happening to the Uyghurs, okay? ... I’m telling you a very hard ugly truth, okay? Of all the things that I care about … yes, it is below my line.”

    Continue reading

Biting the hand that feeds IT © 1998–2022