Zoom in on HDS' Hyper Scale-out Platform kit

Hyper-converged box avoids the NameNode bottleneck


The Hitachi Hyper Scale-Out Platform – announced on Tuesday – is a single system for ingesting, accessing and analyzing big data-type information. Its architecture differs from typical Hadoop systems in that it avoids NameNode resource contention.

The hardware in the 2U rack-enclosure HSP 400 node is this:

  • Two Intel Xeon E5-2620 2GHz processors with 15M of cache.
  • 192GB of RAM.
  • 12 x 3.5” 4TB SAS 7,200rpm HDD (48TB per node).
  • 2 x 2.5” 300GB SAS 10,000rpm HDD for app or OS.
  • 40GbitE dual-port QSFP NIC for internode backend network.
  • 10GbitE single-port SFP + NIC for host connectivity.
  • Brocade ICX 7750-26Q switch.

Twenty nodes will fit in a rack, and nodes are scaled out by being clustered together, starting with a minimum of five nodes. Each node supports compute, a virtual infrastructure, and storage – it being hyper-converged in that sense.

Like hyper-converged infrastructure appliances, capacity and performance scale as nodes are added. Both Hadoop and other types of analytics can run in HSP nodes and access their data.

HSP does not ship with a Hadoop distribution. Instead it has an HDS filesystem, which is object-based with a file front-end, that supports the Hadoop distributed file system (HDFS) via the HDFS API, and this has been certified by Hortonworks.

The HDS filesystem also provides a POSIX-compliant file system interface "that allows other third-party data analytics applications to directly access the data" without it having to be exported to their systems.

There is no CIFS support (it isn't a NAS) and it can support Docker. It can also be bought with a full Hadoop distribution, but it still ships with the HDS filesystem inside it.

It supports "many of the same principals of Hadoop (64MB data chunk size, three-copy policy, data location API)" but is able to avoid the NameNode bottleneck by distributing metadata across the nodes.

There is thus, HDS says, no single point of failure or contention. Data is automatically distributed across nodes in the system, and there is distributed locking.

Every node can serve data without centralized metadata management, and HSP accesses the data via standard POSIX semantics through its distributed file system.

Hadoop analytics jobs work with the HDFS API; others use POSIX, and both have in-place data access.

The nodes run Linux and can run the hypervisor KVM; so starting up analytics is equivalent to spinning up a virtual machine and, hey presto, analytics compute has been brought to the data. In fact Pentaho software can run inside such VMs.

HDS senior veep Sean Moser says Docker containers will be able to run on the HSP's bare-metal. He also says 1,000 nodes have been physically connected in tests, but that was not a final product. So that doesn't mean it supports 1,000 x 48TB = 48,000 TB, meaning 48PB.

The nodes can host different VMs for different big data analytics needs. Network traffic is reduced as data transfers across the network to analytics servers, and a separate analytics data silo, is not needed.

There are connectors to archive systems for archiving of data when it's no longer needed on the analytics platform.

HDS says HSP is self-configuring, self-managing, and self-healing. There is a RESTful API, and Hadoop management tools are supported. It is cloud-ready in that it "provides APIs for OpenStack Glance (image services), Nova (cloud computing fabric controller) and Swift (object storage) projects to enable users to deploy infrastructure as a service (IaaS) functionality."

Customers have pulled HDS along with this; they've been experiencing it for two years. Moser claims the pull on it is phenomenal. ®

Similar topics


Other stories you might like

  • Google keeps legacy G Suite alive and free for personal use
    Phew!

    Google has quietly dropped its demand that users of its free G Suite legacy edition cough up to continue enjoying custom email domains and cloudy productivity tools.

    This story starts in 2006 with the launch of “Google Apps for Your Domain”, a bundle of services that included email, a calendar, Google Talk, and a website building tool. Beta users were offered the service at no cost, complete with the ability to use a custom domain if users let Google handle their MX record.

    The service evolved over the years and added more services, and in 2020 Google rebranded its online productivity offering as “Workspace”. Beta users got most of the updated offerings at no cost.

    Continue reading
  • GNU Compiler Collection adds support for China's LoongArch CPU family
    MIPS...ish is on the march in the Middle Kingdom

    Version 12.1 of the GNU Compiler Collection (GCC) was released this month, and among its many changes is support for China's LoongArch processor architecture.

    The announcement of the release is here; the LoongArch port was accepted as recently as March.

    China's Academy of Sciences developed a family of MIPS-compatible microprocessors in the early 2000s. In 2010 the tech was spun out into a company callled Loongson Technology which today markets silicon under the brand "Godson". The company bills itself as working to develop technology that secures China and underpins its ability to innovate, a reflection of Beijing's believe that home-grown CPU architectures are critical to the nation's future.

    Continue reading
  • China’s COVID lockdowns bite e-commerce players
    CEO of e-tail market leader JD perhaps boldly points out wider economic impact of zero-virus stance

    The CEO of China’s top e-commerce company, JD, has pointed out the economic impact of China’s current COVID-19 lockdowns - and the news is not good.

    Speaking on the company’s Q1 2022 earnings call, JD Retail CEO Lei Xu said that the first two years of the COVID-19 pandemic had brought positive effects for many Chinese e-tailers as buyer behaviour shifted to online purchases.

    But Lei said the current lengthy and strict lockdowns in Shanghai and Beijing, plus shorter restrictions in other large cities, have started to bite all online businesses as well as their real-world counterparts.

    Continue reading
  • Foxconn forms JV to build chip fab in Malaysia
    Can't say when, where, nor price tag. Has promised 40k wafers a month at between 28nm and 40nm

    Taiwanese contract manufacturer to the stars Foxconn is to build a chip fabrication plant in Malaysia.

    The planned factory will emit 12-inch wafers, with process nodes ranging from 28 to 40nm, and will have a capacity of 40,000 wafers a month. By way of comparison, semiconductor-centric analyst house IC Insights rates global wafer capacity at 21 million a month, and Taiwanese TSMC’s four “gigafabs” can each crank out 250,000 wafers a month.

    In terms of production volume and technology, this Malaysian facility will not therefore catapult Foxconn into the ranks of leading chipmakers.

    Continue reading
  • NASA's InSight doomed as Mars dust coats solar panels
    The little lander that couldn't (any longer)

    The Martian InSight lander will no longer be able to function within months as dust continues to pile up on its solar panels, starving it of energy, NASA reported on Tuesday.

    Launched from Earth in 2018, the six-metre-wide machine's mission was sent to study the Red Planet below its surface. InSight is armed with a range of instruments, including a robotic arm, seismometer, and a soil temperature sensor. Astronomers figured the data would help them understand how the rocky cores of planets in the Solar System formed and evolved over time.

    "InSight has transformed our understanding of the interiors of rocky planets and set the stage for future missions," Lori Glaze, director of NASA's Planetary Science Division, said in a statement. "We can apply what we've learned about Mars' inner structure to Earth, the Moon, Venus, and even rocky planets in other solar systems."

    Continue reading
  • The ‘substantial contributions’ Intel has promised to boost RISC-V adoption
    With the benefit of maybe revitalizing the x86 giant’s foundry business

    Analysis Here's something that would have seemed outlandish only a few years ago: to help fuel Intel's future growth, the x86 giant has vowed to do what it can to make the open-source RISC-V ISA worthy of widespread adoption.

    In a presentation, an Intel representative shared some details of how the chipmaker plans to contribute to RISC-V as part of its bet that the instruction set architecture will fuel growth for its revitalized contract chip manufacturing business.

    While Intel invested in RISC-V chip designer SiFive in 2018, the semiconductor titan's intentions with RISC-V evolved last year when it revealed that the contract manufacturing business key to its comeback, Intel Foundry Services, would be willing to make chips compatible with x86, Arm, and RISC-V ISAs. The chipmaker then announced in February it joined RISC-V International, the ISA's governing body, and launched a $1 billion innovation fund that will support chip designers, including those making RISC-V components.

    Continue reading

Biting the hand that feeds IT © 1998–2022