It's a Flink-off: Hazelcast launches own big data munching distributed processing engine

Brilliant, it's about jolly time someone invented one of those

In-memory computing hopeful Hazelcast is launching Hazelcast Jet, a distributed processing engine for the streams of big data which the business expects will be pouring into enterprises soon.

Hazelcast's eponymous in-memory data grid is based on commodity hardware rather than expensive proprietary Oracle Exadata boxes, and will be providing storage functionality for Jet. With both computation and storage kept in-memory, Hazelcast hopes to achieve high speed and low latency.

The company's CEO, Greg Luck, said the business had recognised an “opportunity in the market for Hazelcast as a distributed computing platform to provide something very fast and simple, not just for programming but also for operations.”

Apache Flink-competitor Jet, which the company has licensed as per Apache 2.0, aims to perform parallel execution to enable data-intensive applications to operate in "near real-time".

Unlike Hadoop and Spark, which run from OS processes, Hazelcast Jet doesn't demand operations maintain a whole separate infrastructure. Jet can execute both batch and stream-based data processing applications, which the company believes there to be much market hunger for, especially with applications that require near real-time workloads from sensor updates in IoT architectures (house thermostats, lighting systems), through to in-store e-commerce systems as well as social media platforms.

Luck said: “Hazelcast Jet is a super fast, low latency, next generation directed acyclic graph engine for big data processing. We believe that the Hadoop and Spark ecosystems are too complex to program and to deploy and have set out to bring Hazelcast’s legendary simplicity to big data.”

“We have designed it as a general purpose engine for the intersect of big data programmers and Java programmers,” with a specific focus for existing Hazelcast users.

Hazelcast Jet is built on top of a one-record-per-time architecture (sometimes known as continuous operators), processesing incoming records as soon as possible, as opposed to accumulating records into micro-batches, consequently lowering latency for applications. Jet ingests data at high-velocity (via socket, file, HDFS or Kafka interfaces), and processes the business logic or complex computation on incoming data. ®