This article is more than 1 year old

Who knew? Hadoop is over, says former Hortonworks guru Scott Gnau

Teradata veteran talks up his new home at InterSystems as best of both worlds

Interview Tech history comes in waves. One minute you’re riding a crest, the next you’re wiped out. Just ask Scott Gnau, former CTO of Hortonworks, the company once seen as the figurehead of the big data boom.

Now head of data platforms at InterSystems, Gnau has had time to reflect on the Hadoop legacy and where those with torrential data workloads should go next.

“If you define Hadoop as some of the initial projects are out there, clearly, it's over, right? And the marketplace has moved on,” he told The Register.

But there is a caveat. The “broader ecosystem of open source technologies for data management” has left a valuable legacy, he said.

Just over two years ago, Gnau was promoting Hortonworks Data Platform 3.0 based on Apache Hadoop 3.1 as a "huge step forward, architecturally". But the world has changed.

Gnau joined InterSystems, a proprietary data platform vendor with strengths in the health sector, in 2019, after the merger between Hortonworks and Cloudera. Pitched as a marriage of equals, the deal valued the combined entity as $5.2bn. It now hovers around $3bn.

The demise of Hadoop, as a distributed file system, and as an investment darling, has been linked to the growing popularity of object storage, integrated with the cloud hyperscalers' data offerings in the form of AWS S3, Azure Blob Storage, and Google’s Object Storage, as they are seen as accessible and cheap.

“Clearly object store is having an impact on the entire marketplace,” Gnau said.

Indeed, Teradata, the data warehouse stalwart where Gnau spent nearly 20 years, hitched its wagon to the object storage train in 2019, saying it offered low cost, easy management, and unlimited scalability with its “native” support for the cloud repository.

Object storage

But as a person who has seen some ups and downs, he has a salutary warning for those hyping object storage as an elixir for all data ailments.

“There are advantages to object stores and there are disadvantages,” he said. “To the extent that people believe there's a single answer to all these solutions, they'll be wrong. Object storage is all the craze today, I'm sure there’ll be something that follows on after that.”

While object storage is very flexible and easy to use, it is not specialised to any particular data task, he said. “With generic solutions, you're then also riding the hardware innovation curves to deliver improvements over time. With the continued orders of magnitude increases in data available for managing transactions and analytics, I think that the supply of data will continue to outstrip the supply of money and hardware to process it. So, there will always be kind of this yin and yang between specialised [kit] and technology,” Gnau said.

At his new home, InterSystems, Gnau claims there is the ability to address both structured and unstructured data in the same system with its “unique” proprietary data platform.

Obviously, being company IP he is reluctant to spill the beans on the secret sauce, but Gnau outlined to us a schema-less system that is akin to a “massively intelligent key-value store” with some internal indexing that allows both high volume onboarding of data and efficient analytics. He said it also offers in-memory storage and - having a background of supporting medical systems requiring storage of doctors’ notes and medical images - it also supports unstructured data, Gnau said.

As it strives to broaden its appeal outside its core medical market, InterSystems has signed up Chadwicks Group, a builders’ merchant based in Ireland, to help integrate business systems and provide real-time reporting.

Newly rolled out is Adaptive Analytics, which integrates live data in InterSystems IRIS and popular third-party business intelligence tools such as Tableau, Microsoft Power BI, and Excel. The firm has also made a deal with visualisation specialist AtScale to offer business users the ability to create their own dashboards.

Having spent years at Teradata working with its massively parallel, traditional RDBMS and moving across to manage unstructured big data at Hortonworks, Gnau claims to have found the “best both of the worlds” at InterSystems.

Whether it will stay that way depends of course on how fast those worlds change and whether the proprietary system can keep up. ®

More about

More about

More about


Send us news

Other stories you might like