Linux Foundation projects on AI and data merge – because one of these concepts simply can't exist without the other
Combined org hoping to attract more industry support
Open Source Summit Europe LF AI Foundation and Open Data Platform Initiative (ODPi), until now two orgs under the Linux Foundation umbrella, are merging to become the LF AI and Data Foundation, a home for projects devoted to AI, machine learning (ML), data, and analytics.
The new foundation was unveiled at the Open Source Summit Europe, which is of course virtual-only (though held in timezones friendly to European folks), and taking place this week.
"We've been having an increasing number of data projects join LF AI," said executive director Dr Ibrahim Haddad in a press briefing. "It's the nature of AI, one is dependent on the other."
Amazon hosts a project with us. Microsoft hosts a project with us. Uber hosts three projects with us, and none of them are a member
The LF AI Foundation was set up to address issues like fragmentation, lack of integration, and governance across open-source AI projects, Haddad told us. "We started two-and-a-half years ago with a single project," he said, "and today we are hosting 22 projects."
Several projects cover one of the hottest topics in AI: fairness and explainability.
The Linux AI Foundation Landscape puts its projects into context at https://landscape.lfai.foundation/ (click to enlarge)
On the ODPi side, program director John Mertic said the organisation's activity has shifted since it was formed in 2015 with a focus on Hadoop. It has devoted a lot of attention to Egeria, a standard for exchanging metadata between products and databases.
The core issue is that while different products and projects often store similar entities – such as names, addresses, and social security numbers – they may store them in different ways so exchanging data or applying compliance and security rules across a whole organisation is problematic. There are many existing approaches, but Mertic said Egeria's code-first outlook is distinctive.
Is Egeria winning adoption? "It's being brought into IBM products, it's on the roadmap for SAS, some end users are looking at it," he told us. "It's up and coming. Combining forces with LFAI and creating a larger community will only increase that."
ODPi currently lists seven members including IBM, Cloudera, and SAS. LF AI shows 11 premier members, 14 general members, and 12 associate members. Premier members include AT&T, Baidu, Huawei, Ericsson, IBM, SAS, and Tencent. Some big names, but we noticed that they do not include well-known companies deeply involved in AI and ML, such as Google, AWS, Microsoft, and Nvidia, and that some of the biggest open-source projects in AI and ML, such as Facebook's PyTorch and Google's TensorFlow, are not LF AI projects. Does this make LF AI, and the new merged organisation, less representative than it should be?
There are some 10 companies to be added, said Haddad, but it also felt like we touched a nerve. "There are gaps, but from my perspective they are financial gaps," he said.
"Amazon hosts a project with us. Microsoft hosts a project with us. Uber hosts three projects with us, and none of them are a member. That goes to the specific open-source strategy at these companies and how they see their investment.
"However, from a technical perspective, I don't think there is a gap because all of these companies are using or integrating some of our projects in their cloud offerings, but they are not a member. It's a very interesting question and I would rather you ask that question to those companies – why are you not a member of LF AI although you host a project there?"
It does seem that merging these two entities is in part about trying to increase the scope and clout of the foundation in order to attract more support from some of these giant players with deep pockets. "It is a natural fit," said Mertic, "and the justification for companies like Google becomes a heck of a lot easier." ®