BigInsight, a distribution of Hadoop, is to be dropped by IBM as they are building their own Hadoop platform. They are adopting the Hortonwork INC’s two-way deal that could give users of both companies increased access to enterprise-class capabilities for managing and analyzing big data.

They are working towards migrating their users to the Hortonworks Data Platform (HDP). In return, Hortonworks will resell IBM’s Data Science Experience suite of tools for collaborative analytics, as well as Big SQL, an SQL-on-Hadoop query engine developed by IBM. The two companies will also do joint development to expand the features of Apache Atlas, an open source data governance framework spearheaded by Hortonworks.

By using advanced machine learning and deep learning capabilities, the combination of Hortonworks Data Platform with IBM’s Data Science Experience and the IBM Machine Learning platform can help clients achieve improved analytic results faster and at scale.

IBM and Hortonworks Expand Partnership for Better Analytics

• Hortonworks will resell the IBM Data Science Experience with HDP, a leading Hadoop distribution, and adopt it as its strategic data science platform, giving developers a fast on-ramp to data science capabilities including machine learning, advanced analytics, and statistics. Also, Hortonworks and IBM will create new solution bundles that integrate HDP with IBM Big SQL, IBM’s SQL engine for Hadoop, giving Hortonworks’ legions of clients and users a familiar method of managing their data.
• IBM is adopting HDP for its Hadoop distribution and will fully integrate it with Data Science Experience and Machine Learning. As a result, this solution will combine for users the rich data security, governance and operations functionality provided by HDP, and the advanced analytics and management of the Data Science Experience. IBM will migrate existing IBM Big Insights users to HDP.

They will also be working on Apache Spark, the open source framework for processing and analyzing large data sets across clustered environments.  They will also be working on making the Apache Hadoop framework much more advanced. They also plan on unifying vendors and heterogeneous data environments across data warehouses and databases – ultimately aiming to simplify the environment for better value from all data.

Beyond gaining IBM’s existing customers, Hortonworks will benefit from Big Blue’s vastly larger sales force and longstanding relationships in large enterprise accounts. Notably, BigSQL and DSX won’t part of Hortonworks’ HDP, which is open source. Rather, they’ll be optional bundles, ones that provide important new capabilities for HDP.

Machine Learning

Hortonwork is currently focusing on AI development. The heightened focus on products like DSX and Cloudera’s rival Data Science Workbench aligns with the growing use of machine learning. Machine learning fueled by data science techniques could also enable more proactive and personalized customer service, Pressley said during a user panel discussion.

The huge majority of Hadoop adopters that are currently stalled in attempting to get to broad production use will need to deal with those issues when and if they break the logjam.

As part of the partnership, the companies will also work to advance the development of Unified Governance (IBM BigIntegrate, IBM BigQuality, and IBM Information Governance Catalog) on the Apache Atlas open platform. Through their work, both companies plan to help advance Atlas from its current Incubator status to Apache Top Level Project status, where projects are typically released for open development and deployment.

Duke Energy Corp. is looking to use machine learning and AI tools to identify potential equipment problems by analyzing sensor data from its transmission network, said John Pressley, director of information architecture at the electric utility and natural gas distributor in Charlotte, N.C.

IBM declined to disclose the number of existing BigInsights users that would be affected by the deal.

IBM’s embrace of HDP will increase the size of the Hortonworks customer thus giving Microsoft a bigger target base for moving workloads to the cloud.

For Hortonworks’ own users, DSX offers a more enterprise-ready workbench environment for data scientists than the Apache Zeppelin notebook tool that the company has been offering, Engesser said. The same goes for Big SQL vs. Apache Hive, the SQL query engine that Hortonworks has backed to date.

Microsoft is also in the Hortonworks camp, having built its cloud-based Azure HDInsight managed service for big data users on top of HDP. A Microsoft executive who spoke during the opening keynote session at the DataWorks Summit didn’t mention the Hortonworks-IBM deal or HDInsight; a spokesman for Microsoft later said the company had no comment on the new agreement.

There’s still room for Hadoop, but it now gets a ton of criticisms that it never had to face before, and has a lot of easily-accessible competition.

In addition, it continues the consolidation of the Hadoop market, which is now down to four vendors: Hortonworks and fellow big data specialists Cloudera and MapR Technologies, plus cloud-platform market leader Amazon Web Services.