Data streaming company Confluent has unveiled a major expansion of its partnership with Databricks, bringing together Confluent’s complete Data Streaming Platform and Databricks’ Data Intelligence Platform to empower enterprises with real-time data for AI-driven decision-making. New integrations between Confluent’s Tableflow and Databricks Unity Catalog will seamlessly govern data across operational and analytical systems, allowing businesses to build AI applications more securely and efficiently.

Enterprises are rapidly building AI applications that require reliable, real-time data for better decisions and customer experiences. Yet, only 22 percent of enterprises are confident their current IT infrastructure is able to support these new AI applications. One of the major remaining hurdles they need to overcome is the divide between operational systems where data is generated and analytical systems where that data is turned into insights. Because these tools exist in separate silos, different teams, tools, and processes are applied between the two. The result is that teams are unable to marry real-time data with other systems in a meaningful way, and AI innovation on advanced use cases becomes impossible.

“For companies to maximise returns on their AI investments, they need their data, AI, analytics and governance all in one place,” said Ali Ghodsi, co-founder and CEO, Databricks. “As we help more organisations build data intelligence, trusted enterprise data sits at the centre. We are excited that Confluent has embraced Unity Catalog and Delta Lake as its open governance and storage solutions of choice, and we look forward to working together to deliver long-term value for our customers.”

“Real-time data is the fuel for AI,” said Jay Kreps, co-founder and CEO, Confluent. “But too often, enterprises are held back by disconnected systems that fail to deliver the data they need, in the format they need, at the moment they need it. Together with Databricks, we’re ensuring businesses can harness the power of real-time data to build sophisticated AI-driven applications for their most critical use cases.”

To bridge the divide, Confluent and Databricks are announcing new integrations to ensure real-time interoperability and empower teams across the business to collaborate successfully.  A bidirectional integration between Confluent’s Tableflow with Delta Lake and Databricks’ Unity Catalog, a unified and open governance solution for data and AI, will provide consistent, real-time data across operational and analytical systems that is discoverable, secure, and trustworthy. 

Delta Lake, an open-format storage layer pioneered by Databricks, was originally developed for streaming use cases with fast writes. It has become the most adopted lakehouse format, proven out at a massive scale – processing over 10 exabytes of data daily. Now, Tableflow with Delta Lake makes operational data available immediately to Delta Lake’s rich ecosystem. Confluent and Databricks customers will be able to bring any engine or AI tool such as Apache Spark, Trino, Polars, DuckDB and Daft to their data in Unity Catalog.

Custom integrations between Tableflow and Databricks’ Unity Catalog will also ensure metadata – a critical enabler for AI applications – is automatically applied to data exchanged between platforms. This makes operational data discoverable and actionable for data scientists and analysts working in Databricks while ensuring analytical data is equally accessible and useful for application developers and streaming engineers in Confluent. Additionally, Confluent’s Stream Governance suite will provide upstream governance and metadata to enhance fine-grained governance, end-to-end stream lineage, and automated data quality monitoring in Unity Catalog.

“Leveraging proximity to generation sources is a key factor not just in the energy sector, but also in the field of data,” said Dr. Dora Simroth, Head of Data and AI Engineering, E.ON Digital Technology. “Confluent and Databricks are already essential technologies in our Data and AI stack. These integrations will allow our practitioners to work on a single source of well-defined and timely data for both our operational and analytical plane. By partnering, Confluent and Databricks open up a faster path for us to build data and model-centric digital solutions.”

With these new capabilities, operational data from Confluent becomes a first-class citizen in Databricks, and Databricks data is easily accessible by any processor in the enterprise. The streaming data topics that AI applications consume and the tables that data analysts use will now have consistent views of the same real-time data, enabling faster, smarter AI-driven decision-making across the organisation. This seamless integration between enterprise applications, analytics, and governance is critical for AI innovation at scale.