Data Lakehouse Platforms for Machine Learning: A Comparative Evaluation

Abstract

The data lakehouse is a modern data management architecture that combines the reliability of data warehouses with the flexibility of data lakes. It is promoted as a strong foundation for machine learning workloads, but the architecture itself can be implemented in various ways across different platforms. This thesis examines the data lakehouse architecture and its support for machine learning, and compares how different platforms approach its implementation.

This thesis combines a literature review with empirical research. Five lakehouse features most relevant to the machine learning lifecycle were identified by reviewing relevant literature: ACID transactions, time travel, schema enforcement, feature stores, and pipeline orchestration. An evaluation framework was defined to assess the features along two axes: implementation mechanism and observable behavior. The framework was applied in a case study comparing an identical machine learning pipeline implemented on an open source stack, Databricks, and Snowflake.

The case study showed that all three platforms provided the same essential data lakehouse functionality while utilizing different approaches for implementing the features. The most notable difference was related to the division of responsibility between the platform and the engineer. The managed platforms absorbed configuration and lifecycle decisions, whereas the open source implementation left them to the engineer. The choice of platform therefore depends less on which features are available than on how each platform implements them.