Data Lakehouse: A survey and experimental study

Озеро данных Lakehouse: обзор и экспериментальное исследование
Ahmed A. Harby, Farhana H. Zulkernine
2024-09-26

Analytical query performanceBig data managementData lakeData lakehouseData warehouse
Efficient big data management is a dire necessity to manage the exponential growth in data generated by digital information systems to produce usable knowledge. Structured databases, data lakes, and warehouses have each provided a solution with varying degrees of success. However, a new and superior solution, the data Lakehouse, has emerged to extract actionable insights from unstructured data ingested from distributed sources. By combining the strengths of data warehouses and data lakes, the data Lakehouse can process and merge data quickly while ingesting and storing high-speed unstructured data with post-storage transformation and analytics capabilities. The Lakehouse architecture offers the necessary features for optimal functionality and has gained significant attention in the big data management research community. In this paper, we compare data lake, warehouse, and lakehouse systems, highlight their strengths and shortcomings, identify the desired features to handle the evolving challenges in big data management and analysis and propose an advanced data Lakehouse architecture. We also demonstrate the performance of three state-of-the-art data management systems namely HDFS data lake, Hive data warehouse, and Delta lakehouse in managing data for analytical query responses through an experimental study.
1
An experimental evaluation compares HDFS data lake, Hive data warehouse, and Delta lakehouse systems for analytical query-response data management performance.
2
Data lakehouses combine warehouse-style analytics with lake-style ingestion and storage of high-speed unstructured data from distributed sources.
3
The lakehouse architecture supports rapid data processing and merging, post-storage transformation, and analytics over heterogeneous data.
4
The paper surveys and compares data lakes, data warehouses, and data lakehouses, emphasizing their respective strengths and shortcomings for big data management.
5
The study identifies desired capabilities for addressing evolving big data management and analysis challenges and proposes an advanced lakehouse architecture.

data Lakehouse systems and their comparison with data lakes and data warehouses

big-data management and analytical-query performance, including data ingestion, storage, transformation, analytics, and system strengths and shortcomings

Publication Details
Publication Date
2024-09-26
Journal
Publisher
ISSN
Cited by
76
Access Type
Author Information
Authors
Ahmed A. Harby
Farhana H. Zulkernine
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%