Delta lake: high-performance ACID table storage over cloud object stores
Delta Lake
2020-08-01
SCID: 54.1/p55fe2qu
Discuss with AI
ACID transactionsApache ParquetDelta Lakecloud object storestransaction log
Figures from the paper
Abstract (AI)
Cloud object stores such as Amazon S3 are some of the largest and most cost-effective storage systems on the planet, making them an attractive target to store large data warehouses and data lakes. Unfortunately, their implementation as key-value stores makes it difficult to achieve ACID transactions and high performance: metadata operations such as listing objects are expensive, and consistency guarantees are limited. In this paper, we present Delta Lake, an open source ACID table storage layer over cloud object stores initially developed at Databricks. Delta Lake uses a transaction log that is compacted into Apache Parquet format to provide ACID properties, time travel, and significantly faster metadata operations for large tabular datasets (e.g., the ability to quickly search billions of table partitions for those relevant to a query). It also leverages this design to provide high-level features such as automatic data layout optimization, upserts, caching, and audit logs. Delta Lake tables can be accessed from Apache Spark, Hive, Presto, Redshift and other systems. Delta Lake is deployed at thousands of Databricks customers that process exabytes of data per day, with the largest instances managing exabyte-scale datasets and billions of objects.
Key Findings
1
A compacted transaction log in Apache Parquet provides ACID properties, time travel, and substantially faster metadata operations for large tabular datasets.
2
Delta Lake is an open-source ACID table storage layer designed to add transactional reliability to cloud object stores.
3
Delta Lake supports automatic data-layout optimization, upserts, caching, and audit logs, and integrates with multiple query systems.
4
The design enables rapid identification of relevant partitions among billions, reducing the impact of expensive object-store listing operations.
5
The technology is deployed by thousands of Databricks customers processing exabytes of data daily, with deployments managing exabyte-scale datasets and billions of objects.
Research Object
Delta Lake tables as large tabular datasets stored in cloud object stores
Research Subject
ACID transactions, metadata performance, time travel, and data-management capabilities of the storage layer
Publication Details
Publication Date
2020-08-01
Journal
Publisher
ISSN
Cited by
578
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai2
Cited by9
Data Lakes: A Survey of Functions and Systems2023
Data Lakehouse: A survey and experimental study2024
The Lakehouse: State of the Art on Concepts and Technologies2024
An Empirical Evaluation of Columnar Storage Formats2023
BigLake: BigQuery's Evolution toward a Multi-Cloud Lakehouse2024
The evolution of data storage architectures: examining the secure value of the Data Lakehouse2024
GeoLake: Bringing Geospatial Support to Lakehouses2023
Data Lakehouse: Next Generation Information System2024
The HANA Native Query Engine for Lakehouse Systems2025