Delta lake: high-performance ACID table storage over cloud object stores

Delta Lake
Matei Zaharia, Xiao Li, Michael Armbrust, Ali Ghodsi, Peter A. Boncz, Adrian M. Ionescu, Reynold Xin, Tathagata Das, Mostafa Mokhtar, Liwen Sun, Burak Yavuz, Shixiong Zhu, M Krishna Murthy, Joseph Torres, Herman van Hövell, Alicja Łuszczak, Michał Świtakowski, Michał Szafrański, Takuya Ueshin, Sameer Paranjpye, Pieter Senster, Adrian Ionescu, Takuya Ueshin, Ali Ghodsi
2020-08-01

ACID transactionsApache ParquetDelta Lakecloud object storestransaction log
Cloud object stores such as Amazon S3 are some of the largest and most cost-effective storage systems on the planet, making them an attractive target to store large data warehouses and data lakes. Unfortunately, their implementation as key-value stores makes it difficult to achieve ACID transactions and high performance: metadata operations such as listing objects are expensive, and consistency guarantees are limited. In this paper, we present Delta Lake, an open source ACID table storage layer over cloud object stores initially developed at Databricks. Delta Lake uses a transaction log that is compacted into Apache Parquet format to provide ACID properties, time travel, and significantly faster metadata operations for large tabular datasets (e.g., the ability to quickly search billions of table partitions for those relevant to a query). It also leverages this design to provide high-level features such as automatic data layout optimization, upserts, caching, and audit logs. Delta Lake tables can be accessed from Apache Spark, Hive, Presto, Redshift and other systems. Delta Lake is deployed at thousands of Databricks customers that process exabytes of data per day, with the largest instances managing exabyte-scale datasets and billions of objects.
1
A compacted transaction log in Apache Parquet provides ACID properties, time travel, and substantially faster metadata operations for large tabular datasets.
2
Delta Lake is an open-source ACID table storage layer designed to add transactional reliability to cloud object stores.
3
Delta Lake supports automatic data-layout optimization, upserts, caching, and audit logs, and integrates with multiple query systems.
4
The design enables rapid identification of relevant partitions among billions, reducing the impact of expensive object-store listing operations.
5
The technology is deployed by thousands of Databricks customers processing exabytes of data daily, with deployments managing exabyte-scale datasets and billions of objects.

Delta Lake tables as large tabular datasets stored in cloud object stores

ACID transactions, metadata performance, time travel, and data-management capabilities of the storage layer

Publication Details
Publication Date
2020-08-01
Journal
Publisher
ISSN
Cited by
578
Access Type
Author Information
Authors
Matei Zaharia
Xiao Li
Michael Armbrust
Ali Ghodsi
Peter A. Boncz
Adrian M. Ionescu
Reynold Xin
Tathagata Das
Mostafa Mokhtar
Liwen Sun
Burak Yavuz
Shixiong Zhu
M Krishna Murthy
Joseph Torres
Herman van Hövell
Alicja Łuszczak
Michał Świtakowski
Michał Szafrański
Takuya Ueshin
Sameer Paranjpye
Pieter Senster
Adrian Ionescu
Takuya Ueshin
Ali Ghodsi
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%