Simplify and Scale Data Engineering Pipelines with Delta Lake
3 likes2,418 views
The document outlines a presentation by Amanda Moran from Databricks on simplifying data engineering pipelines with Delta Lake. It discusses the challenges faced by data engineers and how Delta Lake addresses these issues by unifying batch and streaming data processing, offering features like snapshot isolation, time travel, and the ability to handle late arriving data. The presentation also includes a demo and practical guidance on using Delta Lake with Spark APIs.
Simplify and Scale Data Engineering Pipelines with Delta Lake
- 1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
- 2. Amanda Moran, Databricks Simplify and Scale Data Engineering Pipelines with Delta Lake #UnifiedDataAnalytics #SparkAISummit
- 3. ● Solutions Architect @ Databricks ● MS Computer Science, BS Biology ● Previously: HP, Teradata, DataStax, Esgyn ● PMC and Apache Committer on Apache Trafodion ● 5 Different Distributed Systems ● Course with Udacity on Data Engineering Today’s Speaker
- 4. Agenda ● Data Engineers Nightmares and Dreams ● Data Lifecycle vs the Delta Lifecycle ● Transitioning Data Pipeline to Delta ● How Dreams Become True ● DEMO! ● How to use Delta
- 5. Table (Data gets written continuously) AI & Reporting Events Batch Stream Stream The Data Engineer’s Journey…
- 6. Table (Data gets written continuously) AI & Reporting Events Batch Batch Stream Reprocessing Update & Merge Stream Table (Data gets compacted every hour) The Data Engineer’s Journey…
- 7. Table (Data gets written continuously) AI & Reporting Events Batch Batch Stream Unified View Validation Updates & Merge get complex with data lake Reprocessing Update & Merge Stream Table (Data gets compacted every hour) The Data Engineer’s Journey… into a Nightmare
- 8. Table (Data gets written continuously) AI & Reporting Events Batch Batch Stream Unified ViewValidation Updates & Merge get complex with data lake Reprocessing Update & Merge Stream Table (Data gets compacted every hour) The Data Engineer’s Journey… into a Nightmare Can this be simplified?
- 9. A Data Engineer’s Dream... Data Lake CSV, JSON, TXT… Kinesis AI & Reporting Process data continuously and incrementally as new data arrive in a cost efficient way without having to choose between batch or streaming
- 10. What’s missing? 1. Ability to read consistent data while data is being written 2. Ability to read incrementally from a large table with good throughput 3. Ability to rollback in case of bad writes 4. Ability to replay historical data along new data that arrived 5. Ability to handle late arriving data without having to delay downstream processing Data Lake CSV, JSON, TXT… Kinesis AI & Reporting ?
- 11. So… What is the answer? STRUCTURED STREAMING + = The Delta Architecture 1. Unify batch & streaming with a continuous data flow model 2. Infinite retention to replay/reprocess historical events as needed 3. Independent, elastic compute and storage to scale while balancing costs
- 12. Let’s try it instead with
- 13. Data Lake AI & Reporting Streaming Analytics Business-level Aggregates Filtered, Cleaned Augmented Raw Ingestion The Bronze Silver Gold CSV, JSON, TXT… Kinesis Quality Delta Lake allows you to incrementally improve the quality of your data until it is ready for consumption. *Data Quality Levels *
- 14. What does this remind you of?
- 15. Data Lake AI & Reporting Streaming Analytics Business-level Aggregates Filtered, Cleaned Augmented Raw Ingestion The Data Lifecycle of the Past Bronze Silver Gold CSV, JSON, TXT… Kinesis Quality
- 16. Data Lake AI & Reporting Streaming Analytics Business-level Aggregates Filtered, Cleaned Augmented Raw Ingestion The Data Lifecycle of the Past Bronze Silver Gold CSV, JSON, TXT… Kinesis Quality Data Lake
- 17. Data Lake AI & Reporting Streaming Analytics Business-level Aggregates Filtered, Cleaned Augmented Raw Ingestion The Data Lifecycle Bronze Silver Gold CSV, JSON, TXT… Kinesis Quality Data Lake Apache Spark
- 18. Data Lake AI & Reporting Streaming Analytics Business-level Aggregates Filtered, Cleaned Augmented Raw Ingestion The Data Lifecycle Bronze Silver Gold CSV, JSON, TXT… Kinesis Quality Data Lake Apache Spark DW/OLAP
- 19. Transitioning from the Data Lifecycle to the Delta Lake Lifecycle
- 20. Data Lake AI & Reporting Streaming Analytics Business-level Aggregates Filtered, Cleaned Augmented Raw Ingestion The Bronze Silver Gold CSV, JSON, TXT… Kinesis Quality Delta Lake allows you to incrementally improve the quality of your data until it is ready for consumption. *Data Quality Levels *
- 21. Data Lake AI & Reporting Streaming Analytics Business-level Aggregates Filtered, Cleaned Augmented Raw Ingestion The Bronze Silver Gold CSV, JSON, TXT… Kinesis •Dumping ground for raw data •Often with long retention (years) •Avoid error-prone parsing ��
- 22. Data Lake AI & Reporting Streaming Analytics Business-level Aggregates Filtered, Cleaned Augmented Raw Ingestion The Bronze Silver Gold CSV, JSON, TXT… Kinesis Intermediate data with some cleanup applied. Queryable for easy debugging!
- 23. Data Lake AI & Reporting Streaming Analytics Business-level Aggregates Filtered, Cleaned Augmented Raw Ingestion The Bronze Silver Gold CSV, JSON, TXT… Kinesis Clean data, ready for consumption. Read with Spark or Presto*
- 24. Data Lake AI & Reporting Streaming Analytics Business-level Aggregates Filtered, Cleaned Augmented Raw Ingestion The Bronze Silver Gold CSV, JSON, TXT… Kinesis Streams move data through the Delta Lake •Low-latency or manually triggered •Eliminates management of schedules and jobs
- 25. Data Lake AI & Reporting Streaming Analytics Business-level Aggregates Filtered, Cleaned Augmented Raw Ingestion The Bronze Silver Gold CSV, JSON, TXT… Kinesis Delta Lake also supports batch jobs and standard DML UPDATE DELETE MERGE OVERWRITE • GDPR, CCPA • Upserts INSERT
- 26. Data Lake AI & Reporting Streaming Analytics Business-level Aggregates Filtered, Cleaned Augmented Raw Ingestion The Bronze Silver Gold CSV, JSON, TXT… Kinesis Easy to recompute when business logic changes: • Clear tables • Restart streams DELETE DELETE
- 27. How the Dream Becomes True
- 28. Demo Time
- 29. Connecting the dots... Data Lake CSV, JSON, TXT… Kinesis AI & Reporting ?
- 30. Connecting the dots... Snapshot isolation between writers and readers Data Lake CSV, JSON, TXT… Kinesis AI & Reporting ? 1. Ability to read consistent data while data is being written
- 31. Snapshot isolation between writers and readers Optimized file source with scalable metadata handling Connecting the dots... Data Lake CSV, JSON, TXT… Kinesis AI & Reporting ? 1. Ability to read consistent data while data is being written 2. Ability to read incrementally from a large table with good throughput
- 32. Snapshot isolation between writers and readers Optimized file source with scalable metadata handling Time travel Connecting the dots... Data Lake CSV, JSON, TXT… Kinesis AI & Reporting ? 1. Ability to read consistent data while data is being written 2. Ability to read incrementally from a large table with good throughput 3. Ability to rollback in case of bad writes
- 33. Snapshot isolation between writers and readers Optimized file source with scalable metadata handling Time travel Stream the backfilled historical data through the same pipeline Connecting the dots... Data Lake CSV, JSON, TXT… Kinesis AI & Reporting ? 1. Ability to read consistent data while data is being written 2. Ability to read incrementally from a large table with good throughput 3. Ability to rollback in case of bad writes 4. Ability to replay historical data along new data that arrived
- 34. 1. Ability to read consistent data while data is being written 2. Ability to read incrementally from a large table with good throughput 3. Ability to rollback in case of bad writes 4. Ability to replay historical data along new data that arrived 5. Ability to handle late arriving data without having to delay downstream processing Snapshot isolation between writers and readers Optimized file source with scalable metadata handling Time travel Stream the backfilled historical data through the same pipeline Stream any late arriving data added to the table as they get added Connecting the dots... Data Lake CSV, JSON, TXT… Kinesis AI & Reporting ?
- 35. 1. Ability to read consistent data while data is being written 2. Ability to read incrementally from a large table with good throughput 3. Ability to rollback in case of bad writes 4. Ability to replay historical data along new data that arrived 5. Ability to handle late arriving data without having to delay downstream processing Snapshot isolation between writers and readers Optimized file source with scalable metadata handling Time travel Stream the backfilled historical data through the same pipeline Stream any late arriving data added to the table as they get added Connecting the dots... Data Lake CSV, JSON, TXT… Kinesis AI & Reporting
- 36. How do I use ?
- 37. dataframe .write .format("delta") .save("/data") Get Started with Delta using Spark APIs dataframe .write .format("parquet") .save("/data") Instead of parquet... … simply say delta Add Spark Package pyspark --packages io.delta:delta-core_2.12:0.1.0 bin/spark-shell --packages io.delta:delta-core_2.12:0.1.0 Maven
- 38. Build your own Delta Lake at https://delta.io
- 40. Notebook from Today Try the notebook from Databricks Community Edition! Download the notebook at https://dbricks.co/dlw-01
- 41. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT