Explore how Delta brings reliability, performance, and governance to your data lake and all the AI and BI use cases built on top of it
Key Features
- Learn Delta's core concepts, features and what makes it a perfect match for data engineering and analysis
- Using a scenario-based approach, solve business challenges of different industry verticals
- Make optimal choices by understanding the various tradeoffs provided by Delta
Book Description
Delta helps you generate reliable insights at scale and simplifies architecture around data pipelines, allowing you to focus primarily on refining the use cases being worked upon. This is especially important considering the same architecture is reused when onboarding new use cases.
In this book, you'll learn the principles of distributed computing, data modeling techniques, big data design patterns and templates that help solve end-to-end data flow problems for common scenarios and are reusable across use cases and industry verticals. You'll also learn how to recover from errors and the best practices around handling structured, semi-structured, and unstructured data using Delta. Next, you'll get to grips with features such as ACID transactions on big data, disciplined schema evolution, time travel to help rewind a dataset to a different time or version, and unified batch and streaming capabilities that will help you build agile and robust data products.
By the end of this Delta book, you'll be able to use Delta as the foundational block for creating analytics-ready data that fuels all AI/BI use cases.
What you will learn
- Explore the key challenges of traditional data lakes
- Appreciate the unique features of Delta that come out of the box
- Address reliability, performance, and governance concerns using Delta
- Analyze the open data format for an extensible and pluggable architecture
- Handle multiple use cases to support BI, AI, streaming, and data discovery
- Discover how common data and ML design patterns are executed on Delta
- Build and deploy data and ML pipelines at scale leveraging Delta
Who This Book Is For
Individuals in the data domain such as data engineers, data scientists, ML practitioners and BI analysts working with big data will be able to put their knowledge to work with this practical guide to executing pipelines and supporting diverse use cases using the Delta protocol. Basic knowledge of SQL, Python programming, and Spark is required to get the most out of this book.
Table of Contents
- An Introduction to Data Engineering
- Data Modeling and ETL
- Delta - The Foundation Block for Big Data
- Unifying Batch and Streaming with Delta
- Data Consolidation in Delta Lake
- Solving Common Data Pattern Scenarios with Delta
- Delta for Data Warehouse Use Cases
- Handling Different Data Situations
- Delta for Reproducible Machine Learning Pipelines
- Operationalizing Delta Pipelines
- Data As a Service
- Optimizing for Cost and Performance with Delta
- Data Governance with Delta