Azure Data Engineering Cookbook

Over 90 recipes to help you collect and transform data from multiple sources into a single data source which makes it easier to perform analytics on the data

Key Features

• Build data pipelines from scratch and find solutions to common problems in data engineering lifecycle
• Learn how to work with Data Factory, Azure Data Lake, Azure Databricks and Azure Synapse Analytics
• Monitor and troubleshoot your data engineering pipelines using log analytics and Azure monitor

Book Description

Data is the new oil and gaining maximum insights out of data is extremely critical for an organization's success. Building performant data engineering pipelines to ingest, store, process and visualize data is one of the major challenges organizations face in leveraging value out of data.

This book shares 90 useful recipes covering common scenarios in building data engineering pipelines in Azure. The book, a second edition of the immensely successful first edition written by Ahmad Osama, covers several recent enhancements in Azure data engineering.

This edition explores recipes from Azure Synapse Analytics workspaces - gen 2, covering topics like Synapse spark pools, SQL Serverless pools, Synapse integration pipelines and synapse data flows. The book also dives deep into Synapse SQL Pool optimization techniques in the second edition.

Besides Synapse enhancements, it covers building the semantic and visualization layer using Power BI and establishing connectivity of Databricks and Synapse pools with Power BI.

Finally, the book covers overall data engineering pipeline management focusing on areas like tracking impact and data lineage using Azure Purview.

By the end of this book, it will serve as your go-to guide in building excellent data engineering pipelines.

What you will learn

• Perform data ingestion and orchestration using Azure Data Factory
• Move data from on-premise sources to Azure using Data Factory Integration Runtime
• Process your raw data using Azure Databricks and Azure Synapse
• Perform data orchestration and ETL tasks using Azure Synapse analytics
• Implement high availability and monitor performance of Azure SQL Database
• Build effective Synapse SQL pools which can be consumed by Power BI
• Monitor the performance of Synapse SQL and Spark pools using log analytics

Who This Book Is For

This book is targeted at Data engineers, Data architects, database administrators and data professionals who want to get well versed with the Azure data services for building data pipelines. Basic understanding of cloud and data engineering concepts would be beneficial.

Table of Contents

1. Creating and managing data in Azure Data Lake / Azure Blob storage
2. Securing and Monitoring Data in Azure Data Lake
3. Building data ingestion pipelines using Azure Data Factory
4. Configuring Azure Data Factory Integration Runtime
5. Configuring and Securing Azure SQL Database
6. Implementing High Availability and monitoring in Azure SQL Database
7. Processing data using Azure Databricks
8. Processing data using Azure Synapse Analytics
9. Transforming data using Azure Synapse Pipelines
10. Building the serving layer in Azure Synapse SQL Pool
11. Monitoring Synapse SQL and Spark Pools
12. Optimizing and Maintaining Synapse SQL and Spark Pools
13. Monitoring and Maintaining end to end Azure Data engineering pipelines