Introduction xxvii
Part I Azure Data Engineer Certification and Azure Products 1
Chapter 1 Gaining the Azure Data Engineer Associate Certification 3
The Journey to Certification 7
How to Pass Exam DP- 203 8
Understanding the Exam Expectations and Requirements 9
Use Azure Daily 17
Read Azure Articles to Stay Current 17
Have an Understanding of All Azure Products 20
Azure Product Name Recognition 21
Azure Data Analytics 23
Azure Synapse Analytics 23
Azure Databricks 26
Azure HDInsight 28
Azure Analysis Services 30
Azure Data Factory 31
Azure Event Hubs 33
Azure Stream Analytics 34
Other Products 35
Azure Storage Products 36
Azure Data Lake Storage 37
Azure Storage 40
Other Products 42
Azure Databases 43
Azure Cosmos DB 43
Azure SQL Server Products 46
Additional Azure Databases 46
Other Products 47
Azure Security 48
Azure Active Directory 48
Role- Based Access Control 51
Attribute- Based Access Control 53
Azure Key Vault 53
Other Products 55
Azure Networking 56
Virtual Networks 56
Other Products 59
Azure Compute 59
Azure Virtual Machines 59
Azure Virtual Machine Scale Sets 60
Azure App Service Web Apps 60
Azure Functions 60
Azure Batch 60
Azure Management and Governance 60
Azure Monitor 61
Azure Purview 61
Azure Policy 62
Azure Blueprints (Preview) 62
Azure Lighthouse 62
Azure Cost Management and Billing 62
Other Products 63
Summary 64
Exam Essentials 64
Review Questions 66
Chapter 2 CREATE DATABASE dbName; GO 69
The Brainjammer 70
A Historical Look at Data 71
Variety 73
Velocity 74
Volume 74
Data Locations 74
Data File Formats 75
Data Structures, Types, and Concepts 83
Data Structures 83
Data Types and Management 92
Data Concepts 95
Data Programming and Querying for Data Engineers 125
Data Programming 126
Querying Data 143
Understanding Big Data Processing 169
Big Data Stages 169
Etl, Elt, Eltl 174
Analytics Types 175
Big Data Layers 176
Summary 177
Exam Essentials 177
Review Questions 179
Part II Design and Implement Data Storage 181
Chapter 3 Data Sources and Ingestion 183
Where Does Data Come From? 185
Design a Data Storage Structure 189
Design an Azure Data Lake Solution 190
Recommended File Types for Storage 198
Recommended File Types for Analytical Queries 199
Design for Efficient Querying 200
Design for Data Pruning 203
Design a Folder Structure That Represents the Levels of Data Transformation 203
Design a Distribution Strategy 205
Design a Data Archiving Solution 206
Design a Partition Strategy 207
Design a Partition Strategy for Files 209
Design a Partition Strategy for Analytical Workloads 210
Design a Partition Strategy for Efficiency and Performance 211
Design a Partition Strategy for Azure Synapse Analytics 211
Identify When Partitioning Is Needed in Azure Data Lake Storage Gen 2 212
Design the Serving/Data Exploration Layer 213
Design Star Schemas 214
Design Slowly Changing Dimensions 215
Design a Dimensional Hierarchy 219
Design a Solution for Temporal Data 220
Design for Incremental Loading 222
Design Analytical Stores 223
Design Metastores in Azure Synapse Analytics and Azure Databricks 224
The Ingestion of Data into a Pipeline 228
Azure Synapse Analytics 228
Azure Data Factory 268
Azure Databricks 275
Event Hubs and IoT Hub 301
Azure Stream Analytics 303
Apache Kafka for HDInsight 314
Migrating and Moving Data 316
Summary 317
Exam Essentials 317
Review Questions 319
Chapter 4 The Storage of Data 321
Implement Physical Data Storage Structures 322
Implement Compression 322
Implement Partitioning 325
Implement Sharding 328
Implement Different Table Geometries with Azure Synapse Analytics Pools 329
Implement Data Redundancy 331
Implement Distributions 341
Implement Data Archiving 342
Azure Synapse Analytics Develop Hub 346
Implement Logical Data Structures 360
Build a Temporal Data Solution 361
Build a Slowly Changing Dimension 365
Build a Logical Folder Structure 368
Build External Tables 369
Implement File and Folder Structures for Efficient Querying and Data Pruning 372
Implement a Partition Strategy 375
Implement a Partition Strategy for Files 376
Implement a Partition Strategy for Analytical Workloads 377
Implement a Partition Strategy for Streaming Workloads 378
Implement a Partition Strategy for Azure Synapse Analytics 378
Design and Implement the Data Exploration Layer 379
Deliver Data in a Relational Star Schema 379
Deliver Data in Parquet Files 385
Maintain Metadata 386
Implement a Dimensional Hierarchy 386
Create and Execute Queries by Using a Compute Solution That Leverages SQL Serverless and Spark Cluster 388
Recommend Azure Synapse Analytics Database Templates 389
Implement Azure Synapse Analytics Database Templates 389
Additional Data Storage Topics 390
Storing Raw Data in Azure Databricks for Transformation 390
Storing Data Using Azure HDInsight 392
Storing Prepared, Trained, and Modeled Data 393
Summary 394
Exam Essentials 395
Review Questions 396
Part III Develop Data Processing 399
Chapter 5 Transform, Manage, and Prepare Data 401
Chapter 6 Ingest and Transform Data 402
Transform Data Using Azure Synapse Pipelines 404
Transform Data Using Azure Data Factory 410
Transform Data Using Apache Spark 414
Transform Data Using Transact- SQL 429
Transform Data Using Stream Analytics 431
Cleanse Data 433
Split Data 435
Shred JSON 439
Encode and Decode Data 445
Configure Error Handling for the Transformation 450
Normalize and Denormalize Values 451
Transform Data by Using Scala 461
Perform Exploratory Data Analysis 463
Transformation and Data Management Concepts 473
Transformation 473
Data Management 480
Azure Databricks 481
Data Modeling and Usage 485
Data Modeling with Machine Learning 486
Usage 494
Summary 500
Exam Essentials 500
Review Questions 502
Create and Manage Batch Processing and Pipelines 505
Design and Develop a Batch Processing Solution 507
Design a Batch Processing Solution 510
Develop Batch Processing Solutions 512
Create Data Pipelines 538
Handle Duplicate Data 560
Handle Missing Data 569
Handle Late- Arriving Data 571
Upsert Data 572
Configure the Batch Size 578
Configure Batch Retention 581
Design and Develop Slowly Changing Dimensions 582
Design and Implement Incremental Data Loads 583
Integrate Jupyter/IPython Notebooks into a Data Pipeline 590
Chapter 7 Revert Data to a Previous State 591
Handle Security and Compliance Requirements 592
Design and Create Tests for Data Pipelines 593
Scale Resources 593
Design and Configure Exception Handling 593
Debug Spark Jobs Using the Spark UI 594
Implement Azure Synapse Link and Query the Replicated Data 594
Use PolyBase to Load Data to a SQL Pool 595
Read from and Write to a Delta Table 595
Manage Batches and Pipelines 596
Trigger Batches 597
Schedule Data Pipelines 597
Validate Batch Loads 598
Implement Version Control for Pipeline Artifacts 604
Manage Data Pipelines 607
Manage Spark Jobs in a Pipeline 609
Handle Failed Batch Loads 610
Summary 610
Exam Essentials 611
Review Questions 612
Design and Implement a Data Stream Processing Solution 615
Develop a Stream Processing Solution 617
Design a Stream Processing Solution 618
Create a Stream Processing Solution 630
Process Time Series Data 657
Design and Create Windowed Aggregates 658
Process Data Within One Partition 661
Process Data Across Partitions 663
Upsert Data 665
Handle Schema Drift 674
Configure Checkpoints/Watermarking During Processing 680
Replay Archived Stream Data 685
Design and Create Tests for Data Pipelines 688
Monitor for Performance and Functional Regressions 689
Optimize Pipelines for Analytical or Transactional Purposes 689
Scale Resources 690
Design and Configure Exception Handling 691
Handle Interruptions 694
Ingest and Transform Data 694
Transform Data Using Azure Stream Analytics 694
Monitor Data Storage and Data Processing 695
Monitor Stream Processing 695
Summary 695
Exam Essentials 696
Review Questions 697
Part IV Secure, Monitor, and Optimize Data Storage and Data Processing 699
Chapter 8 Keeping Data Safe and Secure 701
Design Security for Data Policies and Standards 702
Design a Data Auditing Strategy 711
Design a Data Retention Policy 716
Design for Data Privacy 717
Design to Purge Data Based on Business Requirements 719
Design Data Encryption for Data at Rest and in Transit 719
Design Row- Level and Column- Level Security 722
Design a Data Masking Strategy 723
Design Access Control for Azure Data Lake Storage Gen 2 724
Implement Data Security 730
Implement a Data Auditing Strategy 731
Manage Sensitive Information 739
Implement a Data Retention Policy 745
Encrypt Data at Rest and in Motion 748
Implement Row- Level and Column- Level Security 749
Implement Data Masking 753
Manage Identities, Keys, and Secrets Across Different Data Platform Technologies 755
Implement Access Control for Azure Data Lake Storage Gen 2 765
Implement Secure Endpoints (Private and Public) 772
Implement Resource Tokens in Azure Databricks 778
Load a DataFrame with Sensitive Information 779
Write Encrypted Data to Tables or Parquet Files 780
Develop a Batch Processing Solution 781
Handle Security and Compliance Requirements 782
Design and Implement the Data Exploration Layer 784
Browse and Search Metadata in Microsoft Purview Data Catalog 784
Push New or Updated Data Lineage to Microsoft Purview 785
Summary 786
Exam Essentials 787
Review Questions 789
Chapter 9 Monitoring Azure Data Storage and Processing 791
Monitoring Data Storage and Data Processing 793
Implement Logging Used by Azure Monitor 793
Configure Monitoring Services 799
Understand Custom Logging Options 821
Measure Query Performance 822
Monitor Data Pipeline Performance 823
Monitor Cluster Performance 824
Measure Performance of Data Movement 824
Interpret Azure Monitor Metrics and Logs 825
Monitor and Update Statistics about Data Across a System 828
Schedule and Monitor Pipeline Tests 830
Interpret a Spark Directed Acyclic Graph 830
Monitor Stream Processing 832
Implement a Pipeline Alert Strategy 832
Develop a Batch Processing Solution 832
Design and Create Tests for Data Pipelines 832
Develop a Stream Processing Solution 837
Monitor for Performance and Functional Regressions 837
Design and Create Tests for Data Pipelines 838
Azure Monitoring Overview 841
Azure Batch 841
Azure Key Vault 842
Azure SQL 843
Summary 844
Exam Essentials 844
Review Questions 846
Chapter 10 Troubleshoot Data Storage Processing 849
Optimize and Troubleshoot Data Storage and Data Processing 851
Optimize Resource Management 854
Compact Small Files 857
Handle Skew in Data 859
Handle Data Spill 860
Find Shuffling in a Pipeline 862
Tune Shuffle Partitions 864
Tune Queries by Using Indexers 869
Tune Queries by Using Cache 876
Optimize Pipelines for Analytical or Transactional Purposes 877
Optimize Pipeline for Descriptive versus Analytical Workloads 886
Troubleshoot a Failed Spark Job 888
Troubleshoot a Failed Pipeline Run 890
Rewrite User- Defined Functions 899
Design and Develop a Batch Processing Solution 901
Design and Configure Exception Handling 902
Debug Spark Jobs by Using the Spark UI 902
Scale Resources 902
Monitor Batches and Pipelines 904
Handle Failed Batch Loads 904
Design and Develop a Stream Processing Solution 905
Optimize Pipelines for Analytical or Transactional Purposes 905
Handle Interruptions 906
Scale Resources 908
Summary 909
Exam Essentials 910
Review Questions 912
Appendix Answers to Review Questions 915
Chapter 1: Gaining the Azure Data Engineer Associate Certification 916
Chapter 2: CREATE DATABASE dbName; GO 916
Chapter 3: Data Sources and Ingestion 917
Chapter 4: The Storage of Data 918
Chapter 5: Transform, Manage, and Prepare Data 918
Chapter 6. Create and Manage Batch Processing and Pipelines 919
Chapter 7: Design and Implement a Data Stream Processing Solution 920
Chapter 8: Keeping Data Safe and Secure 921
Chapter 9: Monitoring Azure Data Storage and Processing 921
Chapter 10: Troubleshoot Data Storage Processing 922
Index 925
