Serverless ETL and Analytics with AWS Glue

Build efficient data lakes that can scale at virtually unlimited size using AWS Glue

Key Features

• Learn how to work with AWS Glue to overcome typical implementation challenges in Data lakes
• Create and manage serverless ETL pipelines that manage data at the big data scale
• A practical guide, written by AWS Glue community members that teaches you how to implement AWS Glue in no time

Book Description

Organizations, these days have gravitated towards services such as AWS Glue that do the undifferentiated heavy lifting and provide serverless spark. AWS Glue helps you to create and manage a Data Lake in a serverless fashion.

This book shows you how AWS Glue can be used to solve real world problems along with learning about data processing, data integration and building data lakes. It teaches you how to perform various aspects of data analysis such as ad hoc queries, visualization and real-time analysis using AWS Glue. A walkthrough of CI/CD for AWS Glue and how to shift left on quality using automated regression tests is also covered. Further, Data security aspects such as access control, encryption, auditing, networking and so on is implemented. Some of the common techniques such as picking the right file format, compression, partitioning, bucketing which is important is also covered. It also takes you through the AWS Glue features such as crawlers, lake formation, governed tables, lineage, Databrew, glue studio, custom connectors and so on. Towards the end, it focuses on providing various troubleshooting and monitoring options.

By the end of this book, you will be able to create, manage, troubleshoot and deploy ETL pipelines using AWS Glue.

What you will learn

• Apply various AWS Glue features for managing and creating Data lakes
• Use Glue DataBrew and Glue Studio for data preparation
• Optimize data layout in cloud storage to accelerate analytic workload
• Manage metadata including database, table, and schema definitions
• Secure your data during access control, encryption, auditing, and network
• Monitor AWS Glue Job for detecting delays and loss of data
• Integrate Spark ML and Sagemaker with AWS Glue to create ML models

Who This Book Is For

This book is helpful for ETL developers, Data engineers and Data analysts who want to understand how AWS Glue can help you solve your business problems. Basic knowledge of AWS data services is assumed.

Table of Contents

1. Data Management: Introduction and Concepts
2. Introduction to Important AWS Glue Features
3. Data Ingestion
4. Data preparation
5. Data Layout
6. Data management
7. Metadata management
8. Data security
9. Data sharing
10. Data Pipeline Management
11. Monitoring
12. Tuning, Debugging, and Troubleshooting
13. Data analysis
14. Machine Learning Integration
15. Architecting data lakes for real world scenarios and edge cases