Fundamentals of Data Observability

Preface
Overview of the Book
Who Should Read This Book
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments

I. Introducing Data Observability

1. Introducing Data Observability

Scaling Data Teams
Challenges of Scaling Data Teams
Segregated Roles and Responsibilities and Organizational Complexity
Anatomy of Data Issues and Consequences
Impact of Data Issues on Data Team Dynamics
Scaling AI Roadblocks
Challenges with Current Data Management Practices
Effects of Data Governance at Scale
Data Observability to the Rescue
The Areas of Observability
How Data Teams Can Leverage Data Observability Now
Low Latency Data Issues Detection
Efficient Data Issues Troubleshooting
Preventing Data Issues
Decentralized Data Quality Management
Complementing Existing Data Governance Capabilities
The Future and Beyond
Conclusion

2. Components of Data Observability

Channels of Data Observability Information
Logs
Traces
Metrics
Observations Model
Physical Space
Server
User
Static Space
Dynamic Space
Expectations
Rules
Automatic Anomaly Detection
Prevent Garbage In, Garbage Out
Conclusion

3. Roles of Data Observability in a Data Organization

Data Architecture
Where Does Data Observability Fit in a Data Architecture?
Data Architecture with Data Observability
How Data Observability Helps with Data Engineering Undercurrents
Security
Data Management
Support for Data Mesh’s Data as Products
Conclusion

II. Implementing Data Observability

4. Generate Data Observations

At the Source
Generating Data Observations at the Source
Low-Level API in Python
Description of the Data Pipeline
Definition of the Status of the Data Pipeline
Data Observations for the Data Pipeline
Generate Contextual Data Observations
Generate Data-Related Observations
Generate Lineage-Related Data Observations
Wrap-Up: The Data-Observable Data Pipeline
Using Data Observations to Address Failures of the Data Pipeline
Conclusion

5. Automate the Generation of Data Observations

Abstraction Strategies
Event Listeners
Aspect-Oriented Programming
High-Level Applications
No-Code Applications
Low-Code Applications
Differences Among Monitoring Alternatives
Conclusion

6. Implementing Expectations

Introducing Expectations
Shift-Left Data Quality
Corner Cases Discovery
Lifting Service Level Indicators
Using Data Profilers
Maintaining Expectations
Overarching Practices
Fail Fast and Fail Safe
Simplify Tests and Extend CI/CD
Conclusion

III. Data Observability in Action

7. Integrating Data Observability in Your Data Stack

Ingestion Stage
Ingestion Stage Data Observability Recipes
Airbyte Agent
Transformation
Transformation Stage Data Observability Recipes
Apache Spark
dbt Agent
Serving
Recipes
BigQuery in Python
Orchestrated SQL with Airflow
Analytics
Machine Learning Recipes
Business Intelligence Recipes
Conclusion

8. Making Opaque Systems Translucent

Data Translucence
Opaque Systems
SaaS
Don’t Touch It; It (Kinda) Works
Inherited Systems
Strategies for Data Translucence
Strategies
The Data Observability Connector
Example: Building a dbt Data Observability Connector (SaaS)
Conclusion

Afterword: Future Observations

Unification of Processing
Generative Milestones
Trustable Expanded Creativity
Conclusion

Index
About the Author