Preface to the First Edition xvii
Preface to the Second Edition xix
1 Introduction 1
1.1 What Data Science Is and Isn¢t 2
1.2 This Book¢s Slogan: Simple Models Are Easier to Work With 3
1.3 How Is This Book Organized? 4
1.4 How to Use This Book? 4
1.5 Why Is It All in Python, Anyway? 4
1.6 Example Code and Datasets 5
1.7 Parting Words 5
Part I The Stuff You¢ll Always Use 7
2 The Data Science Road Map 9
2.1 Frame the Problem 10
2.2 Understand the Data: Basic Questions 11
2.3 Understand the Data: Data Wrangling 12
2.4 Understand the Data: Exploratory Analysis 12
2.5 Extract Features 13
2.6 Model 14
2.7 Present Results 14
2.8 Deploy Code 14
2.9 Iterating 15
2.10 Glossary 15
3 Programming Languages 17
3.1 Why Use a Programming Language? What Are the Other Options? 17
3.2 A Survey of Programming Languages for Data Science 18
3.3 Where to Write Code 20
3.4 Python Overview and Example Scripts 21
3.5 Python Data Types 25
3.6 GOTCHA: Hashable and Unhashable Types 30
3.7 Functions and Control Structures 31
3.8 Other Parts of Python 33
3.9 Python¢s Technical Libraries 35
3.10 Other Python Resources 39
3.11 Further Reading 39
3.12 Glossary 40
3a Interlude: My Personal Toolkit 41
4 Data Munging: String Manipulation, Regular Expressions, and Data Cleaning 43
4.1 The Worst Dataset in the World 43
4.2 How to Identify Pathologies 44
4.3 Problems with Data Content 44
4.4 Formatting Issues 46
4.5 Example Formatting Script 49
4.6 Regular Expressions 50
4.7 Life in the Trenches 53
4.8 Glossary 54
5 Visualizations and Simple Metrics 55
5.1 A Note on Python¢s Visualization Tools 56
5.2 Example Code 56
5.3 Pie Charts 56
5.4 Bar Charts 58
5.5 Histograms 59
5.6 Means, Standard Deviations, Medians, and Quantiles 61
5.7 Boxplots 62
5.8 Scatterplots 64
5.9 Scatterplots with Logarithmic Axes 65
5.10 Scatter Matrices 67
5.11 Heatmaps 68
5.12 Correlations 69
5.13 Anscombe¢s Quartet and the Limits of Numbers 71
5.14 Time Series 72
5.15 Further Reading 75
5.16 Glossary 75
6 Overview: Machine Learning and Artificial Intelligence 77
6.1 Historical Context 77
6.2 The Central Paradigm: Learning a Function from Example 78
6.3 Machine Learning Data: Vectors and Feature Extraction 79
6.4 Supervised, Unsupervised, and In-Between 79
6.5 Training Data, Testing Data, and the Great Boogeyman of Overfitting 80
6.6 Reinforcement Learning 81
6.7 ML Models as Building Blocks for AI Systems 82
6.8 ML Engineering as a New Job Role 82
6.9 Further Reading 83
6.10 Glossary 83
7 Interlude: Feature Extraction Ideas 85
7.1 Standard Features 85
7.2 Features that Involve Grouping 86
7.3 Preview of More Sophisticated Features 86
7.4 You Get What You Measure: Defining the Target Variable 87
8 Machine-Learning Classification 89
8.1 What Is a Classifier, and What Can You Do with It? 89
8.2 A Few Practical Concerns 90
8.3 Binary Versus Multiclass 90
8.4 Example Script 91
8.5 Specific Classifiers 92
8.6 Evaluating Classifiers 102
8.7 Selecting Classification Cutoffs 105
8.8 Further Reading 106
8.9 Glossary 106
9 Technical Communication and Documentation 109
9.1 Several Guiding Principles 109
9.2 Slide Decks 112
9.3 Written Reports 114
9.4 Speaking: What Has Worked for Me 115
9.5 Code Documentation 117
9.6 Further Reading 117
9.7 Glossary 117
Part II Stuff You Still Need to Know 119
10 Unsupervised Learning: Clustering and Dimensionality Reduction 121
10.1 The Curse of Dimensionality 121
10.2 Example: Eigenfaces for Dimensionality Reduction 123
10.3 Principal Component Analysis and Factor Analysis 125
10.4 Skree Plots and Understanding Dimensionality 127
10.5 Factor Analysis 127
10.6 Limitations of PCA 128
10.7 Clustering 128
10.8 Further Reading 133
10.9 Glossary 134
11 Regression 135
11.1 Example: Predicting Diabetes Progression 136
11.2 Fitting a Line with Least Squares 137
11.3 Alternatives to Least Squares 139
11.4 Fitting Nonlinear Curves 139
11.5 Goodness of Fit: R 2 and Correlation 141
11.6 Correlation of Residuals 142
11.7 Linear Regression 142
11.8 LASSO Regression and Feature Selection 144
11.9 Further Reading 145
11.10 Glossary 145
12 Data Encodings and File Formats 147
12.1 Typical File Format Categories 147
12.2 CSV Files 149
12.3 JSON Files 150
12.4 XML Files 151
12.5 HTML Files 153
12.6 Tar Files 154
12.7 GZip Files 155
12.8 Zip Files 155
12.9 Image Files: Rasterized, Vectorized, and/or Compressed 156
12.10 It¢s All Bytes at the End of the Day 157
12.11 Integers 158
12.12 Floats 158
12.13 Text Data 159
12.14 Further Reading 161
12.15 Glossary 161
13 Big Data 163
13.1 What Is Big Data? 163
13.2 When to Use ¢" And not Use ¢" Big Data 164
13.3 Hadoop: The File System and the Processor 165
13.4 Example PySpark Script 165
13.5 Spark Overview 166
13.6 Spark Operations 168
13.7 PySpark Data Frames 169
13.8 Two Ways to Run PySpark 170
13.9 Configuring Spark 170
13.10 Under the Hood 172
13.11 Spark Tips and Gotchas 172
13.12 The MapReduce Paradigm 173
13.13 Performance Considerations 174
13.14 Further Reading 175
13.15 Glossary 176
14 Databases 177
14.1 Relational Databases and MySQL® 178
14.2 Key¢"Value Stores 183
14.3 Wide-Column Stores 183
14.4 Document Stores 184
14.5 Further Reading 186
14.6 Glossary 186
15 Software Engineering Best Practices 187
15.1 Coding Style 187
15.2 Version Control and Git for Data Scientists 189
15.3 Testing Code 191
15.4 Test-Driven Development 193
15.5 AGILE Methodology 194
15.6 Further Reading 194
15.7 Glossary 194
16 Traditional Natural Language Processing 197
16.1 Do I Even Need NLP? 197
16.2 The Great Divide: Language Versus Statistics 198
16.3 Example: Sentiment Analysis on Stock Market Articles 198
16.4 Software and Datasets 200
16.5 Tokenization 201
16.6 Central Concept: Bag-of-Words 201
16.7 Word Weighting: TF-IDF 202
16.8 n-Grams 202
16.9 Stop Words 203
16.10 Lemmatization and Stemming 203
16.11 Synonyms 204
16.12 Part of Speech Tagging 204
16.13 Common Problems 204
16.14 Advanced Linguistic NLP: Syntax Trees, Knowledge, and Understanding 206
16.15 Further Reading 207
16.16 Glossary 207
17 Time Series Analysis 209
17.1 Example: Predicting Wikipedia Page Views 210
17.2 A Typical Workflow 213
17.3 Time Series Versus Time-Stamped Events 213
17.4 Resampling and Interpolation 214
17.5 Smoothing Signals 216
17.6 Logarithms and Other Transformations 217
17.7 Trends and Periodicity 217
17.8 Windowing 217
17.9 Brainstorming Simple Features 218
17.10 Better Features: Time Series as Vectors 219
17.11 Fourier Analysis: Sometimes a Magic Bullet 220
17.12 Time Series in Context: The Whole Suite of Features 222
17.13 Further Reading 222
17.14 Glossary 222
18 Probability 225
18.1 Flipping Coins: Bernoulli Random Variables 225
18.2 Throwing Darts: Uniform Random Variables 226
18.3 The Uniform Distribution and Pseudorandom Numbers 227
18.4 Nondiscrete, Noncontinuous Random Variables 228
18.5 Notation, Expectations, and Standard Deviation 230
18.6 Dependence, Marginal, and Conditional Probability 231
18.7 Understanding the Tails 232
18.8 Binomial Distribution 234
18.9 Poisson Distribution 234
18.10 Normal Distribution 235
18.11 Multivariate Gaussian 236
18.12 Exponential Distribution 237
18.13 Log-Normal Distribution 238
18.14 Entropy 238
18.15 Further Reading 240
18.16 Glossary 240
19 Statistics 243
19.1 Statistics in Perspective 243
19.2 Bayesian Versus Frequentist: Practical Tradeoffs and Differing Philosophies 244
19.3 Hypothesis Testing: Key Idea and Example 245
19.4 Multiple Hypothesis Testing 246
19.5 Parameter Estimation 247
19.6 Hypothesis Testing: t-Test 248
19.7 Confidence Intervals 250
19.8 Bayesian Statistics 252
19.9 Naive Bayesian Statistics 253
19.10 Bayesian Networks 253
19.11 Choosing Priors: Maximum Entropy or Domain Knowledge 254
19.12 Further Reading 255
19.13 Glossary 255
20 Programming Language Concepts 257
20.1 Programming Paradigms 257
20.2 Compilation and Interpretation 264
20.3 Type Systems 266
20.4 Further Reading 267
20.5 Glossary 267
21 Performance and Computer Memory 269
21.1 A Word of Caution 269
21.2 Example Script 270
21.3 Algorithm Performance and Big-O Notation 272
21.4 Some Classic Problems: Sorting a List and Binary Search 273
21.5 Amortized Performance and Average Performance 276
21.6 Two Principles: Reducing Overhead and Managing Memory 277
21.7 Performance Tip: Use Numerical Libraries When Applicable 278
21.8 Performance Tip: Delete Large Structures You Don¢t Need 280
21.9 Performance Tip: Use Built-In Functions When Possible 280
21.10 Performance Tip: Avoid Superfluous Function Calls 280
21.11 Performance Tip: Avoid Creating Large New Objects 281
21.12 Further Reading 281
21.13 Glossary 281
Part III Specialized or Advanced Topics 283
22 Computer Memory and Data Structures 285
22.1 Virtual Memory, the Stack, and the Heap 285
22.2 Example C Program 286
22.3 Data Types and Arrays in Memory 286
22.4 Structs 287
22.5 Pointers, the Stack, and the Heap 288
22.6 Key Data Structures 292
22.7 Further Reading 297
22.8 Glossary 297
23 Maximum-Likelihood Estimation and Optimization 299
23.1 Maximum-Likelihood Estimation 299
23.2 A Simple Example: Fitting a Line 300
23.3 Another Example: Logistic Regression 301
23.4 Optimization 302
23.5 Gradient Descent 303
23.6 Convex Optimization 306
23.7 Stochastic Gradient Descent 307
23.8 Further Reading 308
23.9 Glossary 308
24 Deep Learning and AI 309
24.1 A Note on Libraries and Hardware 310
24.2 A Note on Training Data 310
24.3 Simple Deep Learning: Perceptrons 311
24.4 What Is a Tensor? 314
24.5 Convolutional Neural Networks 315
24.6 Example: The MNIST Handwriting Dataset 317
24.7 Autoencoders and Latent Vectors 318
24.8 Generative AI and GANs 321
24.9 Diffusion Models 323
24.10 RNNs, Hidden State, and the Encoder¢"Decoder 324
24.11 Attention and Transformers 325
24.12 Stable Diffusion: Bringing the Parts Together 326
24.13 Large Language Models and Prompt Engineering 327
24.14 Further Reading 328
24.15 Glossary 329
25 Stochastic Modeling 331
25.1 Markov Chains 331
25.2 Two Kinds of Markov Chain, Two Kinds of Questions 333
25.3 Hidden Markov Models and the Viterbi Algorithm 334
25.4 The Viterbi Algorithm 336
25.5 Random Walks 337
25.6 Brownian Motion 338
25.7 ARIMA Models 339
25.8 Continuous-Time Markov Processes 339
25.9 Poisson Processes 340
25.10 Further Reading 341
25.11 Glossary 341
26 Parting Words: Your Future as a Data Scientist 343
Index 345
