Deep Learning

About the Authors xv

Foreword xvii

Preface xix

Acknowledgment xxi

About the Companion Website xxiii

1 The Multilayer Perceptron 1

1.1 Introduction 1

1.2 The Concept of Neuron 2

1.3 Structure of a Neural Network 14

1.4 Activations 21

1.5 Training a Multilayer Perceptron 22

1.6 Conclusion 37

2 Training Practicalities 41

2.1 Introduction 41

2.2 Generalization and Overfitting 42

2.3 Regularization Techniques 45

2.4 Normalization Techniques 50

2.5 Optimizers 52

2.6 Conclusion 58

3 Deep Learning Tools 61

3.1 Python: An Overview 61

3.2 NumPy 72

3.3 Matplotlib 83

3.4 Scipy 97

3.5 Scikit-Learn 107

3.6 Pandas 116

3.7 Seaborn 125

3.8 Python Libraries for NLP 131

3.9 TensorFlow 138

3.10 Keras 141

3.11 Pytorch 144

3.12 Conclusion 149

4 Convolutional Neural Networks 153

4.1 Introduction 153

4.2 Elements of a Convolutional Neural Network 153

4.3 Training a CNN 160

4.4 Extensions of the CNN 166

4.5 Conclusion 184

5 Recurrent Neural Networks 187

5.1 Introduction 187

5.2 RNN Architecture 188

5.3 Training an RNN 191

5.4 Long-Term Dependencies: Vanishing and Exploding Gradients 199

5.5 Deep RNN 201

5.6 Bidirectional RNN 203

5.7 Long Short-Term Memory Networks 204

5.8 Gated Recurrent Units 218

5.9 Conclusion 221

6 Attention Networks and Transformers 225

6.1 Introduction 225

6.2 Attention Mechanisms 227

6.3 Transformers 242

6.4 BERT 249

6.5 GPT-2 256

6.6.1 Comparison between ViTs and CNNs 264

6.7 Conclusion 269

7 Deep Unsupervised Learning I 273

7.1 Introduction 273

7.2 Restricted Boltzmann Machines 274

7.3 Deep Belief Networks 278

7.4 Autoencoders 279

7.5 Undercomplete Autoencoder 284

7.6 Sparse Autoencoder 285

7.7 Denoising Autoencoders 287

7.8 Convolutional Autoencoder 288

7.9 Variational Autoencoders 291

7.10 Conclusion 297

8 Deep Unsupervised Learning II 301

8.1 Introduction 301

8.2 Elements of GAN 303

8.3 Training a GAN 305

8.4 Wasserstein GAN 309

8.5 DCGAN 312

8.6 cGAN 316

8.7 CycleGAN 318

8.8 StyleGAN 323

8.9 StackGAN 328

8.10 Diffusion Models 333

8.11 Conclusion 338

9 Deep Bayesian Networks 341

9.1 Introduction 341

9.2 Bayesian Models 342

9.3 Bayesian Inference Methods for Deep Learning 344

9.4 Conclusion 352

Problems 353

List of Acronyms 355

Notation 359

Bibliography 365

Index 387