Deep Learning from Scratch

Preface
Understanding Neural Networks Requires Multiple Mental Models
Chapter Outlines
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments
1. Foundations
Functions
Math
Diagrams
Code
Derivatives
Math
Diagrams
Code
Nested Functions
Diagram
Math
Code
Another Diagram
The Chain Rule
Math
Code
A Slightly Longer Example
Math
Diagram
Code
Functions with Multiple Inputs
Math
Diagram
Code
Derivatives of Functions with Multiple Inputs
Diagram
Math
Code
Functions with Multiple Vector Inputs
Math
Creating New Features from Existing Features
Math
Diagram
Code
Derivatives of Functions with Multiple Vector Inputs
Diagram
Math
Code
Vector Functions and Their Derivatives: One Step Further
Diagram
Math
Code
Vector Functions and Their Derivatives: The Backward Pass
Computational Graph with Two 2D Matrix Inputs
Math
Diagram
Code
The Fun Part: The Backward Pass
Diagram
Math
Code
Conclusion
2. Fundamentals
Supervised Learning Overview
Supervised Learning Models
Linear Regression
Linear Regression: A Diagram
Linear Regression: A More Helpful Diagram (and the Math)
Adding in the Intercept
Linear Regression: The Code
Training the Model
Calculating the Gradients: A Diagram
Calculating the Gradients: The Math (and Some Code)
Calculating the Gradients: The (Full) Code
Using These Gradients to Train the Model
Assessing Our Model: Training Set Versus Testing Set
Assessing Our Model: The Code
Analyzing the Most Important Feature
Neural Networks from Scratch
Step 1: A Bunch of Linear Regressions
Step 2: A Nonlinear Function
Step 3: Another Linear Regression
Diagrams
Code
Neural Networks: The Backward Pass
Training and Assessing Our First Neural Network
Two Reasons Why This Is Happening
Conclusion
3. Deep Learning from Scratch
Deep Learning Definition: A First Pass
The Building Blocks of Neural Networks: Operations
Diagram
Code
The Building Blocks of Neural Networks: Layers
Diagrams
Building Blocks on Building Blocks
The Layer Blueprint
The Dense Layer
The NeuralNetwork Class, and Maybe Others
Diagram
Code
Loss Class
Deep Learning from Scratch
Implementing Batch Training
NeuralNetwork: Code
Trainer and Optimizer
Optimizer
Trainer
Putting Everything Together
Our First Deep Learning Model (from Scratch)
Conclusion and Next Steps
4. Extensions
Some Intuition About Neural Networks
The Softmax Cross Entropy Loss Function
Component #1: The Softmax Function
Component #2: The Cross Entropy Loss
A Note on Activation Functions
Experiments
Data Preprocessing
Model
Experiment: Softmax Cross Entropy Loss
Momentum
Intuition for Momentum
Implementing Momentum in the Optimizer Class
Experiment: Stochastic Gradient Descent with Momentum
Learning Rate Decay
Types of Learning Rate Decay
Experiments: Learning Rate Decay
Weight Initialization
Math and Code
Experiments: Weight Initialization
Dropout
Definition
Implementation
Experiments: Dropout
Conclusion
5. Convolutional Neural Networks
Neural Networks and Representation Learning
A Different Architecture for Image Data
The Convolution Operation
The Multichannel Convolution Operation
Convolutional Layers
Implementation Implications
The Differences Between Convolutional and Fully Connected Layers
Making Predictions with Convolutional Layers: The Flatten Layer
Pooling Layers
Implementing the Multichannel Convolution Operation
The Forward Pass
Convolutions: The Backward Pass
Batches, 2D Convolutions, and Multiple Channels
2D Convolutions
The Last Element: Adding “Channels”
Using This Operation to Train a CNN
The Flatten Operation
The Full Conv2D Layer
Experiments
Conclusion
6. Recurrent Neural Networks
The Key Limitation: Handling Branching
Automatic Differentiation
Coding Up Gradient Accumulation
Motivation for Recurrent Neural Networks
Introduction to Recurrent Neural Networks
The First Class for RNNs: RNNLayer
The Second Class for RNNs: RNNNode
Putting These Two Classes Together
The Backward Pass
RNNs: The Code
The RNNLayer Class
The Essential Elements of RNNNodes
“Vanilla” RNNNodes
Limitations of “Vanilla” RNNNodes
One Solution: GRUNodes
LSTMNodes
Data Representation for a Character-Level RNN-Based Language Model
Other Language Modeling Tasks
Combining RNNLayer Variants
Putting This All Together
Conclusion
7. PyTorch
PyTorch Tensors
Deep Learning with PyTorch
PyTorch Elements: Model, Layer, Optimizer, and Loss
Implementing Neural Network Building Blocks Using PyTorch: DenseLayer
Example: Boston Housing Prices Model in PyTorch
PyTorch Elements: Optimizer and Loss
PyTorch Elements: Trainer
Tricks to Optimize Learning in PyTorch
Convolutional Neural Networks in PyTorch
DataLoader and Transforms
LSTMs in PyTorch
Postscript: Unsupervised Learning via Autoencoders
Representation Learning
An Approach for Situations with No Labels Whatsoever
Implementing an Autoencoder in PyTorch
A Stronger Test for Unsupervised Learning, and a Solution
Conclusion
A. Deep Dives
Matrix Chain Rule
Gradient of the Loss with Respect to the Bias Terms
Convolutions via Matrix Multiplication
Index