About the Author xiii
Preface xv
Acknowledgments xvii
About the Companion Website xix
1 Introduction 1
1.1 Artificial Intelligence 1
1.2 A Historical Perspective 1
1.3 Principles of AI 2
1.4 Applications That Are Impossible Without AI 2
1.5 Organization of This Book 3
2 Artificial Neural Networks 7
2.1 Introduction 7
2.2 Applications of ANNs 7
2.3 Components of ANNs 8
2.3.1 Neurons 8
2.3.2 Sigmoid Activation Function 9
2.3.3 Rectilinear Activation Function 9
2.3.4 Weights of Synapses 10
2.4 Training an ANN 11
2.5 Forward Propagation 12
2.5.1 Forward Propagation from Input to Hidden Layer 13
2.6 Back Propagation 13
2.6.1 Back Propagation for a Neuron 13
2.6.2 Back Propagation – from Output to Hidden Layer 15
2.6.3 Back Propagation – from Hidden Layer to Input 16
2.7 Updating Weights 17
2.8 ANN with Input Bias 17
2.9 A Simple Algorithm for ANN Training 18
2.10 Computational Complexity of ANN Training 18
2.11 Normalization of ANN Inputs and Outputs 19
2.12 Concluding Remarks 20
2.13 Exercises for Chapter 2 20
3 ANNs for Optimized Prediction 23
3.1 Introduction 23
3.2 Selection of ANN Inputs 24
3.3 Selection of ANN Outputs 24
3.4 Construction of Hidden Layers 25
3.5 Case Study 1: Sleep-Study Example 25
3.5.1 Using Matrices for ANN Training 26
3.5.2 Forward Propagation 28
3.5.3 Back Propagation 28
3.5.4 Updating Weights 29
3.5.5 Forward Propagation with New Weights 29
3.5.6 Back Propagation with New Weights 30
3.5.7 Using Normalized Input and Output Values 31
3.5.8 Reducing Errors During Training 34
3.5.9 Implementation of Sleep-Study ANN in Python 34
3.5.10 Implementation of Sleep-Study ANN in Matlab 37
3.6 Case Study 2: Prediction of Bike Rentals 41
3.6.1 Python Script for Bike Rentals Using an ANN 41
3.6.2 Matlab Script for Bike Rentals Using an ANN 46
3.7 Concluding Remarks 48
3.8 Exercises for Chapter 3 48
4 ANNs for Financial Stock Trading 51
4.1 Introduction 51
4.2 Programs that Buy and Sell Stocks 51
4.3 Technical Indicators 51
4.3.1 Simple Moving Average 52
4.3.2 Momentum 53
4.3.3 Exponential Moving Average 54
4.3.4 Bollinger Bands 54
4.4 A Simple Algorithmic Trading Policy 55
4.5 A Simple ANN for Algorithmic Stock Trading 57
4.5.1 ANN Inputs and Outputs 57
4.5.2 ANN Architecture 58
4.6 Python Script for Stock Trading Using an ANN 59
4.7 Matlab Script for Stock Trading Using an ANN 63
4.8 Concluding Remarks 65
4.9 Exercises for Chapter 4 65
5 ANNs for Alzheimer’s Disease Prognosis 67
5.1 Introduction 67
5.2 Alzheimer’s Disease 67
5.3 A Simple ANN for AD Prognosis 68
5.4 Python Script for AD Prognosis Using an ANN 71
5.5 Matlab Script for AD Prognosis Using an ANN 75
5.6 Concluding Remarks 80
5.7 Exercises for Chapter 5 81
6 ANNs for Natural Language Processing 83
6.1 Introduction 83
6.2 Impact of Text Messages on Stock Markets 84
6.3 A Simple ANN for NLP 85
6.3.1 ANN Inputs and Outputs 85
6.3.2 Keywords 85
6.3.3 Formation of Training Data 86
6.3.4 ANN Architecture 88
6.4 Python Script for NLP Using an ANN 89
6.5 Matlab Script for NLP Using an ANN 92
6.6 Concluding Remarks 96
6.7 Exercises for Chapter 6 97
7 Convolutional Neural Networks 99
7.1 Introduction 99
7.1.1 Training CNNs 100
7.2 Variations of CNNs 101
7.3 Applications of CNNs 101
7.4 CNN Components 102
7.5 A Numerical Example of a CNN 102
7.6 Computational Cost of CNN Training 108
7.7 Concluding Remarks 112
7.8 Exercises for Chapter 7 112
8 CNNs for Optical Character Recognition 115
8.1 Introduction 115
8.2 A Simple CNN for OCR 115
8.3 Organization of Training and Reference Files 117
8.4 Python Script for OCR Using a CNN 119
8.5 Matlab Script for OCR Using a CNN 124
8.6 Concluding Remarks 130
8.7 Exercises for Chapter 8 130
9 CNNs for Speech Recognition 133
9.1 Introduction 133
9.2 A Simple CNN for Speech Recognition 134
9.3 Organization of Training and Reference Files 136
9.4 Python Script for Speech Recognition Using a CNN 138
9.5 Matlab Script for Speech Recognition Using a CNN 144
9.6 Concluding Remarks 150
9.7 Exercises for Chapter 9 150
10 Recurrent Neural Networks 151
10.1 Introduction 151
10.2 One-to-One Single RNN Cell 153
10.2.1 A Simple Alphabet and One-Hot Encoding 156
10.2.2 Forward and Back Propagation 157
10.3 A Numerical Example 158
10.4 Multiple Hidden Layers 163
10.5 Embedding Layer 165
10.5.1 Forward and Back Propagation with Embedding 167
10.5.2 A Numerical Example with Embedding 168
10.6 Concluding Remarks 172
10.7 Exercises for Chapter 10 172
11 RNNs for Chatbot Implementation 175
11.1 Introduction 175
11.2 Many-to-Many RNN Architecture 175
11.3 A Simple Chatbot 176
11.4 Python Script for a Chatbot Using an RNN 179
11.5 Matlab Script for a Chatbot Using an RNN 183
11.6 Concluding Remarks 188
11.7 Exercises for Chapter 11 189
12 RNNs with Attention 191
12.1 Introduction 191
12.2 One-to-One RNN Cell with Attention 191
12.3 Forward and Back Propagation 193
12.4 A Numerical Example 195
12.5 Embedding Layer 200
12.6 A Numerical Example with Embedding 202
12.7 Concluding Remarks 207
12.8 Exercises for Chapter 12 207
13 RNNs with Attention for Machine Translation 209
13.1 Introduction 209
13.2 Many-to-Many Architecture 210
13.3 Python Script for Machine Translation by an RNN-Att 211
13.4 Matlab Script for Machine Translation by an RNN-Att 216
13.5 Concluding Remarks 223
13.6 Exercises for Chapter 13 223
14 Genetic Algorithms 225
14.1 Introduction 225
14.2 Genetic Algorithm Elements 226
14.3 A Simple Algorithm for a GA 227
14.4 An Example of a GA 230
14.5 Convergence in GAs 231
14.6 Concluding Remarks 232
14.7 Exercises for Chapter 14 232
15 GAs for Dietary Menu Selection 235
15.1 Introduction 235
15.2 Definition of the KP 236
15.3 A Simple Algorithm for the KP 238
15.4 Variations of the KP 239
15.5 GAs for KP Solution 240
15.6 Python Script for Dietary Menu Selection Using a GA 242
15.7 Matlab Script for Dietary Menu Selection Using a GA 245
15.8 Concluding Remarks 248
15.9 Exercises for Chapter 15 248
16 GAs for Drone Flight Control 251
16.1 Introduction 251
16.2 UAV Swarms 251
16.3 UAV Flight Control 252
16.4 A Simple GA for UAV Flight Control 253
16.4.1 Virtual Force-Based Fitness Function 254
16.4.2 FGA Progression 255
16.4.3 Chromosome for FGA 257
16.5 Python Script for UAV Flight Control Using a GA 260
16.6 Matlab Script for UAV Flight Control Using a GA 264
16.7 Concluding Remarks 270
16.8 Exercises for Chapter 16 271
17 GAs for Route Optimization 273
17.1 Introduction 273
17.2 Definition of the TSP 274
17.3 A Simple Algorithm for the TSP 276
17.4 Variations of the TSP 277
17.5 GA Solution for the TSP 277
17.6 Python Script for Route Optimization Using a GA 279
17.7 Matlab Script for Route Optimization Using a GA 284
17.8 Concluding Remarks 287
17.9 Exercises for Chapter 17 289
18 Evolutionary Methods 291
18.1 Introduction 291
18.2 Particle Swarm Optimization 291
18.2.1 Applications of PSO 292
18.2.2 PSO Operation 293
18.2.3 Remarks for PSO 298
18.3 Differential Evolution 298
18.3.1 Different Versions of DE 299
18.3.2 Applications of DE 299
18.3.3 A Simple Algorithm for DE 299
18.3.4 Numerical Example: Maximum of sinc by DE 302
18.3.5 Remarks for DE 305
18.4 Grammatical Evolution 306
18.4.1 A Simple Algorithm for GE 306
18.4.2 Definition of GE 307
18.4.3 A Simple GA to Implement GE 314
18.4.4 Remarks on GE 315
Appendix A ANNs with Bias 317
A.1 Introduction 317
A.2 Training with Bias Input 317
A.3 Forward Propagation 318
A.3.1 Forward Propagation from Input to Hidden Layer 319
A.3.2 Neuron Back Propagation with Bias Input 319
Appendix B Sleep Study ANN with Bias 321
B.1 Inclusion of Bias Term in ANN 321
B.1.1 Inclusion of Bias in Matrices 321
B.1.2 Forward Propagation with Biases 322
Appendix C Back Propagation in a CNN 327
Appendix D Back Propagation Through Time in an RNN 331
D.1 Back Propagation in an RNN 331
D.2 Embedding Layer 335
Appendix E Back Propagation Through Time in an RNN with Attention 337
E.1 Back Propagation in an RNN-Att 337
E.2 Embedding Layer 340
Bibliography 343
Index 353
