Preface xi
About the Companion Website xv
1 Introduction 1
1.1 The Scope of the Book 2
1.1.1 Classification 3
1.1.2 Parameter Estimation 4
1.1.3 State Estimation 5
1.1.4 Relations between the Subjects 7
1.2 Engineering 10
1.3 The Organization of the Book 12
1.4 Changes from First Edition 14
1.5 References 15
2 PRTools Introduction 17
2.1 Motivation 17
2.2 Essential Concepts 18
2.3 PRTools Organization Structure and Implementation 22
2.4 Some Details about PRTools 26
2.4.1 Datasets 26
2.4.2 Datafiles 30
2.4.3 Datafiles Help Information 31
2.4.4 Classifiers and Mappings 34
2.4.5 Mappings Help Information 36
2.4.6 How to Write Your Own Mapping 38
2.5 Selected Bibliography 42
3 Detection and Classification 43
3.1 Bayesian Classification 46
3.1.1 Uniform Cost Function and Minimum Error Rate 53
3.1.2 Normal Distributed Measurements; Linear and Quadratic Classifiers 56
3.2 Rejection 62
3.2.1 Minimum Error Rate Classification with Reject Option 63
3.3 Detection: The Two-Class Case 66
3.4 Selected Bibliography 74
Exercises 74
4 Parameter Estimation 77
4.1 Bayesian Estimation 79
4.1.1 MMSE Estimation 86
4.1.2 MAP Estimation 87
4.1.3 The Gaussian Case with Linear Sensors 88
4.1.4 Maximum Likelihood Estimation 89
4.1.5 Unbiased Linear MMSE Estimation 91
4.2 Performance Estimators 94
4.2.1 Bias and Covariance 95
4.2.2 The Error Covariance of the Unbiased Linear MMSE Estimator 99
4.3 Data Fitting 100
4.3.1 Least Squares Fitting 101
4.3.2 Fitting Using a Robust Error Norm 104
4.3.3 Regression 107
4.4 Overview of the Family of Estimators 110
4.5 Selected Bibliography 111
Exercises 112
5 State Estimation 115
5.1 A General Framework for Online Estimation 117
5.1.1 Models 117
5.1.2 Optimal Online Estimation 123
5.2 Infinite Discrete-Time State Variables 125
5.2.1 Optimal Online Estimation in Linear-Gaussian Systems 125
5.2.2 Suboptimal Solutions for Non-linear Systems 133
5.3 Finite Discrete-Time State Variables 147
5.3.1 Hidden Markov Models 148
5.3.2 Online State Estimation 152
5.3.3 Offline State Estimation 156
5.4 Mixed States and the Particle Filter 163
5.4.1 Importance Sampling 164
5.4.2 Resampling by Selection 166
5.4.3 The Condensation Algorithm 167
5.5 Genetic State Estimation 170
5.5.1 The Genetic Algorithm 170
5.5.2 Genetic State Estimation 176
5.5.3 Computational Issues 177
5.6 State Estimation in Practice 183
5.6.1 System Identification 185
5.6.2 Observability, Controllability and Stability 188
5.6.3 Computational Issues 193
5.6.4 Consistency Checks 196
5.7 Selected Bibliography 201
Exercises 204
6 Supervised Learning 207
6.1 Training Sets 208
6.2 Parametric Learning 210
6.2.1 Gaussian Distribution, Mean Unknown 211
6.2.2 Gaussian Distribution, Covariance Matrix Unknown 212
6.2.3 Gaussian Distribution, Mean and Covariance Matrix Both Unknown 213
6.2.4 Estimation of the Prior Probabilities 215
6.2.5 Binary Measurements 216
6.3 Non-parametric Learning 217
6.3.1 Parzen Estimation and Histogramming 218
6.3.2 Nearest Neighbour Classification 223
6.3.3 Linear Discriminant Functions 230
6.3.4 The Support Vector Classifier 237
6.3.5 The Feedforward Neural Network 242
6.4 Adaptive Boosting – Adaboost 245
6.5 Convolutional Neural Networks (CNNs) 249
6.5.1 Convolutional Neural Network Structure 249
6.5.2 Computation and Training of CNNs 251
6.6 Empirical Evaluation 252
6.7 Selected Bibliography 257
Exercises 257
7 Feature Extraction and Selection 259
7.1 Criteria for Selection and Extraction 261
7.1.1 Interclass/Intraclass Distance 262
7.1.2 Chernoff–Bhattacharyya Distance 267
7.1.3 Other Criteria 270
7.2 Feature Selection 272
7.2.1 Branch-and-Bound 273
7.2.2 Suboptimal Search 275
7.2.3 Several New Methods of Feature Selection 278
7.2.4 Implementation Issues 287
7.3 Linear Feature Extraction 288
7.3.1 Feature Extraction Based on the Bhattacharyya Distance with Gaussian Distributions 291
7.3.2 Feature Extraction Based on Inter/Intra Class Distance 296
7.4 References 300
Exercises 300
8 Unsupervised Learning 303
8.1 Feature Reduction 304
8.1.1 Principal Component Analysis 304
8.1.2 Multidimensional Scaling 309
8.1.3 Kernel Principal Component Analysis 315
8.2 Clustering 320
8.2.1 Hierarchical Clustering 323
8.2.2 K-Means Clustering 327
8.2.3 Mixture of Gaussians 329
8.2.4 Mixture of probabilistic PCA 335
8.2.5 Self-Organizing Maps 336
8.2.6 Generative Topographic Mapping 342
8.3 References 345
Exercises 346
9 Worked Out Examples 349
9.1 Example on Image Classification with PRTools 349
9.1.1 Example on Image Classification 349
9.1.2 Example on Face Classification 354
9.1.3 Example on Silhouette Classification 357
9.2 Boston Housing Classification Problem 361
9.2.1 Dataset Description 361
9.2.2 Simple Classification Methods 363
9.2.3 Feature Extraction 365
9.2.4 Feature Selection 367
9.2.5 Complex Classifiers 368
9.2.6 Conclusions 371
9.3 Time-of-Flight Estimation of an Acoustic Tone Burst 372
9.3.1 Models of the Observed Waveform 374
9.3.2 Heuristic Methods for Determining the ToF 376
9.3.3 Curve Fitting 377
9.3.4 Matched Filtering 379
9.3.5 ml Estimation Using Covariance Models for the Reflections 380
9.3.6 Optimization and Evaluation 385
9.4 Online Level Estimation in a Hydraulic System 392
9.4.1 Linearized Kalman Filtering 394
9.4.2 Extended Kalman Filtering 397
9.4.3 Particle Filtering 398
9.4.4 Discussion 403
9.5 References 406
Appendix A: Topics Selected from Functional Analysis 407
Appendix B: Topics Selected from Linear Algebra and Matrix Theory 421
Appendix C: Probability Theory 437
Appendix D: Discrete-Time Dynamic Systems 453
Index 459
