List of Figures xvii
List of Tables xxiii
Preface xxv
List of Abbreviations xxix
How to Use the Book xxxi
About the Companion Website xxxiii
Prerequisites xxxv
Why are there so many matrixes in this book? xxxvii
1 Manipulations on Matrixes 1
1.1 Matrix Properties 1
1.1.1 Elementary Operations 2
1.2 Eigen-Decomposition 6
1.3 Eigenvectors in Everyday Life 9
1.3.1 Conversations in a Noisy Restaurant 9
1.3.2 Power Control in a Cellular System 12
1.3.3 Price Equilibrium in the Economy 14
1.4 Derivative Rules 15
1.4.1 Derivative with respect to x 16
1.4.2 Derivative with respect to x 17
1.4.3 Derivative with respect to the Matrix X 18
1.5 Quadratic Forms 19
1.6 Diagonalization of a Quadratic Form 20
1.7 Rayleigh Quotient 21
1.8 Basics of Optimization 22
1.8.1 Quadratic Function with Simple Linear Constraint (M=1) 23
1.8.2 Quadratic Function with Multiple Linear Constraints 23
Appendix A: Arithmetic vs. Geometric Mean 24
2 Linear Algebraic Systems 27
2.1 Problem Definition and Vector Spaces 27
2.1.1 Vector Spaces in Tomographic Radiometric Inversion 29
2.2 Rotations 31
2.3 Projection Matrixes and Data-Filtering 33
2.3.1 Projections and Commercial FM Radio 34
2.4 Singular Value Decomposition (SVD) and Subspaces 34
2.4.1 How to Choose the Rank of Afor Gaussian Model? 35
2.5 QR and Cholesky Factorization 36
2.6 Power Method for Leading Eigenvectors 38
2.7 Least Squares Solution of Overdetermined Linear Equations 39
2.8 Efficient Implementation of the LS Solution 41
2.9 Iterative Methods 42
3 Random Variables in Brief 45
3.1 Probability Density Function (pdf), Moments, and Other Useful Properties 45
3.2 Convexity and Jensen Inequality 49
3.3 Uncorrelatedness and Statistical Independence 49
3.4 Real-Valued Gaussian Random Variables 51
3.5 Conditional pdf for Real-Valued Gaussian Random Variables 54
3.6 Conditional pdf in Additive Noise Model 56
3.7 Complex Gaussian Random Variables 56
3.7.1 Single Complex Gaussian Random Variable 56
3.7.2 Circular Complex Gaussian Random Variable 57
3.7.3 Multivariate Complex Gaussian Random Variables 58
3.8 Sum of Square of Gaussians: Chi-Square 59
3.9 Order Statistics for N rvs 60
4 Random Processes and Linear Systems 63
4.1 Moment Characterizations and Stationarity 64
4.2 Random Processes and Linear Systems 66
4.3 Complex-Valued Random Processes 68
4.4 Pole-Zero and Rational Spectra (Discrete-Time) 69
4.4.1 Stability of LTI Systems 70
4.4.2 Rational PSD 71
4.4.3 Paley-Wiener Theorem 72
4.5 Gaussian Random Process (Discrete-Time) 73
4.6 Measuring Moments in Stochastic Processes 75
Appendix A: Transforms for Continuous-Time Signals 76
Appendix B: Transforms for Discrete-Time Signals 79
5 Models and Applications 83
5.1 Linear Regression Model 84
5.2 Linear Filtering Model 86
5.2.1 Block-Wise Circular Convolution 88
5.2.2 Discrete Fourier Transform and Circular Convolution Matrixes 89
5.2.3 Identification and Deconvolution 90
5.3 MIMO systems and Interference Models 91
5.3.1 DSL System 92
5.3.2 MIMO in Wireless Communication 92
5.4 Sinusoidal Signal 97
5.5 Irregular Sampling and Interpolation 97
5.5.1 Sampling With Jitter 100
5.6 Wavefield Sensing System 101
6 Estimation Theory 105
6.1 Historical Notes 105
6.2 Non-Bayesian vs. Bayesian 106
6.3 Performance Metrics and Bounds 107
6.3.1 Bias 107
6.3.2 Mean Square Error (MSE) 108
6.3.3 Performance Bounds 109
6.4 Statistics and Sufficient Statistics 110
6.5 MVU and BLU Estimators 111
6.6 BLUE for Linear Models 112
6.7 Example: BLUE of the Mean Value of Gaussian rvs 114
7 Parameter Estimation 117
7.1 Maximum Likelihood Estimation (MLE) 117
7.2 MLE for Gaussian Model 119
7.2.1 Additive Noise Model with 119
7.2.2 Additive Noise Model with 120
7.2.3 Additive Noise Model with Multiple Observations with Known 121
7.2.3.1 Linear Model 121
7.2.3.2 Model 122
7.2.3.3 Model 123
7.2.4 Model 123
7.2.5 Additive Noise Model with Multiple Observations with Unknown 124
7.3 Other Noise Models 125
7.4 MLE and Nuisance Parameters 126
7.5 MLE for Continuous-Time Signals 128
7.5.1 Example: Amplitude Estimation 129
7.5.2 MLE for Correlated Noise 130
7.6 MLE for Circular Complex Gaussian 131
7.7 Estimation in Phase/Frequency Modulations 131
7.7.1 MLE Phase Estimation 132
7.7.2 Phase Locked Loops 133
7.8 Least Square (LS) Estimation 135
7.8.1 Weighted LS with 136
7.8.2 LS Estimation and Linear Models 137
7.8.3 Under or Over-Parameterizing? 138
7.8.4 Constrained LS Estimation 139
7.9 Robust Estimation 140
8 Cramer-Rao Bound 143
8.1 Cramer-Rao Bound and Fisher Information Matrix 143
8.1.1 CRB for Scalar Problem (P=1) 143
8.1.2 CRB and Local Curvature of Log-Likelihood 144
8.1.3 CRB for Multiple Parameters (p 1) 144
8.2 Interpretation of CRB and Remarks 146
8.2.1 Variance of Each Parameter 146
8.2.2 Compactness of the Estimates 146
8.2.3 FIM for Known Parameters 147
8.2.4 Approximation of the Inverse of FIM 148
8.2.5 Estimation Decoupled From FIM 148
8.2.6 CRB and Nuisance Parameters 149
8.2.7 CRB for Non-Gaussian rv and Gaussian Bound 149
8.3 CRB and Variable Transformations 150
8.4 FIM for Gaussian Parametric Model 151
8.4.1 FIM for with 151
8.4.2 FIM for Continuous-Time Signals in Additive White Gaussian Noise 152
8.4.3 FIM for Circular Complex Model 152
Appendix A: Proof of CRB 154
Appendix B: FIM for Gaussian Model 156
Appendix C: Some Derivatives for MLE and CRB Computations 157
9 MLE and CRB for Some Selected Cases 159
9.1 Linear Regressions 159
9.2 Frequency Estimation 162
9.3 Estimation of Complex Sinusoid 164
9.3.1 Proper, Improper, and Non-Circular Signals 165
9.4 Time of Delay Estimation 166
9.5 Estimation of Max for Uniform pdf 170
9.6 Estimation of Occurrence Probability for Binary pdf 172
9.7 How to Optimize Histograms? 173
9.8 Logistic Regression 176
10 Numerical Analysis and Montecarlo Simulations 179
10.1 System Identification and Channel Estimation 181
10.1.1 Matlab Code and Results 184
10.2 Frequency Estimation 184
10.2.1 Variable (Coarse/Fine) Sampling 187
10.2.2 Local Parabolic Regression 189
10.2.3 Matlab Code and Results 190
10.3 Time of Delay Estimation 192
10.3.1 Granularity of Sampling in ToD Estimation 193
10.3.2 Matlab Code and Results 194
10.4 Doppler-Radar System by Frequency Estimation 196
10.4.1 EM Method 197
10.4.2 Matlab Code and Results 199
11 Bayesian Estimation 201
11.1 Additive Linear Model with Gaussian Noise 203
11.1.1 Gaussian A-priori: 204
11.1.2 Non-Gaussian A-Priori 206
11.1.3 Binary Signals: MMSE vs. MAP Estimators 207
11.1.4 Example: Impulse Noise Mitigation 210
11.2 Bayesian Estimation in Gaussian Settings 212
11.2.1 MMSE Estimator 213
11.2.2 MMSE Estimator for Linear Models 213
11.3 LMMSE Estimation and Orthogonality 215
11.4 Bayesian CRB 218
11.5 Mixing Bayesian and Non-Bayesian 220
11.5.1 Linear Model with Mixed Random/Deterministic Parameters 220
11.5.2 Hybrid CRB 222
11.6 Expectation-Maximization (EM) 223
11.6.1 EM of the Sum of Signals in Gaussian Noise 224
11.6.2 EM Method for the Time of Delay Estimation of Multiple Waveforms 227
11.6.3 Remarks 228
Appendix A: Gaussian Mixture pdf 229
12 Optimal Filtering 231
12.1 Wiener Filter 231
12.2 MMSE Deconvolution (or Equalization) 233
12.3 Linear Prediction 234
12.3.1 Yule-Walker Equations 235
12.4 LS Linear Prediction 237
12.5 Linear Prediction and AR Processes 239
12.6 Levinson Recursion and Lattice Predictors 241
13 Bayesian Tracking and Kalman Filter 245
13.1 Bayesian Tracking of State in Dynamic Systems 246
13.1.1 Evolution of the A-posteriori pdf 247
13.2 Kalman Filter (KF) 249
13.2.1 KF Equations 251
13.2.2 Remarks 253
13.3 Identification of Time-Varying Filters in Wireless Communication 255
13.4 Extended Kalman Filter (EKF) for Non-Linear Dynamic Systems 257
13.5 Position Tracking by Multi-Lateration 258
13.5.1 Positioning and Noise 260
13.5.2 Example of Position Tracking 263
13.6 Non-Gaussian Pdf and Particle Filters264
14 Spectral Analysis 267
14.1 Periodogram 268
14.1.1 Bias of the Periodogram 268
14.1.2 Variance of the Periodogram 271
14.1.3 Filterbank Interpretation 273
14.1.4 Pdf of the Periodogram (White Gaussian Process) 274
14.1.5 Bias and Resolution 275
14.1.6 Variance Reduction and WOSA 278
14.1.7 Numerical Example: Bandlimited Process and (Small) Sinusoid 280
14.2 Parametric Spectral Analysis 282
14.2.1 MLE and CRB 284
14.2.2 General Model for AR, MA, ARMA Spectral Analysis 285
14.3 AR Spectral Analysis 286
14.3.1 MLE and CRB 286
14.3.2 A Good Reason to Avoid Over-Parametrization in AR 289
14.3.3 Cramer-Rao Bound of Poles in AR Spectral Analysis 291
14.3.4 Example: Frequency Estimation by AR Spectral Analysis 293
14.4 MA Spectral Analysis 296
14.5 ARMA Spectral Analysis 298
14.5.1 Cramer-Rao Bound for ARMA Spectral Analysis 300
Appendix A: Which Sample Estimate of the Autocorrelation to Use? 302
Appendix B: Eigenvectors and Eigenvalues of Correlation Matrix 303
Appendix C: Property of Monic Polynomial 306
Appendix D: Variance of Pole in AR(1) 307
15 Adaptive Filtering 309
15.1 Adaptive Interference Cancellation 311
15.2 Adaptive Equalization in Communication Systems 313
15.2.1 Wireless Communication Systems in Brief 313
15.2.2 Adaptive Equalization 315
15.3 Steepest Descent MSE Minimization 317
15.3.1 Convergence Analysis and Step-Size 318
15.3.2 An Intuitive View of Convergence Conditions 320
15.4 From Iterative to Adaptive Filters 323
15.5 LMS Algorithm and Stochastic Gradient 324
15.6 Convergence Analysis of LMS Algorithm 325
15.6.1 Convergence in the Mean 326
15.6.2 Convergence in the Mean Square 326
15.6.3 Excess MSE 329
15.7 Learning Curve of LMS 331
15.7.1 Optimization of the Step-Size 332
15.8 NLMS Updating and Non-Stationarity 333
15.9 Numerical Example: Adaptive Identification 334
15.10 RLS Algorithm 338
15.10.1 Convergence Analysis 339
15.10.2 Learning Curve of RLS 341
15.11 Exponentially-Weighted RLS 342
15.12 LMS vs. RLS 344
Appendix A: Convergence in Mean Square 344
16 Line Spectrum Analysis 347
16.1 Model Definition 349
16.1.1 Deterministic Signals 350
16.1.2 Random Signals 350
16.1.3 Properties of Structured Covariance 351
16.2 Maximum Likelihood and Cramer-Rao Bounds 352
16.2.1 Conditional ML 353
16.2.2 Cramer-Rao Bound for Conditional Model 354
16.2.3 Unconditional ML 356
16.2.4 Cramer-Rao Bound for Unconditional Model 356
16.2.5 Conditional vs. Unconditional Model & Bounds 357
16.3 High-Resolution Methods 357
16.3.1 Iterative Quadratic ML (IQML) 358
16.3.2 Prony Method 360
16.3.3 MUSIC 360
16.3.4 ESPRIT 363
16.3.5 Model Order 365
17 Equalization in Communication Engineering 367
17.1 Linear Equalization 369
17.1.1 Zero Forcing (ZF) Equalizer 370
17.1.2 Minimum Mean Square Error (MMSE) Equalizer 371
17.1.3 Finite-Length/Finite-Block Equalizer 371
17.2 Non-Linear Equalization 372
17.2.1 ZF-DFE 373
17.2.2 MMSE-DFE 374
17.2.3 Finite-Length MMSE-DFE 375
17.2.4 Asymptotic Performance for Infinite-Length Equalizers 376
17.3 MIMO Linear Equalization 377
17.3.1 ZF MIMO Equalization 377
17.3.2 MMSE MIMO Equalization 379
17.4 MIMO-DFE Equalization 379
17.4.1 Cholesky Factorization and Min/Max Phase Decomposition 379
17.4.2 MIMO-DFE 380
18 2D Signals and Physical Filters 383
18.1 2D Sinusoids 384
18.1.1 Moire Pattern 386
18.2 2D Filtering 388
18.2.1 2D Random Fields 390
18.2.2 Wiener Filtering 391
18.2.3 Image Acquisition and Restoration 392
18.3 Diffusion Filtering 394
18.3.1 Evolution vs. Time: Fourier Method 394
18.3.2 Extrapolation of the Density 395
18.3.3 Effect of Phase-Shift 396
18.4 Laplace Equation and Exponential Filtering 398
18.5 Wavefield Propagation 400
18.5.1 Propagation/Backpropagation 400
18.5.2 Wavefield Extrapolation and Focusing 402
18.5.3 Exploding Reflector Model 402
18.5.4 Wavefield Extrapolation 404
18.5.5 Wavefield Focusing (or Migration) 406
Appendix A: Properties of 2D Signals 406
Appendix B: Properties of 2D Fourier Transform 410
Appendix C: Finite Difference Method for PDE-Diffusion 412
19 Array Processing 415
19.1 Narrowband Model 415
19.1.1 Multiple DoAs and Multiple Sources 419
19.1.2 Sensor Spacing Design 420
19.1.3 Spatial Resolution and Array Aperture 421
19.2 Beamforming and Signal Estimation 422
19.2.1 Conventional Beamforming 425
19.2.2 Capon Beamforming (MVDR) 426
19.2.3 Multiple-Constraint Beamforming 429
19.2.4 Max-SNR Beamforming 431
19.3 DoA Estimation 432
19.3.1 ML Estimation and CRB 433
19.3.2 Beamforming and Root-MVDR 434
20 Multichannel Time of Delay Estimation 435
20.1 Model Definition for ToD 440
20.2 High Resolution Method for ToD (L=1) 441
20.2.1 ToD in the Fourier Transformed Domain 441
20.2.2 CRB and Resolution 444
20.3 Difference of ToD (DToD) Estimation 445
20.3.1 Correlation Method for DToD 445
20.3.2 Generalized Correlation Method 448
20.4 Numerical Performance Analysis of DToD 452
20.5 Wavefront Estimation: Non-Parametric Method (L=1) 454
20.5.1 Wavefront Estimation in Remote Sensing and Geophysics 456
20.5.2 Narrowband Waveforms and 2D Phase Unwrapping 457
20.5.3 2D Phase Unwrapping in Regular Grid Spacing 458
20.6 Parametric ToD Estimation and Wideband Beamforming 460
20.6.1 Delay and Sum Beamforming 462
20.6.2 Wideband Beamforming After Fourier Transform 464
20.7 Appendix A: Properties of the Sample Correlations 465
20.8 Appendix B: How to Delay a Discrete-Time Signal? 466
20.9 Appendix C: Wavefront Estimation for 2D Arrays 467
21 Tomography 467
21.1 X-ray Tomography 471
21.1.1 Discrete Model 471
21.1.2 Maximum Likelihood 473
21.1.3 Emission Tomography 473
21.2 Algebraic Reconstruction Tomography (ART) 475
21.3 Reconstruction From Projections: Fourier Method 475
21.3.1 Backprojection Algorithm 476
21.3.2 How Many Projections to Use? 479
21.4 Traveltime Tomography 480
21.5 Internet (Network) Tomography 483
21.5.1 Latency Tomography 484
21.5.2 Packet-Loss Tomography 484
22 Cooperative Estimation 487
22.1 Consensus and Cooperation 490
22.1.1 Vox Populi: The Wisdom of Crowds 490
22.1.2 Cooperative Estimation as Simple Information Consensus 490
22.1.3 Weighted Cooperative Estimation ( ) 493
22.1.4 Distributed MLE ( ) 495
22.2 Distributed Estimation for Arbitrary Linear Models (p>1) 496
22.2.1 Centralized MLE 497
22.2.2 Distributed Weighted LS 498
22.2.3 Distributed MLE 500
22.2.4 Distributed Estimation for Under-Determined Systems 501
22.2.5 Stochastic Regressor Model 503
22.2.6 Cooperative Estimation in the Internet of Things (IoT) 503
22.2.7 Example: Iterative Distributed Estimation 505
22.3 Distributed Synchronization 506
22.3.1 Synchrony-States for Analog and Discrete-Time Clocks 507
22.3.2 Coupled Clocks 510
22.3.3 Internet Synchronization and the Network Time Protocol (NTP) 512
Appendix A: Basics of Undirected Graphs 515
23 Classification and Clustering 521
23.1 Historical Notes 522
23.2 Classification 523
23.2.1 Binary Detection Theory 523
23.2.2 Binary Classification of Gaussian Distributions 528
23.3 Classification of Signals in Additive Gaussian Noise 529
23.3.1 Detection of Known Signal 531
23.3.2 Classification of Multiple Signals 532
23.3.3 Generalized Likelihood Ratio Test (GLRT) 533
23.3.4 Detection of Random Signals 535
23.4 Bayesian Classification 536
23.4.1 To Classify or Not to Classify? 537
23.4.2 Bayes Risk 537
23.5 Pattern Recognition and Machine Learning 538
23.5.1 Linear Discriminant 539
23.5.2 Least Squares Classification 540
23.5.3 Support Vectors Principle 541
23.6 Clustering 543
23.6.1 K-Means Clustering 544
23.6.2 EM Clustering 545
References 549
Index 557
