Digital Signal Processing (DSP) with Python Programming

Preface ix

Notations and Abbreviations xi

A Few Functions of Python^® xiii

Chapter 1 Useful Maths 1

1.1. Basic concepts on probability 1

1.2. Conditional expectation 10

1.3. Projection theorem 11

1.3.1. Conditional expectation 14

1.4. Gaussianity 14

1.4.1. Gaussian random variable 14

1.4.2. Gaussian random vectors 15

1.4.3. Gaussian conditional distribution 16

1.5. Random variable transformation 18

1.5.1. General expression 18

1.5.2. Law of the sum of two random variables 19

1.5.3. δ-method 20

1.6. Fundamental theorems of statistics 22

1.7. A few probability distributions 24

Chapter 2 Statistical Inferences 29

2.1. First step: visualizing data 29

2.1.1. Scatter plot 29

2.1.2. Histogram/boxplot 30

2.1.3. Q-Q plot 32

2.2. Reduction of dataset dimensionality 34

2.2.1. PCA 34

2.2.2. LDA 36

2.3. Some vocabulary 40

2.3.1. Statistical inference 40

2.4. Statistical model 41

2.4.1. Notation 42

2.5. Hypothesis testing 43

2.5.1. Simple hypotheses 45

2.5.2. Generalized likelihood ratio test (GLRT) 50

2.5.3. χ ² goodness-of-fit test 57

2.6. Statistical estimation 58

2.6.1. General principles 58

2.6.2. Least squares method 62

2.6.3. Least squares method for the linear model 64

2.6.4. Method of moments 81

2.6.5. Maximum likelihood approach 84

2.6.6. Logistic regression 100

2.6.7. Non-parametric estimation of probability distribution 103

2.6.8. Bootstrap and others 107

Chapter 3 Inferences on HMM 113

3.1. Hidden Markov models (HMM) 113

3.2. Inferences on HMM 116

3.3. Filtering: general case 117

3.4. Gaussian linear case: Kalman algorithm 118

3.4.1. Kalman filter 118

3.4.2. RTS smoother 127

3.5. Discrete finite Markov case 129

3.5.1. Forward-backward formulas 130

3.5.2. Smoothing formula at one instant 133

3.5.3. Smoothing formula at two successive instants 134

3.5.4. HMM learning using the EM algorithm 135

3.5.5. The Viterbi algorithm 137

Chapter 4 Monte-Carlo Methods 141

4.1. Fundamental theorems 141

4.2. Stating the problem 141

4.3. Generating random variables 144

4.3.1. The cumulative function inversion method 144

4.3.2. The variable transformation method 147

4.3.3. Acceptance-rejection method 149

4.3.4. Sequential methods 151

4.4. Variance reduction 156

4.4.1. Importance sampling 156

4.4.2. Stratification 160

4.4.3. Antithetic variates 164

Chapter 5 Hints and Solutions 167

5.1. Useful maths 167

5.2. Statistical inferences 170

5.3. Inferences on HMM 226

5.4. Monte-Carlo methods 251

Bibliography 261

Index 263