Preface xiii
Acknowledgments xv
About the Companion Website xvii
Part I Probability 1
1 Basic Concepts of Probability Theory 3
1.1 Statistical Regularity and Relative Frequency 3
1.2 Set Theory and Its Applications to Probability 5
1.3 The Axioms and Corollaries of Probability 12
1.4 Joint Probability and Conditional Probability 18
1.5 Statistically Independent Events and Mutually Exclusive Events 21
1.6 Law of Total Probability and Bayes' Theorem 28
1.7 Summary 32
Problems 32
2 Applications in Probability 37
2.1 Odds and Risk 37
2.2 Gambler's Ruin Problem 41
2.3 Systems Reliability 43
2.4 Medical Diagnostic Testing 47
2.5 Bayesian Spam Filtering 50
2.6 Monty Hall Problem 51
2.7 Digital Transmission Error 54
2.8 How to Make the Best Choice Problem 56
2.9 The Viterbi Algorithm 59
2.10 All Eggs in One Basket 61
2.11 Summary 63
Problems 63
3 Counting Methods and Applications 67
3.1 Basic Rules of Counting 67
3.2 Permutations and Combinations 72
3.2.1 Permutations without Replacement 73
3.2.2 Combinations without Replacement 73
3.2.3 Permutations with Replacement 74
3.2.4 Combinations with Replacement 74
3.3 Multinomial Counting 77
3.4 Special Arrangements and Selections 79
3.5 Applications 81
3.5.1 Game of Poker 81
3.5.2 Birthday Paradox 83
3.5.3 Quality Control 86
3.5.4 Best-of-Seven Championship Series 86
3.5.5 Lottery 89
3.6 Summary 90
Problems 90
Part II Random Variables 95
4 One Random Variable: Fundamentals 97
4.1 Types of Random Variables 97
4.2 The Cumulative Distribution Function 99
4.3 The Probability Mass Function 102
4.4 The Probability Density Function 104
4.5 Expected Values 107
4.5.1 Mean of a Random Variable 107
4.5.2 Variance of a Random Variable 110
4.5.3 Moments of a Random Variable 113
4.5.4 Mode and Median of a Random Variable 114
4.6 Conditional Distributions 116
4.7 Functions of a Random Variable 120
4.7.1 pdf of a Function of a Continuous Random Variable 121
4.7.2 pmf of a Function of a Discrete Random Variable 123
4.7.3 Computer Generation of Random Variables 124
4.8 Transform Methods 125
4.8.1 Moment Generating Function of a Random Variable 125
4.8.2 Characteristic Function of a Random Variable 126
4.9 Upper Bounds on Probability 127
4.9.1 Markov Bound 127
4.9.2 Chebyshev Bound 128
4.9.3 Chernoff Bound 128
4.10 Summary 131
Problems 131
5 Special Probability Distributions and Applications 137
5.1 Special Discrete Random Variables 137
5.1.1 The Bernoulli Distribution 137
5.1.2 The Binomial Distribution 138
5.1.3 The Geometric Distribution 140
5.1.4 The Pascal Distribution 142
5.1.5 The Hypergeometric Distribution 143
5.1.6 The Poisson Distribution 144
5.1.7 The Discrete Uniform Distribution 146
5.1.8 The Zipf (Zeta) Distribution 147
5.2 Special Continuous Random Variables 148
5.2.1 The Continuous Uniform Distribution 148
5.2.2 The Exponential Distribution 149
5.2.3 The Gamma Distribution 151
5.2.4 The Erlang Distribution 152
5.2.5 The Weibull Distribution 152
5.2.6 The Beta Distribution 153
5.2.7 The Laplace Distribution 154
5.2.8 The Pareto Distribution 155
5.3 Applications 156
5.3.1 Digital Transmission: Regenerative Repeaters 156
5.3.2 System Reliability: Failure Rate 157
5.3.3 Queuing Theory: Servicing Customers 158
5.3.4 Random Access: Slotted ALOHA 159
5.3.5 Analog-to-Digital Conversion: Quantization 160
5.4 Summary 161
Problems 162
6 Multiple Random Variables 165
6.1 Pairs of Random Variables 165
6.2 The Joint Cumulative Distribution Function of Two Random Variables 167
6.2.1 Marginal Cumulative Distribution Function 169
6.3 The Joint Probability Mass Function of Two Random Variables 170
6.3.1 Marginal Probability Mass Function 170
6.4 The Joint Probability Density Function of Two Random Variables 171
6.4.1 Marginal Probability Density Function 172
6.5 Expected Values of Functions of Two Random Variables 173
6.5.1 Joint Moments 174
6.6 Independence of Two Random Variables 175
6.7 Correlation between Two Random Variables 178
6.8 Conditional Distributions 185
6.8.1 Conditional Expectations 186
6.9 Distributions of Functions of Two Random Variables 188
6.9.1 Joint Distribution of Two Functions of Two Random Variables 191
6.10 Random Vectors 192
6.11 Summary 197
Problems 198
7 The Gaussian Distribution 201
7.1 The Gaussian Random Variable 201
7.2 The Standard Gaussian Distribution 204
7.3 Bivariate Gaussian Random Variables 210
7.3.1 Linear Transformations of Bivariate Gaussian Random Variables 213
7.4 Jointly Gaussian Random Vectors 215
7.5 Sums of Random Variables 217
7.5.1 Mean and Variance of Sum of Random Variables 217
7.5.2 Mean and Variance of Sum of Independent, Identically Distributed Random Variables 218
7.5.3 Distribution of Sum of Independent Random Variables 218
7.5.4 Sum of a Random Number of Independent, Identically Distributed Random Variables 219
7.6 The Sample Mean 220
7.6.1 Laws of Large Numbers 222
7.7 Approximating Distributions with the Gaussian Distribution 223
7.7.1 Relation between the Gaussian and Binomial Distributions 223
7.7.2 Relation between the Gaussian and Poisson Distributions 225
7.7.3 The Central Limit Theorem 226
7.8 Probability Distributions Related to the Gaussian Distribution 230
7.8.1 The Rayleigh Distribution 230
7.8.2 The Ricean Distribution 231
7.8.3 The Log-Normal Distribution 231
7.8.4 The Chi-Square Distribution 232
7.8.5 The Maxwell-Boltzmann Distribution 232
7.8.6 The Student's t-Distribution 233
7.8.7 The F Distribution 234
7.8.8 The Cauchy Distribution 234
7.9 Summary 234
Problems 235
Part III Statistics 239
8 Descriptive Statistics 241
8.1 Overview of Statistics 241
8.2 Data Displays 244
8.3 Measures of Location 249
8.4 Measures of Dispersion 250
8.5 Measures of Shape 255
8.6 Summary 257
Problems 257
9 Estimation 259
9.1 Parameter Estimation 259
9.2 Properties of Point Estimators 260
9.3 Maximum Likelihood Estimators 265
9.4 Bayesian Estimators 270
9.5 Confidence Intervals 272
9.6 Estimation of a Random Variable 274
9.7 Maximum a Posteriori Probability Estimation 275
9.8 Minimum Mean Square Error Estimation 277
9.9 Linear Minimum Mean Square Error Estimation 279
9.10 Linear MMSE Estimation Using a Vector of Observations 282
9.11 Summary 285
Problems 285
10 Hypothesis Testing 287
10.1 Significance Testing 287
10.2 Hypothesis Testing for Mean 291
10.2.1 p-Value 294
10.3 Decision Tests 300
10.4 Bayesian Test 303
10.4.1 Minimum Cost Test 304
10.4.2 Maximum a Posteriori Probability (MAP) Test 305
10.4.3 Maximum-Likelihood (ML) Test 305
10.4.4 Minimax Test 307
10.5 Neyman-Pearson Test 307
10.6 Summary 309
Problems 309
Part IV Random Processes 311
11 Introduction to Random Processes 313
11.1 Classification of Random Processes 313
11.1.1 State Space 314
11.1.2 Index (Time) Parameter 314
11.2 Characterization of Random Processes 318
11.2.1 Joint Distributions of Time Samples 318
11.2.2 Independent Identically Distributed Random Process 319
11.2.3 Multiple Random Processes 320
11.2.4 Independent Random Processes 320
11.3 Moments of Random Processes 320
11.3.1 Mean and Variance Functions of a Random Process 321
11.3.2 Autocorrelation and Autocovariance Functions of a Random Process 321
11.3.3 Cross-correlation and Cross-covariance Functions 324
11.4 Stationary Random Processes 326
11.4.1 Strict-Sense Stationary Processes 326
11.4.2 Wide-Sense Stationary Processes 327
11.4.3 Jointly Wide-Sense Stationary Processes 329
11.4.4 Cyclostationary Processes 331
11.4.5 Independent and Stationary Increments 331
11.5 Ergodic Random Processes 333
11.5.1 Strict-Sense Ergodic Processes 333
11.5.2 Wide-Sense Ergodic Processes 333
11.6 Gaussian Processes 336
11.7 Poisson Processes 339
11.8 Summary 341
Problems 341
12 Analysis and Processing of Random Processes 345
12.1 Stochastic Continuity, Differentiation, and Integration 345
12.1.1 Mean-Square Continuity 345
12.1.2 Mean-Square Derivatives 346
12.1.3 Mean-Square Integrals 347
12.2 Power Spectral Density 347
12.3 Noise 353
12.3.1 White Noise 353
12.4 Sampling of Random Signals 355
12.5 Optimum Linear Systems 357
12.5.1 Systems Maximizing Signal-to-Noise Ratio 357
12.5.2 Systems Minimizing Mean-Square Error 359
12.6 Summary 362
Problems 362
Bibliography 365
Books 365
Internet Websites 368
Answers 369
Index 387
