About the Authors xxi
Preface xxiii
Acknowledgments xxvii
About the Companion Website xxix
1 Choosing Credibility 1
1.1 The Responsibility of an Experimentalist 2
1.2 Losses of Credibility 2
1.3 Recovering Credibility 3
1.4 Starting with a Sharp Axe 3
1.5 A Systems View of Experimental Work 4
1.6 In Defense of Being a Generalist 5
Panel 1.1 The Bundt Cake Story 6
References 6
Homework 6
2 The Nature of Experimental Work 7
2.1 Tested Guide of Strategy and Tactics 7
2.2 What Can Be Measured and What Cannot? 8
2.2.1 Examples Not Measurable 8
2.2.2 Shapes 9
2.2.3 Measurable by the Human Sensory System 10
2.2.4 Identifying and Selecting Measurable Factors 11
2.2.5 Intrusive Measurements 11
2.3 Beware Measuring Without Understanding: Warnings from History 12
2.4 How Does Experimental Work Differ from Theory and Analysis? 13
2.4.1 Logical Mode 13
2.4.2 Persistence 13
2.4.3 Resolution 13
2.4.4 Dimensionality 15
2.4.5 Similarity and Dimensional Analysis 15
2.4.6 Listening to Our Theoretician Compatriots 16
Panel 2.1 Positive Consequences of the Reproducibility Crisis 17
Panel 2.2 Selected Invitations to Experimental Research, Insights from Theoreticians 18
Panel 2.3 Prepublishing Your Experiment Plan 21
2.4.7 Surveys and Polls 22
2.5 Uncertainty 23
2.6 Uncertainty Analysis 23
References 24
Homework 25
3 An Overview of Experiment Planning 27
3.1 Steps in an Experimental Plan 27
3.2 Iteration and Refinement 28
3.3 Risk Assessment/Risk Abatement 28
3.4 Questions to Guide Planning of an Experiment 29
Homework 30
4 Identifying the Motivating Question 31
4.1 The Prime Need 31
Panel 4.1 There’s a Hole in My Bucket 32
4.2 An Anchor and a Sieve 33
4.3 Identifying the Motivating Question Clarifies Thinking 33
4.3.1 Getting Started 33
4.3.2 Probe and Focus 34
4.4 Three Levels of Questions 35
4.5 Strong Inference 36
4.6 Agree on the Form of an Acceptable Answer 36
4.7 Specify the Allowable Uncertainty 37
4.8 Final Closure 37
Reference 38
Homework 38
5 Choosing the Approach 39
5.1 Laying Groundwork 39
5.2 Experiment Classifications 40
5.2.1 Exploratory 40
5.2.2 Identifying the Important Variables 40
5.2.3 Demonstration of System Performance 41
5.2.4 Testing a Hypothesis 41
5.2.5 Developing Constants for Predetermined Models 41
5.2.6 Custody Transfer and System Performance Certification Tests 42
5.2.7 Quality-Assurance Tests 42
5.2.8 Summary 43
5.3 Real or Simplified Conditions? 43
5.4 Single-Sample or Multiple-Sample? 43
Panel 5.1 A Brief Summary of “Dissertation upon Roast Pig” 44
Panel 5.2 Consider a Spherical Cow 44
5.5 Statistical or Parametric Experiment Design? 45
5.6 Supportive or Refutative? 47
5.7 The Bottom Line 47
References 48
Homework 48
6 Mapping for Safety, Operation, and Results 51
6.1 Construct Multiple Maps to Illustrate and Guide Experiment Plan 51
6.2 Mapping Prior Work and Proposed Work 51
6.3 Mapping the Operable Domain of an Apparatus 53
6.4 Mapping in Operator’s Coordinates 57
6.5 Mapping the Response Surface 59
6.5.1 Options for Organizing a Table 59
6.5.2 Options for Presenting the Response on a Scatter-Plot-Type Graph 61
Homework 64
7 Refreshing Statistics 65
7.1 Reviving Key Terms to Quantify Uncertainty 65
7.1.1 Population 65
7.1.2 Sample 66
7.1.3 Central Value 67
7.1.4 Mean, μ or ? 67
7.1.5 Residual 67
7.1.6 Variance, σ2 or S2 68
7.1.7 Degrees of Freedom, Df 68
7.1.8 Standard Deviation, σY or SY 68
7.1.9 Uncertainty of the Mean, δμ 69
7.1.10 Chi?Squared, χ2 69
7.1.11 p?Value 70
7.1.12 Null Hypothesis 70
7.1.13 F?value of Fisher Statistic 71
7.2 The Data Distribution Most Commonly Encountered The Normal Distribution for Samples of Infinite Size 71
7.3 Account for Small Samples: The t?Distribution 72
7.4 Construct Simple Models by Computer to Explain the Data 73
7.4.1 Basic Statistical Analysis of Quantitative Data 73
7.4.2 Model Data Containing Categorical and Quantitative Factors 75
7.4.3 Display Data Fit to One Categorical Factor and One Quantitative Factor 76
7.4.4 Quantify How Each Factor Accounts for Variation in the Data 76
7.5 Gain Confidence and Skill at Statistical Modeling Via the R Language 77
7.5.1 Model and Plot Results of a Single Variable Using the Example Data diceshoe.csv 77
7.5.2 Evaluate Alternative Models of the Example Data hiloy.csv 78
7.5.2.1 Inspect the Data 78
7.5.3 Grand Mean 78
7.5.4 Model by Groups: Group?Wise Mean 78
7.5.5 Model by a Quantitative Factor 78
7.5.6 Model by Multiple Quantitative Factors 78
7.5.7 Allow Factors to Interact (So Each Group Gets Its Own Slope) 79
7.5.8 Include Polynomial Factors (a Statistical Linear Model Can Be Curved) 80
7.6 Report Uncertainty 80
7.7 Decrease Uncertainty (Improve Credibility) by Isolating Distinct Groups 81
7.8 Original Data, Summary, and R 82
References 83
Homework 83
8 Exploring Statistical Design of Experiments 87
8.1 Always Seeking Wiser Strategies 87
8.2 Evolving from Novice Experiment Design 87
8.3 Two?Level and Three?Level Factorial Experiment Plans 88
8.4 A Three?Level, Three?Factor Design 89
8.5 The Plackett–Burman 12?Run Screening Design 93
8.6 Details About Analysis of Statistically Designed Experiments 95
8.6.1 Model Main Factors to Original Raw Data 95
8.6.2 Model Main Factors to Original Data Around Center of Each Factor 96
8.6.3 Model Including All Interaction Terms 97
8.6.4 Model Including Only Dominant Interaction Terms 97
8.6.5 Model Including Dominant Interaction Term Plus Quadratic Term 98
8.6.6 Model All Factors of Example 2, Centering Each Quantitative Factor 99
8.6.7 Refine Model of Example 2 Including Only Dominant Terms 100
8.7 Retrospect of Statistical Design Examples 101
8.8 Philosophy of Statistical Design 101
8.9 Statistical Design for Conditions That Challenge Factorial Designs 102
8.10 A Highly Recommended Tool for Statistical Design of Experiments 103
8.11 More Tools for Statistical Design of Experiments 103
8.12 Conclusion 103
Further Reading 104
Homework 104
9 Selecting the Data Points 107
9.1 The Three Categories of Data 107
9.1.1 The Output Data 107
9.1.2 Peripheral Data 108
9.1.3 Backup Data 108
9.1.4 Other Data You May Wish to Acquire 108
9.2 Populating the Operating Volume 109
9.2.1 Locating the Data Points Within the Operating Volume 109
9.2.2 Estimating the Topography of the Response Surface 109
9.3 Example from Velocimetry 109
9.3.1 Sharpen Our Approach 110
9.3.2 Lessons Learned from Velocimetry Example 111
9.4 Organize the Data 112
9.4.1 Keep a Laboratory Notebook 112
9.4.2 Plan for Data Security 112
9.4.3 Decide Data Format 112
9.4.4 Overview Data Guidelines 112
9.4.5 Reasoning Through Data Guidelines 113
9.5 Strategies to Select Next Data Points 114
9.5.1 Overview of Option 1: Default Strategy with Intensive Experimenter Involvement 115
9.5.1.1 Choosing the Data Trajectory 115
9.5.1.2 The Default Strategy: Be Bold 115
9.5.1.3 Anticipate, Check, Course Correct 116
9.5.1.4 Other Aspects to Keep in Mind 116
9.5.1.5 Endpoints 117
9.5.2 Reintroducing Gosset 118
9.5.3 Practice Gosset Examples (from Gosset User Manual) 119
9.6 Demonstrate Gosset for Selecting Data 120
9.6.1 Status Quo of Experiment Planning and Execution (Prior to Selecting More Samples) 120
9.6.1.1 Specified Motivating Question 120
9.6.1.2 Identified Pertinent Candidate Factors 121
9.6.1.3 Selected Initial Sample Points Using Plackett–Burman 121
9.6.1.4 Executed the First 12 Runs at the PB Sample Conditions 122
9.6.1.5 Analyzed Results. Identified Dominant First-Order Factors. Estimated First-Order Uncertainties of Factors 123
9.6.1.6 Generated Draft Predictive Equation 124
9.6.2 Use Gosset to Select Additional Data Samples 125
9.6.2.1 Example Gosset Session: User Input to Select Next Points 125
9.6.2.2 Example Gosset Session: How We Chose User Input 126
9.6.2.3 Example Gosset Session: User Input Along with Gosset Output 128
9.6.2.4 Example Gosset Session: Convert the Gosset Design to Operator Values 131
9.6.2.5 Results of Example Gosset Session: Operator Plots of Total Experiment Plan 132
9.6.2.6 Execute Stage Two of the Experiment Plan: User Plus Gosset Sample Points 132
9.7 Use Gosset to Analyze Results 133
9.8 Other Options and Features of Gosset 133
9.9 Using Gosset to Find Local Extrema in a Function of Several Variables 134
9.10 Summary 137
Further Reading 137
Homework 137
10 Analyzing Measurement Uncertainty 143
10.1 Clarifying Uncertainty Analysis 143
10.1.1 Distinguish Error and Uncertainty 144
10.1.1.1 Single-Sample vs. Multiple-Sample 145
10.1.2 Uncertainty as a Diagnostic Tool 146
10.1.2.1 What Can Uncertainty Analysis Tell You? 146
10.1.2.2 What is Uncertainty Analysis Good For? 148
10.1.2.3 Uncertainty Analysis Can Redirect a Poorly Conceived Experiment 148
10.1.2.4 Uncertainty Analysis Improves the Quality of Your Work 148
10.1.2.5 Slow Sampling and “Randomness” 149
10.1.2.6 Uncertainty Analysis Makes Results Believable 150
10.1.3 Uncertainty Analysis Aids Management Decision-Making 150
10.1.3.1 Management’s Task: Dealing with Warranty Issues 150
10.1.4 The Design Group’s Task: Setting Tolerances on Performance Test Repeatability 152
10.1.5 The Performance Test Group’s Task: Setting the Tolerances on Measurements 152
10.2 Definitions 153
10.2.1 True Value 153
10.2.2 Corrected Value 153
10.2.3 Data Reduction Program 153
10.2.4 Accuracy 153
10.2.5 Error 154
10.2.6 XXXX Error 154
10.2.7 Fixed Error 154
10.2.8 Residual Fixed Error 154
10.2.9 Random Error 154
10.2.10 Variable (but Deterministic) Error 155
10.2.11 Uncertainty 155
10.2.12 Odds 155
10.2.13 Absolute Uncertainty 155
10.2.14 Relative Uncertainty 155
10.3 The Sources and Types of Errors 156
10.3.1 Types of Errors: Fixed, Random, and Variable 156
10.3.2 Sources of Errors: The Measurement Chain 156
10.3.2.1 The Undisturbed Value 158
10.3.2.2 The Available Value 158
10.3.2.3 The Achieved Value 158
10.3.2.4 The Observed Value 159
10.3.2.5 The Corrected Value 159
10.3.3 Specifying the True Value 160
10.3.3.1 If the Achieved Value is Taken as the True Value 160
10.3.3.2 If the Available Value is Taken as the True Value 163
10.3.3.3 If the Undisturbed Value is Taken as the True Value 166
10.3.3.4 If the Mixed Mean Gas Temperature is Taken as the True Value 167
10.3.4 The Role of the End User 167
10.3.4.1 The End-Use Equations Implicitly Define the True Value 167
10.3.5 Calibration 168
10.4 The Basic Mathematics 170
10.4.1 The Root-Sum-Squared (RSS) Combination 170
10.4.2 The Fixed Error in a Measurement 171
10.4.3 The Random Error in a Measurement 172
10.4.4 The Uncertainty in a Measurement 173
10.4.5 The Uncertainty in a Calculated Result 174
10.4.5.1 The Relative Uncertainty in a Result 176
10.5 Automating the Uncertainty Analysis 178
10.5.1 The Mathematical Basis 178
10.5.2 Example of Uncertainty Analysis by Spreadsheet 179
10.6 Single-Sample Uncertainty Analysis 181
10.6.1 Assembling the Necessary Inputs 184
10.6.2 Calculating the Uncertainty in the Result 185
10.6.3 The Three Levels of Uncertainty: Zeroth-, First-, and Nth-Order 185
10.6.3.1 Zeroth-Order Replication 186
10.6.3.2 First-Order Replication 187
10.6.3.3 Nth-Order Replication 188
10.6.4 Fractional-Order Replication for Special Cases 188
10.6.5 Summary of Single-Sample Uncertainty Levels 189
10.6.5.1 Zeroth-Order 189
10.6.5.2 First-Order 190
10.6.5.3 Nth-Order 190
References 190
Further Reading 191
Homework 191
11 Using Uncertainty Analysis in Planning and Execution 197
11.1 Using Uncertainty Analysis in Planning 197
11.1.1 The Physical Situation and Energy Analysis 198
11.1.2 The Steady?State Method 199
11.1.3 The Transient Method 200
11.1.4 Reflecting on Assumptions Made During DRE Derivations 201
11.2 Perform Uncertainty Analysis on the DREs 202
11.2.1 Uncertainty Analysis: General Form 202
11.2.2 Uncertainty Analysis of the Steady?State Method 203
11.2.3 Uncertainty Analysis – Transient Method 204
11.2.4 Compare the Results of Uncertainty Analysis of the Methods 205
11.2.5 What Does the Calculated Uncertainty Interval Mean? 206
11.2.6 Cross?Checking the Experiment 207
11.2.7 Conclusions 207
11.3 Using Uncertainty Analysis in Selecting Instruments 208
11.4 Using Uncertainty Analysis in Debugging an Experiment 209
11.4.1 Handling Overall Scatter 209
11.4.2 Sources of Scatter 210
11.4.3 Advancing Toward Calibration 211
11.4.4 Selecting Thresholds 212
11.4.5 Iterating Analysis 212
11.4.6 Rechecking Situational Uncertainty 212
11.5 Reporting the Uncertainties in an Experiment 213
11.5.1 Progress in Uncertainty Reporting 214
11.6 Multiple?Sample Uncertainty Analysis 214
11.6.1 Revisiting Single?Sample and Multiple?Sample Uncertainty Analysis 214
11.6.2 Examples of Multiple?Sample Uncertainty Analysis 215
11.6.3 Fixed Error and Random Error 216
11.7 Coordinate with Uncertainty Analysis Standards 216
11.7.1 Describing Fixed and Random Errors in a Measurement 217
11.7.2 The Bias Limit 217
11.7.2.1 Fossilization 218
11.7.2.2 Bias Limits 218
11.7.3 The Precision Index 219
11.7.4 The Number of Degrees of Freedom 220
11.8 Describing the Overall Uncertainty in a Single Measurement 220
11.8.1 Adjusting for a Single Measurement 220
11.8.2 Describing the Overall Uncertainty in a Result 221
11.8.3 Adding the Overall Uncertainty to Predictive Models 222
11.9 Additional Statistical Tools and Elements 222
11.9.1 Pooled Variance 222
11.9.1.1 Student’s t?Distribution – Pooled Examples 223
11.9.2 Estimating the Standard Deviation of a Population from the Standard Deviation of a Small Sample: The Chi?Squared χ2 Distribution 223
References 225
Homework 226
12 Debugging an Experiment, Shakedown, and Validation 231
12.1 Introduction 231
12.2 Classes of Error 231
12.3 Using Time-Series Analysis in Debugging 232
12.4 Examples 232
12.4.1 Gas Temperature Measurement 232
12.4.2 Calibration of a Strain Gauge 233
12.4.3 Lessons Learned from Examples 234
12.5 Process Unsteadiness 234
12.6 The Effect of Time-Constant Mismatching 235
12.7 Using Uncertainty Analysis in Debugging an Experiment 236
12.7.1 Calibration and Repeatability 236
12.7.2 Stability and Baselining 238
12.8 Debugging the Experiment via the Data Interpretation Program 239
12.8.1 Debug the Experiment via the DIP 239
12.8.2 Debug the Interface of the DIP 239
12.8.3 Debug Routines in the DIP 240
12.9 Situational Uncertainty 241
13 Trimming Uncertainty 243
13.1 Focusing on the Goal 243
13.2 A Motivating Question for Industrial Production 243
13.2.1 Agreed Motivating Questions for Industrial Example 244
13.2.2 Quick Answers to Motivating Questions 244
13.2.3 Challenge: Precheck Analysis and Answers 245
13.3 Plackett–Burman 12-Run Results and Motivating Question #3 245
13.4 PB 12-Run Results and Motivating Question #1 247
13.4.1 Building a Predictive Model Equation from R-Language Linear Model 248
13.4.2 Parsing the Dual Predictive Model Equation 249
13.4.3 Uncertainty of the Intercept in the Dual Predictive Model Equation 250
13.4.4 Mapping an Answer to Motivating Question #1 251
13.4.5 Tentative Answers to Motivating Question #1 252
13.5 Uncertainty Analysis of Dual Predictive Model and Motivating Question #2 252
13.5.1 Uncertainty of the Constant in the Dual Predictive Model Equation 252
13.5.2 Uncertainty of Other Factors in the Dual Predictive Model Equation 253
13.5.3 Include All Coefficient Uncertainties in the Dual Predictive Model Equation 254
13.5.4 Overall Uncertainty from All Factors in the Predictive Model Equation 254
13.5.5 Improved Tentative Answers to Motivating Questions, Including Uncertainties 256
13.5.6 Search for Improved Predictive Models 256
13.6 The PB 12-Run Results and Individual Machine Models 256
13.6.1 Individual Machine Predictive Model Equations 258
13.6.2 Uncertainty of the Intercept in the Individual Predictive Model Equations 258
13.6.3 Uncertainty of the Constant in the Individual Predictive Model Equations 259
13.6.4 Uncertainty of Other Factors in the Individual Predictive Model Equation 259
13.6.4.1 Uncertainties of Machine 1 259
13.6.4.2 Uncertainties of Machine 2 260
13.6.4.3 Including Instrument and Measurement Uncertainties 260
13.6.5 Include All Coefficient Uncertainties in the Individual Predictive Model Equations 260
13.6.6 Overall Uncertainty from All Factors in the Individual Predictive Model Equations 261
13.6.7 Quick Overview of Individual Machine Performance Over the Operating Map 262
13.7 Final Answers to All Motivating Questions for the PB Example Experiment 263
13.7.1 Answers to Motivating Question #1 263
13.7.2 Answers to Motivating Question #2 263
13.7.3 Answers to Motivating Question #3 (Expanded from Section 13.3) 263
13.7.4 Answers to Motivating Question #4 264
13.7.5 Other Recommendations (to Our Client) 264
13.8 Conclusions 265
Homework 266
14 Documenting the Experiment: Report Writing 269
14.1 The Logbook 269
14.2 Report Writing 269
14.2.1 Organization of the Reports 270
14.2.2 Who Reads What? 270
14.2.3 Picking a Viewpoint 271
14.2.4 What Goes Where? 271
14.2.4.1 What Goes in the Abstract? 272
14.2.4.2 What Goes in the Foreword? 272
14.2.4.3 What Goes in the Objective? 273
14.2.4.4 What Goes in the Results and Conclusions? 273
14.2.4.5 What Goes in the Discussion? 274
14.2.4.6 References 274
14.2.4.7 Figures 275
14.2.4.8 Tables 276
14.2.4.9 Appendices 276
14.2.5 The Mechanics of Report Writing 276
14.2.6 Clear Language Versus “JARGON” 277
Panel 14.1 The Turbo-Encabulator 278
14.2.7 “Gobbledygook”: Structural Jargon 279
Panel 14.2 U.S. Code, Title 18, No. 793 279
14.2.8 Quantitative Writing 281
14.2.8.1 Substantive Versus Descriptive Writing 281
Panel 14.3 The Descriptive Bank Statement 281
14.2.8.2 Zero-Information Statements 281
14.2.8.3 Change 282
14.3 International Organization for Standardization, ISO 9000 and other Standards 282
14.4 Never Forget. Always Remember 282
Appendix A: Distributing Variation and Pooled Variance 283
A.1 Inescapable Distributions 283
A.1.1 The Normal Distribution for Samples of Infinite Size 283
A.1.2 Adjust Normal Distributions with Few Data: The Student’s t-Distribution 283
A.2 Other Common Distributions 286
A.3 Pooled Variance (Advanced Topic) 286
Appendix B: Illustrative Tables for Statistical Design 289
B.1 Useful Tables for Statistical Design of Experiments 289
B.1.1 Ready-made Ordering for Randomized Trials 289
B.1.2 Exhausting Sets of Two-Level Factorial Designs (≤ Five Factors) 289
B.2 The Plackett–Burman (PB) Screening Designs 289
Appendix C: Hand Analysis of a Two-Level Factorial Design 293
C.1 The General Two-Level Factorial Design 293
C.2 Estimating the Significance of the Apparent Factor Effects 298
C.3 Hand Analysis of a Plackett–Burman (PB) 12-Run Design 299
C.4 Illustrative Practice Example for the PB 12-Run Pattern 302
C.4.1 Assignment: Find Factor Effects and the Linear Coefficients Absent Noise 302
C.4.2 Assignment: Find Factor Effects and the Linear Coefficients with Noise 303
C.5 Answer Key: Compare Your Hand Calculations 303
C.5.1 Expected Results Absent Noise (compare C.4.1) 303
C.5.2 Expected Results with Random Gaussian Noise (cf. C.4.2) 304
C.6 Equations for Hand Calculations 305
Appendix D: Free Recommended Software 307
D.1 Instructions to Obtain the R Language for Statistics 307
D.2 Instructions to Obtain LibreOffice 308
D.3 Instructions to Obtain Gosset 308
D.4 Possible Use of RStudio 309
Index 311
