Preface xi
1 STATIC OPTIMIZATION 1
1.1 Optimization without Constraints 1
1.2 Optimization with Equality Constraints 4
1.3 Numerical Solution Methods 15
Problems 15
2 OPTIMAL CONTROL OF DISCRETE-TIME SYSTEMS 19
2.1 Solution of the General Discrete-Time Optimization Problem 19
2.2 Discrete-Time Linear Quadratic Regulator 32
2.3 Digital Control of Continuous-Time Systems 53
2.4 Steady-State Closed-Loop Control and Suboptimal Feedback 65
2.5 Frequency-Domain Results 96
Problems 102
3 OPTIMAL CONTROL OF CONTINUOUS-TIME SYSTEMS 110
3.1 The Calculus of Variations 110
3.2 Solution of the General Continuous-Time Optimization Problem 112
3.3 Continuous-Time Linear Quadratic Regulator 135
3.4 Steady-State Closed-Loop Control and Suboptimal Feedback 154
3.5 Frequency-Domain Results 164
Problems 167
4 THE TRACKING PROBLEM AND OTHER LQR EXTENSIONS 177
4.1 The Tracking Problem 177
4.2 Regulator with Function of Final State Fixed 183
4.3 Second-Order Variations in the Performance Index 185
4.4 The Discrete-Time Tracking Problem 190
4.5 Discrete Regulator with Function of Final State Fixed 199
4.6 Discrete Second-Order Variations in the Performance Index 206
Problems 211
5 FINAL-TIME-FREE AND CONSTRAINED INPUT CONTROL 213
5.1 Final-Time-Free Problems 213
5.2 Constrained Input Problems 232
Problems 257
6 DYNAMIC PROGRAMMING 260
6.1 Bellman’s Principle of Optimality 260
6.2 Discrete-Time Systems 263
6.3 Continuous-Time Systems 271
Problems 283
7 OPTIMAL CONTROL FOR POLYNOMIAL SYSTEMS 287
7.1 Discrete Linear Quadratic Regulator 287
7.2 Digital Control of Continuous-Time Systems 292
Problems 295
8 OUTPUT FEEDBACK AND STRUCTURED CONTROL 297
8.1 Linear Quadratic Regulator with Output Feedback 297
8.2 Tracking a Reference Input 313
8.3 Tracking by Regulator Redesign 327
8.4 Command-Generator Tracker 331
8.5 Explicit Model-Following Design 338
8.6 Output Feedback in Game Theory and Decentralized Control 343
Problems 351
9 ROBUSTNESS AND MULTIVARIABLE FREQUENCY-DOMAIN TECHNIQUES 355
9.1 Introduction 355
9.2 Multivariable Frequency-Domain Analysis 357
9.3 Robust Output-Feedback Design 380
9.4 Observers and the Kalman Filter 383
9.5 LQG/Loop-Transfer Recovery 408
9.6 H∞ DESIGN 430
Problems 435
10 DIFFERENTIAL GAMES 438
10.1 Optimal Control Derived Using Pontryagin’s Minimum Principle and the Bellman Equation 439
10.2 Two-player Zero-sum Games 444
10.3 Application of Zero-sum Games to H∞ Control 450
10.4 Multiplayer Non-zero-sum Games 453
11 REINFORCEMENT LEARNING AND OPTIMAL ADAPTIVE CONTROL 461
11.1 Reinforcement Learning 462
11.2 Markov Decision Processes 464
11.3 Policy Evaluation and Policy Improvement 474
11.4 Temporal Difference Learning and Optimal Adaptive Control 489
11.5 Optimal Adaptive Control for Discrete-time Systems 490
11.6 Integral Reinforcement Learning for Optimal Adaptive Control of Continuous-time Systems 503
11.7 Synchronous Optimal Adaptive Control for Continuous-time Systems 513
APPENDIX A REVIEW OF MATRIX ALGEBRA 518
A.1 Basic Definitions and Facts 518
A.2 Partitioned Matrices 519
A.3 Quadratic Forms and Definiteness 521
A.4 Matrix Calculus 523
A.5 The Generalized Eigenvalue Problem 525
References 527
Index 535
