| Introduction and Summary | p. 1 |
| Introduction | p. 1 |
| Preliminary Examples | p. 1 |
| Summary of the Following Chapters | p. 6 |
| Continuous-Time Markov Decision Processes | p. 9 |
| Introduction | p. 9 |
| The Control Model | p. 10 |
| Continuous-Time Markov Decision Processes | p. 13 |
| Basic Optimality Criteria | p. 16 |
| Average Optimality for Finite Models | p. 19 |
| Introduction | p. 19 |
| n-bias Optimality Criteria | p. 20 |
| Difference Formulas of n-biases | p. 23 |
| Characterization of n-bias Policies | p. 29 |
| Computation of n-bias Optimal Policies | p. 36 |
| The Policy Iteration Algorithm for Average Optimality | p. 36 |
| The 0-bias Policy Iteration Algorithm | p. 39 |
| n-bias Policy Iteration Algorithms | p. 43 |
| The Linear Programming Approach | p. 46 |
| Linear Programming for Ergodic Models | p. 46 |
| Linear Programming for Multichain Models | p. 49 |
| Notes | p. 52 |
| Discount Optimality for Nonnegative Costs | p. 55 |
| Introduction | p. 55 |
| The Nonnegative Model | p. 55 |
| Preliminaries | p. 56 |
| The Discounted Cost Optimality Equation | p. 60 |
| Existence of Optimal Policies | p. 63 |
| Approximation Results | p. 63 |
| The Policy Iteration Approach | p. 66 |
| Examples | p. 68 |
| Notes | p. 69 |
| Average Optimality for Nonnegative Costs | p. 71 |
| Introduction | p. 71 |
| The Average-Cost Criterion | p. 72 |
| The Minimum Nonnegative Solution Approach | p. 73 |
| The Average-Cost Optimality Inequality | p. 76 |
| The Average-Cost Optimality Equation | p. 80 |
| Examples | p. 81 |
| Notes | p. 84 |
| Discount Optimality for Unbounded Rewards | p. 87 |
| Introduction | p. 87 |
| The Discounted-Reward Optimality Equation | p. 89 |
| Discount Optimal Stationary Policies | p. 95 |
| A Value Iteration Algorithm | p. 98 |
| Examples | p. 98 |
| Notes | p. 102 |
| Average Optimality for Unbounded Rewards | p. 105 |
| Introduction | p. 105 |
| Exponential Ergodicity Conditions | p. 106 |
| The Existence of AR Optimal Policies | p. 109 |
| The Policy Iteration Algorithm | p. 113 |
| Examples | p. 119 |
| Notes | p. 124 |
| Average Optimality for Pathwise Rewards | p. 127 |
| Introduction | p. 127 |
| The Optimal Control Problem | p. 129 |
| Optimality Conditions and Preliminaries | p. 129 |
| The Existence of PAR Optimal Policies | p. 131 |
| Policy and Value Iteration Algorithms | p. 138 |
| An Example | p. 139 |
| Notes | p. 142 |
| Advanced Optimality Criteria | p. 143 |
| Bias and Weakly Overtaking Optimality | p. 143 |
| Sensitive Discount Optimality | p. 147 |
| Blackwell Optimality | p. 159 |
| Notes | p. 160 |
| Variance Minimization | p. 163 |
| Introduction | p. 163 |
| Preliminaries | p. 164 |
| Computation of the Average Variance | p. 164 |
| Variance Minimization | p. 170 |
| Examples | p. 171 |
| Notes | p. 173 |
| Constrained Optimality for Discount Criteria | p. 175 |
| The Model with a Constraint | p. 175 |
| Preliminaries | p. 177 |
| Proof of Theorem 11.4 | p. 182 |
| An Example | p. 184 |
| Notes | p. 186 |
| Constrained Optimality for Average Criteria | p. 187 |
| Average Optimality with a Constraint | p. 187 |
| Preliminaries | p. 188 |
| Proof of Theorem 12.4 | p. 192 |
| An Example | p. 192 |
| Notes | p. 194 |
| p. 195 |
| Limit Theorems | p. 195 |
| Results from Measure Theory | p. 197 |
| p. 203 |
| Continuous-Time Markov Chains | p. 203 |
| Stationary Distributions and Ergodicity | p. 206 |
| p. 209 |
| The Construction of Transition Functions | p. 209 |
| Ergodicity Based on the Q-Matrix | p. 214 |
| Dynkin's Formula | p. 218 |
| References | p. 221 |
| Index | p. 229 |
| Table of Contents provided by Ingram. All Rights Reserved. |
