2nd ed., [rev. and expanded] - New York : Springer, c2013.
xxi, 427 p. : ill. ; 25 cm.
Subtitle of 1st ed.: a singular perturbation approach.
Includes bibliographical references and index.
Prologue and preliminaries. Introduction and overview; Introduction; A brief survey: Markov chains: Singular perturbations; Outline of the book ; Mathematical preliminaries; Introduction; Martingales; Markov chains; piecewise-deterministic processes: Construction of Markov chains; Irreducibility and Quasi-stationary distributions; Gaussian processes and diffusions; Switching diffusions; Notes ; Markovian models; Introduction; Birth and death processes; Finite-state space models: Queues with finite capacity: System reliability: Competing risk theory: Two-time-scale Cox processes: Random evolutions: Seasonal variation models; Stochastic optimization problems: Simulated annealing: Continuous-time stochastic approximation: Systems with Markovian disturbances; Linear systems with jump Markov disturbance: Linear Quadratic control problems: Singularly perturbed LQ systems with wide-band noise: Large-scale systems : decomposition and aggregation; Time-scale separation; Notes
Two-time-scale Markov chains. Asymptotic expansions of solutions for forward equations; Introduction; Irreducible case: Asymptotic expansions: Outer expansion: Initial-layer correction: Exponential decay of ...: Asymptotic validation: Examples: Two-time-scale expansion; Markov chains with multiple weakly irreducible classes: Asymptotic expansions: Analysis of remainder: Computational procedure : user's guide: Summary of results: An example; Inclusion of absorbing states; Inclusion of transient states; Remarks on countable-state-space cases: Countable-state spaces : Part I: Countable-state spaces : Part II: A remarks on finite-dimensional approximation; Remarks on singularly perturbed diffusions; Notes ; Occupation measures : Asymptotic properties and ramification; Introduction; The irreducible case: Occupation measure: Conditions and preliminary results: Exponential bounds: Asymptotic normality: Extensions; Markov chains with weak and strong interactions: Aggregation of Markov chains: Exponential bounds: Asymptotic distributions; Measurable generators; Remarks on inclusion of transient and absorbing states: Inclusion of transient states: Inclusion of absorbing states; Remarks on a stability problem: Notes ; Asymptotic expansions of solutions for backward equations; Introduction; Problem formulation: A preliminary lemma: Formulation; Construction of asymptotic expansions: Leading term ... and zero-order terminal-layer term ...: Higher-order terms; Error estimates; Asymptotic expansions including transient states; Remarks: Related problems; Notes
Applications : MDPs, near-optimal controls, numerical methods, and LQG with switching. Markov decision problems; Introduction; Problem formulation; Limit problem; Asymptotic optimality; Convergence rate and error bound; Long-run average cost; Computational procedures; Notes ; Stochastic control of dynamical systems; Introduction; Problem formulation; Properties of the value functions; Asymptotic optimal controls; Convergence rate; Weak convergence approach: Problem setup: Relaxed control formulation: Near optimality; Notes ; Numerical methods for control and optimization; Introduction; Numerical methods for optimal control; Optimization under threshold policy: Stochastic optimization formulation: Convergence: Examples: Error bounds; Notes ; Hybrid LQG problems; Introduction; Problem formulation; Optimal controls; Two-time-seale approximation : recurrent states: Limit riccati equations: Nearly optimal controls; Two-time-scale approximation : inclusion of transient states; Two-time-scale approximation : inclusion of absorbing states; A numerical example; Remarks on indefinite control weights; Notes ; Background materials; Properties of generators; Weak convergence; Relaxed control; Viscosity solutions of HJB equations; Value functions and optimal controls; Miscellany ; Bibliography ; Index.