Continuoustime Markov chains and applications : a twotimescale approach
 Author/Creator
 Yin, George, 1954
 Language
 English.
 Edition
 2nd ed., [rev. and expanded]
 Imprint
 New York : Springer, c2013.
 Physical description
 xxi, 427 p. : ill. ; 25 cm.
 Series
 Stochastic modelling and applied probability 37.
Access
Contributors
 Contributor
 Zhang, Qing, 1959
Contents/Summary
 Bibliography
 Includes bibliographical references and index.
 Contents

 Prologue and preliminaries. Introduction and overview; Introduction; A brief survey: Markov chains: Singular perturbations; Outline of the book ; Mathematical preliminaries; Introduction; Martingales; Markov chains; piecewisedeterministic processes: Construction of Markov chains; Irreducibility and Quasistationary distributions; Gaussian processes and diffusions; Switching diffusions; Notes ; Markovian models; Introduction; Birth and death processes; Finitestate space models: Queues with finite capacity: System reliability: Competing risk theory: Twotimescale Cox processes: Random evolutions: Seasonal variation models; Stochastic optimization problems: Simulated annealing: Continuoustime stochastic approximation: Systems with Markovian disturbances; Linear systems with jump Markov disturbance: Linear Quadratic control problems: Singularly perturbed LQ systems with wideband noise: Largescale systems : decomposition and aggregation; Timescale separation; Notes
 Twotimescale Markov chains. Asymptotic expansions of solutions for forward equations; Introduction; Irreducible case: Asymptotic expansions: Outer expansion: Initiallayer correction: Exponential decay of ...: Asymptotic validation: Examples: Twotimescale expansion; Markov chains with multiple weakly irreducible classes: Asymptotic expansions: Analysis of remainder: Computational procedure : user's guide: Summary of results: An example; Inclusion of absorbing states; Inclusion of transient states; Remarks on countablestatespace cases: Countablestate spaces : Part I: Countablestate spaces : Part II: A remarks on finitedimensional approximation; Remarks on singularly perturbed diffusions; Notes ; Occupation measures : Asymptotic properties and ramification; Introduction; The irreducible case: Occupation measure: Conditions and preliminary results: Exponential bounds: Asymptotic normality: Extensions; Markov chains with weak and strong interactions: Aggregation of Markov chains: Exponential bounds: Asymptotic distributions; Measurable generators; Remarks on inclusion of transient and absorbing states: Inclusion of transient states: Inclusion of absorbing states; Remarks on a stability problem: Notes ; Asymptotic expansions of solutions for backward equations; Introduction; Problem formulation: A preliminary lemma: Formulation; Construction of asymptotic expansions: Leading term ... and zeroorder terminallayer term ...: Higherorder terms; Error estimates; Asymptotic expansions including transient states; Remarks: Related problems; Notes
 Applications : MDPs, nearoptimal controls, numerical methods, and LQG with switching. Markov decision problems; Introduction; Problem formulation; Limit problem; Asymptotic optimality; Convergence rate and error bound; Longrun average cost; Computational procedures; Notes ; Stochastic control of dynamical systems; Introduction; Problem formulation; Properties of the value functions; Asymptotic optimal controls; Convergence rate; Weak convergence approach: Problem setup: Relaxed control formulation: Near optimality; Notes ; Numerical methods for control and optimization; Introduction; Numerical methods for optimal control; Optimization under threshold policy: Stochastic optimization formulation: Convergence: Examples: Error bounds; Notes ; Hybrid LQG problems; Introduction; Problem formulation; Optimal controls; Twotimeseale approximation : recurrent states: Limit riccati equations: Nearly optimal controls; Twotimescale approximation : inclusion of transient states; Twotimescale approximation : inclusion of absorbing states; A numerical example; Remarks on indefinite control weights; Notes ; Background materials; Properties of generators; Weak convergence; Relaxed control; Viscosity solutions of HJB equations; Value functions and optimal controls; Miscellany ; Bibliography ; Index.
 Publisher's Summary
 This book gives a systematic treatment of singularly perturbed systems that naturally arise in control and optimization, queueing networks, manufacturing systems, and financial engineering. It presents results on asymptotic expansions of solutions of Komogorov forward and backward equations, properties of functional occupation measures, exponential upper bounds, and functional limit results for Markov chains with weak and strong interactions. To bridge the gap between theory and applications, a large portion of the book is devoted to applications in controlled dynamic systems, production planning, and numerical methods for controlled Markovian systems with largescale and complex structures in the realworld problems. This second edition has been updated throughout and includes two new chapters on asymptotic expansions of solutions for backward equations and hybrid LQG problems. The chapters on analytic and probabilistic properties of twotimescale Markov chains have been almost completely rewritten and the notation has been streamlined and simplified. This book is written for applied mathematicians, engineers, operations researchers, and applied scientists. Selected material from the book can also be used for a one semester advanced graduatelevel course in applied probability and stochastic processes.
(source: Nielsen Book Data)
Subjects
Bibliographic information
 Publication date
 2013
 Responsibility
 G. George Yin, Qing Zhang.
 Series
 Stochastic modelling and applied probability, 01724568 ; 37
 Note
 Subtitle of 1st ed.: a singular perturbation approach.
 ISBN
 9781461443452
 1461443458