15 Markov Chains, Decision Processes and Replacement
A customer switches brands, an account moves from current to overdue to paid, a machine wears from good to poor to broken. In each case the system moves among a few states at random, and the probability of the next move depends only on the present state. Such a system is a Markov chain, and its long-run behaviour is computed by linear algebra. When a decision maker can influence the moves, by advertising, chasing debts or replacing machines, the chain becomes a Markov decision process, solved by the dynamic programming of the section on that subject extended to an infinite horizon. The classical replacement models of OR are among its first applications.
15.1 Markov chains
15.2 Absorbing chains
15.3 Replacement of deteriorating equipment
15.4 Markov decision processes
15.5 Markov models in software
15.6 Practice Problems
15.2 Absorbing chains
15.3 Replacement of deteriorating equipment
15.4 Markov decision processes
15.5 Markov models in software
15.6 Practice Problems
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.