2.2 Markov Processes
Definition 2.2.1. A process \(\{X(t) : t\in T\}\) with countable state space is a Markov process if, for all times \(t_0<t_1<\cdots <t_n<t\) in \(T\) and all states \(x_0, \dots , x_n, x\), \[P\left (X(t) = x\mid X(t_n) = x_n, X(t_{n-1}) = x_{n-1}, \dots , X(t_0) = x_0\right ) = P\left (X(t) = x\mid X(t_n) = x_n\right ).\] In discrete time this reads \[P(X_{n + 1} = j\mid X_n = i, X_{n-1} = i_{n-1}, \dots , X_0 = i_0) = P(X_{n + 1} = j\mid X_n = i)\] for all \(n\) and all states.
In words: given the present state, the future is independent of the past. The process has no memory beyond its current state; how it got there is irrelevant to where it goes next.
Example 2.2.2. Let \(Y_1, Y_2, \dots \) be independent with \(P(Y_i = 1) = p\) and \(P(Y_i = -1) = 1 - p\).
- (a)
- Is \(S_n = Y_1 + \cdots + Y_n\) (a random walk) a Markov chain?
- (b)
- Is \(M_n = \max (S_0, S_1, \dots , S_n)\), with \(S_0 = 0\), a Markov chain?
Solution.
- (a)
- Yes. \(S_{n + 1} = S_n + Y_{n + 1}\) and \(Y_{n + 1}\) is independent of \(S_0, \dots , S_n\), so given \(S_n = i\) the next value is \(i + 1\) with probability \(p\) and \(i - 1\) with probability \(1 - p\), whatever the earlier values.
- (b)
- No. Whether the maximum can increase depends on whether \(S_n\) is currently at the maximum, which \(M_n\) alone does not reveal. Take \(p = \frac 12\). The histories with \(M_1 = M_2 = M_3 = 1\) are the paths \(S_1, S_2, S_3 = 1, 0, 1\) and \(1, 0, -1\), equally likely; \(M_4 = 2\) needs \(S_4 = 2\), possible only from the first, with probability \(\frac 12\). So \[P(M_4 = 2\mid M_3 = 1, M_2 = 1, M_1 = 1) = \tfrac 12\cdot \tfrac 12 = \tfrac 14 .\] The only path with \(M_1 = M_2 = 0\) and \(M_3 = 1\) is \(-1, 0, 1\), which ends at its maximum, so \[P(M_4 = 2\mid M_3 = 1, M_2 = 0, M_1 = 0) = \tfrac 12 .\] The two conditional probabilities differ although the present value \(M_3 = 1\) is the same, so \(\{M_n\}\) is not Markov. The pair \((S_n, M_n)\), however, is a Markov chain: enlarging the state space can restore the Markov property.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.