3.8 Expected Number of Visists to a Transient State
Consider a finite States Markov chain and suppose that the States are numbered so that \(T = \{1, \, 2,\, \cdots \, , \, t\}\) denotes the set of transient States.
Let \[P_T = \begin {pmatrix} P_{11} & P_{12} & \cdots & P_{1t}\\ P_{21} & P_{22} & \cdots & P_{2t}\\ \vdots & \vdots & & \vdots \\ P_{t1} & P_{t2} & \cdots & P_{tt}\\ \end {pmatrix}\] denotes the set of transient States denote the matrix of transition probabilities associated with transient States only. Hence the sum of elements of ites rows may be less than 1.
For transient States \(i\) and \(j\) , let \(S_{ij}\) denote the expected number of visits to State \(j\) given that the chain started from State \(i\). We define a variable \(\delta _{ij}\) as follows: \[\delta _{ij} = \begin {cases} 1 & \text {if}\hspace {0.3cm} i = j\\ 0 & \text {if} \hspace {0.3cm} i\neq j\\ \end {cases}\] \[S_{ij} = \delta _{ij} + \sum ^{\infty }_{k = 1} S_{kj}\, P_{ik}.\]
Since a transient State is not accessible from a recurrent State \(P_{ik} = 0\) if \(k\) not transient. \[S_{ij} = \delta _{ij} + \sum ^t_{k = 1} S_{kj}\, P_{ik}\] \(S_{kj} = \) expected visits to State \(k\)(Recurrent) for State \(j\) \(\, \{ k\) recurrent, \(j\) transient\(\}\).
\(S_{kj} = \) for \(k\) is recurrent because a transient state is Not visited from a recurrent state.
Let \(\, S = (S_{ij})\hspace {0.3cm} i, j = 1, \, 2, \, \cdots \, , \, t.\)
Hence the above equation is written in the matrix form \[S = I + P_TS\] \[S = \left (I - P_T\right )^{-1}\] Let \[P_T = \bordermatrix {~ & E_0 & E_1 & E_2 & E_3 & E_4 & E_5\cr E_0 & 0 & 0.4 & 0 & 0 & 0 & 0\cr E_1 & 0.6 & 0 & 0.4 & 0 & 0 & 0\cr E_2 & 0 & 0.6 & 0 & 0.4 & 0 & 0\cr E_3 & 0 & 0 & 0.6 & 0 & 0.4 & 0\cr E_4 & 0 & 0 & 0 & 0.6 & 0 & 0.4\cr E_5 & 0 & 0 & 0 & 0 & 0.6 & 0\cr }\]
\(S = (I - P_T)^{-1}\)
\(S_{35} = 0.9228\,\, \hspace {0.3cm}\) expected number of visits to state 5.
\(S_{32} = 2.3677\)
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.