3.7 Expected Number of Visits

Let \(V^{(n)}_{ij}\) denote the total number of visits that the chain makes to State \(j\) in the first \(n\) steps from step 0 to step \(n- 1\) starting from State \(i\).
Given \(X_0 = i\), we define an indicator variable at step \(k\geq \) as \[I_{ij}^{(k)} = \begin {cases} 1 & \text {if}\hspace {0.3cm} X_k = j\hspace {0.5cm}\text {i.e}\hspace {0.5cm} P^{(k)}_{ij} > 0\\ 0 & \text {otherwise}\\ \end {cases}\] Hence \(E\left (I_{ij}^{(k)}\right ) = P_{ij}^{(k)}\).

Clearly \[V_{ij}^{(n)} = \sum ^{n - 1}_{k = 0} I_{ij}^{(k)}\]

\[E\left (V_{ij}^{(n)}\right ) = \sum ^{n-1}_{k = 0} E\left (I_{ij}^{(k)}\right ) = \sum ^{n -1}_{k = 0} P^{(k)}_{ij}\hspace {0.3cm} \cdots \cdots \hspace {0.5cm} (*)\] Let \[V^{(n)} = \begin {pmatrix} E\left (V_{ij}^{(n)}\right )\\ \end {pmatrix}_{(i,j)\in S}\]

Hence in matrix form equation \((*)\) can be written as \[V^n = I + P + P^2 + \, \cdots \, + P^{n -1}\]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.