3.1 Introduction
Consider a stochastic process \(\{X_n , \, n = 0, \, 1, \, \cdots \}\). Let the statespace be \(\, S = \{0, \, 1,\, \cdots \}\).
Assume \(\, \{X_n\, , \, n = 0\, , \, 1\, , \, 2\, , \, \cdots \cdots \}\) is homogeneous Markov process. Then
\[P\left (X_{n + 1} = j\, /\, X_n = i\right ) \, = \, P\left (X_1 = j\, /\, X_0 = i\right )\]
Let \(\, P\left (X_1 = j\, /\, X_0 = i\right )\, = \, P_{ij}\). We call \(P_{ij}\) as one step transition probability from state \(i\) to state \(j\).
The probability satisfies the following:
- i).
- \(\, P_{ij}\geq 0\hspace {0.2cm} \forall \, i \, \& \, j \, = 0\, , \, 1\, , \, 2\, , \, \cdots \cdots \)
- ii).
- \(\displaystyle {\sum ^{\infty }_{j = 0} P_{ij} = 1}\)
We arrange these one step transition probabilities in a matrix \(P\) as follows:
\[ P = \begin {pmatrix} P_{00} & P_{01} & P_{02} & \cdots \cdots \\ P_{10} & P_{11} & P_{12} & \cdots \cdots \\ P_{20} & P_{21} & P_{22} & \cdots \cdots \\ \vdots & \vdots & \vdots & \\ \end {pmatrix}\]
We note that every element in \(P\) must be \(\geq 0\) and also sum of elements of each row of \(P\) is
1.
- a).
- Weather conditions, rain or no rain previous day and not part weather conditions
determines if it will rain next day or not.
Let \(\alpha \) be the probability that if it rains today, then it will rain tomorrow. If it does not rain today, it will rain tomorrow with probability \(\beta \). Describe this experiment as a \(\,\{X_n\, :\, n = 0\, , \, 1\, , \, 2\, , \, \cdots \cdots \}\,\) Markov chain.
Let random variable \(X_n\) denote the weather of the \(n^{\text {th}}\) day \(\, \forall \, n = 0\, , \, 1\, , \cdots \)
\(X_n\) can take only tow values Rain or not Rain
\[ P = \bordermatrix {~ & \text {Rain} & \text {NotRain} \cr \text {Rain} & \alpha & 1 - \alpha \cr \text {NotRain} & \beta & 1 - \beta \cr }\]
- b).
- Suppose that whether or not it rains today depends on previous weather conditions through the last two days. Specifically assume that if it has rained for the past two days, then it will rain tomorrow with probability 0.7; if it rained today but not yesterday then it will rain tomorrow with probability 0.5; if it rained yesterday but not today then it will rain tomorrow with probability 0.4; if it has not rained in the past two days then it will rain tomorrow with probability 0.2. Describe this experiment as a Markov chain. \[ P = \bordermatrix {~ & RR & R\overline {R} & \overline {R}R & \overline {R}\, \overline {R}\cr RR & .7 & .3 & 0 & 0\cr R\overline {R} & 0 & 0 & .4 & .6\crcr \overline {R}R & .5 & .5 & 0 & 0\cr \overline {R}\, \overline {R} & 0 & 0 & .2 & .8\cr }\]
- c).
- Balls are thrown one at a time and must land in one of the \(k\) cells with equal probability. Let \(X_n\) be the number of occupied cells after the \(n^{\text {th}}\) throw. Find the transition matrix for the chain \(\,\{X_n\, :\, n = 0\, , \, 1\, , \, 2\, , \, \cdots \cdots \}\,\).
- d).
- An Urn contains four balls, some are black, remaining are white. A ball is drawn at random. If it is black there is a probability \( 1- p\) that it will be returned to the urn, there is a probability \(p\) that a white ball be returned to the urn instead. If it is white, it will be returned to the urn. This procedure is repeated indefinitely. Let \(X_n\) be the number of black balls in the urn after the \(n^{\text {th}}\) draw. Find the transition matrix for the chain \(\,\{X_n\, :\, n = 0\, , \, 1\, , \, 2\, , \, \cdots \cdots \}\,\).
- e).
- A company has three machines. Each day, independent of each other, a machine breaks down with probability \(p\). Each night, there is one repair person who can repair only one machine. Let \(X_n\) be the number of machines available at the beginning of the \(n^{\text {th}}\) day. Find the transition matrix for the chain \(\,\{X_n\, :\, n = 0\, , \, 1\, , \, 2\, , \, \cdots \cdots \}\,\).
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.