Solution.

(a)

With states \(R\) (rain) and \(N\) (no rain), \[P = \bordermatrix {~ & R & N\cr R & \alpha & 1 - \alpha \cr N & \beta & 1 - \beta \cr } .\]

(b)

Tomorrow’s weather depends on two days, so the weather alone is not Markov. Take the state to be the pair (yesterday, today); then the next state is (today, tomorrow), which depends only on the current pair. Writing \(\bar R\) for no rain, \[P = \bordermatrix {~ & RR & R\bar R & \bar RR & \bar R\bar R\cr RR & 0.7 & 0.3 & 0 & 0\cr R\bar R & 0 & 0 & 0.4 & 0.6\cr \bar RR & 0.5 & 0.5 & 0 & 0\cr \bar R\bar R & 0 & 0 & 0.2 & 0.8\cr } .\] For instance from \(R\bar R\) (rain yesterday, none today) the next state is \(\bar RR\) if it rains tomorrow, with probability \(0.4\), and \(\bar R\bar R\) otherwise. Enlarging the state space to recover the Markov property is a standard device.

(c)

With \(i\) cells occupied, the next ball lands in an occupied cell with probability \(\frac ik\) and in an empty one with probability \(\frac {k - i}{k}\). So \(P_{i,i} = \frac ik\) and \(P_{i,i + 1} = \frac {k - i}{k}\) for \(i = 0, 1, \dots , k\), all other entries being zero; state \(k\) is absorbing.

(d)

With \(i\) black balls the number falls to \(i - 1\) only if a black ball is drawn and replaced by a white one, with probability \(\frac i4p\). So for \(i = 0, \dots , 4\), \[P_{i,i - 1} = \frac {ip}{4},\qquad P_{i,i} = 1 - \frac {ip}{4} .\]

(e)

If \(i\) machines work at the start of a day, the number \(B\) that break down is binomial with parameters \(i\) and \(p\). At night one broken machine is mended if any machine is broken, so the next state is \(\min (3, i - B + 1)\). The states are \(1, 2, 3\) (if the chain starts at \(3\), at least one machine is always working the next morning), and \[P = \bordermatrix {~ & 1 & 2 & 3\cr 1 & p & 1 - p & 0\cr 2 & p^2 & 2p(1 - p) & (1 - p)^2\cr 3 & p^3 & 3p^2(1 - p) & (1 - p)^3 + 3p(1 - p)^2\cr } .\] Each row sums to \(1\); in the last row, for example, it is the binomial expansion of \(\left [p + (1 - p)\right ]^3\).