1.3 Definitions and Axioms of Probability

Definition 1.3.1 (Classical probability). If the sample space \(S\) consists of \(N\) equally likely outcomes, the probability of an event \(E\) is \[P(E) = \frac {n(E)}{N},\] where \(n(E)\) is the number of outcomes in \(E\).

Example 1.3.2. Two coins are tossed. Find the probability of (i) at least one head and (ii) exactly one tail.

Solution. \(S = \{HH, HT, TH, TT\}\) with \(N = 4\) equally likely outcomes. At least one head is \(A = \{HH, HT, TH\}\), so \(P(A) = \frac 34\). Exactly one tail is \(B = \{HT, TH\}\), so \(P(B) = \frac 24 = \frac 12\).

The classical definition needs equally likely outcomes. Another approach repeats the experiment many times and counts.

Definition 1.3.3 (Relative frequency). Suppose the experiment is repeated \(M\) times and the event \(E\) occurs \(M(E)\) times. The ratio \(\frac {M(E)}{M}\) is the relative frequency of \(E\). Experience shows that it settles down as \(M\) grows, and the frequency interpretation takes its limiting value \[P(E) = \lim _{M\to \infty }\frac {M(E)}{M}\] as the probability of \(E\).

Neither definition is fully general, so modern probability starts instead from a few rules that any sensible assignment of probabilities must obey. A probability is a function \(P\) that assigns to each event \(E\subset S\) a real number \(P(E)\), satisfying the following axioms.

1.3.1 Axioms of probability

(A1)
\(P(E)\ge 0\) for every event \(E\).
(A2)
\(P(S) = 1\).
(A3)
If \(E_1, E_2, \dots \) are mutually exclusive events, then \[P\left (\bigcup _{j=1}^\infty E_j\right ) = \sum _{j=1}^\infty P(E_j).\]

Axiom (A3) includes the case of finitely many events, by taking all but finitely many of the \(E_j\) to be empty; this uses \(P(\emptyset ) = 0\), proved below. Both the classical and relative-frequency definitions satisfy the axioms.

Theorem 1.3.4. For any events \(A\) and \(B\):

(i)
\(P(\emptyset ) = 0\);
(ii)
\(P(A') = 1 - P(A)\), where \(A'\) is the complement of \(A\);
(iii)
\(P(A\cup B) = P(A) + P(B) - P(A\cap B)\);
(iv)
if \(A\subset B\) then \(P(A)\le P(B)\); in particular \(P(A)\le 1\).

Proof.

(i)
Take \(E_1 = S\) and \(E_j = \emptyset \) for \(j\ge 2\) in (A3). These are mutually exclusive and their union is \(S\), so \(P(S) = P(S) + \sum _{j\ge 2}P(\emptyset )\). Since \(P(\emptyset )\ge 0\), this forces \(P(\emptyset ) = 0\).
(ii)
\(A\) and \(A'\) are mutually exclusive with union \(S\), so by (A3) and (A2), \(P(A) + P(A') = 1\).
(iii)
Split each set into disjoint pieces: \[A\cup B = (A\cap B')\cup (A\cap B)\cup (A'\cap B),\qquad A = (A\cap B')\cup (A\cap B),\qquad B = (A'\cap B)\cup (A\cap B).\] By (A3), \begin {align*} P(A\cup B) &= P(A\cap B') + P(A\cap B) + P(A'\cap B)\\ &= \left [P(A) - P(A\cap B)\right ] + P(A\cap B) + \left [P(B) - P(A\cap B)\right ]\\ &= P(A) + P(B) - P(A\cap B). \end {align*}
(iv)
If \(A\subset B\), then \(B = A\cup (A'\cap B)\) with the two pieces disjoint, so \(P(B) = P(A) + P(A'\cap B)\ge P(A)\) by (A1). Taking \(B = S\) gives \(P(A)\le 1\).
□

Example 1.3.5. Events \(A\) and \(B\) have \(P(A) = 0.3\), \(P(B) = 0.6\) and \(P(A\cup B) = 0.8\). Find \(P(A')\), \(P(A\cap B)\), \(P(A'\cup B')\) and \(P(A'\cap B')\).

Solution. By theorem 1.3.4,

  • \(P(A') = 1 - 0.3 = 0.7\);
  • \(P(A\cap B) = P(A) + P(B) - P(A\cup B) = 0.3 + 0.6 - 0.8 = 0.1\);
  • by De Morgan’s law \(A'\cup B' = (A\cap B)'\), so \(P(A'\cup B') = 1 - 0.1 = 0.9\);
  • \(A'\cap B' = (A\cup B)'\), so \(P(A'\cap B') = 1 - 0.8 = 0.2\).

Example 1.3.6. Of a make of mobile phone, \(2\%\) have a defective screen only, \(4\%\) have a defective charging system only, and \(90\%\) have neither defect. What percentage have both defects?

Solution. Let \(A\) be a defective screen and \(B\) a defective charging system. Then \(P(A\cap B') = 0.02\), \(P(A'\cap B) = 0.04\) and \(P\left ((A\cup B)'\right ) = 0.90\). So \(P(A\cup B) = 0.10\), and since \(A\cup B\) is the disjoint union of \(A\cap B'\), \(A'\cap B\) and \(A\cap B\), \[P(A\cap B) = 0.10 - 0.02 - 0.04 = 0.04 .\] So \(4\%\) have both defects.

Example 1.3.7. In a class of \(40\) students, \(22\) study French, \(18\) study German and \(9\) study both. A student is chosen at random. Find the probability that the student studies (i) at least one of the two languages, (ii) exactly one of them, (iii) neither.

Solution. Let \(F\) and \(G\) be the events of studying French and German. Then \(P(F) = \frac {22}{40}\), \(P(G) = \frac {18}{40}\) and \(P(F\cap G) = \frac {9}{40}\).

(i)
\(P(F\cup G) = \frac {22 + 18 - 9}{40} = \frac {31}{40} = 0.775\).
(ii)
Exactly one is \(F\cup G\) without \(F\cap G\): \(\frac {31 - 9}{40} = \frac {22}{40} = 0.55\).
(iii)
\(P\left ((F\cup G)'\right ) = 1 - \frac {31}{40} = \frac {9}{40} = 0.225\).

1.3.2 Practice Problems

Problems on sample spaces and the axioms. Try each one before opening the solution.

Problem 1.3.1. Consider the experiment of tossing four coins simultaneously and observing the side facing up for each coin.

(a).
Write down the sample space for the experiment, the event \(A\) of observing at least one head, the event \(B\) of observing at most two heads, and the event \(C\) of observing at least two tails.
(b).
Calculate \(P(A\cup B)\), \(P(A\cap B)\) and \(P(A^c\cup C)\).

Show solution

Solution. (a). Each coin lands two ways, so \(|S| = 2^4 = 16\) equally likely outcomes: \begin {align*} S = \{&HHHH,\ HHHT,\ HHTH,\ HTHH,\ THHH,\ HHTT,\ HTHT,\ HTTH,\\ &THHT,\ THTH,\ TTHH,\ HTTT,\ THTT,\ TTHT,\ TTTH,\ TTTT\}. \end {align*}

\(A\) excludes only \(TTTT\), so \(|A| = 15\). \(B\) collects the outcomes with \(0\), \(1\) or \(2\) heads, so \(|B| = 1+4+6 = 11\). \(C\) collects those with at least two tails, so \(|C| = 6+4+1 = 11\).

It is worth noticing that \(B\) and \(C\) are the same event. With four coins, at most two heads means at least two tails – the two descriptions pick out exactly the same 11 outcomes.

(b). \[P(A) = \frac {15}{16},\qquad P(B) = P(C) = \frac {11}{16}.\] \(A\cup B = S\), because the only outcome outside \(A\) is \(TTTT\), and \(TTTT\) is in \(B\): \[P(A\cup B) = 1.\] \(A\cap B\) is the set of outcomes with one or two heads, \(4+6 = 10\) of them: \[P(A\cap B) = \frac {10}{16} = \frac {5}{8}.\] \(A^c = \{TTTT\}\), which already lies in \(C\), so \(A^c\cup C = C\): \[P(A^c\cup C) = \frac {11}{16}.\]

Problem 1.3.2. For any events \(A\), \(B\) and \(C\), prove that \[P(A\cup B\cup C) = P(A)+P(B)+P(C)-P(A\cap B)-P(A\cap C)-P(B\cap C)+P(A\cap B\cap C).\]

Show solution

Solution. Apply the two-event addition rule twice. Treat \(A\cup B\) as a single event and add \(C\) to it: \[P((A\cup B)\cup C) = P(A\cup B) + P(C) - P((A\cup B)\cap C).\] The first term expands by the same rule, \(P(A\cup B) = P(A)+P(B)-P(A\cap B)\). For the last term, distribute the intersection over the union: \[(A\cup B)\cap C = (A\cap C)\cup (B\cap C),\] and apply the addition rule once more to that: \[P((A\cap C)\cup (B\cap C)) = P(A\cap C)+P(B\cap C)-P((A\cap C)\cap (B\cap C)).\] Now \((A\cap C)\cap (B\cap C) = A\cap B\cap C\). Substituting the two expansions, \begin {align*} P(A\cup B\cup C) &= [P(A)+P(B)-P(A\cap B)] + P(C)\\ &\quad - [P(A\cap C)+P(B\cap C)-P(A\cap B\cap C)]\\ &= P(A)+P(B)+P(C)-P(A\cap B)-P(A\cap C)-P(B\cap C)\\ &\quad +P(A\cap B\cap C), \end {align*}

as required.

The pattern is worth remembering: adding the three single probabilities counts each pairwise overlap twice, so the pairwise terms are subtracted; that subtraction removes the triple overlap three times having added it three times, so it must be added back once.

Problem 1.3.3. A game is played as follows: a player selects a point at random from the square \(0\leq x\leq 1\), \(0\leq y\leq 1\). There is a number \(\beta \) such that the player wins if at least one of the coordinates of the chosen point is greater than \(\beta \). Find the value of \(\beta \) that makes the game fair.

Show solution

Solution. “At random” from the square means the probability of landing in a region equals that region’s area, since the whole square has area 1.

It is easier to work with the losing event. The player loses when both coordinates are at most \(\beta \), that is when the point falls in the smaller square \(0\leq x\leq \beta \), \(0\leq y\leq \beta \), which has area \(\beta ^{2}\). Hence \[P(\text {win}) = 1 - \beta ^{2}.\] The game is fair when the player wins half the time: \[1-\beta ^{2} = \frac {1}{2} \implies \beta ^{2} = \frac {1}{2} \implies \beta = \frac {1}{\sqrt {2}} = \frac {\sqrt {2}}{2} \approx 0.707.\] (The negative root is discarded because \(\beta \) must lie in \([0,1]\).)

Problem 1.3.4. You wish to choose two different numbers from \(1, 2, 3, 4, 5\), and do not care about the order of selection.

(a).
Find the sample space for this experiment.
(b).
What is the probability that the numbers chosen differ by 1?

Show solution

Solution. (a). Order does not matter, so an outcome is a pair \(\{a,b\}\) with \(a\neq b\). There are \(\binom {5}{2}=10\) of them: \[S = \{12,\,13,\,14,\,15,\,23,\,24,\,25,\,34,\,35,\,45\}.\]

(b). The pairs differing by 1 are \(12, 23, 34, 45\) – four of the ten. All outcomes are equally likely, so \[P(\text {differ by }1) = \frac {4}{10} = \frac {2}{5}.\]

Problem 1.3.5. From a set of cards numbered 1 to 20 a card is drawn at random. Find the probability that the number is

(a).
divisible by 4;
(b).
greater than 15;
(c).
divisible by 4 and greater than 15.

If the card is divisible by 4 and it is not replaced, find the probability that the second card drawn is

(d).
even;
(e).
odd.

Show solution

Solution. There are 20 equally likely cards.

(a). The multiples of 4 are \(4, 8, 12, 16, 20\), so \(P = \frac {5}{20} = \frac {1}{4}\).

(b). The numbers greater than 15 are \(16, 17, 18, 19, 20\), so \(P = \frac {5}{20} = \frac {1}{4}\).

(c). Both conditions hold for 16 and 20 only, so \(P = \frac {2}{20} = \frac {1}{10}\).

For parts (d) and (e), the card already drawn is divisible by 4 and so is even. It is not replaced, which leaves 19 cards: 9 even and 10 odd.

(d). \(P(\text {second is even}) = \frac {9}{19}\).

(e). \(P(\text {second is odd}) = \frac {10}{19}\).

Problem 1.3.6. An ordinary die and two coins are thrown together. Find the probability that

(a).
two heads and a number less than 3 are obtained;
(b).
the coins show different faces and an even number is shown on the die;
(c).
the die shows an odd number and the coins show the same face;
(d).
a 6 and at least one head are obtained.

Show solution

Solution. The die and the two coins are independent, and the sample space has \(6\times 2\times 2 = 24\) equally likely outcomes.

(a). Two heads has probability \(\tfrac 14\); a number less than 3 (that is, 1 or 2) has probability \(\tfrac 26=\tfrac 13\). Hence \[P = \frac {1}{4}\times \frac {1}{3} = \frac {1}{12}.\]

(b). The coins differ in 2 of the 4 ways, probability \(\tfrac 12\); the die is even with probability \(\tfrac 36=\tfrac 12\). Hence \(P = \tfrac 12\times \tfrac 12=\tfrac 14\).

(c). The die is odd with probability \(\tfrac 12\) and the coins match with probability \(\tfrac 12\), so \(P = \tfrac 14\).

(d). The die shows 6 with probability \(\tfrac 16\); at least one head has probability \(1-\tfrac 14 = \tfrac 34\). Hence \[P = \frac {1}{6}\times \frac {3}{4} = \frac {1}{8}.\]

Problem 1.3.7. A coin is biased so that the probability that it lands showing heads is \(\frac {2}{3}\). The coin is tossed three times. Find the probability that

(a).
no heads are obtained;
(b).
more heads than tails are obtained.

Show solution

Solution. The tosses are independent with \(P(H)=\tfrac 23\) and \(P(T)=\tfrac 13\).

(a). No heads means three tails: \[P = \left (\frac {1}{3}\right )^{3} = \frac {1}{27}.\]

(b). With three tosses, more heads than tails means two heads or three heads: \begin {align*} P &= \binom {3}{2}\left (\frac {2}{3}\right )^{2}\left (\frac {1}{3}\right ) + \left (\frac {2}{3}\right )^{3}\\ &= 3\cdot \frac {4}{9}\cdot \frac {1}{3} + \frac {8}{27} = \frac {12}{27}+\frac {8}{27} = \frac {20}{27}. \end {align*}

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.