CMSE11624 Statistics for Analytics
Session 3: Probability and Sampling
Experiment & Sample Space
A random process is a situation in which we know what outcomes could happen, but we don’t know which particular outcome will happen.
Examples: coin tosses, die rolls, iTunes shuffle, whether the stock market goes up or down tomorrow, etc.
It can be helpful to model a process as random even if it is not truly random.
An experiment is any process that generates well-defined outcomes.
The sample space for an experiment is the set of all experimental outcomes.
An experimental outcome is called a sample point.
Example: Boring Investments
Boring has invested in two stocks, Marrley Oil and Clean Energy. Boring has determined that the possible outcomes of these investments three months from now are as follows:
image
A Counting Rule for Multiple-Step Experiments
Boring Investments can be viewed as a two-step experiment. It involves two stocks, each with a set of experimental outcomes:
Marrley Oil: \(n_1 = 4\)
Clean Energy: \(n_2 = 2\)
Total Number of Experimental Outcomes: \(n_1 \times n_2 = 8\)
Tree Diagram
A tree diagram can represent the sample space for a multiple-step experiment:
image
Probability
Probability is a numerical measure of the likelihood that an event/outcome will occur.
There are several possible interpretations of probability but they (almost) completely agree on the mathematical rules probability must follow.
\(P(A)\) = Probability of event A
\(0 \le P(A) \le 1\)
A probability of near zero indicates an event is unlikely to occur;
A probability near 1 indicates an event is almost certain to occur.
Other probabilities between 0 and 1 represent degrees of likelihood that an event will occur.
![]()
image The sum of probabilities of all possible outcomes is 1
Probability
The following arguments are TRUE or FALSE
If an event with probability 1, it will definitely happen.
If an event with probability 0, it will definitely not happen.
If an event will definitely happen, its probability is 1
If an event will definitely not happen, its probability is 0
image
Law of large numbers (cont.)
When tossing a fair coin, if heads comes up on each of the first 10 tosses, what do you think the chance is that another head will come up on the next toss? 0.5, less than 0.5, or more than 0.5?
\[\underline{H} \hspace{1mm} \underline{H} \hspace{1mm} \underline{H} \hspace{1mm} \underline{H} \hspace{1mm} \underline{H} \hspace{1mm} \underline{H} \hspace{1mm} \underline{H} \hspace{1mm} \underline{H} \hspace{1mm} \underline{H} \hspace{1mm} \underline{H} \hspace{1mm} \underline{?}\]
The probability is still 0.5, or there is still a 50% chance that another head will come up on the next toss. \[P(H \text{ on 11}^{th} \text{ toss}) = P(T \text{ on 11}^{th} \text{ toss}) = 0.5\]
The coin is not “due" for a tail.
The common misunderstanding of the LLN is that random processes are supposed to compensate for whatever happened in the past; this is just not true and is also called gambler’s fallacy (or law of averages).
Subjective Method
Applying the subjective method, an analyst made the following probability assignments.
image
Events and Their Probabilities
An event is a collection of sample points.
The probability of any event is equal to the sum of the probabilities of the sample points in the event.
If we can identify all the sample points of an experiment and assign a probability to each, we can compute the probability of an event.
For example, let event M = Marrley Oil Profitable. Then: \[M = \{(10, 8), (10, -2), (5, 8), (5, -2)\}\]
\[\begin{aligned} \mathbb{P}(M) & = \mathbb{P}(10, 8) + \mathbb{P}(10, -2) + \mathbb{P}(5, 8) + \mathbb{P}(5, -2) \\ & = 0.20 + 0.08 + 0.16 + 0.26 \\ & = 0.70 \end{aligned}\]
Complement of an Event
The complement of event A is defined to be the event consisting of all sample points that are not in A.
The complement of \(A\) is denoted by \(A^{\complement}\)
Union of Two Events
The union of events A and B is the event containing all sample points that are in A or B or both.
The union of events A and B is denoted by \(A \cup B\).
Union of Two Events
Event M = Marrley Oil Profitable
Event C = Clean Energy Profitable
\(M \cup C\) = Marrley Oil Profitable or Clean Energy Profitable
In mathematics: \[M \cup C = \{(10, 8), (10, -2), (5, 8), (5, -2), (0, 8), (-20, 8) \}\]
\[\begin{aligned} \mathbb{P}(M \cup C) & = \mathbb{P}(10, 8) + \mathbb{P}(10, -2) + \mathbb{P}(5, 8) + \mathbb{P}(5, -2) + \mathbb{P}(0, 8) + \mathbb{P}(-20, 8) \\ & = 0.20 + 0.08 + 0.16 + 0.26 + 0.10 + 0.02 \\ & = 0.82 \end{aligned}\]
Intersection of Two Events
The intersection of events A and B is the set of all sample points that are in both A and B.
The intersection of events A and B is denoted by \(A \cap B\).
Intersection of Two Events
Event M = Marrley Oil Profitable
Event C = Clean Energy Profitable
\(M \cap C\) = Marrley Oil Profitable and Clean Energy Profitable
In mathematics: \[M \cap C = \{(10, 8), (5, 8) \}\]
\[\begin{aligned} \mathbb{P}(M \cap C) & = \mathbb{P}(10, 8) + \mathbb{P}(5, 8) \\ & = 0.20 + 0.16 \\ & = 0.36 \end{aligned}\]
Addition Law
Event M = Marrley Oil Profitable
Event C = Clean Energy Profitable
\(M \cup C\) = Marrley Oil Profitable or Clean Energy Profitable
As we know, \[\mathbb{P}(M) =0.70, \mathbb{P}(C) = 0.48, \mathbb{P}(M \cap C) = 0.36\]
Thus \[\begin{aligned} \mathbb{P}(M \cup C) & =\mathbb{P}(M) + \mathbb{P}(C) - \mathbb{P}(M \cap C) \\ & = 0.70 + 0.48 - 0.36 \\ & = 0.82 \end{aligned}\]
This result is the same as that obtained earlier using the definition of the probability of an event.
Mutually Exclusive Events
Two events are said to be mutually exclusive if the events have no sample points in common.
Two events are mutually exclusive if, when one event occurs, the other cannot occur.
![]()
image If events A and B are mutually exclusive, \[P(A \cap B) = 0\]
The addition law for mutually exclusive events is \[P(A \cup B) = P(A) + P(B)\]
Conditional Probability
Event M = Marrley Oil Profitable
Event C = Clean Energy Profitable
\(\mathbb{P}(C|M)\) = Clean Energy Profitable given Marrley Oil Profitable
As we know, \[\mathbb{P}(M) =0.70, \mathbb{P}(M \cap C) = 0.36\]
Thus \[\mathbb{P}(C | M) = \frac{\mathbb{P}(M \cap C) }{ \mathbb{P}(M)} = \frac{0.36}{0.70} = 0.5143\]
Multiplication Law
Event M = Marrley Oil Profitable
Event C = Clean Energy Profitable
\(M \cap C\) = Clean Energy Profitable and Marrley Oil Profitable
As we know, \[\mathbb{P}(M) =0.70, \mathbb{P}(C \cap M) = 0.5143\]
Thus \[\mathbb{P}(M \cap C) = \mathbb{P}(M) \mathbb{P}(M|C) = 0.70 \times 0.5143 = 0.36\]
This result is the same as that obtained earlier using the definition of the probability of an event.
Multiplication Law for Independent Events
Event M = Marrley Oil Profitable
Event C = Clean Energy Profitable
Are events M and C independent?
Does \(\mathbb{P}(M \cap C) = \mathbb{P}(M) \mathbb{P}(C)\) ?
As we know, \[\mathbb{P}(M \cap C) = 0.36, \mathbb{P}(M) =0.70, \mathbb{P}(C) =0.48\]
But \[\mathbb{P}(M) \times \mathbb{P}(C) = 0.70 \times 0.48 = 0.34 \neq 0.36 = \mathbb{P}(M \cap C)\]
Hence, M and C are not independent.
Law of Total Probability: Example
Let’s look at a question:
The students who complete all assignments can probably pass the exam (probability is 0.95).
But if the students didn’t complete all assignments, they have less chance to pass the exam (probability is 0.3).
Now, in our class, 90% students have completed all assignments at the end of the term.
Question: what is the probability that a student passes the exam?
Diagram 
Solution: Let \(B\) denote student passes exam, let \(A\) denote student completed all assignments
\[\begin{aligned} \mathbb{P}(B) & = \mathbb{P}(BA \cup BA^{\complement}) =\mathbb{P}(BA) + \mathbb{P}(BA^\complement) \\ & = \mathbb{P}(B|A)\mathbb{P}(A) + \mathbb{P}(B|A^{\complement})\mathbb{P}(A^{\complement}) \\ & = 0.95 \times 0.9 + 0.3 \times 0.1 \\ & = 0. 885 \end{aligned}\]
Law of Total Probability: Further
Diagram 
Solution: Let \(B\) denote student passes exam, let \(A_k\) denote that the student completed \(k\) assignments, \[\begin{aligned} \mathbb{P}(B) & = \mathbb{P}(B|A_0)\mathbb{P}(A_0) + \cdots \\ & \quad + \mathbb{P}(B|A_k)\mathbb{P}(A_k) + \cdots \\ & \quad + \mathbb{P}(B|A_N)\mathbb{P}(A_N) \end{aligned}\]
Bayes’ Theorem: Example
Let’s look at a question:
The students who complete all assignments can probably pass the exam (probability is 0.95).
But if the students didn’t complete all assignments, they have less chance to pass the exam (probability is 0.3).
Now, in our class, 90% students have completed all assignments at the end of the term.
Question: suppose that Peter has passed the exam, what is the probability that Peter completed all assignments?
Diagram 
Solution: \[\begin{aligned} \mathbb{P}(A | B) & = \frac{\mathbb{P}(AB)}{\mathbb{P}(B)} \\ & = \frac{\mathbb{P}(B|A) P(A)}{\mathbb{P}(B)} \\ & = \frac{0.95 \times 0.9}{0.885} \\ & \approx 0.97 \end{aligned}\]
Bayes’ Theorem: Further
Diagram 
Solution: \[\begin{aligned} \mathbb{P}(A_k | B) & = \frac{\mathbb{P}(A_k B)}{\mathbb{P}(B)} \\ & = \frac{\mathbb{P}(B|A_k) P(A_k)}{\mathbb{P}(B)} \\ & = \frac{\mathbb{P}(B|A_k) P(A_k)}{\sum_{i=0}^N \mathbb{P}(B|A_i)\mathbb{P}(A_i)} \end{aligned}\]
Causality and Bayes’ Theorem
Total Probability Formula \[\mathbb{P}(\text{Effect}) = \sum_{i=0}^n \mathbb{P}(\text{Effect}|C_i)\mathbb{P}(C_i)\]
Bayes’ Formula \[\mathbb{P}(C_j |\text{Effect}) = \frac{\mathbb{P}(\text{Effect}|C_j) \mathbb{P}(C_j)}{ \sum_{i=0}^n \mathbb{P}(\text{Effect}|C_i)\mathbb{P}(C_i)}\]
image
Expected value of a discrete random variable (cont.)
Below is a visual representation of the probability distribution of winnings from this game:
Continuous Distributions
Below is a histogram of the distribution of heights of US adults.
The proportion of data that falls in the shaded bins gives the probability that a randomly sampled US adult is between 180 cm and 185 cm.
From histograms to continuous distributions
Since height is a continuous numerical variable, its probability density function is a smooth curve.
Probabilities from continuous distributions
Therefore, the probability that a randomly sampled US adult is between 180 cm and 185 cm can also be estimated as the shaded area under the curve.
By definition...
Since continuous probabilities are estimated as “the area under the curve", the probability of a person being exactly 180 cm (or any exact value) is defined as 0.
