CMSE11624 Statistics for Analytics
Session 5: Foundations of Inference
Table of Contents
Statistical Inference
Statistical Inference
Inference: Two Formats
Statistical inference is the process by which we draw conclusions about an entire population based on a sample. It’s fundamentally what allows statisticians to make claims about broader trends based on smaller sets of observations.
Estimation
Calculating a Population Parameter from a Sample Statistic,
e.g., Average Income
Point Estimation vs Interval Estimation
Testing
Judgement on a Statement about a Population (Hypothesis) based on Sample.
Population has average salary = £15k
Estimation
Point Estimation
Point estimation focuses on providing the most ‘likely’ value of an unknown parameter.
Error in the estimate = difference between population parameter and sample statistic
Bias is systematic tendency to over- or under-estimate the true population parameter.
Sampling error describes how much an estimate will tend to vary from one sample to the next.
Much of statistics is focused on understanding and quantifying sampling error, and sample size is helpful for quantifying this error.
Good Estimators
Unbiased: On average (expectation), the estimator should equal the true parameter value across multiple samples. \[\mathbb{E}(\hat{\theta}) = \theta\]
Consistent: As the sample size grows, the estimator should converge to the true parameter value. \[\lim_{n \to \infty } \hat{\theta}_n = \theta\]
Efficient (Precise): Among all unbiased estimators, the efficient one has the smallest sampling variance or error. \[\min Var(\hat{\theta})\]
Bias and Precision trade-off
Commonly estimated parameters
Mean (\(\mu\)) : point estimator is \(\bar{X}\)
Variance (\(\sigma^2\)), point estimator is \(s^2\)
Proportion (\(p\)), point estimator is \(\hat{p}\)
Sampling Distribution
If repeated samples are taken from a population and the same statistic (e.g. mean) is calculated from each sample, the statistics will vary, that is, it is a random variable and it has a distribution.
The probability distribution of a statistic (“sampling distribution”) indicates the chance of getting certain values.
However, in real-world applications, we never actually observe the sampling distribution, yet it is useful to always think of a point estimate as coming from such a hypothetical distribution.
- Understanding the sampling distribution will help us characterise and make sense of the point estimates that we do observe.
Example: Spending in Online Retail Store
Let’s say you’re interested in the monthly spending of individual customers at an online retail store.
the distribution of customer spending at an online retail store is not normal
skewed to the right (most people spend a little, but a few spend a lot).
Now, instead of looking at individual customers,
look at groups of 50 customers at a time
calculate the average spending for each group
Similar to "bell curve," or a normal distribution
Spending vs Average Spending
Central Limit Theorem
Central Limit Theorem
Let \(X_1, X_2, \ldots, X_n\) be a sequence of i.i.d. random variables with \(E[X_i] = \mu\) and \(\text{Var}(X_i) = \sigma^2 < \infty\).
Then, as \(n \to \infty\), the random variable \[Z = \frac{(X_1 + X_2 + \ldots + X_n) - n\mu}{\sigma\sqrt{n}} = \frac{\sqrt{n}(\bar{X}_n - \mu)}{\sigma}\] converges in distribution to a standard normal distribution \(N(0, 1)\).
In other words, \[\lim_{{n \to \infty}} P\left( Z \leq z \right) = \Phi(z)\] where \(\Phi(z)\) is the cumulative distribution function of the standard normal distribution. Simple version: \[\lim_{n \to \infty} \bar{X}_n \sim \text{A Normal Distribution}\]
Sampling Distribution of \(\bar{X}\)
Sampling Distribution of \(\bar{X}\)
The sampling distribution of \(\bar{X}\) is the probability distribution of all possible values of the sample mean \(\bar{X}\).
Sampling Distribution: Expectation and Variance
Expected Value of \(\bar{X}\) \[\mathbb{E}(\bar{X}) = \mu\]
Standard Deviation
Finite Population: \[\sigma_X = \frac{\sigma}{\sqrt{n}} \sqrt{\frac{N-n}{N-1}}\]
Infinite Population: \[\sigma_X = \frac{\sigma}{\sqrt{n}}\]
A finite population is treated as being infinite if \(n/N \leq 0.05\)
\(\sqrt{(N-n)/(N-1)}\) is the finite population correlation factor.
\(\sigma_X\) is referred as the standard error(SE) of the mean
Sampling Distribution of \(\bar{X}\)
Distribution of Sample Mean
When the population has a normal distribution, the sampling distribution of \(\bar{X}\) is normal in shape for any sample size.
In most applications, the sampling distribution of can be approximated by a normal distribution whenever the sample is size 30 or more.
In cases where the population is highly skewed or outliers are present, samples of size 50 may be needed.
Sampling Distribution of \(\hat{p}\)
When sampling from a binary or "Bernoulli" population (where the outcome is either a success or failure), the sample proportions also tend to follow a normal distribution as the sample size increases, provided that certain conditions are met.
Conditions:
Sample size \(n\) should be large (CLT).
\(np\) and \(n(1-p)\) should be greater than 5 or 10 (Binomial to Normal).
Sampling Distribution of Proportion: $$
\[\begin{aligned} \text{Mean: } \mu_{\hat{p}} &= p \\ \text{Standard Deviation (infinite): } \sigma_{\hat{p}} &= \sqrt{\frac{p(1-p)}{n}} \end{aligned}\]$$
For large \(n\), \(\hat{p}\) is approximately \(N(\mu_{\hat{p}}, \sigma_{\hat{p}}^2)\).
\(\sigma_{\hat{p}}\) is referred as the standard error(SE) of the proportion.
Sampling Distribution of Proportion
Sampling Distribution of Proportion
Making inferences about a Population Proportion: we want to know the Population with proportion \(p\).
A simple random sample of \(n\) elements is selected from the population
The sample data provide a value for the sample proportion \(\hat{p}\)
The value of \(\hat{p}\) is used to make inferences about the value of \(p\)
Point Estimates vs Interval Estimates
Point Estimates – Single Number
Confidence Interval Estimates – provide much more information and are preferred when making inferences.
Point estimate: no information about how close the estimate is likely to be the parameter
Interval estimate: add a margin of error which helps us to understand the accuracy of the point estimate
Interval Estimation: How to do it?
The purpose of an interval estimate is to provide information about how close the point estimate is to the value of the parameter.
An interval estimate can be computed by adding and subtracting a margin of error to the point estimate. \[\text{Point Estimate} \pm \text{Margin of Error}\]
E.g, the general form of an interval estimate of a population mean is: \[\bar{X} \pm \text{Margin of Error}\]
Margin of Error:
Population Mean with \(\sigma\) Known
Population mean: \(\sigma\) Known
- The population standard deviation \(\sigma\) is rarely known exactly, but often a good estimate can be obtained based on historical data or other information
Population mean: \(\sigma\) Known
The interval estimate of \(\mu\): \[\bar{X} \pm z_{\alpha/2}\frac{\sigma}{\sqrt{n}},\]
\(\bar{X}\) is the sample mean
\(1- \alpha\) is the confidence coefficient, and \(\alpha\) is significant level.
\(z_{\alpha/2}\) is the \(z\) (critical) value providing an area of \(\alpha/2\) in the upper tail of the standard normal probability distribution
- If \(\alpha=5\%\), \(z_{\alpha/2} = 1.96\) (in some cases, just using 2)
\(\sigma\) is the population standard deviation
\(n\) is the sample size
Adequate Sample Size
In most applications, a sample size of n = 30 is adequate.
If the population distribution is highly skewed or contains outliers, a sample size of 50 or more is recommended.
If the population is not normally distributed but is roughly symmetrical, a sample size as small as 15 will suffice.
If the population is believed to be at least approximately normal, a sample size of less than 15 can be used.
According to CLT, normal distribution is assumed if the sample size is large
Example: Average Savings Account Balance
A bank wants to estimate the average balance of all its savings accounts. Knowing the average balance can help the bank in resource allocation, risk assessment, and other decision-making processes.
Scenario:
Sample of 100 accounts.
Sample mean \(\bar{X} = £1,500\).
Population standard deviation \(\sigma = £300\).
95% confidence interval.
Steps:
Z-score for 95% confidence: \(z_{\alpha/2} = 1.96\).
Margin of error \(E\): \[E = z_{\alpha/2} \times \frac{\sigma}{\sqrt{n}} = 1.96 \times \frac{300}{\sqrt{100}} = £58.8\]
Confidence Interval: \[\bar{X} \pm E = £1,500 \pm £58.8 = (£1,441.2, £1,558.8)\]
Interpretation:
- 95% confident that the true average balance is between £1,441.2 and £1,558.8.
Population Mean with \(\sigma\) unknown
Population mean: \(\sigma\) unknown
If an estimate of the population standard deviation \(\sigma\) cannot be obtained prior to sampling, we use the sample standard deviation \(s\) to estimate \(\sigma\).
This is the case of unknown \(\sigma\)
In this case, the interval estimate for \(\mu\) is based on the \(t\) distribution.
(We shall assume for now that the population is normally distributed.)
\(t\) Distribution
\(t\) Distribution also called Student’s \(t\)-distribution, is a probability distribution that is used for estimating population parameters when the sample size is small and/or the population variance is unknown.
The PDF of the \(t\)-distribution with \(\nu\) degrees of freedom is: \[f(t; \nu) = \frac{\Gamma\left(\frac{\nu + 1}{2}\right)}{\sqrt{\nu \pi} \, \Gamma\left(\frac{\nu}{2}\right)} \left(1 + \frac{t^2}{\nu}\right)^{-\frac{\nu + 1}{2}}\] where \(\Gamma(z)\) is the gamma function.
Symmetry around mean zero.
Heavier tails than normal distribution.
Defined by degrees of freedom \(\nu = n - 1\).
Converges to normal distribution as \(n\) increases.
Normal distribution vs t-distribution
t-Table
Population mean: \(\sigma\) unknown
The interval estimate of \(\mu\): \[\bar{X} \pm t_{\alpha/2}\frac{s}{\sqrt{n}},\]
\(\bar{X}\) is the sample mean
\(1- \alpha\) is the confidence coefficient, and \(\alpha\) is significant level.
\(t_{\alpha/2}\) is the \(t\) (critical) value providing an area of \(\alpha/2\) in the upper tail of a \(t\) distribution with \(n-1\) degree of freedom
\(s\) is the sample standard deviation
\(n\) is the sample size
Example in Finance
Suppose a financial analyst wants to estimate the average return on a particular stock. The analyst gathers data for the monthly returns over the past year (12 months).
Sample mean return \(\bar{X} = 8\%\)
Sample standard deviation \(s = 4\%\)
Sample size \(n = 12\), \(\nu = 11\)
t-score for 95% CI \(t = 2.201\)
Check t-table
Using Python
> from scipy.stats import t > t_lecture = t.ppf(1 - 0.05/2, 11) > print(t_lecture) 2.200985160082949Margin of Error: \[E = t_{5\%/2} \times \frac{s}{\sqrt{n}} = 2.201 \times \frac{4}{\sqrt{12}} \approx 2.52\%\]
Confidence Interval: \((5.48\%, 10.52\%)\)
Summary of Workflow for a Population Mean
Population Proportion
Interval Estimation: Population Proportion
The confidence interval for a population proportion \(p\) is given by: \[\hat{p} \pm z_{\frac{\alpha}{2}} \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}\] Where:
\(\hat{p}\) is the sample proportion.
\(z_{\alpha/2}\) is the critical z-value (normal distribution).
\(1- \alpha\) is the confidence coefficient, and \(\alpha\) is significant level.
\(n\) is the sample size.
Requirements:
The sampling distribution of P can be approximated by a normal distribution whenever \(np \geq 5\) and \(n(1-p) \geq 5\)
or sometimes the value is 10.
Example in Banking and Finance
Suppose a bank surveys 1,000 customers and finds that 800 are satisfied. \[\text{Sample proportion, } \hat{p} = \frac{800}{1000} = 0.8\] The 95% confidence interval is then: \[\text{CI} = 0.8 \pm 1.96 \sqrt{\frac{0.8 \times 0.2}{1000}}\]
Take Home Messages
Take Home Messages
Point Estimation
Concept of Central Limit Theorem
Distribution of sample mean (sample size or population distribution)
Distribution of sample proportion (sample size, \(np \& n(1-p) \geq 5\) or 10)
Interval Estimation
Focus on the population mean with known \(\sigma\) \[\bar{X} \pm z_{\alpha/2}\frac{\sigma}{\sqrt{n}}\]
Focus on the population mean with unknown \(\sigma\) (assuming population is normally distributed) \[\bar{X} \pm t_{\alpha/2}\frac{s}{\sqrt{n}}\]
Focus on population proportion (sample size, \(np \& n(1-p) \geq 5\) or 10) \[\hat{p} \pm z_{\alpha/2}\sqrt{\frac{\hat{p} (1- \hat{p})}{n}}\]
