Statistics for Analytics

Session 2: Exploring Data & Introducing Probability

Dr Zexun Chen

Today’s route

Use summaries and plots to describe data, then define the uncertain outcome.

The Banking Question

What does this bank’s customer data look like?

Question: Which balances are typical, and which records deserve investigation?

Inspect

Rows, types, missingness

Summarise

Centre and spread

Visualise

Shape and anomalies

Question

Explain what you see

Six real campaign records

Record Age Balance (€) Housing loan Personal loan
1 58 2,143 Yes No
2 44 29 Yes No
3 33 2 Yes Yes
4 47 1,506 Yes No
5 33 1 No No
6 35 231 Yes No

The table gives detail; summaries reveal structure.

Source: first six anonymised rows of Moro, Rita & Cortez, UCI Bank Marketing.

Sample vs Population

Illustrative bank: What is the mean balance of its current-account customers on 30 June?

A banking population of all 800 current-account customers, with highlighted members selected at random into a sample of 30 customers. The portraits are schematic and do not show every customer.

  • Population: the whole group of interest; all 800 customers (\(N=800\)).
  • Sample: the subset we observe; 30 randomly selected customers (\(n=30\)).

Statistic vs Parameter

A statistic summarises a sample; a parameter summarises a population.

For the illustrative bank, the 30 sampled customers have mean balance €1,450.

Quantity Statistic: 30 sampled customers Parameter: all 800 customers
Mean balance \(\bar{x}=€1{,}450\) \(\mu\), the population mean
Balance variance \(s^2\) \(\sigma^2\)
Balance standard deviation \(s\) \(\sigma\)

We use a sample statistic to estimate a population parameter.

Descriptive vs Inferential

Descriptive statistics

Summarise the data we observed using tables, plots and numerical summaries.

“The 30 sampled customers have a mean balance of €1,450.”

Inferential statistics

Use sample data to draw conclusions about a population and quantify uncertainty.

“Use the sample to estimate the mean balance of all 800 customers, with a confidence interval.”

Describing a sample is descriptive; generalising to the population is inferential.

Sample, statistic or parameter?

A bank randomly samples 30 of its 800 current-account customers. Their mean balance is €1,450. Which statement is correct?

  1. €1,450 is a population parameter because it is a mean.
  2. €1,450 is a sample statistic; estimating the mean for all 800 is inference.
  3. The 30 selected customers form the population of interest.
  4. The mean balance of all 800 customers must be €1,450.

Centre & Spread

Numerical measures: three questions

Question Type of measure Examples
Where do values lie? Location Mean, median, mode, percentiles, quartiles
How far apart are values? Variability / dispersion Variance, standard deviation, range, IQR
What about asymmetry and extremes? Shape Skewness, kurtosis

Describe a typical balance, its spread, and the shape of the distribution.

Location: Mean

The arithmetic mean is the sum of the values divided by their number.

Sample: \(n\) observations Population: all \(N\) values
\(\displaystyle\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i\) \(\displaystyle\mu=\frac{1}{N}\sum_{i=1}^{N}x_i\)

For the six campaign balances: \(\bar{x}=€3{,}912/6=€652\).

  • Uses every value, so large balances pull the mean upwards.
  • Balances deviations: \(\sum_{i=1}^{n}(x_i-\bar{x})=0\).
  • Connects to the total: total balance \(=n\bar{x}\).

Location: Mode

The mode is the value or category with the greatest frequency.

For the 45,211 campaign records:

Last contact method Frequency
Cellular 29,285
Unknown 13,020
Telephone 2,906

Mode = Cellular. A mode can summarise categorical as well as numerical data.

Source: UCI Bank Marketing, contact variable.

One mode or several?

For a numerical distribution, modes are local peaks.

Three illustrative review-time densities have one, two and three peaks, labelled unimodal, bimodal and multimodal.

Several peaks can suggest different customer groups or review processes.

Illustrative densities generated in Python; these are hypothetical review times.

Location: Median

Sort the observations first. The median divides the ordered data into two halves.

Number of observations Ordered review times (minutes) Median
Odd: \(n=5\) 2, 4, 7, 9, 30 \(7\)
Even: \(n=6\) 2, 4, 7, 9, 12, 30 \((7+9)/2=8\)
  • Odd \(n\): take the middle value.
  • Even \(n\): average the two middle values.

The median is resistant to extreme values because it depends on ordered position.

What changes when one balance changes?

Use the six recorded balances: €2,143, €29, €2, €1,506, €1, €231.

  1. Calculate the mean and median.
  2. In a hypothetical version, replace €2,143 with €21,430.
  3. Predict which summary changes most, then check your calculation.

Which result would you use for a typical customer, and which for the total balance?

Mean or median? Real balances

The recorded balance is strongly right-skewed.

Mean = €1,362; median = €448.

  • For a typical recorded customer, lead with the median and the distribution.
  • For the total recorded balance, use the mean multiplied by the record count.

Source: UCI Bank Marketing. Figure generated from all 45,211 records; histogram truncated at the 99th percentile and boxplot outlier marks hidden for legibility.

Location: Percentiles

The \(p\)th percentile is a cut-off with approximately \(p\%\) of the observations at or below it.

Illustrative banking example: the 90th percentile of application review time is 45 minutes.

About 90% of recorded reviews took 45 minutes or less; about 10% took longer.

  • The percentile level tells us where a value sits in the ordered data.
  • The percentile value has the variable’s units: here, minutes.

Location: Quartiles

Quartiles are the 25th, 50th and 75th percentiles; \(Q_2\) is the median.

Ordered observations divided into four bands of approximately 25 percent, with Q1 at 25 percent, Q2 or the median at 50 percent, and Q3 at 75 percent.

Recorded bank balance \(Q_1\) \(Q_2\) \(Q_3\)
Value (€) 72 448 1,428

The middle 50% lies between \(Q_1\) and \(Q_3\).

Source: UCI Bank Marketing. Band widths show shares of records, not euro distances.

Same mean, same spread?

These illustrative branches both have a mean review time of 10 minutes. Which has more consistent observed review times?

Branch Review times (minutes)
A 8, 9, 10, 11, 12
B 2, 6, 10, 14, 18
  1. Branch A
  2. Branch B
  3. Both are equally consistent because their means are equal.
  4. We cannot compare consistency from these observations.

Measures of variability

Variability, or dispersion, describes how far the observations spread out.

Illustrative branch A review times are 8, 9, 10, 11 and 12 minutes; branch B times are 2, 6, 10, 14 and 18. Both means are 10 minutes, but branch B has greater spread.

  • Variance and standard deviation: use deviations from the mean.
  • Range and IQR: measure the width of all or part of the data.

Variability: Variance and SD

Variance uses squared deviations from the mean. Standard deviation (SD) is its square root.

Measure Population: all \(N\) values Sample: \(n\) observations
Variance \(\displaystyle\sigma^2=\frac{\sum_{i=1}^{N}(x_i-\mu)^2}{N}\) \(\displaystyle s^2=\frac{\sum_{i=1}^{n}(x_i-\bar{x})^2}{n-1}\)
SD \(\sigma=\sqrt{\sigma^2}\) \(s=\sqrt{s^2}\)

For review times, variance is in minutes squared; SD is in minutes.

SD expresses spread around the mean in the original measurement units.

Calculating variance and SD

For Branch B’s five review times, \(\bar{x}=(2+6+10+14+18)/5=10\) minutes.

Review time \(x_i\) Deviation \(x_i-\bar{x}\) Squared deviation \((x_i-\bar{x})^2\)
2 −8 64
6 −4 16
10 0 0
14 4 16
18 8 64
Total 0 160

\[s^2=\frac{160}{5-1}=40\text{ minutes}^2, \qquad s=\sqrt{40}\approx6.32\text{ minutes}.\]

Why divide by \(n-1\)?

The expected variance estimate divided by the true variance equals one when dividing squared deviations by n minus one, but equals (n minus one) over n when dividing by n. The latter rises from 0.5 at n equals 2 towards one.

  • Deviations from \(\bar{x}\) sum to zero: only \(n-1\) vary freely.
  • Dividing by \(n\) underestimates population variance on average.
  • Dividing by \(n-1\) corrects this bias.

An unbiased estimate is correct on average over repeated samples: \(\mathbb{E}[s^2]=\sigma^2\).

Theoretical expectations for independent observations from a common distribution with positive finite variance.

Variability: Range and IQR

Schematic boxplot with quartiles Q1, Q2 and Q3. IQR spans Q1 to Q3. Whiskers end at observed values inside the 1.5 IQR fences; an outlier beyond the upper fence contributes to the full range.

Measure Calculation for recorded balances (€) What it captures
Range \(102{,}127-(-8{,}019)=110{,}146\) Full span; determined by two extremes
IQR \(Q_3-Q_1=1{,}428-72=1{,}356\) Width of the middle 50%

The IQR is resistant to extreme values; the range is highly sensitive to them.

Diagram: schematic boxplot. Numerical example: UCI Bank Marketing, recorded balances.

Outliers: a flag, not a verdict

An outlier is unusually small or large for the variable and setting.

Screening rule Flag values Context
IQR rule Below \(Q_1-1.5\,IQR\) or above \(Q_3+1.5\,IQR\) Based on the central half of the data
Three-SD rule Outside \(\bar{x}\pm3s\) A heuristic for roughly bell-shaped data

For recorded balances, the IQR fences are −€1,962 and €3,462.

Investigate recording errors, unusual genuine cases and different subgroups.

A five-number summary

Statistic Recorded balance (€)
Minimum −8,019
\(Q_1\) 72
Median 448
\(Q_3\) 1,428
Maximum 102,127

For the central balances, report median €448 and IQR €1,356; report the extremes alongside them.

Source: UCI Bank Marketing, balance variable in euros.

Shape matters for risk

Name the long tail: right = positive skew; left = negative skew.

Three illustrative distributions mark the mean, median and mode. In the right-skewed positive example, mode is left of median, which is left of mean. In the symmetric example all three coincide. In the left-skewed negative example, mean is left of median, which is left of mode.

For risk management, inspect the adverse tail: right for losses; left for returns.

Schematic reused from the original Week 2 slides; the marked order describes these examples.

Skewness: two useful formulas

Population moment skewness: the third standardised moment.

\[\gamma_1=\mathbb{E}\!\left[\left(\frac{X-\mu}{\sigma}\right)^3\right] =\frac{\mathbb{E}[(X-\mu)^3]}{\sigma^3}.\]

Pearson’s median coefficient: a summary calculated from the data.

\[S_{\mathrm{P}}=\frac{3(\bar{x}-\tilde{x})}{s}, \qquad \tilde{x}=\text{median},\quad s=\text{SD}.\]

Bank balances: \(S_{\mathrm{P}}\approx3(1362.27-448)/3044.77\approx0.90\).

Both have no units; they are different summaries and their values need not agree.

Definitions: NIST. Balance example: UCI Bank Marketing.

Kurtosis: a glance at tail risk

The fourth power gives large standardised deviations much greater weight.

\[\kappa=\mathbb{E}\!\left[\left(\frac{X-\mu}{\sigma}\right)^4\right], \qquad \text{excess kurtosis}=\kappa-3.\]

A normal model with kurtosis 3 and a heavier-tailed model with kurtosis 9 both have mean zero, standard deviation one and skewness zero. An enlarged right-tail panel shows greater density beyond three standard deviations for the heavier-tailed model.

Normal benchmark: \(\kappa=3\), so excess kurtosis is \(0\).

Same mean and SD can hide different extreme-loss risks.

NIST definitions; illustrative theoretical curves generated in Python.

Statistical Graphics

Bar charts: compare categories

Bar chart of campaign records: cellular 29,285, telephone 2,906, and unknown 13,020.

  • Begin the value axis at zero.
  • Sort intentionally when order is not inherent.
  • Prefer position and length over area or 3D effects.

Source: UCI Bank Marketing. Bars show counts; a chart of group means must label that different quantity.

Line charts: change over time

Question: How does a branch’s review workload change from month to month?

Illustrative completed loan reviews: January 120, February 135, March 128, April 150, May 160 and June 172. A line connects the six monthly counts in time order.

Put time on the horizontal axis; look for trends, spikes and gaps.

Illustrative monthly counts for one hypothetical branch, generated in Python.

Frequency distributions: group the values

A frequency distribution records the count in each group or interval.

Age band (years) Records Relative frequency
10 to under 30 5,273 11.66%
30 to under 50 29,744 65.79%
50 to under 70 9,640 21.32%
70 to under 90 545 1.21%
90 to under 110 9 0.02%

Relative frequency = count / 45,211.

Next: turn these same age bands and counts into a histogram.

Source: UCI Bank Marketing. Displayed percentages are rounded.

Histograms: from the table to the plot

The same age bands and counts, drawn as adjoining bars.

Histogram of the five age intervals from the preceding frequency table: 10 to under 30 has 5,273 records; 30 to under 50 has 29,744; 50 to under 70 has 9,640; 70 to under 90 has 545; and 90 to under 110 has 9. All intervals are 20 years wide.

Each bar spans a numerical interval; with equal widths, height shows count.

Source: UCI Bank Marketing, all 45,211 records; the same age bands as the preceding table.

Histograms: choose the bin width

The same bank balances shown using €1,000 and €250 histogram bins; narrower bins reveal more detail near zero.

Bins that are too wide hide structure; bins that are too narrow amplify noise.

UCI Bank Marketing, balances up to the 99th percentile. Densities are normalised over that displayed subset.

Density plots: a smooth view

Two kernel density estimates of the same bank balances, with different bandwidths, show how smoothing changes the apparent peak.

The total area is 1. Density is not a count; bandwidth controls smoothness.

UCI Bank Marketing, balances up to the 99th percentile; each fitted density integrates to one over its full support.

Boxplots and violins: compare groups

Boxplots and violin plots compare credit limits for clients with and without recorded next-month default.

Boxplots summarise quartiles and spread; violins reveal the smoothed distribution shape.

UCI Default of Credit Card Clients. Boxplot outlier markers are hidden; all values contribute to summaries and violins.

Mosaic plots: two categories

A mosaic plot uses group width for contact volume and orange height for subscription rate: 14.9% cellular, 13.4% telephone, and 4.1% unknown.

Width shows group size; orange height shows the subscription rate within each group.

Source: UCI Bank Marketing. This extends the comparison introduced in Week 1.

Scatterplots: two numerical variables

Credit limit and September bill amount show a weak positive relationship.

Source: Yeh, UCI Default of Credit Card Clients. Plot shows a reproducible 3,500-record sample; the fitted line uses all 30,000 records.

Chart guide: one variable

Variable Banking question Chart What to check
Categorical How many records use each contact method? Bar chart Counts or proportions; zero baseline
Numerical Where do balances cluster, and how long is the tail? Histogram Bin width; any omitted extremes
Numerical What is the smooth shape of the balance distribution? Density plot Bandwidth; area represents proportion
Numerical What are the typical balance and central spread? Boxplot Median, IQR, whiskers and outliers

Choose the question first, then the chart. Label the variable and its units.

Chart guide: comparisons and change

Variables Banking question Chart Reading cue
Numerical + categorical How do credit limits differ by default group? Side-by-side boxplots or violins Compare centre and spread; violins also show shape
Two categorical Do subscription rates differ by contact method? Mosaic plot Width = group size; height = within-group proportion
Two numerical Are credit limit and bill amount related? Scatterplot Direction, form, strength and unusual points
Numerical over time How does review workload change each month? Line chart Time order, trends, spikes and gaps

Use the same units and scales for comparisons; make group sizes clear.

Avoid chart decoration

Keep

Readable labels, units, source, honest scale

Remove

3D effects, heavy grids, redundant legends, clip art

The purpose of a chart is comparison, not decoration.

Choose a useful summary

A small number of very large balances pull the mean above the median. Which would you lead with to describe a typical customer?

  1. Mean alone
  2. Median, supported by a distribution plot
  3. Maximum balance alone
  4. Correlation

Build the risk dashboard

The recorded bank balances have mean €1,362, median €448, and maximum €102,127. Which summaries and plots would you report, and what would you check before treating negative balances as errors?

Explore before modelling

Validate

Types and ranges

Summarise

Centre and spread

Visualise

Shape and groups

Explain

Findings and limits

From Data to Probability

Random experiment and sample space

Question: Will a loan default during the next year?

  • Experiment: observe its status over that fixed period.
  • Outcome: repayment or default.
  • Sample space: \(S=\{R,D\}\).

Conceptual illustration of a bank risk analyst reviewing loan files with uncertain outcomes.

The possible outcomes need not have equal probabilities.

Events are sets

For two loans, record the first loan’s status and then the second:

\[S=\{RR,RD,DR,DD\}.\]

Event Outcomes
\(A\): first loan defaults \(\{DR,DD\}\)
\(B\): second loan defaults \(\{RD,DD\}\)
\(A\cap B\): both default \(\{DD\}\)
\(A\cup B\): at least one defaults \(\{RD,DR,DD\}\)

Exactly one default

For two loans over the next year, R means no default and D means default. Which set represents exactly one loan defaulting?

  1. \(\{RR,DD\}\)
  2. \(\{DD\}\)
  3. \(\{RD,DR,DD\}\)
  4. \(\{RD,DR\}\)

Probability rules

For any event \(A\):

\[0\le P(A)\le1, \qquad P(S)=1, \qquad P(A^c)=1-P(A)\]

Impossible events have probability 0; certain events have probability 1.

Where do probabilities come from?

Approach Basis Banking example
Classical Equally likely outcomes: \(m/N\) Choose one of 10 audit files uniformly; 3 are overdue, so \(P=.3\)
Empirical Observed frequency: count / total Estimate a default rate from historical records
Judgement Belief informed by available evidence Assign probabilities to stress scenarios

State the source and assumptions behind a probability.

From an observed rate to an uncertain outcome

The campaign records contain 5,289 subscriptions among 45,211 contacts.

  1. Calculate the observed subscription proportion.
  2. Define the event for a new customer’s contact.
  3. Name one assumption needed to use the historical rate for that contact.

The observed proportion describes these records; a probability model concerns uncertain outcomes.

Session 2 takeaway

  • Describe shape, centre, spread and unusual observations.
  • Choose charts and summaries for the question.
  • Read relationships visually from scatterplots.
  • Define outcomes, events and the basis of a probability.

Next: combine events and model counts of uncertain outcomes.