CMSE11624 Statistics for Analytics

Session 1: Data & Data Collection

Dr Zexun Chen
Zexun.Chen@ed.ac.uk

Table of Contents

    About Data

    What is Data?

    image

    What is Data?

    An answer from ChatGPT:

    Data refers to raw, unprocessed facts and figures that are collected for a specific purpose, such as analysis or decision-making. It can exist in various forms, including numerical, textual, visual, etc.

    Data serves as the foundation for generating information and knowledge, particularly in fields like predictive analytics, management science, and business economics.

    In a structured format, data can be organised into tables, graphs, or spreadsheets, making it easier to analyse and interpret.

    Unstructured data, on the other hand, may include text, images, or any other form of data that doesn’t fit into a traditional database.

    The collection, analysis, and interpretation of data are central to various disciplines and industries, including but not limited to, healthcare, finance, marketing, and scientific research.

    With the advent of technologies like big data and machine learning, the ability to process and analyse large volumes of data has become increasingly important.

    Statistics for Data

    Stents are devices put inside blood vessels that assist in patient recovery after cardiac events and reduce the risk of an additional heart attack or death. Many doctors have hoped that there would be similar benefits for patients at risk of stroke.

    Question: Does the use of stents reduce the risk of stroke?

    The researchers who asked this question conducted an experiment with 451 at-risk patients. Each volunteer patient was randomly assigned to one of two groups:

    • Treatment group. Patients in the treatment group received a stent and medical management. The medical management included medications, management of risk factors, and help in lifestyle modification.

    • Control group. Patients in the control group received the same medical management as the treatment group, but they did not receive stents.

    Researchers randomly assigned 224 patients to the treatment group and 227 to the control group.

    Results

    Researchers studied the effect of stents at two time points: 30 days after enrollment and 365 days after enrollment.

    0-30 days   0-365 days
    2-3 stroke no event stroke no event
    treatment 33 191 45 179
    control 13 214 28 199
    Total 46 405 73 378
    Descriptive statistics for the stent study.

    Let’s compare the condition after one year!

    • Proportion who had a stroke in the treatment (stent) group:
      \(45/224 = 0.20 = 20\%\).

    • Proportion who had a stroke in the control group:
      \(28/227 = 0.12 = 12\%\).

    Understanding the results

    Do the data show a “real" difference between these two groups?

    • Suppose you flip a coin 100 times. While the chance a coin lands heads in any given coin flip is 50%, we probably won’t observe exactly 50 heads. This type of fluctuation is part of almost any type of data generating process.

    • The observed difference between the two groups (20 - 12 = 8%) may be real, or may be due to natural variation.

    • It’s hard to say, 8% is a big number or small number.

    • We need statistical tools to determine if the difference is large or not.

    About Data Collection

    Data Collection

    In general, there are three types of consumer data:

    • First-party data, which is collected from first-hand experience and is not used in the past.

      • Raw

      • Unstructured

      • Careful reprocessing is required

    • Second-party data, which is data shared by another organisation about its customers (or its first-party data)

      • Usually it is basically processed

      • Further processed is required according to your own analysis aim

      • Follow the data sharing policy

    • Third-party data, which is data that’s been aggregated and rented or sold by organisations that don’t have a connection to your company or users

      • Usually it is not cheap

      • Follow data protection scheme

      • Privacy concerns

    Data Collection Methods

    • Surveys

    • Polls

    • Interviews

    • Delphi Technique

    • Focus Groups

    • Questionnaire

    • Experiments/ Observation

    • Transactional Tracking

    • Online Tracking

    • Forms

    • Social Media Monitoring

    • etc

    Where we can collect financial data

    • Traditional financial data providers

      • Bloomberg

      • Capital IQ

      • Moody

      • ...

    • Government websites

      • Office for Statistics

      • ...

    • Financial reports in exchange website

    • Social media

    • ...

    About Data Basics

    What we usually do with Data

    image

    Classroom survey

    A survey was conducted on students in an introductory statistics course. Below are a few of the questions on the survey, and the corresponding variables the data from the responses were stored in:

    • gender: What is your gender?

    • intro_extra: Do you consider yourself introverted or extraverted?

    • sleep: How many hours do you sleep at night, on average?

    • bedtime: What time do you usually go to bed?

    • countries: How many countries have you visited?

    • dread: On a scale of 1-5, how much do you dread being here?

    Data matrix

    Data collected on students in a statistics class on a variety of variables:

    variable
    \(\downarrow\)
    1-5 Stu. gender intro_extra \(\cdots\) dread
    1-5 1 male extravert \(\cdots\) 3
    2 female extravert \(\cdots\) 2
    3 female introvert \(\cdots\) 4 \(\leftarrow\)
    4 female extravert \(\cdots\) 2 observation
    \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\)
    86 male extravert \(\cdots\) 3
    1-5

    • The elements are the entities on which data are collected.

    • A variable is a characteristic of interest for the elements.

    • The set of measurements collected for a particular element is called an observation.

    • The total number of data values in a data set is the number of elements multiplied by the number of variables.

    Types of variables

    image

    Types of variables (cont.)

    image

    Types of variables

    Ultimately, the data can be of FOUR forms

    • Nominal: A type of data in which objects fall into unordered categories, number (category) used as a label.

    • Ordinal: A type of data in which categories can be ordered, number (category) used to rank.

    • Interval: Numerical (arithmetic) value, meaningful interval, but no absolute zero.

    • Ratio: Numerical (arithmetic) value with absolute zero, meaningful ratio.

    Types of variables (cont.)

    gender sleep bedtime countries dread
    1 male 5 12-2 13 3
    2 female 7 10-12 7 2
    3 female 5.5 12-2 1 4
    4 female 7 12-2 2
    5 female 3 12-2 1 3
    6 female 3 12-2 9 4

    • gender: qualitative, categorical

    • sleep: quantitative, numerical, continuous

    • bedtime: qualitative, categorical, ordinal

    • countries: quantitative, numerical, discrete

    • dread: qualitative, categorical, ordinal - could also be used as numerical

    Likert Scale

    Respondent specifies a level of agreement or disagreement with statements that express a favourable or unfavourable attitude towards the concept under study

    For example: Interviewing vampires Blue (Aristocratic) Blood as compared to Common Blood.

    Strongly Disagree No Agree Strongly
    disagree feeling agree
    ...has stronger hit 1 2 3 4 5
    ...has stronger flavour 1 2 3 4 5
    of tradition
    ...has more attractive 1 2 3 4 5
    colour

    image

    Cross-sectional Data and Time Series Data

    Cross-sectional data are collected at the same or approximately the same point in time.

    • Example: data detailing the number of building permits issued in January 2020 in each of the UK counties

    Time series data are collected over several time periods.

    • Example: data detailing the number of building permits issued in London, UK in each of the last 36 months

    Panel Data

    image

    About Relationships in Data

    Relationships among variables

    Does there appear to be a relationship between total income and loan amount?

    image

    Explanatory and response variables

    • To identify the explanatory variable in a pair of variables, identify which of the two is suspected of affecting the other:

      explanatory variable \(\xrightarrow{might~affect}\)response variable

    • Labeling variables as explanatory and response does not guarantee the relationship between the two is actually causal, even if there is an association identified between the two variables. We use these labels only to keep track of which variable we suspect affects the other.

    Re-look at the types of data collection

    • Observational studies: Collect data in a way that does not directly interfere with how the data arise (e.g. surveys).

      • Can provide evidence of a naturally occurring association between variables, but they cannot by themselves show a causal connection.

    • Experiment: Researchers randomly assign subjects to various treatments in order to establish causal connections between the explanatory and response variables.

    Association vs. causation

    • When two variables show some connection with one another, they are called associated variables.

      • Associated variables can also be called dependent variables and vice-versa.

    • If two variables are not associated, i.e. there is no evident connection between the two, then they are said to be independent.

    • In general, association does not imply causation, and causation can only be inferred from a randomised experiment.

    About Data Ethics

    Data Bias

    Most organically produced datasets are biased (except for data generated by carefully designed randomised experiments).

    • Data does not include variables that properly capture the phenomenon we want to predict

    • Data includes content produced by humans which may contain bias against groups of people

    Data Bias (cont.)

    Data bias occurs due to structural characteristics of the systems that produce or collect the data,

    • Response or Activity Bias: the content generated by humans: reviews on Amazon, Twitter tweets, Facebook posts, Wikipedia entries,etc.

    • Selection bias due to feedback loops: Systems for online advertising, content personalization, recommendations, all have built-in feedback loops.

    • Bias due to system drift: drift refers to changes over time to the system generating the data.

    • Omitted variable bias: critical attributes that influence the outcome are missing (especially in human input cases)

    • Societal bias: human generated content on the web and in social media abound in biases.

    GDPR

    General Data Protection Regulation The General Data Protection Regulation is a Regulation in EU law on data protection and privacy in the EU and the European Economic Area. The GDPR is an important component of EU privacy law and of human rights law, in particular Article 8 of the Charter of Fundamental Rights of the European Union.

    Also, now we have UK GDPR!

    There are three aspects we need to focus

    • Privacy

    • Fairness

    • Transparency/accountability/explainability

    Data Sharing

    Think how likely would you be willing to share these types of data.

    image