02 - Exploratory Data Analysis - DATS

Statistical analysis of experiments starts with graphical and non-graphical Explanatory Data Analysis (EDA)

EDA always precedes formal (confirmatory) data analysis.

Characteristic:

Numeric
Factor (qualitative) - ex. Gender {1,2,3} -->
- Categories (Levels in R) - Absolute or relative

Univariant Explanatory Data Analysis

We take each column one by one and we summarize it.

How can we summarize different variables types

Factor variable
- Numbers - counting the occurrences of each category
- Charts - Bar plot, Pie chart - Always use a new structure that contains, at leas, the counting (ex barplot(table(var)) )
Numeric variable
- Numbers - We distinguish the central and the dispersion (spread)
  - Summarize(var) --> into 5 numbers:
    - min
    - Q1 (first quartile)
    - Mean
    - Q2 (percentile of 50%) = Median
    - Q3 (third quartile)
    - max
  - Charts: Histogram, boxplot (helps identify outliers)

Robustness of an indicator

Classic Statistic

classic statistic

A classic statistic or indicator is one that is higly influenced by the presence of one or more outliers

Robust Statistic

robust statistic

A Robust statistic or indicator is one that is NOT higly influenced by the presence of one or more outliers

Numeric indicators

#Central Tendency
- #Mean
- #Median
#Dispersion or Spread
- #Variance
- #Standard Deviation
#Skewness
#Kurtosis
#Quartiles
- #Median

Central Tendency

Mean

#Classic Statistic

mean ($\mu$)

Simple mean (Also known as average although it is not a mathematical concept)
$\overset{―}{x} = \frac{x_{1} + \dots + x_{n}}{n}$
It indicates some sort of central trend. It is also known as the expected value: It's the outcome of a variable when the observations get close to infinity.

Quartiles

Median

Median is a #Robust Statistic

It's the second quartile
median = Q2

median or 2° quartile

It's the value that splits the sample in 2 parts:

50% of the observation are less than the median
50% of the observation are greater than the median

Q1 - Q3

#Robust Statistic

Q1: 25% of observation are less than Q1 & 75% are greater than Q1
Q3: 75% of observation are less than Q3 & 25% are greater than Q3

Dispersion or spread

The spread of a distribution mostly refers to the #Variance or the #Standard Deviation.

Variance

It's a #Classic Statistic (it's very affected by outliers)

variance ($s_{x}^{2}$ or $\sigma^{2}$)

$σ^{2} = S_{x}^{2} = \frac{1}{n - 1} \sum (x_{i} - \overset{―}{x})^{2}$

Variance has squared units so it's difficult to interprete. So we use standard deviation

Standard Deviation

It's a #Classic Statistic (it's very affected by outliers)

standard deviation ($s_{x}$ or $\sigma$)

$σ = S_{x} = \sqrt{S_{x}^{2}}$

IQR - Inter Quartile Range

It's a #Robust Statistic

inter quartile range ($iqr$)

It indicates how spread apart the observation are.
$I Q R = Q_{3} - Q_{1}$

Skewness and Kurtosis

Skewness

skewness ($\gamma_{2}$)

The skewness is a measure of asymmetry of a distribution.

$γ_{2} = 0 ⟹$ Perfect symmetry.

Kurtosis

kurtosis ($\gamma_{1}$)

The Kurtosis of a distribution measures how far away a distribution is from a Gaussian distribution in terms of peakedness vs flatness.

$γ_{1} < 0 :$ Rounder shoulders and thin tails compared to a Gaussian distribution
$γ_{1} > 0 :$ More sharply shaped picks and fat tails compared to a Gaussian distribution

Moments

Numeric charts

Boxplot

The boxplot is a particular plot useful to understand the central tendency of the data.

BOX
- Bottom is the Q1 (first quartile)
- Top is the Q3 (third quartile)
- Line in the middle is the #Median
Whiskers - the lines that exit the box
- They indicate the minimum and maximum value of the data, without considering the #Outliers
- Every data point that is over 1.5 the distance between the box edge and the whisker end, is considered an outlier and is plotted on its own

Assuming there are no outliers, the whiskers indicate the min and the max.

Quantile-Quantile plot - QQ plot

The QQ plot is used to see how well a particular data sample follows a particular theoretical distribution.

quantile-quantile plot (qq plot)

A Quantile-Quantile Plot (QQ Plot) is a graph of the standard model quantiles $\hat{F^{- 1}} (q_{i})$ where
$q_{i} = \frac{i - 0.5}{n} i = 1, 2, . . ., n$
versus $x_{(i)}, i = 1, 2, . . ., n$ , so that $x_{(1)} < x_{(2)} < \dots < x_{(n)}$ is the ordered sample data.

If ${\hat{F}}^{- 1} (q_{i})$ and $x_{(n)}$ are close together than the QQ plot will also be approximately linear with an intercept meaning location and a slope meaning location. Also, it is a measurement of the goodness of fit of the proposed distribution.

Bivariate Explanatory Data Analysis

2 variables: $X, Y$ (numbers)

we are interested in:

Coefficient of correlation
- Pearson: $ρ_{x y} = \frac{c o v (Y, X)}{S_{X} S_{Y}} = r_{X Y}$
- Spearman (suitable with integral data, order values not normally distributed: non-parametric statistics)

Correlation

It's adimensional.
the coefficient of correlation ranges between:

- 1 \leq ρ \leq 1

If $| ρ | \to 1$ , we can assume a perfectly linear relation between the variables: $Y = a X + b$

The sign indicates how the relation is working.

The pearson coefficient is given by:

r_{X Y} = ρ = \frac{\sum_{i = 1}^{n} \sum_{j = 1}^{n} (X_{i} - \overset{―}{X}) (Y_{i} - \overset{―}{Y})}{S_{X} S_{Y}}

Charts

Scatter plot
Scatter graph

❗❗❗❗❗❗❗❗❗❗❗❗
❗❗❗ COMPLETARE ❗❗❗
❗❗❗❗❗❗❗❗❗❗❗❗ tutta la parte da pag 16 delle slide in poi.

mosaic plot