--- title: "Exploring Numerical Data" subtitle: | | IMS2 Ch. 5 | Math 115 author: "Yurk" format: revealjs: theme: beige embed-resources: true editor: source metadata: license: CC BY-SA --- ## Life Expectancies ```{r} #| include: false #| echo: false library(tidyverse) library(openintro) library(tidytuesdayR) library(knitr) library(kableExtra) ``` ```{r} #| include: false #| echo: false life <- read_csv("data/life_exp.csv") |> mutate( state = str_to_title(state), county = str_to_title(county) ) ``` - Every county in the US (3,142 counties) - Variables include county name, state, average life expectancy (expectancy), median income (income) --- - First 15 rows of `life_exp` data ::: {style="font-size: 85%;"} ```{r} #| include: true #| echo: false library(tidyverse) life %>% slice_head(n = 15) |> kbl(booktabs = TRUE, escape = FALSE) |> kable_styling(font_size = 18) ``` ::: ## Life Expectancies in MA Counties - For now, we will focus on life expectancy in Massachusetts (14 counties) - Since the `life_exp` data include all US counties, we need to filter the data to retain just the Massachusetts counties - You can filter a data set using Jamovi (or another spreadsheet program) ```{r} #| include: false #| echo: false life_ma <- life |> filter(state == "Massachusetts") ``` --- - First 15 rows of filtered `life_exp` data ::: {style="font-size: 85%;"} ```{r} #| include: true #| echo: false library(tidyverse) life_ma %>% slice_head(n = 15) |> kbl(booktabs = TRUE, escape = FALSE) |> kable_styling(font_size = 18) ``` ::: --- - Let's take a quick peak at a dotplot of life expectancies for the MA counties ```{r} #| include: true #| echo: false life_ma <- life |> filter(state == "Massachusetts") |> select(county, expectancy) |> mutate(expectancy = round(expectancy)) life_ma |> ggplot(aes(x = expectancy)) + geom_dotplot(stackratio = 1.2) + scale_y_continuous(NULL, breaks = NULL) + # hide meaningless y scale theme_minimal() + theme(text = element_text(size = 20)) ``` ## Summaries of numerical data - Measures of center - mean - median - Percentiles/quantiles - quartiles - other percentiles - Measures of spread - interquartile range - standard deviation ## Mean - If there are $n$ cases in a sample then the **sample mean** of the numeric variable $x$ is $$\bar{x}=\frac{x_1+x_2+\cdots+x_n}{n}$$ - The sample mean is a measure of the center of the distribution of the data - The sample mean $\bar{x}$ (a statistic) gives us a point estimate of the population mean $\mu$ (a parameter) --- The mean life expectancy in Massachusetts counties is 79.9 years ```{r} #| include: true #| echo: false life_ma |> summarize(mean = mean(expectancy)) |> kbl(booktabs = TRUE, escape = FALSE) |> kable_styling(font_size = 28) ``` ## Median - The **median** is the value that splits the data in half - 50% of the data fall below the median - We can also compute the median of the life expectancy data ```{r} #| include: true #| echo: false life_ma |> summarize(mean = mean(expectancy), median = median(expectancy)) |> kbl(booktabs = TRUE, escape = FALSE) |> kable_styling(font_size = 28) ``` --- - Let's see where the mean and median fall on the dotplot - The mean is red and dashed. Note that it is pulled toward the thicker left tail of the distribution. ```{r} #| include: true #| echo: false lma_mean <- mean(life_ma$expectancy) lma_med <- median(life_ma$expectancy) life_ma |> ggplot(aes(x = expectancy)) + geom_dotplot(stackratio = 1.2) + geom_vline(xintercept = lma_mean, color = "red", linetype = "dashed") + geom_vline(xintercept = lma_med, color = "blue") + scale_y_continuous(NULL, breaks = NULL) + # hide meaningless y scale theme_minimal() + theme(text = element_text(size = 20)) ``` ## Group means - We can also compare means between different groups in the data - Let's compare the mean of the life expectancy variable between counties in West Coast states (California, Oregon, Washington) and counties that are not in West Coast states - Is life expectancy higher in counties in west coast states? ```{r} #| include: true #| echo: false life <- life |> mutate(west_coast = if_else(state %in% c("California", "Oregon", "Washington"), "yes", "no")) life |> group_by(west_coast) |> summarize(mean = mean(expectancy), median = median(expectancy)) |> kbl(booktabs = TRUE, escape = FALSE) |> kable_styling(font_size = 28) ``` ## Percentiles - The *X*th **percentile** is the value below which *X*% of the data fall - The median is the 50th percentile - For example, the 90th percentile of the life expectancy variable in the Massachusetts data is 81 years, meaning that 90% of the counties have an average life expectancy that is less than 81 years ```{r} #| include: false #| echo: false quantile(life_ma$expectancy, 0.9) ``` ## Quartiles - The **first quartile** (Q1) is the 25th percentile, the value below which 25% of the data fall - The **third quartile** (Q3) is the 75th percentile, the value below which 75% of the data fall - The median is sometimes described as the second quartile (Q2) - Quartiles are often included in numerical summaries of a data set --- - Let's add quartiles to our summary of the Massachusetts county life expectancy data ```{r} #| include: true #| echo: false life_ma |> summarize(Q1 = quantile(expectancy, 0.25), median = median(expectancy), Q3 = quantile(expectancy, 0.75), mean = mean(expectancy)) |> kbl(booktabs = TRUE, escape = FALSE) |> kable_styling(font_size = 28) ``` ## Maximum and minimum values - Maximum and minimum values in a data set are often included numerical summaries as well - Let's add them to our summary of the Massachusetts county life expectancy data ```{r} #| include: true #| echo: false life_ma |> summarize(min = min(expectancy), Q1 = quantile(expectancy, 0.25), median = median(expectancy), Q3 = quantile(expectancy, 0.75), max = max(expectancy), mean = mean(expectancy)) |> kbl(booktabs = TRUE, escape = FALSE) |> kable_styling(font_size = 28) ``` ## Range - The simplest measure of spread/variability of a distribution of data is the **range** - It is simply the difference between the largest and smallest values - The range is $81-78=3$ years for the Massachusetts county life expectancy data ```{r} #| include: true #| echo: false life_ma |> summarize(range = max(expectancy) - min(expectancy)) |> kbl(booktabs = TRUE, escape = FALSE) |> kable_styling(font_size = 28) ``` ## Interquartile range - The **interquartile range** (IQR) is the difference Q3-Q1 - The IQR will never be larger than the range! - Th IQR is $80-79.25=0.75$ years for the Massachusetts county life expectancy data ```{r} #| include: true #| echo: false life_ma |> summarize(iqr = IQR(expectancy), range = max(expectancy) - min(expectancy)) |> kbl(booktabs = TRUE, escape = FALSE) |> kable_styling(font_size = 28) ``` ## Standard deviation - The most commonly used measure of variability is the **standard deviation** - The **deviation** of a single observation $i$ is the difference between the observed value and the mean, $x_i - \bar{x}$ - The standard deviation describes the typical deviation of the data from the mean --- - The **sample variance** is the average squared deviance $$s^2=\frac{(x_1-\bar{x})^2 + (x_2-\bar{x})^2 \cdots (x_n-\bar{x})^2}{n-1}$$ - We divide by $n-1$ rather than $n$ (the sample size) to obtained an **unbiased estimate** of the **population variance** $\sigma^2$. Otherwise $s^2$ tends to underestimate $\sigma^2$ - The **sample standard deviation** is $$s = \sqrt{s^2} = \sqrt{\frac{\sum_{i=1}^n(x_i-\bar{x})^2}{n-1}}$$ --- - For many numeric variables the following rules of thumb apply: - Roughly 68% of the data fall within 1 standard deviation of the mean - Roughly 95% of the data fall within 2 standard deviations of the mean --- - The standard deviation for the Massachusetts county life expectancy data is 0.864 years ```{r} #| include: true #| echo: false life_ma |> summarize(sd = sd(expectancy), iqr = IQR(expectancy), range = max(expectancy) - min(expectancy)) |> kbl(booktabs = TRUE, escape = FALSE) |> kable_styling(font_size = 28) ``` ## Identifying outliers using IQR - An **outlier** is an observation that is extreme relative to the rest of the data - There is no universally accepted method for identifying outliers - One common method that is also simple uses the IQR - With this method, a point is considered an outlier if it is larger than Q3 or smaller than Q1 by more than $1.5\times$IQR. --- - Let's use this method to identify outliers in the life expectancy data for the whole country - For the whole country, $Q1=75.70$ years and $Q3=78.77$ years - Thus, $IQR = 78.77-75.70=3.07$ - Any county with a life expectancy less than $75.70-1.5\times3.07=71.09$ years or greater than $78.77+1.5\times3.07=80.31$ years is considered an outlier ```{r} #| include: false #| echo: false iqr <- IQR(life$expectancy) q1 <- quantile(life$expectancy, 0.25) q3 <- quantile(life$expectancy, 0.75) q1 q3 iqr life <- life |> mutate(is_outlier = case_when(expectancy < q1 - 1.5*iqr ~ TRUE, expectancy <= q3 + 1.5*iqr ~ FALSE, TRUE ~ TRUE)) ``` --- - Using the $1.5\times$IQR rule, 12 counties identified as outliers - All have low life expectancies ```{r} #| include: true #| echo: false life |> filter(is_outlier) |> select(-income, -west_coast, -is_outlier) |> arrange(desc(expectancy)) |> kbl(booktabs = TRUE, escape = FALSE) |> kable_styling(font_size = 22) ``` ## Cars ```{r} #| include: true #| echo: false cars <- read_csv("data/cars04.csv", col_types = cols( msrp = col_integer(), dealer_cost = col_integer(), ncyl = col_integer(), horsepwr = col_integer(), city_mpg = col_integer(), hwy_mpg = col_integer(), weight = col_integer(), wheel_base = col_integer(), length = col_integer(), width = col_integer() )) ``` - data on all new car models (428) in a certain year - 19 variables - includes weight, highway mpg (hwy_mpg), msrp, whether a pickup or not (pickup) - we will explore a variety of visualizations involving numerical variables --- - Here are the first 8 rows ::: {style="font-size: 75%;"} ```{r} #| include: true #| echo: false cars |> slice_head(n = 8) |> kbl(booktabs = TRUE, escape = FALSE) |> kable_styling(font_size = 16) ``` ::: ## Dotplot - A **dotplot** represents each case with a dot - Dots are stacked on top of each other at the appropriate location on the x-axis --- - Dot plot of vehicle weights ```{r} #| include: true #| echo: false cars |> ggplot(aes(x = weight)) + geom_dotplot(dotsize = 0.2, stackratio = 1.2) + scale_y_continuous(NULL, breaks = NULL) + # hide meaningless y scale theme_minimal() + theme(text = element_text(size = 20)) ``` ## Skew and Symmetry - Note that the distribution of weights is **skewed right**, meaning that it has a thicker tail on the right side - A distribution with a thicker tail on the left is **skewed left** - A distribution that is roughly equal in both directions is called **symmetric** ## Histogram - In a **histogram** data are aggregated into bins on the x-axis - The height of each bar is proportional to the number of cases in the bin --- - Histogram of vehicle weights ```{r} #| include: true #| echo: false cars |> ggplot(aes(x = weight)) + geom_histogram() + theme_minimal() + theme(text = element_text(size = 20)) ``` ## Density plot - In a **density plot** the shape of the distribution is represented using a smooth line (think of this as a smoothed out histogram) --- - Density plot of vehicle weights ```{r} #| include: true #| echo: false cars |> ggplot(aes(x = weight)) + geom_density() + theme_minimal() + theme(text = element_text(size = 20)) ``` ## Box plot - A **box plot** takes a different approach to visualizing the distribution of a numerical variable - Boxplots are constructed using summary statistics: - The box extends from Q1 to Q3 with a vertical line at the median (Q2) - Whiskers extend from the box to the smallest and largest values that are not outliers - Outliers are plotted as individual points --- - Box plot of vehicle weights. Are there any outliers? ```{r} #| include: true #| echo: false cars |> ggplot(aes(x = weight)) + geom_boxplot() + theme_minimal() + theme(text = element_text(size = 20)) ``` --- - It is important to note that box plots can miss important characteristics of the distributions - For example, if the distribution is bimodal, then the box plot won't show it ## Scatter plot - visualizing 2 numerical variables - We can visualize two numeric variables using a scatter plot --- - A scatter plot of highway gas mileage vs. vehicle weight ```{r} #| include: true #| echo: false cars |> ggplot(aes(x = weight, y = hwy_mpg)) + geom_point() + theme_minimal() + theme(text = element_text(size = 20)) ``` ## Faceted histograms - We can visualize two variables, where one is numeric and the other is categorical using **faceted histograms** - We simply plot a separate histogram for each level of the categorical variable --- - Faceted histogram showing the distributions of highway mileage for vehicles that are pickups and vehicles that are not pickups ```{r} #| include: true #| echo: false cars |> ggplot(aes(x = hwy_mpg)) + geom_histogram() + facet_wrap(~pickup, ncol = 1, scales = "free_y") + theme(text = element_text(size = 20)) ``` ## Colored density plots - We can use colored density plots for a similar purpose- - By coloring the density plots according to the levels of the categorical variable, we can plot them on the same axes and still distinguish between the distributions --- - Colored density plots to visualize the distributions of highway mileage for vehicles that are pickups and vehicles that are not pickups ```{r} #| include: true #| echo: false cars |> ggplot(aes(x = hwy_mpg, fill = pickup)) + geom_density(alpha = 0.5) + theme_minimal() + theme(text = element_text(size = 20)) ``` ## Transforming data - Sometimes it is helpful to **transform** a variable - For example, a *log* transformation is commonly applied to distributions that are strongly skewed to the right - The transformed variable is often more appropriate for analyses that use a mathematical model to approximate the distribution of the data - The `msrp` data are skewed right --- - Histogram of MSRP ```{r} #| include: true #| echo: false cars |> ggplot(aes(x = msrp)) + geom_histogram() + theme_minimal() + theme(text = element_text(size = 20)) ``` --- - We can create a new variable by taking the (natural) *log* of the `msrp` variable - The transformed variable has a more symmetric, bell-shaped distribution --- - Histogram of log of MSRP ```{r} #| include: true #| echo: false cars <- cars |> mutate(msrp_log = log(msrp)) cars |> ggplot(aes(x = msrp_log)) + geom_histogram() + theme_minimal() + theme(text = element_text(size = 20)) ```