2. Exploring Data

Math 115

Yurk and Martin

Comic Characters Dataset

  • 15,128 comic characters from DC and Marvel comics
  • Key variables:
    • name: Character name
    • identity (id): Personal identity status (secret, public, no dual)
    • alignment (align): Good, bad, neutral, etc.

Frequency Tables

We can summarize a categorical variable using a frequency table:

identity count
No Dual 1,470
Public 5,958
Secret 7,691
Unknown 9
Total 15,128

Proportion Calculations

Example 1: What proportion of comic characters have a secret identity?

\[\text{Proportion} = \frac{\text{Count}}{\text{Total}} = \frac{7,691}{15,128} = 0.508\]

Example 2: What percentage of comic characters have a secret identity?

\[\text{Percentage} = \text{Proportion} \times 100 = 0.508 \times 100 = 50.8\%\]

Proportions

Including proportions reveals patterns more clearly than raw counts:

identity count proportion
No Dual 1,470 0.097
Public 5,958 0.394
Secret 7,691 0.508
Unknown 9 0.001
Total 15,128 1.000

Secret identities (50.8%) are most common.

Visualizing Categorical Data

We can visualize categorical data using a bar plot:

Bar plot showing frequencies of identity variable levels

Life Expectancy Dataset

Moving to numerical data:

  • Michigan counties (83 counties)
  • Variables: county name, average life expectancy

Why numerical data needs different tools:

  • Categorical: Count frequencies in each category
  • Numerical: Describe center, variability (spread), and shape of distribution

Dotplot Visualization

Each dot = one Michigan county’s life expectancy

Mean and Median

Two measures of center:

Mean: Arithmetic average

\[\bar{x}=\frac{x_1+x_2+\cdots+x_n}{n}\]

  • Uses every data point

Median: Middle value when data are ordered

  • 50% above, 50% below

Comparing Center Measures

Michigan life expectancy data:

mean median
78.04108 77.85

Mean (red, dashed) vs. Median (blue, solid)

Percentiles and Quartiles

Percentile: The Xth percentile is a value where X% of observations fall below.

Quartiles divide data into four equal parts:

  • Q1 (25th percentile): 25% of data below this value
  • Q2 (50th percentile): The median - splits data in half
  • Q3 (75th percentile): 75% of data below this value
Q1 median Q3
77.3 77.85 78.8425

Finding the median

Consider the data set: 13, 16, 12, 14, 19, 12, 14, 13, 14

  • Write in order: 12, 12, 13, 13, 14, 14, 14, 16, 19

There are 9 points in the data set, n = 9

  • The median is the midpoint at (n+1)/2 = (9+1)/2 = 5
  • The median is the 5th point = 14

Drop the 9th point in the set so n = 8 and (n+1)/2 = 4.5

  • Find the 4th and 5th point and take the average
  • The median = (13+14)/2 = 27/2 = 13.5

Finding the quartiles

Find the 1st quartile of 9-point set

  • 12, 12, 13, 13, 14, 14, 14, 16, 19
  • Take the first half of the set: 12, 12, 13, 13, 14
  • n = 5, the midpoint of the first half is at (5+1)/2 = 3
  • The 1st quartile is 13

The 3rd quartile is the midpoint of the set’s second half

Range and IQR

Measures of spread (variability):

Range: Maximum - minimum

  • For MI: 81.875 - 75.075 = 6.8 years
  • Uses only two data points (extremes)

Interquartile Range (IQR): Q3 - Q1

  • For MI: 78.8425 - 77.3 = 1.5425 years
  • Spread of middle 50%

Standard Deviation

Most important spread measure:

\[s = \sqrt{\frac{\sum_{i=1}^n(x_i-\bar{x})^2}{n-1}}\]

Interpretation: Typical distance from the mean

Comparing Spread Measures

Michigan life expectancy data:

range IQR SD
6.8 1.54 1.21

Complete summary:

min Q1 median Q3 max mean SD
75.08 77.3 77.85 78.84 81.88 78.04 1.21

Outliers

Outlier: Data point unusually far from other observations

1.5×IQR rule: A point is an outlier if it is more than 1.5×IQR beyond Q1 or Q3

For Michigan counties:

  • Q1 = 77.3, Q3 = 78.84, IQR = 1.54
  • Lower bound: 77.3 - 1.5×1.54 = 74.99
  • Upper bound: 78.84 + 1.5×1.54 = 81.16

Outliers Identified

Result: 2 Michigan counties identified as outliers (both with high life expectancy)

County Life Expectancy (years)
Leelanau County 81.875
Ottawa County 81.290

Outliers shown in red

Skewness: Mean vs Median

Car prices (MSRP): Mean = $29,783 (red-dashed); Median = $26,350 (blue-solid)

Mean > median indicates right skew (high prices pull mean upward)

Note: Choosing Appropriate Measures

When to use mean vs median:

  • Mean: Best for symmetric distributions; uses all data
  • Median: Better when extreme values present; resistant to outliers

When to use SD vs IQR:

  • SD: Pair with mean for symmetric data
  • IQR: Pair with median when extreme values present

Key principle: Choose measures that are resistant when data contain extreme values

Visualization Types for Numerical Data

Four common visualizations:

  1. Dotplot: See every individual data point
  2. Histogram: Group data into bins to reveal shape
  3. Density plot: Smooth curve showing distribution shape
  4. Box plot: Display numerical summaries (Q1, median, Q3, outliers)

Each serves different purposes and reveals different aspects of the same data

Dotplot

Best for: Small datasets, seeing individual values

Histogram

Best for: Large datasets, revealing overall shape

Density Plot

Best for: Smooth representation of distribution shape

Box Plot

Best for: Showing numerical summaries visually

Box Plot Construction

Box construction:

  • Box spans IQR (Q1 to Q3)
  • Center line = median
  • Whiskers extend to min/max (excluding outliers)
  • Dots = outliers (beyond 1.5×IQR)

References