17. Simple Linear Regression

Three Open Questions

In 17a we fit a line and learned how to read it. Three things we didn’t settle:

1. How do we pick the “best” line? Infinitely many lines pass through a cloud of points — some rule has to choose one. That rule is least squares.

2. How well does the line fit? Strong and weak relationships both produce a line; we need numbers that tell them apart. Those are r and R^2.

3. Is a line the right tool at all? Some patterns aren’t straight, and a line summarizes them badly. Residual plots are how we check.

Why the Blue Line?

Try it — the dashed green lines also pass through the cloud and slope upward. What would you say to defend the blue one as best?

Eyeballing narrows it down, but it can’t settle it — two people can prefer different lines and neither can prove the other wrong. “Best” needs a criterion: a number every candidate line can be scored on, so the winner is decided by arithmetic instead of opinion.

Residuals

For each data point, the residual is the vertical gap between observed and predicted:

e_i = y_i - \hat{y}_i = \text{observed} - \text{predicted}

Try it — our line is \widehat{wgt} = -105 + 1.02 \times hgt. Someone 175 cm tall weighs 90 kg. Find their predicted weight, then their residual. Which side of the line are they on?

Predicted: -105 + 1.02 \times 175 = 73.5 kg. Residual: e = 90 - 73.5 = 16.5 kg. Positive → the point sits above the line, so the model underpredicted this person. A negative residual is the mirror image: below the line, model overpredicted.

The Least Squares Criterion

The least squares regression line is the one that minimizes the sum of squared residuals:

\sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2

Try it — why square them? Why not just add up the residuals, \sum e_i?

Positive and negative residuals cancel, so the plain sum cannot distinguish a good line from a bad one — even a badly tilted line can sum to zero. Squaring makes every residual positive, so residuals in both directions add together. It also gives more weight to large residuals: a residual of 4 contributes 16, while a residual of 2 contributes 4.

How Strong Is the Linear Pattern? — Correlation r

The least-squares line always exists — even when the linear pattern is barely there. We need a separate number to summarize strength (and direction) of the linear relationship.

The correlation coefficient r does that:

-1 \leq r \leq 1

Key properties: r is unit-free; swapping x and y doesn’t change r; r only captures linear patterns.

Visualizing r

Scatter plots with different correlations. From OpenIntro IMS2, Figure 7.10.

The sign matches the slope direction; the magnitude reflects how tightly the points hug a straight line.

r for Height vs Weight

Try it — from the scatter alone, which value is r closest to?

(e) r = 0.717. The cloud slopes upward, so r must be positive: (a), (b), and (c) are out before you judge strength at all. Of the rest, 0.95 would hug the line far more tightly, and 0.3 would be much more scattered, with only a slight upward trend.

R^2 — How Much Variability Does the Line Explain?

The coefficient of determination is just r squared:

R^2 = r^2 \quad \text{(for simple linear regression).}

Interpretation: R^2 is the proportion of variability in y that is explained by x through the linear model.

• R^2 near 1 → the line accounts for most of the variability in y.
• R^2 near 0 → a straight line in x explains little. Careful — that doesn’t always mean x is irrelevant; a strong curved relationship also leaves R^2 near 0.

For our model: R^2 = (0.717)^2 \approx 0.514.

Height explains about 51\% of the variability in weight. The remaining 49\% is due to other factors (build, muscle mass, lifestyle, …).

Residual Plots — Is the Line Even Right?

A residual plot puts the residual e_i = y_i - \hat{y}_i on the vertical axis and the predicted value \hat{y}_i (or x_i) on the horizontal axis.

What we want to see: points scattered randomly around zero, with no obvious pattern.

Try it: does this plot show random scatter, or a pattern? Is a line the right tool here?

Random scatter around zero — no curve, no fan, roughly constant spread. The linear model is appropriate for these data.

Warning Patterns in Residuals

Each column is one dataset: the data with its fitted line on top, the residual plot for those same points directly below.

• Curve → the relationship isn’t linear. A line is the wrong summary, and the slope and R^2 don’t describe what’s actually there.
• Fan → the line is still a reasonable summary, but the spread of y grows with x, so the model is far more precise at one end of the range than the other. Constant variance is one of the conditions checked in topic 18.

Outliers, Leverage, Influence

An outlier is a point that stands out from the rest of the data. All three above are outliers — what separates them is how much they affect the line:

• High-leverage point — an outlier with an extreme x-value (far from the mean of x). Has potential to pull the line.
• Influential point — a high-leverage point whose removal would visibly change the slope. Usually = high leverage + doesn’t follow the pattern.

Practical check: if you suspect a point is influential, refit the model with it removed and compare the slopes.

What We Have, What’s Still Missing

By the end of 17b, the regression toolkit describes the relationship in a sample:

Still missing: is the slope real in the population, or could it be sampling noise from a flat true relationship? That’s inference for the slope — topic 18.

References

• Introduction to Modern Statistics (2e) textbook by Mine Çetinkaya-Rundel and Johanna Hardin
• Chapter 7
• Heinz, G. et al. (2003). Exploring relationships in body dimensions. Journal of Statistics Education, 11(2). (bdims in openintro.)
• Gift aid data: random sample of 50 students from the 2011 entering class at Elmhurst College, from “What Students Really Pay to Go to College,” The Chronicle of Higher Education. (elmhurst in openintro.)
• Ezekiel, M. (1930). Methods of Correlation Analysis. Wiley. (cars in base R — speeds and stopping distances recorded in the 1920s.)