Math 115: Introduction to Statistics – 17. Simple Linear Regression

Three Open Questions

In 17a we fit a line and learned how to read it. Three things we didn’t settle:

How do we pick the “best” line? Infinitely many lines pass through a cloud of points — some rule has to choose one. That rule is least squares.
How well does the line fit? Strong and weak relationships both produce a line; we need numbers that tell them apart. Those are $r$ and $R^2$.
Is a line the right tool at all? Some patterns aren’t straight, and a line summarizes them badly. Residual plots are how we check.

Recap: The Linear Model

In 17a we fit a line to the body-measurements data ($n = 507$):

\[\widehat{wgt} = -105 + 1.02 \times hgt\]

Slope: each extra cm of height raises predicted weight by about 1.02 kg. Intercept: sets the line’s vertical position in the cloud — not a meaningful prediction at height 0.

A recap of seventeen A. This is the body-measurements data: five hundred seven people, with height in centimeters on the horizontal axis and weight in kilograms on the vertical axis. The points slope upward, which tells us that taller people tend to weigh more. The blue line is the model we fit to that pattern. Its equation is weight hat equals minus one hundred five, plus one point oh two times height. The two numbers in that equation do different jobs. The slope, one point oh two, sets the tilt of the line. Each additional centimeter of height raises the predicted weight by about one point oh two kilograms. The intercept, minus one hundred five, sets the vertical position. It slides the line up or down until it runs through the middle of the points rather than above them or below them. Taken literally, the intercept is the predicted weight at a height of zero, which is not a meaningful quantity.

Why the Blue Line?

Try it — the dashed green lines also pass through the cloud and slope upward. What would you say to defend the blue one as best?

Eyeballing narrows it down, but it can’t settle it — two people can prefer different lines and neither can prove the other wrong. “Best” needs a criterion: a number every candidate line can be scored on, so the winner is decided by arithmetic instead of opinion.

Here is the same scatterplot, with two more lines drawn in green dashes. Both of them pass through the cloud of points, and both slope upward. The blue line is the one we have been calling the line of best fit. Before we say why, look at all three and ask yourself what you would actually say to defend the blue one. What makes it better than the green ones? Eyeballing gets you a long way. Most people can rule out a line that misses the cloud entirely. But eyeballing cannot settle it. Two reasonable people can look at this picture, prefer different lines, and neither one can prove the other wrong. To call a line the best, we need a criterion: a number we can compute for any candidate line, so that comparing two lines becomes arithmetic instead of opinion. Least squares is that criterion, and it is built out of one simple quantity called the residual.

Residuals

For each data point, the residual is the vertical gap between observed and predicted:

\[e_i = y_i - \hat{y}_i = \text{observed} - \text{predicted}\]

Try it — our line is $\widehat{wgt} = -105 + 1.02 \times hgt$. Someone 175 cm tall weighs 90 kg. Find their predicted weight, then their residual. Which side of the line are they on?

Predicted: $-105 + 1.02 \times 175 = 73.5$ kg. Residual: $e = 90 - 73.5 = 16.5$ kg. Positive → the point sits above the line, so the model underpredicted this person. A negative residual is the mirror image: below the line, model overpredicted.

For each data point, the residual is the vertical gap between the observed y value and the predicted y value — observed minus predicted. Geometrically, residuals are the vertical segments from data points to the line. To illustrate, we’ve drawn the residuals for eight random data points as red segments. Now work one out. Our line is weight hat equals minus one hundred five, plus one point oh two times height. Someone one hundred seventy-five centimeters tall weighs ninety kilograms. Work out what the line predicts for them first, then turn that into their residual. And which side of the line do they sit on? Start with the prediction. One point oh two times one hundred seventy-five is one hundred seventy-eight point five. Subtract one hundred five from that, and the line predicts seventy-three point five kilograms. Now the residual: observed minus predicted. Ninety minus seventy-three point five is sixteen point five kilograms. The residual is positive, which means the observed weight is bigger than the predicted weight, so the point sits above the line — the model underpredicted this person by about sixteen and a half kilos. A negative residual is the mirror image: the point sits below the line and the model overpredicted. And the size of the residual tells you how far from the line the point is, vertically. Residuals are the building block for everything else today — they’re how we’ll measure how well a line fits.

The Least Squares Criterion

The least squares regression line is the one that minimizes the sum of squared residuals:

\[\sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2\]

Try it — why square them? Why not just add up the residuals, $\sum e_i$?

Positive and negative residuals cancel, so the plain sum cannot distinguish a good line from a bad one — even a badly tilted line can sum to zero. Squaring makes every residual positive, so residuals in both directions add together. It also gives more weight to large residuals: a residual of 4 contributes 16, while a residual of 2 contributes 4.

The least-squares regression line is the one that minimizes the sum of squared residuals — the sum over all data points of e sub i squared. Before we accept the squaring, it is worth asking why it is there. Why not simply add up the residuals themselves? Because positive and negative residuals cancel each other. A line that overpredicts some points and underpredicts others can sum to zero, and a good line also sums to zero, so the plain sum cannot distinguish between them. Squaring solves this. Every squared residual is positive, so residuals in both directions add together instead of cancelling. Squaring also gives more weight to large residuals. A residual of two contributes four, while a residual of four contributes sixteen. This gives us the criterion we needed. Every candidate line can be assigned a score, and the least-squares line is the one with the smallest score.

Software Finds $b_0$ and $b_1$

You won’t compute least squares by hand — Jamovi does it for you. The output:

Term	Estimate
Intercept	-105.01
Height (hgt)	1.02

These are the same $b_0 = -105$ and $b_1 = 1.02$ from 17a — the line that minimizes $\sum e_i^2$.

How Strong Is the Linear Pattern? — Correlation $r$

The least-squares line always exists — even when the linear pattern is barely there. We need a separate number to summarize strength (and direction) of the linear relationship.

The correlation coefficient $r$ does that:

\[-1 \leq r \leq 1\]

Value of $r$	Meaning
$r$ close to $+1$	strong positive linear relationship
$r$ close to $-1$	strong negative linear relationship
$r$ close to $0$	weak (or no) linear relationship

Key properties: $r$ is unit-free; swapping $x$ and $y$ doesn’t change $r$; $r$ only captures linear patterns.

The least-squares line always exists. Software will fit a line to any data — including data with no real linear pattern. So we need a separate number to summarize how strong the linear pattern actually is. That number is the correlation coefficient, written lowercase r. It’s bounded between negative one and positive one. R close to plus one means a strong positive linear relationship — as x increases, y reliably increases. R close to negative one means a strong negative linear relationship. R close to zero means weak or no linear relationship — even if the points form a clear curved pattern, r might be near zero, because r only captures linear patterns. There are three key properties. R is unit-free — it’s the same whether you measure height in centimeters or inches. R is symmetric — swapping x and y gives the same r. And critically, r only summarizes linear relationships — a clear curved pattern can have r near zero, and r alone can’t tell you whether a line is the right model.

Visualizing $r$

Scatter plots with different correlations. From OpenIntro IMS2, Figure 7.10.

The sign matches the slope direction; the magnitude reflects how tightly the points hug a straight line.

$r$ for Height vs Weight

Try it — from the scatter alone, which value is $r$ closest to?

(a) $-0.95$

(d) $0.3$

(b) $-0.7$

(e) $0.7$

(c) $-0.3$

(f) $0.95$

(e) $r = 0.717$. The cloud slopes upward, so $r$ must be positive: (a), (b), and (c) are out before you judge strength at all. Of the rest, $0.95$ would hug the line far more tightly, and $0.3$ would be much more scattered, with only a slight upward trend.

Here is the height-and-weight scatter again, with its least-squares line. Before we look up the number, calibrate your eye. There are six choices on the screen. Which one is r closest to? The closest value is positive zero point seven; the actual correlation is zero point seven one seven. Start with the sign. The cloud slopes upward, so r has to be positive, and that rules out the three negative choices before you judge strength at all. That leaves zero point three, zero point seven, and zero point nine five. Zero point nine five would look much tighter, with the points hugging the line closely. Zero point three would look much more scattered, with only a slight upward trend. What we have is in between. The points cluster around the line, but there is plenty of noise. At any given height there is a meaningful range of weights.

$R^2$ — How Much Variability Does the Line Explain?

The coefficient of determination is just $r$ squared:

\[R^2 = r^2 \quad \text{(for simple linear regression).}\]

Interpretation: $R^2$ is the proportion of variability in $y$ that is explained by $x$ through the linear model.

$R^2$ near 1 → the line accounts for most of the variability in $y$.
$R^2$ near 0 → a straight line in $x$ explains little. Careful — that doesn’t always mean $x$ is irrelevant; a strong curved relationship also leaves $R^2$ near 0.

For our model: $R^2 = (0.717)^2 \approx 0.514$.

Height explains about $51\%$ of the variability in weight. The remaining $49\%$ is due to other factors (build, muscle mass, lifestyle, …).

Read the Output

A random sample of 50 students from one entering class at Elmhurst College. Both variables are in thousands of dollars.

\[\widehat{\text{Aid}} = 24.32 - 0.0431 \times \text{Income}, \qquad r = -0.499\]

Interpret the slope in context — then convert it to dollars. Each additional $1,000 of family income is associated with about 0.0431 thousand dollars less gift aid, on average — roughly $43 less.
Predict gift aid for a family income of $100,000. $24.32 - 0.0431 \times 100 = 20.01$ thousand — about $20,000.
Compute $R^2$ and interpret. $R^2 = (-0.499)^2 \approx 0.249$. About 25% of the variability in gift aid is explained by family income; the other 75% comes from merit awards, athletics, program of study, and need beyond raw income.

Your turn, with a real regression from a different setting. These are fifty students drawn at random from one entering class at Elmhurst College. The response is gift aid — grant money the student does not repay — and the predictor is family income. Both are measured in thousands of dollars, which is going to matter for the interpretation. The fitted line is aid hat equals twenty-four point three two minus zero point oh four three one times income, and r is minus zero point four nine nine. Three questions: interpret the slope, make a prediction, and compute R squared. Notice the slope is negative this time. Advance when you’re ready. The slope interpretation. For each additional thousand dollars of family income, we expect gift aid to decrease by about zero point oh four three one thousand dollars, on average. That’s an awkward number until you convert it. Zero point oh four three one thousand dollars is about forty-three dollars. So every extra thousand dollars of family income is associated with roughly forty-three dollars less gift aid. For the prediction, a family income of one hundred thousand dollars means income equals one hundred in the model’s units. Twenty-four point three two minus zero point oh four three one times one hundred is twenty point oh one — so about twenty thousand dollars of gift aid. Square the correlation: minus zero point four nine nine, squared, is about zero point two four nine. So about twenty-five percent of the variability in gift aid is explained by family income. The other seventy-five percent comes from everything else the aid formula responds to — merit awards, athletics, program of study, and need factors beyond raw income. And notice that squaring removed the sign. R squared is always positive, even when the relationship is negative.

Residual Plots — Is the Line Even Right?

A residual plot puts the residual $e_i = y_i - \hat{y}_i$ on the vertical axis and the predicted value $\hat{y}_i$ (or $x_i$) on the horizontal axis.

What we want to see: points scattered randomly around zero, with no obvious pattern.

Try it: does this plot show random scatter, or a pattern? Is a line the right tool here?

Random scatter around zero — no curve, no fan, roughly constant spread. The linear model is appropriate for these data.

Warning Patterns in Residuals

Each column is one dataset: the data with its fitted line on top, the residual plot for those same points directly below.

Curve → the relationship isn’t linear. A line is the wrong summary, and the slope and $R^2$ don’t describe what’s actually there.
Fan → the line is still a reasonable summary, but the spread of $y$ grows with $x$, so the model is far more precise at one end of the range than the other. Constant variance is one of the conditions checked in topic 18.

Three datasets, one per column. The top row shows the data with its fitted line; the bottom row shows the residual plot for those same points. Only the first column is what we want to see. There the points fall along a line, and the residuals scatter randomly around zero with roughly constant spread. That is the case where a line is an appropriate summary — the same picture we saw for height and weight. The middle column shows the curve pattern. The data bend, but a straight line was fit anyway, so the residuals are negative at both ends and positive in the middle. That systematic shape is what to look for. When the residuals show it, the relationship is not linear, a straight line is the wrong summary, and the slope and R squared do not describe what is actually there. The third column shows the fan pattern. Here the line itself is reasonable — the points do follow a straight trend. What changes across the range is how far the points fall from the line. At the low end they sit close to it; at the high end they are much farther away. The model is more precise for small x than for large x. That does not make the slope wrong, but it matters in topic eighteen, where constant variance is one of the conditions we check before trusting a test or a confidence interval about the slope.

Read Two Residual Plots

For each plot: is a line an appropriate summary? If not, what is the problem?

A — random scatter around zero, spread roughly constant. A line is a reasonable summary. B — the fan pattern: residuals sit tightly around zero at the low end, then spread about twice as far once predicted distance gets large. The line still captures the trend, but the model is noticeably more precise for short stopping distances than long ones.

Two residual plots, from two real regressions. Plot A is the Elmhurst gift aid data you worked with a few slides ago — aid predicted from family income. Plot B is a different dataset entirely: the stopping distance of fifty cars, predicted from the speed they were travelling. For each plot, decide whether a line is an appropriate summary, and if it is not, name the problem. Plot A is the healthy case. The points fall on both sides of zero, there is no curve, and the spread stays about the same from left to right. A line is a reasonable summary for the gift aid data. Plot B is the fan pattern from the previous slide. At the low end, where predicted stopping distances are short, the residuals sit close to zero. Once the predicted distance gets large, they spread about twice as far in both directions. The line still captures the trend — stopping distance really does grow with speed — but the model is noticeably more precise for short stopping distances than for long ones. That uneven spread is what to notice.

Outliers, Leverage, Influence

An outlier is a point that stands out from the rest of the data. All three above are outliers — what separates them is how much they affect the line:

High-leverage point — an outlier with an extreme $x$-value (far from the mean of $x$). Has potential to pull the line.
Influential point — a high-leverage point whose removal would visibly change the slope. Usually = high leverage + doesn’t follow the pattern.

Practical check: if you suspect a point is influential, refit the model with it removed and compare the slopes.

Residual plots ask whether a line is the right shape for the data as a whole. There is a second way a regression can mislead you, and it comes down to a single point. An outlier is a point that stands out from the rest of the data. All three circled points are outliers. What separates them is leverage — how unusual the point’s x value is, regardless of where its y falls. In the left panel the circled point sits well below the trend, but its x value is in the middle of the range, so its leverage is low. In the middle panel the circled point sits far to the right, so its leverage is high, but its y value agrees with the trend the other points already show, so the fit is nearly the same with or without it. Leverage is only potential. An influential point is one whose removal would visibly change the slope. That takes both high leverage and a y value that misses the pattern, which is the right-hand panel. This matters because when a single point is driving the slope, your conclusion rests on that one observation rather than on the data as a whole — and a data entry error, or one case that does not belong with the rest, can change the number you report. So if you suspect a point is influential, refit without it and compare the slopes. If the slope barely moves, the point was not driving the result.

Influential or Not?

Blue line = all points. Dashed line = flagged point (red) removed. Which flagged point actually moved the line — and why?

Only C is influential.

A — an outlier at a typical $x$, so low leverage. The line barely moves.

B — an outlier with high leverage, but its $y$ agrees with the trend, so the fit is nearly the same with or without it.

C — high leverage and off the pattern. That combination is what makes it influential: the slope falls from $1.12$ to $0.77$.

Each panel has the same forty background points plus one flagged point, circled in red. The blue line is fit to all the points; the dashed gray line is fit with the flagged point removed. Pause and decide which panel’s flagged point actually moved the line, and why. The answer is panel C. Take them in order. In panel A, the flagged point is an outlier — it badly misses the trend — but its x value sits in the middle of the range, so its leverage is low. The line barely rotates, and the two lines are indistinguishable. In panel B, the flagged point is an outlier with high leverage, because its x value is extreme. But its y value agrees with the trend the rest of the data already show, so the fit is nearly the same with or without it, and again the two lines coincide. Panel C has both. The x value is extreme, so leverage is high, and the y value misses the pattern. That combination is what makes a point influential. The slope falls from about one point one two to about zero point seven seven, and the gap between the solid and dashed lines is clearly visible. High leverage on its own is not enough, and missing the pattern on its own is not enough. Influence takes both.

What We Have, What’s Still Missing

By the end of 17b, the regression toolkit describes the relationship in a sample:

Component	Question it answers
$b_0, b_1$	What’s the equation of the line?
$\hat{y}$ from $b_0 + b_1 x$	What’s the predicted $y$?
$r$	How strong is the linear pattern, and in which direction?
$R^2$	What fraction of variability does $x$ explain?
Residual plot	Is a line even the right tool?
Refit without a point	Is one point driving the slope?

Still missing: is the slope real in the population, or could it be sampling noise from a flat true relationship? That’s inference for the slope — topic 18.

Standing back, we’ve built a full descriptive toolkit for simple linear regression in a sample. The intercept and slope give us the equation of the line. Plugging in x values gives predictions. The correlation coefficient tells us how strong the linear pattern is and whether it runs in the positive or negative direction. R squared tells us what fraction of the variability in y the line explains. Residual plots tell us whether a line is the right summary at all, and refitting without a suspicious point tells us whether that one point is driving the slope. All of this lives in the sample world. The question we still can’t answer: is a slope like the one we observed real in the population, or could it just be sampling noise from a population where the true slope is zero? That’s the inference question — same pattern we’ve used for proportions, means, and group comparisons. Topic eighteen A picks it up right after this lecture: hypothesis test for the slope.

References

Introduction to Modern Statistics (2e) textbook by Mine Çetinkaya-Rundel and Johanna Hardin
Chapter 7
Heinz, G. et al. (2003). Exploring relationships in body dimensions. Journal of Statistics Education, 11(2). (bdims in openintro.)
Gift aid data: random sample of 50 students from the 2011 entering class at Elmhurst College, from “What Students Really Pay to Go to College,” The Chronicle of Higher Education. (elmhurst in openintro.)
Ezekiel, M. (1930). Methods of Correlation Analysis. Wiley. (cars in base R — speeds and stopping distances recorded in the 1920s.)

17. Simple Linear Regression