Linear Regression, Single Predictor

Body Measurements

• bdims body measurement dataset, available here
• 507 physically active individuals (247 men, 260 women)
• age, weight (wgt), height (hgt), sex, 21 body girth variables (e.g., hip girth)

Weight vs. Height

Scatter plot of weight vs. height.

It appears that the data fall roughly along a line.

Linear Model

Scatter plot of weight vs. height with line of best fit.

We can add a line of best fit to the scatter plot.

• Equation for line: \[y = b_0 + b_1 x\]
• \(b_0\) and \(b_1\) are coefficients
  • \(b_0\) = intercept
  • \(b_1\) = slope
• \(b_0\) and \(b_1\) are statistics (fit using sample)
• \(\beta_0\) and \(\beta_1\) are the corresponding parameters
• The fitted values are \(b_0=-105.0\), \(b_1=1.018\)

Variable Roles

• wgt = outcome/response (dependent variable, \(y\))
• hgt = predictor (independent variable, \(x\))
• We use a hat to indicate an estimate or prediction \[\widehat{wgt} = -105.0 + 1.02 \times hgt\]

Using a Model to Make Predictions

• Use the model to predict the weight of a person with a given height
• The predicted weight of a 170 cm tall individual is \[\begin{array}{rcl}\widehat{wgt} &=& -105.0 + 1.018 \times hgt\\ &=& -105.0 + 1.02 \times 170 \\ &=& 68.4\, kg\end{array}\]

Correlation

• The correlation coefficient describes strength and direction of a linear relationship
• Denoted \(r\) for a sample, \(\rho\) for a population
• \(-1\leq r\leq1\)

• Direction and strength of linear relationship
  • Direction
    • \(r>0\) indicates a positive association
    • \(r<0\) indicates a negative association.
  • Strength
    • Values close to 0 indicate a weak linear association
    • Values close to -1 or 1 indicate a strong linear association
• Correlation does not depend on the units of measurement of the variables
• Correlation does not depend on the roles of the varibles

Some scatter plots and their correlations. IMS 2 Figure 7.10.

• Let \((x_i,y_i)\) be the \(i\)th observation of the numeric variables \(x\) and \(y\)
• Then \(r\) is \[r=\frac{1}{n-1}\sum_{i=1}^n\frac{x_i-\bar{x}}{s_x}\cdot\frac{y_i-\bar{y}}{s_y}\]
• Here \(\bar{x}\) and \(\bar{y}\) are the sample means, and \(s_x\) and \(s_y\) are the sample standard deviations of the \(x\) and \(y\)
• \(r\) is independent of the units of measurement of \(x\) and \(y\)

Scatter plot of weight vs. height with line of best fit.

Correlation between height and weight: \(r=0.717\)

• Properties of least squares line

  • The line passes through the point \((\bar{x},\bar{y})\)
  • The slope is \[\boxed{b_1=\frac{s_y}{s_x}r}\]

• We can use these properties to compute the slope and intercept if we know the means, SDs, and correlation

Calculating the Slope

We use \(b_1=\frac{s_y}{s_x}r=\frac{13.3}{9.41}\cdot0.717\approx 1.02\) to calculate the slope

Calculating the Intercept

\(b_0=69.1-1.02\cdot171 \approx -105.01\)

Interpretation of coefficients

\[\widehat{wgt} = -105.0 + 1.02 \times hgt\]

• Slope: for each additional centimeter of height, we expect weight to increase by 1.02 kg
• Intercept: we would predict a 0 cm tall individual to weigh -105.0 kg
• In many cases, this intercept interpretation is not useful
• Better to think of intercept as positioning line vertically so it passes through the data cloud

Extrapolation

• Predicting response variable outside of the range of the observed data is an example of extrapolation
• We should not expect the model to apply reliably outside of this range
• Extrapolation can lead to nonsensical predictions (0 cm tall individuals with negative weight) or inaccurate ones
• Predicting response variable inside of the range of the observed data is called interpolation

Least Squares Regression

• How is the best fit line determined?
• Slope and intercept chosen to minimize the error between the observed and predicted response

Plot highlighting three residuals. IMS2 Figure 7.8.

The residual (error) for the \(i\)th observation \((x_i,y_i)\) is \[e_i = y_i - \hat{y}_i\]

• The least squares regression line minimizes the sum of the squared residuals, \[e_1^2+e_2^2+\cdots+e_n^2\]

Using Software

• Typically we will use Jamovi or other Statistical software to compute the coefficients of the least squares line
• We will learn to do this in J Lab 3
• Results are usually presented in a regression table

 MODEL SPECIFIC RESULTS

 MODEL 1

 Model Coefficients - wgt                                              
 ───────────────────────────────────────────────────────────────────── 
   Predictor    Estimate       SE            t            p            
 ───────────────────────────────────────────────────────────────────── 
   Intercept    -105.011254    7.53940919    -13.92831    < .0000001   
   hgt             1.017617    0.04398680     23.13459    < .0000001   
 ───────────────────────────────────────────────────────────────────── 

Coefficient of determination (\(R^2\))

• The coefficient of determination, also known as R-squared (\(R^2\)) is used to measure how well a model describes the data
• \(R^2\) is the proportion of variation in the outcome/response variable that is explained by the model
• For simple linear regression, \(R^2 = r^2\)

• \(R^2\) will always have values between 0 and 1
• Value close to 1: linear model fits the data well (describes nearly 100% of the variability in outcomes)
• Value close to 0 indicates that it does not fit well

• We can compute \(R^2\) using Jamovi or other Statistical Software
• Height explains about 51.5% of the variability in weights

 Model Fit Measures                  
 ─────────────────────────────────── 
   Model    R            R²          
 ─────────────────────────────────── 
       1    0.7173011    0.5145208   
 ─────────────────────────────────── 
   Note. Models estimated using
   sample size of N=507

Coefficient of determination (\(R^2\))

• The coefficient of determination, also known as R-squared (\(R^2\)) is used to measure how well a model describes the data
• \(R^2\) is the proportion of variation in the outcome/response variable that is explained by the model
• For simple linear regression (one numeric predictor), \(R^2 = r^2\)

• \(R^2\) will always have values between 0 and 1
• Value close to 1: linear model fits the data well (describes nearly 100% of the variability in outcomes)
• Value close to 0 indicates that it does not fit well

Residual plots

• residual plot is a plot of residuals vs. predicted values (scatter plot with points \((\hat{y}_i,e_i)\))
• Useful for diagnosing problems with the linear models
• If there is a pattern in the residual plot, then a linear model, single predictor is most likely not appropriate.
• A more complicated model (e.g., a nonlinear model or a model that includes more predictors) may be more appropriate

There are no obvious patterns in the height vs. weight residual plot.

Residual plot for weight vs. height with horizontal line at \(e=0\) for reference.

More residual plots

• Which one(s) appear to have a pattern?

Some scatter plots (top) and corresponding residual plots (bottom). From IMS2.

Outliers

• Outliers are observations that fall far from the point cloud
• high leverage points fall horizontally far from the center of the point cloud
• high leverage points have more pull on the regression line
• influential points have a strong influence on the slope of the regression line
• influential points can be identified by fitting a line with the point removed. If the slope is very different than when the point is included, then the point is influential.

Each of the following plots has an outlier. Which are high leverage? Influential?

More residual plots. From IMS2 Example 7.3

• A: There is one outlier far from the other points, though it only appears to slightly influence the line.
• B: There is one outlier on the right though it is quite close to the least squares line, which suggests it wasn’t very influential. However, it is a point of high leverage.
• C: There is one point far away from the cloud (so it is a high leverage point), and this outlier appears to pull the least squares line up on the right (so it is influential).

And even more residual plots

Continuation of residual plots from IMS2 Example 7.3

• D: There is a primary cloud and then a small secondary cloud of four outliers. The secondary cloud appears to be influencing the line somewhat strongly, making the least square line fit poorly almost everywhere. There might be an interesting explanation for the dual clouds, which is something that could be investigated.

• E: There is no obvious trend in the main cloud of points and the outlier on the right appears to largely control the slope of the least squares line. So it is an influential and also high leverage point

• F: There is one outlier far from the cloud (and, therefore a point of high leverage). However, it falls close to the least squares line and does not appear to be very influential.

Example

More residual plots. (From IMS2 Exercise 32). 7.2

• The average percentage of children receiving reduced-fee lunches is 30.833% with a standard deviation of 26.724% and the average percentage of bike riders wearing helmets is 30.883% with a standard deviation of 16.948%.
• Let us find coefficients \(b_1\) and \(b_0\) for this data set.

Total Sum of Squares (Optional)

• total sum of squares, denoted SST, describes the total variation in the outcome \[SST = (y_1-\bar{y})^2 + (y_2-\bar{y})^2 + \cdots + (y_n-\bar{y})^2\]
• Note that SST does not involve the model at all
• However, can think of a null model that uses the sample mean as the prediction
• SST is the sum of the squared residuals for the null model

Sum of Squared Errors

• sum of squared errors, denoted SSE, quantifies the variation in outcomes that the model fails to describe \[\begin{array}{rcl}SSE &=& (y_1-\hat{y}_1)^2 + (y_2-\hat{y}_2)^2 + \cdots + (y_n-\hat{y}_n)^2 \\ &=& e_1^2 + e_2^2 + \cdots + e_n^2\end{array}\]
• Given by the sum of the squared residuals, which we have encountered before

Regression Sum of Squares

• regression sum of squares, denoted SSR, measures the variation that is accounted for by the model \[SSR = SST - SSE\]
• Hence, the proportion of variation in the outcome that is described by the model is \[R^2 = \frac{SST - SSE}{SST} = 1 - \frac{SSE}{SST}\]

Categorical predictor with 2 levels (Optional)

• If the independent variable is categorical can we still use linear regression?
• We will consider categorical predictors with 2 levels
• Can have more than 2 (chapter 8)
• Linear model only makes sense if \(x\) is a number, so we need to recode the levels of the predictor as numbers

• In the bdims data, the sex variable has two levels: 0 for female, and 1 for male
• This variable already has indicator coding
• We can code any variable with two levels this way
• Assign one level as 0 and the other as 1

The equation for the regression line is \[\widehat{wgt}=60.6 + 17.5\times sex\]

• Females (sex = 0) \[\widehat{wgt} = 60.6\,kg\]
• Males (sex = 1) \[\widehat{wgt} = 60.6 + 17.5 = 88.1\,kg\]
• intercept is predicted female height
• slope adjusts height to get predicted male height

The model predicts that each female will have the mean weight for females and each male will have the mean eight for males!

• We can have R compute \(R^2\)
• Height explains about 51.5% of the variability in weights
• Sex explains about 43.3% of the variability in weights