Chapter 8 Applied Problems: Guided Assignment

Published

October 29, 2026

library(tidyverse)
library(tidymodels)
library(ISLR2)
library(rpart.plot)

Variable-importance helpers

The vip package provides a convenient common interface for variable-importance plots, and you may see it used in older examples. However, vip is no longer available from CRAN. To avoid requiring installation of an archived package, these applied problems extract variable importance directly from the fitted model engines instead:

  • randomForest::importance() for bagging and random-forest models;
  • xgboost::xgb.importance() for XGBoost models.

The helper functions below use those engine-specific importance values and draw the plots with ggplot2. This gives us the same statistical information needed for these problems without depending on vip.

rf_importance_plot <- function(fit, top_n = 10) {
  rf_engine <- fit |>
    extract_fit_engine()

  imp <- randomForest::importance(rf_engine)

  imp_tbl <- as.data.frame(imp) |>
    rownames_to_column("variable") |>
    as_tibble()

  importance_col <- if ("%IncMSE" %in% names(imp_tbl)) {
    "%IncMSE"
  } else if ("MeanDecreaseAccuracy" %in% names(imp_tbl)) {
    "MeanDecreaseAccuracy"
  } else {
    names(imp_tbl)[ncol(imp_tbl)]
  }

  imp_tbl |>
    transmute(
      variable,
      importance = .data[[importance_col]]
    ) |>
    arrange(desc(importance)) |>
    slice_head(n = top_n) |>
    ggplot(aes(x = reorder(variable, importance), y = importance)) +
    geom_col() +
    coord_flip() +
    labs(x = NULL, y = "Variable importance") +
    theme_minimal()
}

xgb_importance_plot <- function(fit, top_n = 10) {
  xgb_engine <- fit |>
    extract_fit_engine()

  xgboost::xgb.importance(model = xgb_engine) |>
    as_tibble() |>
    slice_head(n = top_n) |>
    ggplot(aes(x = reorder(Feature, Gain), y = Gain)) +
    geom_col() +
    coord_flip() +
    labs(x = NULL, y = "Gain") +
    theme_minimal()
}

Introduction

Use the Chapter 8 lab workflow to complete the following applied problems. Continue to use tidymodels and tidyverse tools.

For tuned models:

  • set a reproducible random seed;
  • tune only on resamples created from the training data;
  • keep the test set untouched until the final model has been selected;
  • inspect tuning plots rather than relying only on one numerical optimum;
  • finalize and fit the chosen workflow using the full training set;
  • report final test-set performance.

The first problem is a regression problem using Carseats. The second is a classification problem using Caravan.

Problem 1: Predicting Sales in Carseats

Problem 1 — Chapter 8, Problem 8 with the assignment modifications

Predict the quantitative response Sales using the remaining variables in the Carseats data.

You will compare:

  1. a regression tree,
  2. a tuned regression tree,
  3. bagging,
  4. random forest,
  5. XGBoost.

Use test RMSE as the main performance measure. For bagging, random forest, and XGBoost, also examine variable importance.

Part (a): Prepare the data

Guidance

  • Convert Carseats to a tibble and examine the variables.
  • Split the observations into training and test sets, stratifying by Sales.
  • Create cross-validation folds from the training data.
  • Use the same split and folds for all models so the comparisons are fair.

Useful functions: initial_split(), training(), testing(), and vfold_cv().

Part (b): Fit an initial regression tree

Guidance

Fit a regression tree predicting Sales ~ . using decision_tree() with the rpart engine.

Then:

  • plot the fitted tree with rpart.plot();
  • compute the test RMSE;
  • briefly interpret the first few splits in the tree;
  • explain what a terminal node prediction represents in a regression tree.

Do not tune the tree yet.

Part (c): Tune tree complexity

Guidance

Tune the tree’s cost_complexity parameter using the cross-validation folds from part (a).

  • Mark cost_complexity = tune().
  • Create a reasonable grid of possible values.
  • Use tune_grid().
  • Plot the tuning results with autoplot().
  • Select the value with the smallest cross-validation RMSE.
  • Finalize the workflow and fit the final tree to the complete training set.
  • Plot the final tree and compute its test RMSE.

Then choose a hypothetical observation with values for the predictors used near the top of the final tree and explain, split by split, how the observation would travel through the tree to a terminal-node prediction.

Part (d): Bagging

Guidance

Fit a bagging model by using a random forest with all available predictors considered at each split.

  • Use rand_forest().
  • Set mtry equal to the number of available predictor columns.
  • Tune min_n.
  • Use randomForest as the engine with variable importance enabled.
  • Use autoplot() to show how min_n affects RMSE and \(R^2\).
  • Select the best model using RMSE.
  • Compute test RMSE.
  • Display variable importance using rf_importance_plot().

Identify the predictors that appear most important.

Part (e): Random forest

Guidance

Now allow the random forest to consider only a subset of predictors at each split.

Tune both mtry and min_n using a space-filling grid such as grid_latin_hypercube().

Then:

  • plot the tuning results;
  • discuss how the two tuning parameters appear related to RMSE and \(R^2\);
  • select the best model by RMSE;
  • finalize and fit the model;
  • compute test RMSE;
  • display variable importance with rf_importance_plot().

Compare the test RMSE and variable-importance pattern with the bagging model.

Part (f): XGBoost

Guidance

Use XGBoost instead of BART.

Tune the model similarly to the Chapter 8 lab. A reasonable specification is to tune:

  • mtry,
  • min_n,
  • tree_depth,
  • learn_rate,
  • loss_reduction,
  • sample_size.

Keep trees fixed at a reasonably large value such as 1000.

Because Carseats contains categorical predictors, explicitly preprocess them with a recipe using dummy variables before finalizing the range of mtry.

Use plots to examine how the tuning parameters relate to RMSE and \(R^2\). Select the final model using RMSE, then compute test RMSE, plot variable importance with xgb_importance_plot(), and compare XGBoost with the regression tree, bagging, and random forest.

Part (g): Model comparison

Guidance

Create a formatted table with one row for each method:

  • Regression tree
  • Tuned regression tree
  • Bagging
  • Random forest
  • XGBoost

Include model name, main tuned parameter(s), and test RMSE.

Use the table to answer:

  1. Which model has the smallest test RMSE?
  2. How much improvement is obtained over the single regression tree?
  3. Do bagging, random forest, and XGBoost identify similar important predictors?
  4. Is the most complex model clearly better, or are some methods essentially competitive?

Problem 2: Predicting Purchase in Caravan

Problem 2 — Chapter 8, Problem 11 with the assignment modifications

Use the Caravan data to predict whether a customer makes a purchase.

You will:

  1. split the data into 75% training and 25% testing;
  2. tune an XGBoost classifier;
  3. evaluate ROC AUC, accuracy, a confusion matrix, and an ROC curve;
  4. compare classification at thresholds 0.5 and 0.2;
  5. compare XGBoost with a simpler KNN model tuned by ROC AUC.

Part (a): Prepare the data

Guidance

  • Convert Caravan to a tibble.
  • Create a 75% training / 25% testing split.
  • Stratify by Purchase.
  • Create cross-validation folds from the training set, also stratifying by Purchase.

Check the class counts before modeling. Purchase = "Yes" is relatively uncommon, so overall accuracy alone may be misleading.

Part (b): Tune XGBoost

Guidance

Build an XGBoost classification workflow.

Use a preprocessing recipe that handles unseen factor levels, creates dummy variables, removes zero-variance predictors, and normalizes numeric predictors.

Tune the same general XGBoost parameters used in Problem 1.

Use cross-validation and produce plots showing the effect of the tuning parameters on ROC AUC and accuracy. Select the final model using ROC AUC, not accuracy.

Then evaluate the final model on the test set:

  • test ROC AUC;
  • test accuracy using the usual 0.5 threshold;
  • confusion matrix using the 0.5 threshold;
  • ROC curve;
  • variable-importance plot.

Use event_level = "second" when treating "Yes" as the event, and use .pred_Yes for ROC calculations.

Part (c): Change the XGBoost threshold to 0.2

Guidance

The default class prediction uses a threshold of 0.5. Create a new predicted class using

\[ \widehat{P}(Purchase = Yes) > 0.2. \]

Then compute the confusion matrix and accuracy.

Compare the 0.2 confusion matrix with the 0.5 confusion matrix. Focus on how lowering the threshold changes the number of predicted purchases, false positives, and false negatives. Explain why a lower threshold may be useful when the positive class is rare.

Part (d): Fit a simpler KNN model

Guidance

Fit a KNN classifier as the simpler comparison model.

  • Use nearest_neighbor(neighbors = tune()).
  • Normalize predictors.
  • Tune \(K\) using ROC AUC.
  • Plot the tuning results.
  • Finalize and fit the selected model.

For the test set, report ROC AUC; accuracy and confusion matrix using threshold 0.5; accuracy and confusion matrix using threshold 0.2; and the ROC curve.

Compare the KNN results with XGBoost.

Part (e): Compare thresholds and models

Guidance

Create a compact comparison table containing at least:

  • XGBoost at threshold 0.5;
  • XGBoost at threshold 0.2;
  • KNN at threshold 0.5;
  • KNN at threshold 0.2.

Include accuracy, number of true positives, number of false positives, number of false negatives, and ROC AUC for each underlying model.

Use the table and ROC curves to answer:

  1. Which model ranks observations better according to ROC AUC?
  2. What is gained and lost by lowering the threshold from 0.5 to 0.2?
  3. Which model/threshold combination would you prefer if identifying potential purchasers is more important than avoiding false positives?
  4. Can the simpler KNN model be made more useful by changing the threshold?

Submission checklist

Before submitting, verify that your report includes reproducible seeds, untouched test sets, cross-validation for every tuned model, tuning plots, final test metrics, tree plots where requested, variable-importance plots for ensemble models, ROC curves for the Caravan classifiers, confusion matrices at both 0.5 and 0.2, formatted comparison tables, and written interpretations.