set.seed(...)
default_split <- Default %>%
initial_split(prop = 0.75, strata = default)
default_train <- training(default_split)
default_validation <- testing(default_split)Chapter 5 Applied Problems: Guided Instructions
Introduction
Use the resampling approaches introduced in the Chapter 5 lab. Build models with parsnip, recipes, and workflows where requested, and produce graphs with ggplot2.
The guidance below explains why each step is useful and identifies relevant functions, variables, and code structures. It deliberately leaves some objects and arguments incomplete. The displayed R fragments are not evaluated when this instruction file is rendered.
Useful packages include tidyverse, tidymodels, and ISLR2. The tidymodels package collection loads the resampling, tuning, and performance packages used here, including rsample, dials, tune, and yardstick.
Problem 1: Validation-set estimates for credit-card default
Complete Problem 5 from Chapter 5 of ISLR (pp. 220-221) using the modifications below. The response default is highly imbalanced, so always examine its class counts before interpreting accuracy.
Part 1(a): Fit the logistic model to all observations
Why this matters. This initial fit establishes the model and provides an apparent, or resubstitution, accuracy. Since the same observations are used for fitting and evaluation, it is not an unbiased estimate of future performance.
Convert Default to a tibble and inspect it with glimpse() and count(default). Specify logistic regression with logistic_reg(), classification mode, and the "glm" engine.
Fit default ~ balance + income using parsnip. Attach predictions to the full data with augment(new_data = Default) and pass the resulting tibble to accuracy(truth = default, estimate = .pred_class).
Use tidy() to inspect coefficient signs and p-values. Briefly explain why high accuracy can occur even for an unhelpful classifier when defaults are rare.
Part 1(b): One 75/25 validation split
Why this matters. Evaluating observations that were not used for fitting provides a more honest estimate of prediction performance.
Set a random seed, then use initial_split() with 75% assigned to training. Since default is imbalanced, use strata = default so both subsets retain similar class proportions.
The main splitting pattern is:
Fit default ~ balance + income using only default_train. Use augment(new_data = default_validation); this automatically supplies predicted probabilities and .pred_class. Compute validation accuracy with yardstick. Validation error can be obtained as \(1-\text{accuracy}\).
Part 1(c): Repeat the validation split
Why this matters. A single validation estimate depends on which observations happen to enter the training and validation sets. Repetition reveals this split-to-split variability.
Repeat Part 1(b) two additional times, using a different seed for every split. Keep the training proportion, stratification variable, model formula, and metric unchanged.
You may repeat the code explicitly or write a short function whose input is a seed and whose output is an accuracy tibble. If you use a function, it should perform these steps internally: set the seed, split, extract training and validation data, fit, augment, and calculate accuracy.
Display all three validation accuracies together. Comment on their variation and report the approximate validation error rate.
Part 1(d): Add student status with a recipe
Why this matters. This part tests whether student improves held-out prediction after balance and income are already included. A recipe also demonstrates how categorical predictors are prepared without manually creating dummy variables.
Use a recipe with default ~ balance + income + student and add step_dummy(all_nominal_predictors()). Combine the recipe and the existing logistic specification in a workflow.
default_rec <- recipe(
default ~ balance + income + student,
data = default_train
) %>%
step_dummy(all_nominal_predictors())
default_wf <- workflow() %>%
add_recipe(...) %>%
add_model(...)Fit the workflow on the training set and evaluate it on the corresponding validation set. For a fair comparison, use the same split for the model with and without student. Compare the two accuracies and state whether student status meaningfully reduces validation error.
Problem 2: Bootstrap standard errors for Boston home values
Complete Problem 9 from Chapter 5 of ISLR (p. 223). The variable medv is the median value of owner-occupied homes in a census tract, measured in thousands of dollars.
Parts 2(a)-2(b): Mean and formula-based standard error
Why this matters. The sample mean estimates the population mean, while its standard error measures how much that estimate would vary across repeated samples.
Use one summarize() call to create a one-row, three-column tibble containing:
- the sample mean of
medv; - the sample standard deviation of
medv; and - the estimated standard error \(s/\sqrt{n}\).
Name the columns clearly. Inside summarize(), later expressions may refer to columns created earlier, so a pattern such as se = sd_value / sqrt(n()) can be used after defining sd_value.
Explain that the standard deviation describes variation among census tracts, whereas the standard error describes uncertainty in the estimated mean.
Part 2(c): Bootstrap standard error of the mean
Why this matters. The bootstrap estimates sampling variability by repeatedly resampling the observed tracts rather than relying only on a theoretical formula.
Set a seed and create 1,000 bootstrap resamples with bootstraps(times = 1000). Each row contains an rsplit object. Use analysis() to recover the resampled data.
Write a small function that accepts one split and returns the mean of medv, or use an anonymous function inside map_dbl(). Add the 1,000 bootstrap means as a column of the bootstrap tibble. The standard deviation of that column is the bootstrap estimate of the standard error.
boston_boots <- Boston %>%
bootstraps(times = 1000)
mean_results <- boston_boots %>%
mutate(boot_mean = map_dbl(splits, function(one_split) {
# extract the resample and calculate mean(medv)
}))Compare the bootstrap standard error with the formula-based estimate from Parts 2(a)-2(b).
Part 2(d): Approximate confidence interval for the mean
Why this matters. Combining the point estimate with its bootstrap standard error gives an approximate range of plausible values for the population mean.
Construct the textbook’s approximate 95% interval using \(\widehat{\mu}\pm2\widehat{SE}_{boot}\). Report the lower and upper endpoints in the original medv units and interpret the interval in context.
Part 2(e): Estimate the median
Why this matters. The median is less affected by unusually expensive or inexpensive census tracts and describes the center differently from the mean.
Use median(medv) to obtain \(\widehat{\mu}_{med}\). Compare it briefly with the sample mean and connect any difference to the shape of the medv distribution.
Part 2(f): Bootstrap standard error of the median
Why this matters. Unlike the mean, the median does not have the same simple introductory standard-error formula, so the bootstrap is especially convenient.
Reuse the same 1,000 bootstrap splits from Part 2(c), but compute the median within every resample. Take the standard deviation of the 1,000 bootstrap medians. Do not create a new set of resamples unless you have a specific reason to do so.
Part 2(g): Estimate the tenth percentile
Why this matters. The tenth percentile describes the lower end of the home-value distribution rather than its center.
Use quantile(medv, probs = 0.10) to estimate the population’s tenth percentile. Explain the estimate as a value below which approximately 10% of census-tract medv values fall.
Part 2(h): Bootstrap standard error of the tenth percentile
Why this matters. Quantile estimates can have sampling distributions that are difficult to approximate with a simple formula, making bootstrap resampling useful.
For every bootstrap split, compute quantile(medv, probs = 0.10). Store the 1,000 estimates and use their standard deviation as the bootstrap standard error. Report the point estimate and its estimated standard error together.
Problem 3: Tune KNN with 10-fold cross-validation
This problem revisits Problem 14(h) from Chapter 4. The goal is to choose the number of neighbors using cross-validation rather than selecting \(K\) from test-set results. Area under the ROC curve is the primary performance metric.
Part 3(a): Recreate mpg01
Why this matters. The same binary response used in Chapter 4 is needed so that KNN can classify automobiles as above- or below-median fuel economy.
Compute the median of mpg, then create mpg01 = 1 when mpg is above the median and 0 otherwise. Convert it to a factor with levels ordered 0, then 1, and check count(mpg01). Do not use continuous mpg as a predictor.
Part 3(b): Create a final training/test split
Why this matters. Cross-validation will tune \(K\) using only the training data; the untouched test set supplies a final performance assessment.
Set a seed and use initial_split(prop = 2/3, strata = mpg01). Extract the training and test tibbles with training() and testing(). Check their row counts and class proportions.
Do not examine test performance while selecting \(K\).
Part 3(c): Tunable model, recipe, and workflow
Why this matters. Marking the neighbor count with tune() tells tidymodels that cross-validation should compare multiple values instead of fitting one fixed KNN model.
Begin the specification with nearest_neighbor(neighbors = tune()), then set classification mode and the "kknn" engine.
Use the Chapter 4 predictor set cylinders + displacement + horsepower + weight. Since KNN is distance based, normalize all numeric predictors in the recipe. Combine the specification and recipe in a workflow.
knn_spec <- nearest_neighbor(neighbors = tune()) %>%
set_mode("classification") %>%
set_engine("kknn")
knn_rec <- recipe(
mpg01 ~ cylinders + displacement + horsepower + weight,
data = Auto_train
) %>%
step_normalize(all_numeric_predictors())Part 3(d): Ten folds and the K grid
Why this matters. Ten-fold cross-validation repeatedly validates each candidate \(K\) on observations excluded from fitting, reducing dependence on one validation split.
Set a new seed and create vfold_cv(v = 10, strata = mpg01) from the training data. Stratification helps maintain the two class proportions across folds.
Create every integer \(K\) from 1 through 100 with grid_regular(neighbors(range = c(1L, 100L)), levels = 100). Verify the grid before tuning.
Use tune_grid() with the workflow, folds, grid, and metric_set(roc_auc, accuracy). You are evaluating 100 values across 10 assessment folds, so explain why this requires 1,000 fitted models.
Part 3(e): Examine cross-validation performance
Why this matters. The performance curve shows whether the apparent best \(K\) is isolated or lies in a broad region of similarly performing models.
Use autoplot() to display the tuning results and collect_metrics() to inspect the numerical estimates and standard errors. Use show_best(metric = "roc_auc", n = 5) for the five largest cross-validated AUC estimates.
Describe how performance changes as \(K\) increases and whether several nearby values perform similarly.
Part 3(f): Apply the one-standard-error rule
Why this matters. The one-standard-error rule chooses a simpler model whose estimated performance is statistically indistinguishable from the best observed performance.
For KNN, larger \(K\) means a smoother, simpler decision boundary. Use select_by_one_std_err(metric = "roc_auc", desc(neighbors)); the descending ordering tells the function to consider larger, simpler values first.
The function is named select_by_one_std_err() in current versions of tune. Use the tuning plot and metric table to explain why the selected \(K\) is within one standard error of the maximum AUC.
Part 3(g): Finalize and fit the workflow
Why this matters. After \(K\) has been selected without using the test data, the chosen model can be refitted using all available training observations.
Pass the one-standard-error result to finalize_workflow(). Fit the finalized workflow to the complete training set. Print the selected neighbor count and finalized workflow so that the final model is documented.
Part 3(h): Final test-set evaluation
Why this matters. This is the first and only evaluation of the selected model on the untouched test set, so it estimates performance on genuinely unseen automobiles.
Augment the test data once and save the prediction tibble. Use .pred_class with accuracy(). If level 0 is the event, use .pred_0 with roc_auc() and roc_curve(), specifying event_level = "first" consistently.
Plot the ROC curve with autoplot(). Compare test accuracy and AUC with the corresponding cross-validation estimates for the selected \(K\). Small differences are expected because the test set and CV assessment folds contain different observations.
Submission reminder
Report seeds and keep validation/test observations out of model fitting and tuning. Explain why each resampling method is used, not merely which function was called. Include concise interpretations of accuracy, bootstrap standard errors, the selected \(K\), and the comparison between cross-validation and final test performance.