Chapter 2 Applied Problems: Guided Instructions

Introduction

In addition to labs we will complete Applied Problem sets for each chapter of the ISLR book. These will build on concepts introduced in the labs. Many of the problems will be from the Applied Exercises that are provided at the end of each chapter in the textbook. Some of these will be modified to use functions from the tidyverse and tidymodels packages or to incorporate additional topics. Other problems will be original problems that are not from the textbook.

There are guidelines for Applied Problem submissions on our Moodle page that you should read before you begin working on this problem set.

The suggestions below point toward useful functions and code structures. They are not complete solutions. The displayed R chunks are examples and are not evaluated when this instruction file is rendered. Your submission should contain your completed R code, the relevant output or plots, and written answers to all questions that ask you to describe, compare, explain, or comment.

Problems

Problem 1: College data

Complete Problem 8 from Chapter 2 of the ISLR book (pp. 54–56) with the following modifications and suggestions.

Parts 1(a) and 1(b): Reading and viewing the data

Use the read_csv() function instead of read.csv(). Read the data directly from the website using the URL rather than downloading a copy first.

Do not set the row names, since tibbles do not have row names. In addition to using View(), use glimpse() to look at the data. The first column, which contains the university names, will initially have a name similar to …1. Use rename() to change its name to UniName.

There is a small trick here. Although R prints the column name as …1, the name must be surrounded by backticks when it is used in code. Therefore, part of your import pipeline should look like rename(UniName = \…1`). Connectread_csv()andrename()with a pipe and assign the result tocollege`.

After importing, use glimpse(college) and View(college) to confirm that the university names and variable types look reasonable. The dataset URL is "https://www.statlearning.com/s/College.csv".

Part 1(c)(i): Summaries and counts

Use summary() to produce numerical summaries of the variables. Also use count() to find the numbers of private and public institutions. The variable that identifies these two groups is Private, so begin with a pipeline of the form college %>% count(...).

In your written response, report the two counts and briefly identify a few interesting features of the numerical summaries. For example, look for variables with a large range or observations that might be unusually high or low.

Part 1(c)(ii): Scatterplot matrix

Use select() to select columns 2 through 11. Because the university names remain in the tibble, these column numbers differ from those in the textbook. Your pipeline should begin college %>% select(2:11).

Pass the selected data to ggpairs() from the GGally package rather than using pairs(). You may find it helpful to enlarge the figure by placing the following options at the top of the R chunk:

Look for variables whose points show a noticeable upward or downward pattern. In your discussion, name at least two relationships and describe their direction. Also mention any nonlinear patterns, clusters, or unusual observations that seem important.

Part 1(c)(iii): Comparing tuition by institution type

Use ggplot2 to create side-by-side boxplots of Outstate versus Private, and use theme_classic(). When plotting Variable 1 versus Variable 2, place Variable 1 on the vertical axis. Thus, think carefully about which variable belongs in x = and which belongs in y =.

Useful pieces of the plot are ggplot(aes(...)), geom_boxplot(), and theme_classic(). Assemble these pieces into a complete plot rather than using the base-R plot() function from the textbook.

Use the graph to compare the centers, spreads, overlap, and possible outliers of the two groups. Include this interpretation in your answer.

Part 1(c)(iv): Creating new variables

Use mutate() to add three variables to college:

Elite, indicating whether more than 50% of students come from the top 10% of their high-school class;
Selective, with categories Less, More, and Most, corresponding to less than 20%, between 20% and 50%, and more than 50%; and
Accept.Perc, the percentage of applicants who are accepted.

For Elite, the useful structure is if_else(condition, value_if_true, value_if_false). You will need to supply the condition involving Top10perc and the two category labels.

Creating Selective is more difficult because it has three categories. Use this structure as a guide, replacing each comment with the appropriate condition:

mutate(
  Selective = case_when(
    # condition for the least selective group ~ "Less",
    # condition for the middle group          ~ "More",
    # all remaining schools                   ~ "Most"
  )
)

Remember that case_when() checks conditions from top to bottom. Make sure that schools at the category boundaries are assigned consistently.

The acceptance percentage will use the Accept and Apps columns. Since the requested result is a percentage rather than a proportion, remember the factor of 100.

After creating the variables, use select() with the original columns and the new columns to check several rows in View(). Then use count(Elite) and count(Selective) to check the category totals.

Create side-by-side boxplots of Outstate versus Selective. The plot from Part 1(c)(iii) provides a useful model; change the grouping variable. Describe how out-of-state tuition differs across the three groups.

Finally, save the modified tibble in your clean_data directory. The function call will have the form saveRDS(object_name, "clean_data/file_name.rds").

Part 1(c)(v): Histograms

Use ggplot2 to create histograms for several quantitative variables. Instead of using par(mfrow = c(2, 2)), create at least one faceted plot in which the histograms are separated by Selective and arranged in one column.

The main components you will need are:

geom_histogram() for the histogram;
facet_wrap(~ Selective, ncol = 1) for three vertically arranged panels; and
scale_fill_viridis_d() if fill = Selective is mapped inside aes().

You will need to choose the quantitative variable placed on the x-axis and an appropriate number of bins. Decide whether fixed scales or scales = "free_y" make the comparison easier. Explain your choice and describe the shapes, centers, spreads, and unusual values visible in the distributions.

Part 1(c)(vi): Further exploration

Explore several variables and relationships using plots and numerical summaries. Create at least one additional variable whose meaning you can explain.

Your exploration must include:

a scatterplot comparing two quantitative variables, with both point color and shape determined by Selective;
a set of boxplots with individual observations added using geom_jitter(); and
a written summary of your findings.

For the scatterplot, the mapping will have a form similar to aes(x = ..., y = ..., color = Selective, shape = Selective). Choose the two quantitative variables based on an interesting relationship you noticed in the scatterplot matrix.

For the second graph, combine geom_boxplot() and geom_jitter(). A small value for the width argument in geom_jitter() keeps points close to their group. You may also use outlier.shape = NA in the boxplot layer so that unusual observations are not drawn twice.

A grouped summary can strengthen your interpretation. Consider a pipeline using group_by(Selective) followed by summarize(), and calculate one or two useful group statistics such as a median or mean.

Your written summary should connect each conclusion to a plot or statistic. Describe associations rather than claiming that one variable causes another.

Problem 2: Boston housing data

Complete Problem 10 from Chapter 2 of the ISLR book (p. 57). Convert the Boston data to a tibble using Boston <- Boston %>% as_tibble(). Use functions from the tidyverse to work with the data, ggplot2 and ggpairs() to create plots, and functions from corrr to help answer Part 2(c).

Part 2(a): Understanding the data

Load the ISLR2 package and access the Boston data. Use glimpse() or dim() to determine the numbers of observations and variables. Remember that dim() reports the number of rows first.

Use ?Boston to open the help page. The abbreviated variable names are not always self-explanatory, so use this documentation to explain what the rows and columns represent. Do not simply copy the output of glimpse() as your answer.

Part 2(b): Pairwise scatterplots

Use ggpairs() to investigate pairwise relationships among the variables. A plot containing every variable may be crowded, so you may first use select() to create a smaller group of related variables and then make more than one scatterplot matrix.

The basic pipeline is Boston %>% select(...) %>% ggpairs(). Choose the columns yourself. Make sure that your selections allow you to investigate several different relationships rather than repeatedly showing nearly the same variables.

When describing a relationship, name both variables, state whether the direction is positive or negative, and comment on its apparent strength. Also notice curvature, clusters, and unusual observations. For the 0/1 variable chas, a boxplot after converting it to a factor may be more informative than treating it as a continuous variable.

Part 2(c): Relationships with crime

Determine whether any predictors appear to be associated with per-capita crime rate, crim. Use correlations to identify promising variables, but use scatterplots to decide whether the numerical correlations describe the relationships well.

The following short construction may be helpful because the corrr workflow is less familiar:

cor_Boston <- Boston %>% correlate()

cor_Boston %>% focus(crim)

You can send the correlation object to rplot() for a graphical overview. To locate stronger relationships, arrange the focused correlations by their absolute values. The expression abs(crim) ignores whether a correlation is positive or negative when measuring its size.

After selecting a few candidate predictors, make separate scatterplots with crim on the vertical axis. Consider adding a smooth curve with geom_smooth(se = FALSE) to help reveal a nonlinear pattern. Explain the direction of each important relationship and whether clusters or extreme observations affect your conclusion.

Part 2(d): Unusually high values

Investigate whether any census tracts have particularly high values of crim, tax, or ptratio. Histograms or boxplots are useful for seeing whether a few observations are separated from the rest. Numerical summaries such as the minimum, quartiles, and maximum help support what you see.

Rather than writing three almost identical summaries, try select(crim, tax, ptratio) %>% summary(). If you prefer a tidy plot with separate panels, pivot_longer() can put the three selected columns into a pair of columns such as variable and value, which can then be faceted.

State what you mean by “particularly high” and comment on the range of each variable. Do not base the conclusion only on the maximum.

Part 2(e): Charles River tracts

Find how many census tracts border the Charles River. According to the documentation, chas is an indicator variable, so first determine which value means that a tract borders the river.

The simplest useful output is Boston %>% count(chas). Select the appropriate count from that table and report it in context. An alternative is to use sum() on a logical condition such as chas == ....

Part 2(f): Median pupil–teacher ratio

Find the median pupil–teacher ratio among the census tracts. Identify the relevant variable using the documentation, then use summarize() with median().

Your pipeline can begin Boston %>% summarize(...). Give the calculated column a descriptive name and report the result in context, rather than displaying an unlabeled number.

Part 2(g): Tract with the lowest home value

Identify the census tract with the lowest median value of owner-occupied homes. Report the values of the other predictors for this tract, compare them with the overall ranges, and comment on your findings.

The response variable for this question is medv. Two useful approaches for locating the row are which.min() or slice_min(). If you use slice_min(), its order_by argument should identify the variable being minimized. Consider using with_ties = TRUE so that tied minimum values would not be missed.

To preserve the original row number, you can first add an identifier with mutate(tract = row_number()). This construction is slightly more advanced, so the beginning of the pipeline could be:

Boston %>%
  mutate(tract = row_number()) %>%
  slice_min(order_by = ..., n = 1, with_ties = TRUE)

Use summary(Boston) or a suitable summarize(across(...)) calculation to obtain reference values for the full dataset. Compare the selected tract with quartiles or medians as well as with minima and maxima. Treat chas as a category when interpreting it.

Part 2(h): Tracts with many rooms

Find how many tracts average more than seven rooms per dwelling and how many average more than eight. Pay attention to the phrase more than: your conditions should use > rather than >=.

One approach is to use filter(rm > ...) followed by count() or summarize(n = n()). Change the cutoff to answer the second count. You could also use sum(rm > ...), since R treats a true condition as 1 when it is summed.

Next, create a separate tibble containing only the tracts with more than eight rooms. A useful beginning is many_room_tracts <- Boston %>% filter(...). Use numerical summaries and several carefully chosen plots to describe this group.

To make comparisons easier, you might create a two-level variable using if_else(rm > 8, "More than 8", "8 or fewer"). Then use that variable as the grouping or faceting variable in plots of medv, crim, ptratio, lstat, or other variables that you find informative. You do not need to plot every variable, but your description should address several important characteristics and should acknowledge if the more-than-eight-room group is small.

Submission reminder

For each part, include enough output to support your answer, but avoid filling the submission with unused output. Every requested description or comparison should be written in complete sentences and should refer to specific features of the displayed results.