--- title: "Confidence Intervals with Bootstrapping" subtitle: | | IMS2 Ch. 12 | Math 115 author: "Yurk" format: revealjs: theme: beige df-print: paged embed-resources: true editor: source metadata: license: CC BY-SA --- ## Sampling Distribution ```{r} #| include: false #| echo: false library(tidyverse) library(infer) library(openintro) ``` - A **sampling distribution** is the distribution we would obtain if we could select samples of the same size again and again from the same population, calculating the value of the statistic of interest each time - Much of inferential statistics is based on being able to approximate sampling distributions ------------------------------------------------------------------------ - We rarely have the ability to select many samples from the same population (if we did we would usually just select a larger sample!) - However, we can *make up* a population and repeatedly sample from it to test different statistical ideas ## Candidate X ```{r} #| include: false #| echo: false true_prop_yes <- 0.6 set.seed(12345) all_polls <- tibble(poll = rep(1:100, each = 30), vote = sample(c("no","yes"), size = 3000, replace = TRUE, prob = c(1 - true_prop_yes, true_prop_yes))) all_props <- all_polls |> group_by(poll) |> summarise(prop_yes = mean(vote == "yes")) ``` - Candidate X is running for president in the US - Candidate X's campaign can only afford to poll a sample of 30 voters - They would like to use that sample to estimate the proportion of voters who support Candidate X in the whole population ($p$) ---- - Let's make up a population in which 60% of voters support Candidate X for president (parameter: p = 0.6) - The corresponding statistic is the proportion of voters *in the sample* who support Candidate X (statistic: $\hat{p}$) - Using a computer, we can repeatedly select random samples of 30 from the theoretical population to see how $\hat{p}$ might vary from sample to sample - Let's do this 100 times - What does the sampling distribution of $\hat{p}$ look like? ------------------------------------------------------------------------ ```{r} #| include: true #| echo: false #| fig-cap: "Sampling distribution. Proportions for 100 samples of 30 from a population with 60% (dashed vertical line) support for Candidate X." all_props |> ggplot(aes(x = prop_yes)) + geom_dotplot(dotsize = 0.8, fill = "white") + geom_vline(xintercept = true_prop_yes, color = "red", linetype = "dashed", linewidth = 1) + scale_y_continuous(NULL, breaks = NULL) + labs(x = "proportion of Candidate X supporters (samples of 30)") + theme_minimal() + theme(text = element_text(size = 18)) ``` ## Describing the Sampling Distribution ```{r} #| include: true #| echo: false se_p <- all_props |> summarize(se = sd(prop_yes)) %>% pull() ``` - We can summarize the sampling distribution with a mean and standard deviation - The mean of this distribution is the true proportion of yes votes for Candidate X (0.6) - The standard deviation of a statistic's sampling distribution has a special name, the **standard error** (SE) - The standard error for the proportion of yes votes for Candidate X is SE = `r round(se_p,3)`. ## A more realistic scenario (one sample) ```{r} #| include: false #| echo: false one_poll <- tibble(vote = sample(c(rep("yes", 21), rep("no", 9)))) ``` - Suppose you work for Candidate X's campaign and want to estimate the proportion of US voters that support Candidate X - You do not know that the true value is $p=0.6$ - You conduct a poll in which you select a *single* sample of 30 voters ------------------------------------------------------------------------ Results of the poll ```{r} #| include: true #| echo: false one_poll ``` ------------------------------------------------------------------------ - You find that 21 people in the sample support Candidate X. Thus, $\hat{p} =21/30 = 0.7$ - This gives you a **point estimate** for the proportion of voters who support Candidate X ($p\approx 0.7$) - However, we know that there will be variability from sample to sample, creating uncertainty in our estimate - We can express that uncertainty by making an **interval estimate** instead - E.g., we might estimate Candidate X's support to be between 0.6 and 0.8 based on how much the statistic is expected to vary from sample to sample ------------------------------------------------------------------------ - Calculating a **95% confidence interval** is one way to find such an estimate - To calculate one from a single sample, we need to approximate the sampling distribution - We can use a randomization-based approach called **bootstrapping** ## Bootstrapping - In practice, our single sample is the best approximation we have of the population - We can simulate repeated random sampling from the population by randomly sampling from the sample *with replacement* - We select bootstrap samples that are the same size as the original sample - Variability tends to be very close to the variability in the sampling distribution ------------------------------------------------------------------------ - Let's select 100 bootstrapped sampled using the results from our one poll - Later, we will learn how to do this using Jamovi ```{r} #| include: true #| echo: false #| message: false source("code/jamferi_functions.R") set.seed(8675309) one_poll_boot_samples <- one_poll |> specify(response = vote, success = "yes") |> generate(reps = 100, type = "bootstrap") one_poll_boot <- one_poll_boot_samples |> calculate(stat = "prop") ``` ------------------ 100 Bootstrap samples ```{r} #| include: true #| echo: false one_poll_boot_samples %>% ungroup() %>% mutate(person = rep(1:30,100)) %>% pivot_wider(names_from = replicate, values_from = vote, names_prefix = "boot_") %>% select(-person) ``` ## Bootstrap distribution - Calculate $\hat{p}$ for each bootstrap sample - The resulting distribution is called a **bootstrap distribution** for $\hat{p}$ - The variability of this distribution is approximately the same as the variability of the sampling distribution ------------------------------------------------------------------------ ```{r} #| include: true #| echo: false #| fig-cap: "Distribution of bootstrapped proportions. Each dot represents the proportion of support for Candidate X from a single bootstrap sample of 30." # one_poll_boot |> # ggplot(aes(x = stat)) + # geom_dotplot(dotsize = 0.1) + # scale_y_continuous(NULL, breaks = NULL) + # labs(x = "bootstrapped proportion of yes votes") + # theme_minimal() + # theme(text = element_text(size = 18)) singleProp_bootPlot(one_poll_boot, 21/30, dotHist = "dotplot", confLevel = 95, ciType = "bootperc", plotCI = FALSE) ``` ## Bootstrap estimate of SE ```{r} #| include: false #| echo: false se_p_boot <- one_poll_boot |> summarize(se_boot = sd(stat)) |> pull() ``` - We can estimate the standard error using the standard deviation of the bootstrap distribution - Using the bootstrap distribution we get a standard error estimate of $SE_{boot}=$ `r round(se_p_boot,3)` - Compare this with the standard error we measured earlier using the actual sampling distribution $SE=$ `r round(se_p,3)` ## Bootstrap 95% Confidence Interval - The **95 % bootstrap percentile confidence interval** for a parameter $p$ is obtained by calculating the 2.5% and 97.5% percentiles for the bootstrapped statistics. ---- ```{r} #| include: true #| echo: false #| fig-cap: "Distribution of bootstrapped proportion with 95% boostrap CI." # one_poll_boot |> # ggplot(aes(x = stat)) + # geom_dotplot(dotsize = 0.1) + # scale_y_continuous(NULL, breaks = NULL) + # labs(x = "bootstrapped proportion of yes votes") + # theme_minimal() + # theme(text = element_text(size = 18)) singleProp_bootPlot(one_poll_boot, 21/30, dotHist = "dotplot", confLevel = 95, ciType = "bootperc", plotCI = TRUE) ``` ---- ```{r} #| include: false #| echo: false ci_tib <- one_poll_boot |> summarize(lower = quantile(stat, 0.025), upper = quantile(stat, 0.975)) cil <- ci_tib$lower ciu <- ci_tib$upper ``` - These values are `r round(cil,3)` and `r round(ciu,3)` for the bootstrap distribution of proportions of yes votes for Candidate X - We say we are 95% confident that the value of the true proportion of yes votes is between `r round(cil,3)` and `r round(ciu,3)` - Does the interval include the true value of $p=0.6$? ------------------------------------------------------------------------ - The confidence interval reflects what we think are **plausible** values for the parameter - For example, it is plausible that 80% of people plan to vote for Candidate X, because 0.8 is between `r round(cil,3)` and `r round(ciu,3)` - It is not plausible that 50% (or less) of people plan to vote for Candidate X, because 0.5 is not between `r round(cil,3)` and `r round(ciu,3)` - This would be good news for Candidate X. ## Other Confidence Levels ```{r} #| include: false #| echo: false ci_tib90 <- one_poll_boot |> summarize(lower = quantile(stat, 0.05), upper = quantile(stat, 0.95)) cil90 <- ci_tib90$lower ciu90 <- ci_tib90$upper ``` - A 90% CI is between 5% and 95% percentiles of the bootstrap distribution (the middle 90%) --- ```{r} #| include: true #| echo: false #| fig-cap: "Distribution of bootstrapped proportion with 90% boostrap CI." # one_poll_boot |> # ggplot(aes(x = stat)) + # geom_dotplot(dotsize = 0.1) + # scale_y_continuous(NULL, breaks = NULL) + # labs(x = "bootstrapped proportion of yes votes") + # theme_minimal() + # theme(text = element_text(size = 18)) singleProp_bootPlot(one_poll_boot, 21/30, dotHist = "dotplot", confLevel = 90, ciType = "bootperc", plotCI = TRUE) ``` ---- - The 90% boostrap percentile confidence interval for the proportion of yes votes for Candidate X is between `r round(cil90,3)` and `r round(ciu90,3)` - 90% CI is narrower than 95% CI - 95% CI: (`r round(cil,3)`, `r round(ciu,3)`) - 90% CI: (`r round(cil90,3)`, `r round(ciu90,3)`) - With a higher confidence level (95%), the interval needs to be wider for us to be more confident that it contains the true value of the parameter ## Properties of Confidence Intervals - The confidence interval will contain the observed value of the statistic (usually near the center of the interval) - Larger sample sizes result in narrower confidence intervals (we are more confident that the parameter is close to the point estimate if the point estimate comes from a larger sample) - If we were to repeatedly sample from the population and calculate a 95% confidence interval from each sample, about 95% of the confidence intervals would contain the true value of the parameter