--- title: "Decision Errors" subtitle: | | IMS2 Ch. 14 | Math 115 author: "Yurk" format: revealjs: theme: beige df-print: paged embed-resources: true editor: source metadata: license: CC BY-SA --- ## CPR Study ```{r} #| include: false #| echo: false library(tidyverse) library(infer) library(openintro) ``` - **Research question:** Do blood thinners affect survival rate in heart attack patients that have received CPR? - `cpr` data set is available [here](https://www.openintro.org/data/csv/cpr.csv) - 2 variables - *group*: treatment (received blood thinner) or control (did not) - *outcome*: died or survived (for at least 24 hours) - 90 patients (40 treatment, 50 control, randomly assigned) ## Hypotheses - Blood thinners can be administered to treat a clot that is causing a heart attack - CPR can cause internal injuries - Blood thinners can make it more difficult for these injuries to heal - Do blood thinners affect survival in a positive or negative way? - Alternative hypothesis reflects the fact that we don't have expectations about the direction of the relationship ------------------------------------------------------------------------ Two-sided hypothesis test - $H_0$: Blood thinners do not affect survival rate. $p_T-p_C = 0$ - $H_A$: Blood thinners affect survival rate. $p_T-p_C \neq 0$ Example of one-sided hypothesis test - $H_0$: Blood thinners do not affect survival rate. $p_T-p_C = 0$ - $H_A$: Blood thinners increase survival rate. $p_T-p_C > 0$ ## Results (EDA) ::: panel-tabset ### Counts ```{r} #| include: false #| echo: false cpr |> count(group, outcome) |> pivot_wider(names_from = outcome, values_from = n) |> mutate(total = died + survived) ``` | group | died | survived | total | |-----------|:----:|:--------:|------:| | control | 39 | 11 | 50 | | treatment | 26 | 14 | 40 | | total | 65 | 25 | 90 | ### Bar Plot ```{r} #| include: true #| echo: false #| fig-cap: "Standardized barplot showing proportions of patients that survived and died" cpr |> ggplot(aes(x = group, fill = outcome)) + geom_bar(position = "fill") + labs(y = "proportion") + theme_minimal() + theme(text = element_text(size = 18)) ``` ::: ## Difference in proportions ```{r} #| include: false #| echo: false cpr |> group_by(group) |> summarize(prop = mean(outcome == "survived")) |> summarize(dif = diff(prop)) #treatment - control ``` - Success: outcome = "survived" - Statistic of interest: difference in proportions $$\hat{p}_T-\hat{p}_C$$ - Observed difference: $$\frac{14}{40}-\frac{11}{50}=0.13$$ ## Z Score - SE = 0.0955 (more on estimating this in Ch. 17) - Thus, $$Z = \frac{0.13-0}{0.0955} = 1.36$$ ## Hypothesis test ```{r} #| include: false #| echo: false cpr |> group_by(group) |> summarize(n = n(), prop = mean(outcome == "survived")) |> mutate(varpi = prop*(1-prop)/n) |> summarize(se = sqrt(sum(varpi))) ``` - Technical conditions met for using a normal approximation for the null distribution - Since we calculated $Z$, we can use the standard normal distribution $N(0,1)$ - We will use significance level $\alpha=0.05$ ## p-value for 2-sided test - As with a 1-sided test, the p-value is the probability of observing a value of the statistic that is at least as extreme as the observed value, assuming the null hypothesis is true - With a 2-sided test, extreme values are in both tails of the null distribution --- - To calculate the p-value for a 2-sided hypothesis test we start by calculating two areas - The area to the left of the observed statistic - The area to the right of the observed statistic - The p-value is twice the smaller area ------------------------------------------------------------------------ ```{r} #| include: true #| echo: false #| fig-cap: "Density function standard normal distribution with left tail shaded below Z=1.36." #normTail(m = 0, s = 0.0955, L = -0.13, U = 0.13) obs_stat <- 1.36 m <- 0 s <- 1 xl <- 3.5 * s if(abs(obs_stat) > xl) xl <- obsStat xl <- xl + xl/2 ggplot(data.frame(x = c(-xl, xl)), aes(x)) + stat_function(fun = dnorm, args = list(mean = m, sd = s)) + stat_function(fun = dnorm, args = list(mean = m, sd = s), xlim = c(-xl, obs_stat), geom = "area", fill = "red", alpha = 0.5) + geom_hline(yintercept = 0, color = "black") + #geom_text(data = tibble(x = 0, y = 2, label = "Area = 0.895", size = 18), aes(x, y, label = label)) + annotate("text", label = "Area = 0.9131", x = 0, y = 0.1, size = 5) + theme_classic() + scale_y_continuous(NULL, breaks = NULL) + xlab("") + theme(text = element_text(size = 18)) ``` --- ```{r} #| include: true #| echo: false #| fig-cap: "Density function standard normal distribution with right tail shaded above Z=1.36." #normTail(m = 0, s = 0.0955, L = -0.13, U = 0.13) obs_stat <- 1.36 m <- 0 s <- 1 xl <- 3.5 * s if(abs(obs_stat) > xl) xl <- obsStat xl <- xl + xl/2 ggplot(data.frame(x = c(-xl, xl)), aes(x)) + stat_function(fun = dnorm, args = list(mean = m, sd = s)) + stat_function(fun = dnorm, args = list(mean = m, sd = s), xlim = c(obs_stat, xl), geom = "area", fill = "red", alpha = 0.5) + geom_hline(yintercept = 0, color = "black") + #geom_text(data = tibble(x = 0, y = 2, label = "Area = 0.895", size = 18), aes(x, y, label = label)) + annotate("text", label = "Area = 0.0869", x = 2.4, y = 0.025, size = 5) + theme_classic() + scale_y_continuous(NULL, breaks = NULL) + xlab("") + theme(text = element_text(size = 18)) ``` --- - p-value is $2\times0.0869 = 0.1738$ - Since the p-value is greater than 0.05, we do not reject the null hypothesis - It is plausible that there is no difference in survival rates between the two groups - Note: The p-value is twice as large for a two-sided test than for a one-sided test ---- - With a symmetric null distribution (like the normal distribution) that is centered at 0, the two tails are symmetric - Thus, for example, the p-value is also equal to the sum of the areas in the two tails that are to the left of -1.36 to the right of 1.36 ---- ```{r} #| include: true #| echo: false #| fig-cap: "Density function standard normal distribution tails shaded below Z=-1.36 and above Z=1.36." #normTail(m = 0, s = 0.0955, L = -0.13, U = 0.13) obs_stat <- 1.36 m <- 0 s <- 1 xl <- 3.5 * s if(abs(obs_stat) > xl) xl <- obsStat xl <- xl + xl/2 ggplot(data.frame(x = c(-xl, xl)), aes(x)) + stat_function(fun = dnorm, args = list(mean = m, sd = s)) + stat_function(fun = dnorm, args = list(mean = m, sd = s), xlim = c(obs_stat, xl), geom = "area", fill = "red", alpha = 0.5) + geom_hline(yintercept = 0, color = "black") + stat_function(fun = dnorm, args = list(mean = m, sd = s), xlim = c(-xl,-obs_stat), geom = "area", fill = "red", alpha = 0.5) + geom_hline(yintercept = 0, color = "black") + #geom_text(data = tibble(x = 0, y = 2, label = "Area = 0.895", size = 18), aes(x, y, label = label)) + annotate("text", label = "Area = 0.0869", x = 2.4, y = 0.025, size = 5) + annotate("text", label = "Area = 0.0869", x = -2.4, y = 0.025, size = 5) + theme_classic() + scale_y_continuous(NULL, breaks = NULL) + xlab("") + theme(text = element_text(size = 18)) ``` ## Decision Errors - There are two types of errors we can make in a hypothesis test - **Type 1 Error** occurs if we conclude that the null hypothesis is false when it is not (false alarm) - **Type 2 Error** occurs if we fail to reject the null hypothesis even though it is false (missed opportunity) ## Controlling Type 1 Errors - Type 1 Error is typically considered more severe - The probability of making a type 1 error (assuming the null is true) is equal to the significance level - Can reduce $\alpha$ to make type 1 errors less likely ## Controlling Type 2 Errors - There is a trade-off between the two types of errors - Decreasing probability of type 1, increases the probability of type 2 - **Power** is the probability of rejecting the null hypothesis if the alternative is true - Higher power reduces the chance of making a type 2 error - Power is related to effect size (easier to detect larger effects), sample size (larger sample results in more power), among other things ## Relationship between CI and Hypothesis tests - A significance level of $\alpha$ for a 2-sided hypothesis test corresponds to a confidence level of $1-\alpha$ - E.g., $\alpha = 0.05$ corresponds to a 95% confidence interval - If we would reject the null hypothesis at $\alpha=0.05$, then the 95% confidence interval would not contain the null value - Conversely, if we calculate the 95% confidence interval first and find that it does not contain the null value, then we would reject the null hypothesis at $\alpha=0.05$ with a two-sided test --- - In the CPR study, we did not reject the null hypothesis at $\alpha=0.05$ (p-value = 0.1738) - Even though we did not calculate a confidence interval for the difference in survival proportions, we know that the 95% confidence interval would contain 0 (the value under the null hypothesis)