Inference with Mathematical Models

Chapter 13
Math 115

Opportunity Costs

Research question: Does reminding someone of opportunity cost impact purchase decision?
opportunity_cost data set is available here
2 variables
- group: treatment or control
- decision: buy or not buy
150 students (75 assigned to treatment, 75 control)

All students given the following statement:

“Imagine that you have been saving some extra money on the side to make some purchases, and on your most recent visit to the video store you come across a special sale on a new video. This video is one with your favorite actor or actress, and your favorite type of movie (such as a comedy, drama, thriller, etc.). This particular video that you are considering is one you have been thinking about buying for a long time. It is available for a special sale price of $14.99. What would you do in this situation? Please circle one of the options below.”

Control and treatment group given different options.

Control

“(A) Buy this entertaining video. (B) Not buy this entertaining video.”

Treatment

“(A) Buy this entertaining video. (B) Not buy this entertaining video. Keep the $14.99 for other purchases.”

group	buy video	not buy video	total
treatment	41	34	75
control	56	19	75
total	97	53	150

Standardized barplot showing proportions of students that buy or do not buy

Difference in proportions

Success: decision = “not buy video”
Statistic of interest: difference in proportions \[\hat{p}_T-\hat{p}_C\]
Observed difference: \[\frac{34}{75}-\frac{19}{75}=0.2\]

Hypotheses

In words:

    $H_0:$ How the options are
    presented has no impact
    on decision.

    $H_A:$ Higher proportion
    will choose not to buy if
    opportunity cost
    highlighted.

In symbols:

$H_0: p_T -p_C = 0$
$H_A: p_T -p_C > 0$

Null Distribution - Simulation

We want to simulate the assumption of the null hypothesis: \[H_0: p_T -p_C = 0\]
- It assumes that highlighting the opportunity cost (treatment) had no effect on the decision
- Under this assumption we can randomly shuffle 97 “buy video” and 53 “not buy video” between control and treatment groups
- We also will need to keep the groups’ sizes (both are 75 in our example)
- After each simulation we calculate the difference of sample proportions of “buy video” decisions

Simulate 1,000 samples assuming true null hypothesis
Random permutation of response (decision)
Calculate statistic for each permutation
Plot these statistics on a dotplot

Histogram of 1,000 differences in randomized proportions, test statistic (0.2) is the dashed vertical line.

p-value - Simulation

The p-value is the probability that the value of the statistic would be at least as extreme as the observed value if the null hypothesis is true
Out of 1,000 simulations of a true null hypotheses, 6 had differences $\hat{p}_T-\hat{p}_C \geq 0.2.$
Thus, the p-value is $\frac{6}{1000} = 0.006$

Standard error

We quantify the variability of a sampling distribution with so called the standard error (SE).
The standard error is equal to the standard deviation associated with the statistic.
Usually, The standard error is itself an estimate, calculated from the sample of data.
Here we use the null distribution to compute a standard error.
$SE = 0.0791$

Null Distribution – Mathematical Model

It may be possible to use a mathematical model of the null distribution instead of simulation
For a single proportion or difference in proportions, we may be able to use a normal distribution

Central Limit Theorem for proportions

The sample proportion (or difference in proportions) will follow a bell-shaped curve called the normal distribution if the following technical conditions are met:

Independent observations
Large enough sample: for proportions, at least 10 successes and 10 failures in each group

Normal Distributions

$N(\mu, \sigma)$ denotes a normal distribution with mean $\mu$ and standard deviation $\sigma$
Normal distributions are bell-shaped and symmetric
The center of the distribution is determined by mean $\mu$
The width is determined by the standard deviation $\sigma$

Two examples of normal distributions with different means and standard deviations

Normal Approximation of Null Distribution

The data in the opportunity cost example satisfy the techincal conditions
We expect the distribution to be centered at the null hypothesis value and we calculated the standard deviation $0.0791$
We can approximate the null distribution using $N(\mu = 0, \sigma = 0.0791)$

How good is the approximation?

Null distribution from randomly permuted data and normal approximation

We can find the p-value by calculating the area under the curve in the same region ($\ge 0.2$)

For a probability density function (pdf), the area under the curve (integral) is a probability
The total area under the curve is exactly 1

Shaded area is probability that value is less than 0.1

The probability that the value is “less than 0.1” is 0.895

The probability that the value is “at least 0.1” is $1 - 0.895 = 0.105$

Computing a p-value from a Normal Distribution

We can compute a p-value using the normal approximation of the null distribution
Find the area in the tail is beyond the observed value of the statistic
For the opportunity cost example, the observed difference in proportions is 0.2
So the p-value is the area of the region “at least as large as 0.2”

The p-value is 0.00572

Conclusion

So p-value from simulations ($0.006$) or from the mathematical model using normal distribution ($0.00572$) are very close to each other because the validity conditions are satisfied (at least 10 successes and 10 failures in each group)
Either p-value is smaller than the significance level $\alpha=0.05$
Based on these p-values we reject null hypothesis and conclude that highlighting the opportunity cost resulted in significantly higher probability of not buying video than in the control group

Z score

The Z score of an observation is the number of standard deviations that the observation falls above or below the mean. \[Z = \frac{x-\mu}{\sigma}\]
The value $\mu$ is the value of the mean of the null hypothesis distribution (in this case it is 0)

We can standardize the observed difference in proportions in the opportunity costs problem \[Z = \frac{0.2 - 0}{0.0791} = 2.528\]

If the $X$ is distributed according to $N(\mu,\sigma)$, then $Z$ will be distributed according to $N(0,1)$
$N(0,1)$ is called the standard normal distribution

Standard normal distribution

We can use the Z score to calculate the p-value using the standard normal distribution
Using the standard normal distribution, we get exactly the same p-value (0.00572)

This calculation can be performed using a custom Jamovi module that is available on our Moodle page
In order to access the Module click on “Randomize” tab and open “Model Based Inference”
Let’s try it!

Comparing z-scores

Example: SAT scores follow a nearly normal distribution with a mean of 1500 points and a standard deviation of 300 points. ACT scores also follow a nearly normal distribution with mean of 21 points and a standard deviation of 5 points. Suppose Nel scored 1800 points on their SAT and Sian scored 24 points on their ACT. Who performed better?

Nel’s z-score is \[Z_{Nel} = \frac{1800-1500}{300}=1\] Sian’s z-score is \[Z_{Sian} = \frac{24-21}{5}=0.6\]

So Nel performed better

68-95-99.7 rule

About 68% of normally distributed data fall within 1 SD of the mean
About 95% fall within 2 SD (1.96 to be more precise)
About 99.7% fall within 3 SD

This rule can be used to bound p-values
If Z = 2.528, we know the value is outside of the middle 95% of the distribution

Standard normal 95% of the distribution in the middle

Thus it is some where in the right tail that includes 2.5% of the data
This means we know the p-value is less than 0.025, just based on the Z score

Using a Normal Distribution to Construct a 95% Confidence Interval

If sampling distribution is reasonably normal then we can construct a 95% confidence interval from a point estimate using the 68-95-99.7 rule
95% of the values will fall within 1.96 SD of the true value
95% confidence interval: \[\textrm{point estimate}\pm1.96\times SE\]

0.2 is a point estimate of the difference in proportions of students who would not buy a video ($p_T-p_C$)
SE = 0.0791
A 95% confidence interval for the difference in proportions is \[0.2 \pm 1.96\times0.0791 = 0.2 \pm 0.155\]
The quantity 0.155 is called the margin of error

Interpretation of the Confidence Interval

We can also write the 95% confidence interval as \[(0.045, 0.355)\]
We are 95% confident that in the population the probability of not buying video when the opportunity cost is highlighted is from 4.5% to 35.5% higher that not buying video when the opportunity cost is not highlighted.
Note that the value of the null hypothesis (i.e $0$) is not in the confidence interval which is consistent with the fact that we rejected the null hypothesis

Other Confidence Levels

For other confidence levels, the confidence interval can be computed as \[\textrm{point estimate}\pm z^{\ast}\times SE\]
To determine $z^{\ast}$, need to know how many standard deviations needed to contain X% of the data, where X% corresponds to the confidence level

Critical Values

Confidence Level	Critical Value ($z^{\ast}$)
90%	1.645
95%	1.96
99%	2.576

For a 99% CI, we want to find the cuttoff that includes 99.5% of the values, leaving 0.5% in the right tail and 0.5% in the left tail
Thus, a 99% CI for the difference in proportions is \[0.2\pm 2.576 \times 0.0791=0.2 \pm 0.203\]
Note that the margin of error is larger with a higher confidence level

Meaning of the level of confidence

Figure below shows twenty five 95% confidence intervals for a proportion that were constructed from 25 different datasets that all came from the same population where the true proportion was p=0.3.
However, 1 of these 25 confidence intervals (or about 5% of the intervals) happened not to include the true value

From IMS2 Figure 13.11.