Inference with Mathematical Models

Opportunity Costs

• Research question: Does reminding someone of opportunity cost impact purchase decision?
• opportunity_cost data set is available here
• 2 variables
  • group: treatment or control
  • decision: buy or not buy
• 150 students (75 assigned to treatment, 75 control)

All students given the following statement:

“Imagine that you have been saving some extra money on the side to make some purchases, and on your most recent visit to the video store you come across a special sale on a new video. This video is one with your favorite actor or actress, and your favorite type of movie (such as a comedy, drama, thriller, etc.). This particular video that you are considering is one you have been thinking about buying for a long time. It is available for a special sale price of $14.99. What would you do in this situation? Please circle one of the options below.”

Control and treatment group given different options.

• Control

“(A) Buy this entertaining video. (B) Not buy this entertaining video.”

• Treatment

“(A) Buy this entertaining video. (B) Not buy this entertaining video. Keep the $14.99 for other purchases.”

Results (EDA)

Difference in proportions

• Success: decision = “not buy video”
• Statistic of interest: difference in proportions \[\hat{p}_T-\hat{p}_C\]
• Observed difference: \[\frac{34}{75}-\frac{19}{75}=0.2\]

Hypotheses

Null Distribution - Simulation

• Simulate 1,000 samples assuming true null hypothesis
• Random permutation of response (decision)
• Calculate statistic for each permutation

Histogram of 1,000 differences in randomized proportions (null distribution), showing observed difference as dashed vertical line.

p-value - Simulation

• The p-value is the probability that the value of the statistic would be at least as extreme as the observed value if the null hypothesis is true
• Out of 1,000 simulations of a true null hypotheses, 5 had differences \(\hat{p}_T-\hat{p}_C \geq 0.2.\)
• Thus, the p-value is \(\frac{5}{1000} = 0.005\)

Standard error - Simulation

• We can also use the null distribution to compute a standard error (standard deviation of a sampling distribution)
• The standard error for the differences in proportions is \(SE = 0.0797\)

Null Distribution – Mathematical Model

• It may be possible to use a mathematical model of the null distribution instead of simulation
• For a single proportion or difference in proportions, we may be able to use a normal distribution

Normal Distributions

\(N(\mu, \sigma)\) denotes a normal distribution with mean \(\mu\) and standard deviation \(\sigma\)

Two examples of normal distributions with different means and standard deviations

Normal Approximation of Null Distribution

• The data in the opportunity cost example satisfy the techincal conditions
• We can approximate the null distribution using \(N(\mu = 0, \sigma = 0.0797)\)
• Why would we use these values?

How good is the approximation?

Null distribution from randomly permuted data and normal approximation

Computing Probabilities Using a Normal Distribution

For a probability density function, the area under the curve (integral) is a probability

Shaded area is probability that value is less than 0.1

The probability that the value is less than 0.1 is 0.895

The probability that the value is at least 0.1 is 0.105

Computing a p-value from a Normal Distribution

• We can compute a p-value using the normal approximation of the null distribution
• Find the area in the tail is beyond the observed value of the statistic
• For the opportunity cost example, the observed difference in proportions is 0.2

The p-value is 0.006.

Z score

The Z score of an observation is the number of standard deviations that the observation falls above or below the mean. \[Z = \frac{x-\mu}{\sigma}\]

We can standardize the observed difference in proportions in the opportunity costs problem \[Z = \frac{0.2 - 0}{0.0797} = 2.509\]

• If the \(X\) is distributed according to \(N(\mu,\sigma)\), then \(Z\) will be distributed according to \(N(0,1)\)
• \(N(0,1)\) is called the standard normal distribution

Standard normal distribution

• We can use the Z score to calculate the p-value using the standard normal distribution
• Using the standard normal distribution, we get exactly the same p-value (0.006)

• This calculation can be performed using a custom Jamovi module that is available on our Moodle page
• Let’s try it!

68-95-99.7 rule

From IMS2 Figure 13.8.

• This rule can be used to bound p-values
• Since Z = 2.509, we know the value is outside of the middle 95% of the distribution
• Thus it is some where in the right tail that includes 2.5% of the data
• This means we know the p-value is less than 0.025, just based on the Z score

Using a Normal Distribution to Construct a 95% Confidence Interval

• If sampling distribution is reasonably normal then we can construct a 95% confidence interval from a point estimate using the 68-95-99.7 rule

• 95% of the values will fall within 1.96 SD of the true value

• 95% confidence interval: \[\textrm{point estimate}\pm1.96\times SE\]

• 0.2 is a point estimate of the difference in proportions of students who would not buy a video (\(p_T-p_C\))

• SE = 0.0797

• A 95% confidence interval for the difference in proportions is \[0.2 \pm 1.96\times0.0797 = 0.2 \pm 0.156\]

• The quantity 0.156 is called the margin of error

• We can also write the 95% confidence interval as \((0.0438, 0.356)\)

Other Confidence Levels

• For other confidence levels, the confidence interval can be computed as \[\textrm{point estimate}\pm z^{\ast}\times SE\]
• To determine \(z^{\ast}\), need to know how many standard deviations needed to contain X% of the data, where X% corresponds to the confidence level
• This critical value \(z{^\ast}\) is sometimes called a multiplier
• The Jamovi module can be used to find multipliers for different confidence levels

• For a 99% CI the multiplier is 2.58

• Thus, a 99% CI for the difference in proportions is \[0.2\pm 2.58 \times 0.0797=0.2 \pm 0.205\]

• Note that the margin of error is larger with a higher confidence level