Testing claims about data with statistical rigor
Hypothesis testing is HOW we make statistical decisions. CORE CONCEPT: Null hypothesis (H₀): Boring, no effect, status quo Alternative hypothesis (H₁): Interesting, something changed PROCESS:
CRITICAL INSIGHT: We never prove H₀ is true, we just fail to reject it THIS WEEK YOU'LL LEARN: ✓ H₀ vs H₁ hypotheses ✓ Type I and Type II errors ✓ P-values and significance ✓ One-sample, two-sample tests ✓ T-tests (parametric) ✓ ANOVA (3+ groups) ✓ Chi-square (categorical) ✓ Non-parametric alternatives ✓ Effect sizes (not just p-values) ✓ Power analysis (sample size)
HYPOTHESIS TESTING LOGIC:
# EXAMPLE: Is new teaching method better? # H₀: New method NOT better (mean gain = 0) # H₁: New method IS better (mean gain ≠ 0) # Collect data control_group <- c(5, 3, 2, 4, 3) treatment_group <- c(8, 7, 9, 6, 8) # Test t.test(treatment_group, control_group) # Output: # t = 2.598, df = 8, p-value = 0.03217 # mean of x = 7.6, mean of y = 3.4 # Interpretation: # IF H₀ were true (no difference), probability of seeing # this much difference (or more) by chance = 3.2% # This is unusual! Reject H₀ # Conclusion: New method appears better
STEPS IN HYPOTHESIS TEST:
# H₀: μ = 100 (population mean equals 100) # H₁: μ ≠ 100 (population mean differs from 100)
α = 0.05 # Standard (5% chance of false positive) # Less common: α = 0.01 (more stringent), α = 0.10 (more lenient)
t_statistic <- (sample_mean - hypothesized_mean) / (se)
# P-value = probability of data this extreme IF H₀ true p_value <- 2 * pt(-abs(t_statistic), df=n-1) # Two-tailed
if (p_value < α) { "Reject H₀" # Significant result } else { "Fail to reject H₀" # Not significant }
# Connect back to research question
"We have sufficient evidence that..."TYPES OF ERRORS:
# Type I Error (False Positive) # Reject H₀ when it's actually true # Probability = α (significance level) # "Crying wolf" # Type II Error (False Negative) # Fail to reject H₀ when it's actually false # Probability = β # "Missing real effect" # Trade-off: Decrease α → increase β
P-VALUES - MOST MISUNDERSTOOD CONCEPT:
WRONG interpretations: ✗ "P-value is probability H₀ is true" ✗ "P-value = probability result is due to chance" ✗ "P < 0.05 means we found important effect" ✗ "P > 0.05 means H₀ is true"
CORRECT:
# EXAMPLE: Testing fairness of coin # H₀: p = 0.5 (fair coin) # H₁: p ≠ 0.5 (unfair coin) # Flip 100 times, get 65 heads # p-value = P(≥65 heads out of 100 | fair coin) # p-value ≈ 0.002 (very unlikely if fair) # Reject H₀: Coin appears unfair # NOT: "2% chance coin is unfair" (wrong!) # NOT: "98% confidence coin is unfair" (wrong!) # CORRECT: "Prob of 65+ heads if fair = 0.002" (right!)
EFFECT SIZE - WHY IT MATTERS:
# Large p-value doesn't mean no effect, just insufficient evidence # Small p-value doesn't mean large effect # EXAMPLE: # Study 1: Small sample, huge difference, p = 0.05 # Study 2: Huge sample, tiny difference, p = 0.001 # Which one matters more? Depends on effect size! # Cohen's d (standardized difference) # d = (mean1 - mean2) / pooled_sd # |d| < 0.2 = small effect # |d| = 0.5 = medium effect # |d| > 0.8 = large effect library(effsize) cohen.d(treatment_group, control_group) # Cohen's d: 1.65 (large effect!)
ONE-SAMPLE T-TEST:
# Test if sample mean differs from known value data <- c(98, 102, 100, 99, 101, 98, 103, 101) # H₀: μ = 100 # H₁: μ ≠ 100 t.test(data, mu = 100) # Output: # t = 0.9428, df = 7, p-value = 0.3822 # mean of x = 100.375 # Interpretation: p = 0.38 > 0.05 # Fail to reject H₀ # Data consistent with μ = 100 # 95% CI for mean: t.test(data)$conf.int # [1] 99.08 101.67
TWO-SAMPLE T-TEST:
# Compare two independent groups group1 <- c(85, 90, 88, 92, 87) group2 <- c(78, 80, 82, 79, 81) # H₀: μ₁ = μ₂ (groups equal) # H₁: μ₁ ≠ μ₂ (groups differ) t.test(group1, group2) # Output: # t = 3.7747, df = 8, p-value = 0.005 # mean of x = 88.4, mean of y = 80 # Interpretation: p = 0.005 < 0.05 # Reject H₀ # Groups significantly different # Check assumptions # 1. Normality shapiro.test(group1) # p > 0.05 → normal shapiro.test(group2) # 2. Equal variances var.test(group1, group2) # p > 0.05 → equal variances # If unequal variances: Welch's t-test t.test(group1, group2, var.equal = FALSE)
PAIRED T-TEST:
# Same subjects, before/after before <- c(85, 78, 90, 82, 88) after <- c(92, 85, 95, 88, 94) # H₀: No difference (μ_diff = 0) # H₁: Difference exists (μ_diff ≠ 0) t.test(before, after, paired = TRUE) # Output: # t = -5.5678, df = 4, p-value = 0.0073 # mean difference = -7 # Interpretation: Significant improvement