Advanced Quantitative Methods

Doctoral students often learn statistics as a bag of disconnected tests: a t-test here, an ANOVA there, a correlation somewhere else, chosen from a flowchart. This course begins by dismantling that view. Nearly every technique you will use is a special case of one framework, the general linear model (GLM). Learning the GLM once, deeply, is far more powerful than memorizing a dozen tests, because it tells you what the tests share and where they differ.

One dataset runs through every lesson of this course, and it is worth memorizing now. The Brightwater tutoring study enrolled 240 fourth-graders across 12 elementary schools of the Brightwater district in an after-school reading program, recording for each student a fall reading score, a spring reading score, and the number of tutoring hours received per week. Scores run from 0 to 100, the spring scores average 70 points with a standard deviation of 12, and tutoring averages 3.0 hours per week with a standard deviation of 1.5. We fit a regression to these data in Lesson 2, decompose its variance in Lesson 3, and keep returning to the same 240 students through logistic regression, multilevel models, mediation, and power analysis, because seeing one study analyzed twelve ways teaches more than seeing twelve studies analyzed once.

Key idea: The tests you learned as separate tools are one model wearing different clothes, so learning the model gives you all of the tools at once.

In plain terms

Here is the whole idea in a paragraph. You have an outcome you care about, and you have several things you believe predict it, so the model tells you to weight each predictor, add the weighted pieces together, and treat that sum as your best guess. Whatever the guess misses is called error, and the model makes no attempt whatsoever to explain it.

Everything after that is bookkeeping, and the bookkeeping has an order. Lesson 2 explains how the weights get chosen, by a criterion called least squares. Lesson 3 explains how to judge whether the guesses were any good, using a quantity called R-squared. Standard errors and confidence intervals then tell you whether an individual weight is large enough to take seriously. The reason a t-test and an analysis of variance look like different procedures in software is that the output tables package identical arithmetic under different headings, while a single estimation engine runs underneath both.

The equation

The GLM expresses an outcome as a weighted sum of predictors plus error. For a single case i:

Y_i = β₀ + β₁X_1i + β₂X_2i + ... + β_kX_ki + ε_i

The right-hand side has two parts. The systematic (deterministic) component, everything except the error, is the model's prediction: the part of Y explained by the predictors. The random component, the error term ε_i, is everything the model does not capture for case i. Statistical modeling is the disciplined splitting of an outcome into a part we can explain and a residual we treat as noise.

• β₀ is the intercept: the model's predicted value of Y when every predictor equals zero.
• Each β_j is a slope or partial regression coefficient: the predicted change in Y for a one-unit increase in X_j, holding the other predictors fixed.
• ε_i is the residual error for case i, the vertical distance from the observation to the model's prediction.

Put Brightwater numbers into that skeleton to make it concrete. Suppose the fitted model is Spring = 58 + 4.0(tutoring hours), so that a student receiving 3 hours carries a predicted score of 58 + 4.0(3) = 70. If she actually scored 77, her residual is 77 minus 70, or +7 points, meaning the model underpredicted her performance by seven points. Every one of the 240 students carries such a residual, and the entire enterprise of regression consists of choosing coefficients that make those residuals small in a precisely defined sense.

Read the phrase "linear model" carefully, because it is narrower than people assume and also wider. It means linear in the parameters, not linear in the predictors. The model Y = b0 + b1X + b2X-squared + e is still a general linear model, because the coefficients enter additively even though the curve bends. What would break the framework is something like Y = b0 X-to-the-b1, where a coefficient sits in an exponent. So the GLM handles curves, thresholds, and categories with ease. It handles them by transforming the predictors, not by abandoning the equation.

Key idea: "Linear" describes how the coefficients enter the equation, not whether the picture is a straight line, which is why the GLM can fit curves.

Why the classic tests are all the GLM

What makes a predictor "count" is not whether it is a number or a category, because categories can be represented numerically (a topic we treat in Module 3). Once you accept that, the familiar tests collapse into one family:

Classic test	As a GLM
Pearson correlation	Regression of Y on one continuous X (standardized)
Independent-samples t-test	Regression of Y on one two-level group indicator
One-way ANOVA	Regression of Y on a set of group indicators
Multiple regression	Regression of Y on several predictors of any type
ANCOVA	Regression of Y on group indicators plus a continuous covariate

Every entry in that table estimates coefficients by the same criterion and tests them the same way. A t-test comparing two group means is numerically identical to a regression with one dummy predictor: the slope equals the difference in means, and the test of the slope is the t-test. This is not an analogy; it is an identity.

Key idea: A t-test and a one-predictor regression are the same computation, so the choice between them is a choice of vocabulary, not of method.

Worked example: the t-test IS a regression

A claim of numerical identity deserves arithmetic rather than assertion, so take ten Brightwater students, five who received no tutoring and five who did. The numbers are deliberately small enough that you can verify every step by hand.

• Control group (5 students): 60, 64, 66, 68, 72, which sum to 330 and therefore average 66.
• Tutored group (5 students): 70, 72, 76, 78, 84, which sum to 380 and therefore average 76.

First, run it as a t-test. The deviations from 66 in the control group are -6, -2, 0, +2, +6, and their squares sum to 36 + 4 + 0 + 4 + 36 = 80. The deviations from 76 in the tutored group are -6, -4, 0, +2, +8, and their squares sum to 36 + 16 + 0 + 4 + 64 = 120. The pooled variance is (80 + 120) / (5 + 5 - 2) = 200 / 8 = 25, so the pooled standard deviation is 5.

t = (76 - 66) / (5 × square root of (1/5 + 1/5)) = 10 / (5 × 0.6325) = 10 / 3.1623 = 3.16, with df = 8.

Now run the identical data as a regression by coding a dummy variable D = 0 for control and D = 1 for tutored. The mean of D is 0.5 and the mean of Y is 71, the sum of squared deviations of D is 10 × 0.25 = 2.5, and the cross-product sum works out to 25 because the control side contributes 12.5 and the tutored side another 12.5.

• Slope b₁ = 25 / 2.5 = 10, which is exactly the difference between the two group means.
• Intercept b₀ = 71 - 10(0.5) = 66, which is exactly the control-group mean.
• Residual sum of squares = 80 + 120 = 200, so the residual mean square becomes 200 / 8 = 25.
• Standard error of b₁ = square root of (25 / 2.5) = square root of 10 = 3.1623, giving t = 10 / 3.1623 = 3.16 on the same 8 degrees of freedom as before.

Push the demonstration one step further. The total sum of squares is 450 and the residual sum of squares is 200, so the model accounts for 250 of it, giving R-squared = 250 / 450 = 0.556. That figure can be recovered directly from the t statistic, because t-squared / (t-squared + df) = 10 / (10 + 8) = 0.556, and the overall F statistic for the model is simply t-squared = 10.0. Three procedures that carry different names in a methods textbook have produced one set of numbers.

One precision note is worth carrying forward. The regression reproduces the pooled-variance t-test, which assumes the two groups share a common error variance, whereas Welch's t-test, the version most statistical software now runs by default, relaxes that assumption and will therefore return a slightly different value. The identity described here holds for the classical equal-variances version of the test.

Key idea: Ten numbers produced t = 3.16 under two procedural names because the underlying procedures were never actually distinct.

The assumptions travel with the model

Because these tests are one model, they share one set of assumptions, which we develop across this course: the relationship is linear in the parameters, the errors are independent, have constant variance (homoscedasticity), and are approximately normal, and the predictors are measured without gross error. When you later read that a t-test "assumes normality," you are seeing one instance of the GLM's error assumption. Learn the assumptions once and you have learned them for the whole family.

Be exact about what the normality assumption covers, because this is the single most misremembered point in applied statistics. The assumption is about the errors, that is, about Y conditional on the predictors. It is not an assumption that Y itself is normally distributed, and it is certainly not an assumption that the predictors are normal. Brightwater spring scores could be strongly bimodal, with a cluster of struggling readers and a cluster of strong ones, and the residuals could still be beautifully normal once tutoring and fall score are in the model. Testing the raw outcome for normality before fitting a regression is a widespread habit and a misdirected one.

Notice also which assumptions carry the most weight, because they are not interchangeable in importance. Linearity protects the coefficient estimates themselves, while independence and constant variance protect the standard errors that surround them, and normality mostly governs small-sample inference. With the 240 students of the Brightwater study, the central limit theorem carries the sampling distributions of the coefficients toward normality even when the residuals are moderately skewed, whereas with a sample of twelve schools it provides no such protection.

Key idea: Normality is an assumption about residuals, not about the outcome and not about the predictors, and it is the assumption that matters least in a large sample.

Where people get stuck

• Thinking "linear" means "straight line." It means linear in the coefficients, so adding a squared term, a logarithm, or a spline basis leaves you inside the general linear model even though the fitted picture visibly bends.
• Reading a slope without its holding-constant clause. In a one-predictor model b₁ captures the entire association, whereas in a three-predictor model it captures only the association net of the other two, which means the number changes meaning whenever the model changes.
• Testing the raw outcome for normality. Because the assumption governs the errors rather than the outcome, the sequence has to be fit the model first and inspect the residuals afterward.
• Trusting the flowchart. Decision trees that route you from "two groups, continuous outcome" to "independent-samples t-test" are not incorrect, but they conceal the underlying unity, and a researcher who can write the equation never needs one.

Key idea: Most confusion in applied statistics comes from treating these outputs as separate species rather than as one animal photographed from several angles.

Throughout the course, when a new technique appears, ask the orienting question: how does this extend the GLM? Logistic regression changes the outcome's distribution and adds a link function; multilevel models relax the independence-of-errors assumption; factor analysis models the covariance among many indicators. Each is a principled modification of the same skeleton, and seeing the skeleton keeps you from getting lost.

Recap

• The GLM splits an outcome into a systematic part (predictors times coefficients) and a random part (the error term), and every technique in this course is a variation on that split.
• "Linear" refers to how coefficients enter the equation, so squared terms, logs, and dummy variables all stay inside the family.
• Correlation, the pooled t-test, one-way ANOVA, ANCOVA, and multiple regression are the same model with different predictor types.
• Ten Brightwater scores gave t = 3.16 from a t-test and t = 3.16 from a regression slope, with F = t-squared = 10.0 and R-squared = 0.556.
• The assumptions are shared: linearity in the parameters, independent errors, constant error variance, approximately normal errors, and predictors measured without gross error.
• Normality applies to residuals, not to the outcome or the predictors, and it matters most when the sample is small.
• Later techniques are principled edits to this skeleton: a link function for logistic regression, random effects for clustering, a measurement model for latent constructs.

Sources

1. NIST/SEMATECH. (2012). What terminology do statisticians use to describe process models? In e-Handbook of statistical methods. itl.nist.gov
2. Diez, D. M., Cetinkaya-Rundel, M., & Barr, C. D. (2019). OpenIntro statistics (4th ed.). OpenIntro. openintro.org
3. Penn State Eberly College of Science. (n.d.). STAT 501: Regression methods. online.stat.psu.edu
4. Navarro, D. (2019). Learning statistics with R. learningstatisticswithr.com
5. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning (2nd ed.). Springer. statlearning.com
6. Gelman, A., Hill, J., & Vehtari, A. (2020). Regression and other stories. Cambridge University Press. avehtari.github.io
7. Bhattacherjee, A. (2012). Social science research: Principles, methods, and practices (2nd ed.). University of South Florida. digitalcommons.usf.edu

The GLM leaves one question open: given data, how do we choose the coefficients? The dominant answer is ordinary least squares (OLS), and understanding its logic clarifies both what regression delivers and where it can mislead.

The question is sharper than it sounds, because infinitely many lines can be drawn through a cloud of points and every one of them will fit some observations well and others badly. What OLS supplies is a single, unambiguous rule for preferring one line over all the rest, together with a closed-form recipe for computing it. Once you understand the rule, you also understand precisely which kinds of data will break it, which is the more valuable half of the lesson.

Key idea: Regression does not discover the line hiding in the data; it applies a stated criterion for what counts as the best line, and the criterion has consequences.

In plain terms

Picture the scatterplot of Brightwater students, with tutoring hours running across the bottom and spring reading score climbing the side. You want to draw one straight line through the cloud, but which one? The procedure is to measure how far each dot sits above or below whichever line you are considering, square all of those distances, and add them together, because the line that makes this total as small as arithmetic permits is the least-squares line that every regression command returns.

Two consequences follow immediately from that recipe, and both are consequential enough to organize the remainder of the lesson. Because squaring means an observation that misses by 10 points contributes a hundred times what an observation missing by 1 point contributes, a single unusual case can drag the entire line perceptibly toward itself. And because nothing in the criterion inquires whether a straight line was ever a defensible shape for these particular data, statistical software will cheerfully report a slope for a relationship that curves, plateaus, or does not exist at all.

The least-squares criterion

For each case the residual is the observed value minus the predicted value, e_i = Y_i minus Y-hat_i. OLS chooses the coefficients that minimize the sum of squared residuals, the quantity Σe_i². Why square them? Two reasons. First, residuals above and below the line have opposite signs; squaring prevents them from canceling, so the criterion measures total misfit rather than net misfit. Second, squaring penalizes large misses disproportionately, which yields a unique, smooth solution obtainable in closed form. A consequence worth remembering is that OLS is sensitive to outliers, because one far-off point contributes its squared distance to the total.

The simple-regression formulas

For one predictor, the least-squares slope and intercept are:

b₁ = Σ(X_i - X̄)(Y_i - Ȳ) / Σ(X_i - X̄)²    and    b₀ = Ȳ - b₁X̄

The slope numerator is the covariance of X and Y (up to a constant); the denominator is the variance of X (up to the same constant). So the slope is covariance divided by the variance of the predictor. The intercept formula guarantees the line passes through the point of means (X̄, Ȳ), a fact you can use to sanity-check any fitted line.

Two algebraic consequences of these formulas are worth committing to memory, because they let you audit software output in your head. First, whenever a model contains an intercept the residuals sum to exactly zero, which is why the raw sum of residuals is useless as a measure of fit. Second, the residuals are uncorrelated with every predictor by construction, so a residual plot that shows a relationship with a predictor is telling you the model is misspecified rather than that the arithmetic went wrong.

Key idea: The slope is a covariance divided by a variance, the line always passes through the point of means, and the residuals always sum to zero when an intercept is present.

Worked example: fitting the Brightwater line by hand

Take six Brightwater students, recorded as weekly tutoring hours X paired with spring reading score Y: (1, 60), (2, 66), (3, 70), (3, 74), (4, 74), and (5, 76). The tutoring hours sum to 18, giving X̄ = 3, while the reading scores sum to 420, giving Ȳ = 70.

The deviations of X from its mean are -2, -1, 0, 0, +1, +2, whose squares total 4 + 1 + 0 + 0 + 1 + 4 = 10, and the corresponding deviations of Y from its mean are -10, -4, 0, +4, +4, +6, so that multiplying the paired deviations produces 20, 4, 0, 0, 4, and 12, which sum to 40. Substituting those two totals into the formulas gives:

b₁ = 40 / 10 = 4.0    and    b₀ = 70 - 4.0(3) = 58.

The fitted line is Spring = 58 + 4.0(hours), which is precisely the Brightwater equation quoted in Lesson 1. Because its predictions at X = 1, 2, 3, 3, 4, 5 are 62, 66, 70, 70, 74, and 78, the residuals come out as -2, 0, 0, +4, 0, and -2, which sum to zero exactly as the algebra promised, and squaring them yields 4 + 0 + 0 + 16 + 0 + 4 = 24 as the residual sum of squares.

Now examine the claim that no alternative line performs better. A steeper candidate through the same point of means, Spring = 55 + 5.0(hours), predicts 60, 65, 70, 70, 75, and 80, generating residuals of 0, +1, 0, +4, -1, -4 whose squares total 34, while a flatter candidate, Spring = 61 + 3.0(hours), generates squared residuals totalling 34 as well. Both alternatives are demonstrably worse than 24, and you could continue testing candidate lines indefinitely without ever beating the least-squares solution, because differential calculus guarantees that it is the unique minimum of a convex criterion.

The remainder of the regression table descends from these same six observations. Since the total sum of squares is 100 + 16 + 0 + 16 + 16 + 36 = 184, the model accounts for 184 - 24 = 160 of it, giving R-squared = 160 / 184 = 0.870, and with n - 2 = 4 residual degrees of freedom the residual mean square becomes 24 / 4 = 6, so the standard error of the slope is the square root of (6 / 10) = 0.775. Those figures produce t = 4.0 / 0.775 = 5.16 together with a 95 percent confidence interval of 4.0 plus or minus 2.776(0.775), running from 1.85 to 6.15 points per tutoring hour, which means six students purchase a confident sign and a badly imprecise magnitude.

Key idea: Every quantity in a regression output - slope, intercept, R-squared, standard error, t, and interval - descends from two sums you can compute with a pencil.

A worked example

Suppose we regress exam score (Y) on hours studied (X) for five students and, from the data, obtain Σ(X - X̄)(Y - Ȳ) = 40 and Σ(X - X̄)² = 10, with X̄ = 4 and Ȳ = 70. Then:

• b₁ = 40 / 10 = 4. Each additional hour of study predicts a 4-point higher exam score.
• b₀ = 70 minus (4)(4) = 70 minus 16 = 54. A student who studied zero hours is predicted to score 54.
• Prediction for 6 hours: Y-hat = 54 + 4(6) = 78.

Notice the interpretive caution the intercept demands: predicting Y at X = 0 is only meaningful if X = 0 is within, or near, the observed range. Predicting an exam score for negative study hours would be nonsense, and predicting far outside the data range is extrapolation, which the model cannot justify.

What OLS does and does not promise

OLS gives the line that minimizes squared vertical error in this sample. Under the GLM assumptions it also has attractive theoretical properties: the Gauss-Markov theorem shows OLS is the best linear unbiased estimator (BLUE), meaning that among all linear unbiased estimators it has the smallest variance. But "best linear unbiased" is conditional on the assumptions holding.

When errors are heteroscedastic or correlated, OLS remains unbiased but is no longer most efficient and its standard errors are wrong, a theme we return to in Module 3. For now, hold two ideas together: OLS is a precise, well-defined recipe, and its guarantees are only as good as the assumptions behind them.

Key idea: Gauss-Markov optimality is conditional, so quoting BLUE without checking homoscedasticity and independence is citing a warranty whose terms you have not read.

Worked example: how one bad value moves the line

Return to those six students and suppose that the final score of 76 was mistyped as 46 during data entry. The scores now sum to 390, so Ȳ falls to 65 and the deviations become -5, +1, +5, +9, +9, and -19, which when multiplied by the unchanged X deviations produce 10, -1, 0, 0, 9, and -38, summing to -20 and therefore giving:

b₁ = -20 / 10 = -2.0    and    b₀ = 65 - (-2.0)(3) = 71.

A single keystroke reversed the slope from +4.0 to -2.0, converting an apparently beneficial tutoring effect into an apparently harmful one. The reason is structural rather than accidental, because the mistyped observation sits at the extreme right of the predictor range and therefore possesses high leverage, which least squares compounds by weighting its squared miss so heavily. This single demonstration is the most persuasive argument in the entire course for plotting your data before believing any coefficient, and Lesson 6 develops the diagnostic machinery that would have caught it.

Anscombe's celebrated quartet establishes the same warning from the opposite direction, since four small datasets with visibly incompatible shapes - one genuinely linear, one curved, one linear apart from a single outlier, and one in which a lone extreme predictor value dictates everything - nevertheless share identical means, identical variances, an identical correlation, and an identical fitted regression line. Matching summary statistics therefore provide no assurance of comparable data, and the only inexpensive defence available to an analyst remains the scatterplot.

Key idea: Least squares is exquisitely sensitive to extreme points because it squares misses, so one recording error can invert a substantive conclusion.

Where people get stuck

• Interpreting the intercept when zero is impossible. The Brightwater intercept of 58 is the predicted score for a student with zero tutoring, which is a real category here, whereas an intercept for zero years of age or zero body mass is an arithmetic artifact that should be reported and then ignored.
• Extrapolating past the data. The fitted line says nothing about 20 hours of tutoring per week, because no student in the sample received anything close to that, and a straight line is the least defensible thing to trust far from the observed range.
• Confusing the residual with the error. The error is the unobservable deviation of a case from the true population line, while the residual is its observable estimate from the fitted sample line, and the whole logic of diagnostics rests on using the second to reason about the first.
• Treating a small residual sum of squares as evidence of a correct model. A curved relationship fitted with a straight line can still produce a modest residual total, so the shape of the residuals matters more than their size.
• Deleting inconvenient points. An extreme observation deserves investigation, documentation, and a sensitivity analysis reported both ways, not quiet removal.

Key idea: Most OLS disasters are not computational failures but interpretive ones - reading an intercept, an extrapolation, or an outlier as though the model had earned the right to speak.

Recap

• OLS selects the coefficients that minimize the sum of squared residuals, which is a stated criterion rather than a discovered truth.
• Squaring prevents positive and negative misses from cancelling and yields a unique closed-form solution, at the price of heavy sensitivity to extreme observations.
• The slope equals the cross-product sum divided by the predictor sum of squares, and the intercept forces the line through the point of means.
• Six Brightwater students gave b₁ = 40 / 10 = 4.0 and b₀ = 58, with residual sum of squares 24; competing lines with slopes of 3.0 and 5.0 both scored 34.
• From those same numbers, R-squared = 160 / 184 = 0.870, the slope standard error is 0.775, t = 5.16, and the 95 percent interval runs from 1.85 to 6.15.
• Mistyping one score changed the slope from +4.0 to -2.0, which is why plotting the data is a methodological requirement rather than a courtesy.
• Gauss-Markov guarantees OLS is the best linear unbiased estimator only while the error assumptions hold; under heteroscedasticity or dependence it stays unbiased but its standard errors mislead.

Sources

1. NIST/SEMATECH. (2012). e-Handbook of statistical methods. National Institute of Standards and Technology. itl.nist.gov
2. Penn State Eberly College of Science. (n.d.). STAT 462: Applied regression analysis. online.stat.psu.edu
3. Diez, D. M., Cetinkaya-Rundel, M., & Barr, C. D. (2019). OpenIntro statistics (4th ed.). OpenIntro. openintro.org
4. Anscombe, F. J. (1973). Graphs in statistical analysis. The American Statistician, 27(1), 17-21. find source ↗
5. Wikipedia contributors. (2024). Anscombe's quartet. Wikipedia. en.wikipedia.org
6. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning (2nd ed.). Springer. statlearning.com
7. Cetinkaya-Rundel, M., & Hardin, J. (2024). Introduction to modern statistics (2nd ed.). OpenIntro. openintro-ims.netlify.app

Once a line is fitted, we ask how well it fits. The answer rests on partitioning the outcome's variability into a part the model explains and a part it does not, an idea that recurs in ANOVA, multilevel models, and beyond.

This lesson also introduces the first genuine inference of the course, because partitioning variance leads directly to the F-test and therefore to the p-value. Since p-values are the most systematically misreported quantity in the empirical literature, the final section states carefully what one does and does not mean, following the American Statistical Association's formal statement on the subject.

Key idea: Fit is a question about how the outcome's variability divides, and inference is a question about how surprising that division would be if the predictors carried no information at all.

In plain terms

Before you fit anything at all, the Brightwater spring scores already vary, since some students scored 55 while others scored 85, and that existing spread is the raw material the model is given to work with. Its entire job is to account for as much of that spread as it can using the predictors you supplied.

So imagine splitting the spread into two bins. The first bin holds the variation your predictors successfully track, meaning the portion where higher tutoring hours really do line up with higher scores, while the second bin holds everything left over that the model cannot account for and therefore treats as noise. R-squared reports what fraction of the original spread landed in the first bin, and the F-test asks whether that fraction is larger than randomly chosen predictors would typically have delivered.

The three sums of squares

Consider the deviation of each observation from the mean of Y. That total deviation splits into two pieces: the part captured by the fitted line and the part left over.

• Total sum of squares (SS_total) = Σ(Y_i - Ȳ)². The total variability in the outcome around its mean.
• Regression (model) sum of squares (SS_reg) = Σ(Y-hat_i - Ȳ)². The variability the model explains, the spread of the predictions around the mean.
• Residual (error) sum of squares (SS_res) = Σ(Y_i - Y-hat_i)². The variability the model fails to explain.

The identity SS_total = SS_reg + SS_res holds exactly for OLS with an intercept. This clean decomposition is the arithmetic heart of the GLM.

Key idea: Total variability equals explained plus unexplained, exactly and always, provided the model contains an intercept and was fitted by least squares.

Worked example: partitioning Brightwater

Scale the arithmetic up from six students to the full sample of 240. Because the spring scores have a standard deviation of 12, their variance is 144, and the total sum of squares is therefore (n - 1) times the variance, or 239 × 144 = 34,416. That single number is the entire supply of variability the models below are competing to explain.

Fit the one-predictor model first, regressing spring score on weekly tutoring hours alone. It returns R-squared = 0.250, so the regression sum of squares is 0.250 × 34,416 = 8,604 and the residual sum of squares is the remainder, 25,812. With one predictor and 240 cases the degrees of freedom are 1 for the model and 240 - 1 - 1 = 238 for the residual, which makes MS_reg = 8,604 and MS_res = 25,812 / 238 = 108.45, so the overall test statistic is F(1, 238) = 8,604 / 108.45 = 79.3.

Now add the fall reading score as a second predictor. R-squared jumps to 0.635, meaning the regression sum of squares becomes 0.635 × 34,416 = 21,854 while the residual falls to 12,562, and with two model degrees of freedom and 237 residual degrees of freedom the statistic becomes F(2, 237) = 10,927 / 53.0 = 206.2. Notice how much of the story the baseline score was carrying all along, which is precisely the confounding problem Lesson 4 takes up.

Key idea: Every entry in an analysis-of-variance table is a sum of squares divided by its degrees of freedom, and every F statistic is one such quotient divided by another.

R-squared

The coefficient of determination is the fraction of total variance the model explains:

R² = SS_reg / SS_total = 1 - (SS_res / SS_total)

If SS_total = 200 and SS_res = 50, then R² = 1 minus (50/200) = 0.75: the model accounts for 75 percent of the variance in Y. In simple regression R² equals the squared Pearson correlation, which is why r = 0.5 corresponds to only 25 percent of variance explained, a useful antidote to overreading modest correlations.

The problem R-squared cannot solve, and adjusted R-squared

R² never decreases when you add a predictor, even a useless one, because OLS can always exploit chance associations to shave a little off SS_res. This makes raw R² a poor guide for comparing models with different numbers of predictors. Adjusted R² corrects for this by penalizing model complexity, using the degrees of freedom:

R̄²_adj = 1 - (1 - R²) · (n - 1) / (n - k - 1)

where n is the sample size and k the number of predictors. Adjusted R² can fall when a predictor adds nothing, which is exactly the behavior we want for model comparison.

Watch the correction operate on Brightwater. For the two-predictor model, adjusted R-squared = 1 - (1 - 0.635)(239 / 237) = 1 - 0.365(1.0084) = 0.632, barely below the raw 0.635, because two predictors in 240 cases is a modest amount of flexibility. Now append a genuinely irrelevant third predictor, the calendar month in which each student was born, and suppose raw R-squared creeps up to 0.636 by chance. Adjusted R-squared becomes 1 - (1 - 0.636)(239 / 236) = 1 - 0.364(1.0127) = 0.631, which is lower than before. The raw statistic rewarded the useless variable and the adjusted statistic penalized it, which is exactly the diagnostic behavior you want.

Two cautions belong with R-squared itself. It is not a measure of whether the model is correctly specified, since a badly curved relationship forced through a straight line can still return a respectable value, and it is not a measure of whether the model is useful, since a value of 0.05 in a field where outcomes are dominated by unmeasured causes may represent an important discovery while a value of 0.95 obtained by regressing a variable on a near-copy of itself represents nothing at all.

Key idea: R-squared answers how much variance the model absorbed, not whether the model is right, whether the effect matters, or whether the finding will replicate.

The overall F-test

The F-test asks whether the model as a whole explains significantly more variance than an intercept-only (mean) model. It compares mean squares, each a sum of squares divided by its degrees of freedom:

F = (SS_reg / k) / (SS_res / (n - k - 1)) = MS_reg / MS_res

Under the null hypothesis that all slopes are zero, and provided the model's assumptions hold, this ratio follows an F distribution. A large F, with its correspondingly small p-value, means the data would be surprising if the predictors carried no explanatory power whatsoever; it does not by itself establish that the model is correct, that the effect is large, or that the finding will replicate. Note the division of labor: the F-test evaluates the model globally, whereas the t-tests on individual coefficients evaluate each predictor's unique contribution. It is possible, especially under collinearity (Module 3), for the F-test to be significant while no single coefficient is, because the predictors share explanatory work.

The same machinery compares nested models, which is the workhorse test of applied modeling. To ask whether adding the fall reading score improved the Brightwater model, compute the incremental F from the two R-squared values:

F = [(0.635 - 0.250) / 1] / [(1 - 0.635) / 237] = 0.385 / 0.00154 = 250.0, on 1 and 237 degrees of freedom.

That is overwhelming evidence that the baseline score belongs in the model. The identical formula with q predictors in the numerator tests whether any block of variables adds explanatory power, which is how you evaluate a set of dummy variables, an interaction, or a whole theoretical construct in one test rather than a scatter of separate ones.

Key idea: The incremental F-test compares two nested models directly from their R-squared values, and it is the correct way to test a block of predictors as a unit.

What a p-value is, and the four things it is not

Because every F and t in this course terminates in a p-value, the definition deserves to be stated with care. The American Statistical Association put it this way in its 2016 statement: a p-value is the probability, under a specified statistical model that includes the null hypothesis and every assumption of the analysis, of obtaining a summary of the data at least as extreme as the one observed. Everything in that sentence is load-bearing, particularly the phrase "and every assumption," because a small p-value can signal an assumption violation just as easily as a real effect.

Four misreadings account for most of the damage in published work.

• A p-value is not the probability that the null hypothesis is true. Because it is computed while assuming the null, it cannot simultaneously be a probability statement about that same null, and producing the probability that the null is true given the data would require a prior probability the p-value never consults.
• A p-value is not the probability that the results are due to chance. This phrasing appears constantly in abstracts and press releases yet means nothing coherent, since the calculation has already conditioned on chance being the only mechanism in operation.
• A p-value is not a measure of effect size or importance. With the 240 Brightwater students, a tutoring effect of 0.3 points per hour could comfortably reach p below .05 while remaining educationally irrelevant, whereas with only 20 students an effect of 6 points per hour could miss the threshold while being enormous.
• A non-significant p-value does not establish the null. It licenses only the statement that the data were compatible with the null alongside many alternatives, so interpreting p = .21 as "no effect" confuses absence of evidence with evidence of absence.

A numerical illustration makes the first point concrete. Imagine a field in which 1,000 hypotheses are tested and 100 of them are genuinely true, with studies powered at 0.80 and tested at alpha = .05. The true hypotheses yield 100 × 0.80 = 80 significant results, while the 900 false ones yield 900 × 0.05 = 45 significant results anyway, so 125 findings cross the threshold and 45 of them, or 36 percent, are false. The p-value was .05 in every one of those 125 studies, yet the chance of being wrong was seven times higher than the naive reading suggests, because it depends on the base rate of true hypotheses and on power, neither of which the p-value knows anything about.

Two further disciplines follow. Report the effect size and its confidence interval alongside the test, because the interval communicates precision that a p-value cannot, and resist treating .05 as a boundary between real and unreal, since the difference between p = .04 and p = .06 is itself statistically indistinguishable and substantively trivial.

Key idea: A p-value measures compatibility between data and a fully specified model, and it answers neither "is the null true" nor "is this effect large."

Where people get stuck

• Treating a high R-squared as validation. Fit and correctness are separate questions, so a badly curved relationship or a serious omitted confounder can coexist quite comfortably with an R-squared of 0.85.
• Comparing raw R-squared across models with different predictor counts. Because the raw statistic can never fall when a variable is added, only adjusted R-squared or an incremental F-test makes such a comparison meaningful.
• Comparing R-squared across different samples. The statistic depends on how variable the predictors happen to be in the sample at hand, so a restricted-range sample reports a smaller value for exactly the same underlying relationship.
• Reading a significant F as evidence that every predictor matters. The omnibus test is a statement about the set as a whole, and under collinearity it can be highly significant while no individual coefficient reaches significance.
• Saying "there was a 5 percent chance the result was a fluke." This is the second misreading listed above, and reviewers at careful journals now flag the phrasing routinely.

Key idea: Nearly every error in this lesson comes from asking a fit statistic or a p-value to answer a question it was never constructed to address.

Recap

• SS_total = SS_reg + SS_res exactly, for any least-squares model containing an intercept.
• With 240 Brightwater students and a spring-score standard deviation of 12, the total sum of squares is 239 × 144 = 34,416, which the one-predictor model splits into 8,604 explained and 25,812 unexplained.
• That split gives R-squared = 0.250 and F(1, 238) = 8,604 / 108.45 = 79.3; adding the fall score raises R-squared to 0.635 and F(2, 237) to 206.2.
• Adjusted R-squared fell from 0.632 to 0.631 when an irrelevant month-of-birth predictor was added, which is the behavior that makes it usable for model comparison.
• The incremental F of 250.0 on 1 and 237 degrees of freedom tests one nested model against another and generalizes to testing a whole block of predictors at once.
• A p-value is the probability of data at least this extreme under the null model and all its assumptions, nothing more.
• It is not the probability the null is true, not the probability the result is due to chance, not a measure of size, and a large p-value does not establish the null.
• In a field with 10 percent true hypotheses, 80 percent power, and alpha = .05, fully 45 of 125 significant findings - 36 percent - are false.

Sources

1. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. amstat.org
2. Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337-350. pmc.ncbi.nlm.nih.gov
3. Goodman, S. (2008). A dirty dozen: Twelve p-value misconceptions. Seminars in Hematology, 45(3), 135-140. pubmed.ncbi.nlm.nih.gov
4. Sterne, J. A. C., & Davey Smith, G. (2001). Sifting the evidence: What's wrong with significance tests? BMJ, 322(7280), 226-231. pmc.ncbi.nlm.nih.gov
5. NIST/SEMATECH. (2012). How can I tell if a model fits my data? In e-Handbook of statistical methods. itl.nist.gov
6. Gelman, A., & Stern, H. (2006). The difference between "significant" and "not significant" is not itself statistically significant. The American Statistician, 60(4), 328-331. find source ↗
7. Penn State Eberly College of Science. (n.d.). STAT 501: Regression methods. online.stat.psu.edu

The move from one predictor to several is where regression becomes a workhorse of the social and biomedical sciences, because it lets us estimate the association of one variable with an outcome while statistically holding others constant. Reading these coefficients correctly, and knowing the limits of the control they provide, is the central interpretive skill of this module.

The stakes are practical rather than pedantic, since almost every substantive claim in an observational literature takes the form of a partial coefficient, and almost every dispute about such a literature is a dispute about whether the covariate set was adequate. A researcher who cannot state precisely what a coefficient holds constant, and what it necessarily fails to hold constant, is not in a position to defend the claim it supports.

Key idea: A partial coefficient is defined by the company it keeps, so the same predictor can carry three different numbers in three different models without any of them being wrong.

In plain terms

The Brightwater district wants to know whether after-school tutoring raises reading scores, but the families who signed up were not a random slice of the district, because parents who arrange extra tutoring also tend to have children who were already reading well in the fall. So the raw comparison between heavily tutored and lightly tutored students is contaminated by a head start that had nothing to do with the program.

Adding the fall reading score to the regression is an attempt to repair this, and the repair works by comparing students who entered the year at the same level, so that any remaining difference in spring scores can no longer be attributed to where they started. That comparison is what the phrase "controlling for baseline" actually denotes, and it is genuinely useful, but it repairs only the contamination you measured and named.

The meaning of a partial coefficient

In the model Y = b₀ + b₁X₁ + b₂X₂ + e, the coefficient b₁ is the predicted change in Y for a one-unit increase in X₁ with X₂ held constant. It is a partial coefficient: it isolates the part of X₁'s association with Y that is not shared with X₂.

Equivalently, b₁ is what you would get by removing X₂'s linear influence from both Y and X₁ and then regressing the two residualized variables on each other. This is why adding a predictor can change, shrink, or even flip the sign of an existing coefficient: the coefficient answers a different question once the set of things held constant changes.

That residualization identity, known as the Frisch-Waugh-Lovell theorem, is worth carrying around because it converts an abstract phrase into a three-step procedure you could execute by hand. Regress tutoring hours on the fall score and keep the residuals, which represent the portion of a student's tutoring that the fall score could not anticipate; regress the spring score on the fall score and keep those residuals as well; then regress the second set of residuals on the first. The slope you obtain is numerically identical to the tutoring coefficient from the two-predictor model, which demonstrates that "adjusting for the fall score" means literally subtracting out everything the fall score could have explained on both sides.

Key idea: Statistical adjustment is subtraction, not magic - it removes the linear shadow a covariate casts over both the predictor and the outcome, then examines what remains.

A concrete reading

Suppose a wage regression yields Wage = 20000 + 2500·(years of education) + 800·(years of experience). The education coefficient of 2500 says: among people with the same years of experience, each additional year of education is associated with 2500 more dollars of predicted annual wage. The phrase "among people with the same experience" is not optional decoration; it is the coefficient's definition. Strip it away and you have misstated the result.

Worked example: what adjustment does to the tutoring effect

Lesson 2 fitted the simple Brightwater regression and obtained Spring = 58 + 4.0(tutoring hours), a headline that a district newsletter would happily render as "each hour of tutoring is worth four points." Adding the fall reading score changes the picture substantially, because the two-predictor model estimates:

Spring = 18.8 + 2.2(tutoring hours) + 0.72(fall score)

The tutoring coefficient has fallen from 4.0 to 2.2, a drop of nearly half, and the drop is not mysterious once you know that students who received more tutoring had entered the year reading better. In these data each additional weekly tutoring hour is associated with a fall score 2.5 points higher, and the omitted-variable bias formula makes the accounting exact:

b_simple = b_adjusted + (b_fall × slope of fall on tutoring) = 2.2 + (0.72 × 2.5) = 2.2 + 1.8 = 4.0

So 1.8 of the original 4.0 points, or 45 percent of the apparent tutoring effect, was baseline reading ability arriving in disguise. The adjusted estimate of 2.2 points per hour comes with a standard error of 0.33, giving t = 6.58 and a 95 percent interval from 1.54 to 2.86, so the effect survives adjustment comfortably even though its magnitude was substantially overstated before.

Notice which direction the bias ran and why, because the sign is predictable rather than arbitrary. When an omitted variable is positively related both to the predictor and to the outcome, leaving it out inflates the coefficient; when it is positively related to one and negatively to the other, leaving it out deflates the coefficient and can even reverse its sign. This is the arithmetic behind every dispute in which one team reports a strong association and another team reports none after "adding controls."

Key idea: The gap between an unadjusted and an adjusted coefficient equals the omitted covariate's effect on the outcome multiplied by its association with the predictor, which makes the direction of confounding calculable rather than mysterious.

Statistical control is not experimental control

It is tempting to treat "controlling for X₂" as if it recreated an experiment in which X₂ was held fixed by design. It does not. Statistical control adjusts only for the variables you include and measure, and only insofar as the linear form is correct. Three limits deserve emphasis:

• Only measured confounders are controlled. A variable you did not include cannot be adjusted for. An omitted variable correlated with both predictor and outcome biases the coefficients that remain (omitted-variable bias, treated in Module 6).
• Control is only as good as measurement. If X₂ is measured with error, adjusting for it is incomplete, and residual confounding survives.
• The functional form must be right. Adjusting linearly for a variable whose true relationship is curved leaves part of its influence unremoved.

These limits are why a regression coefficient, however many covariates accompany it, is not automatically a causal effect. Module 6 develops the conditions under which it can be. For now, the discipline is to state control claims exactly: "adjusting for the measured covariates in the model," never "holding all else equal" as though the model were omniscient.

A fourth limit deserves equal billing, and Lesson 15 will develop it formally: adding covariates is not uniformly protective, because certain variables actively introduce bias when adjusted for. A variable that lies on the causal pathway from tutoring to reading, such as minutes of independent reading, absorbs part of the very effect under study, while a variable that is a common consequence of both tutoring and reading achievement can manufacture an association where none existed. The instinct to "control for everything available" is therefore not merely inefficient but potentially harmful, and the choice of covariate set must be argued from a causal model rather than from convenience.

Key idea: Randomization balances the confounders you never thought of, whereas regression balances only the ones you measured, modelled correctly, and were right to include.

Suppression: a cautionary pattern

Occasionally a predictor that correlates near zero with the outcome becomes a strong, significant predictor once another variable is added. This suppression effect is not a paradox; it happens when the added variable removes irrelevant variance from the first predictor, sharpening its relationship with Y. Suppression is a vivid reminder that a coefficient's value is a property of the whole model, not of the predictor in isolation. You cannot interpret any single coefficient without knowing what else is in the equation.

Brightwater supplies a clean instance in the teacher-experience variable. Raw correlation between a teacher's years of experience and her students' spring scores is essentially nil, with an unadjusted coefficient of 0.05 points per year that no reader would look at twice. Once the fall score enters the model, however, the experience coefficient rises to roughly 0.45 points per year and becomes clearly distinguishable from zero, because the district had for years been assigning its most experienced teachers to the classrooms with the weakest incoming readers. The raw association was flattened by that assignment policy, and adjusting for baseline reading removed the flattening rather than creating an effect from nothing.

The practical lesson is uncomfortable but important: a coefficient near zero in a bivariate table is not evidence that a variable is irrelevant, just as a large bivariate coefficient is not evidence that a variable matters. Both readings ignore the covariate structure that determines what either number means.

Key idea: Suppression shows that adjustment can strengthen a coefficient as easily as weaken it, so "the effect shrank after controls" is a finding about the covariates, not a verdict on the predictor.

Where people get stuck

• Dropping the holding-constant clause. Writing "each tutoring hour raises scores by 2.2 points" instead of "among students with the same fall score" is the single most common misstatement in applied writing, and it converts a conditional association into an unconditional causal claim.
• Reading "controlled for" as "equivalent to randomized." Adjustment addresses the covariates in the model, measured with whatever error they carry and modelled in whatever functional form you chose, so residual confounding is the default expectation rather than an exotic failure.
• Interpreting a sign flip as an error. When a coefficient reverses after a covariate is added, the two numbers answer different questions, and the honest response is to report both and explain the causal reasoning that selects one.
• Adding covariates indiscriminately. Mediators and colliders both do damage when adjusted for, and no amount of statistical sophistication substitutes for an explicit argument about which variables belong.
• Assuming a bivariate null means no relationship. Suppression demonstrates the opposite, and the Brightwater teacher-experience variable moved from 0.05 to 0.45 on the strength of one added covariate.

Key idea: Every difficulty in this lesson traces back to forgetting that a coefficient is a conditional quantity whose meaning is fixed by the rest of the equation.

Recap

• In a multiple regression, each slope is partial: the predicted change in the outcome per unit of that predictor, among cases equal on the others in the model.
• The Frisch-Waugh-Lovell residualization procedure reproduces a partial coefficient exactly and shows that adjustment is literal subtraction of a covariate's linear influence from both sides.
• Brightwater's tutoring coefficient fell from 4.0 to 2.2 once the fall reading score entered, with the adjusted estimate carrying a standard error of 0.33 and a 95 percent interval from 1.54 to 2.86.
• The omitted-variable bias formula accounts for the gap exactly: 2.2 + (0.72 × 2.5) = 4.0, so 45 percent of the raw effect was baseline ability.
• The direction of confounding is predictable from the signs of the two associations the omitted variable holds with predictor and outcome.
• Statistical control reaches only measured covariates, only to the extent they are measured accurately, and only if their functional form is right.
• Suppression runs the opposite way: teacher experience moved from a negligible 0.05 to about 0.45 points per year once baseline reading was held constant.

Sources

1. Penn State Eberly College of Science. (n.d.). STAT 501: Regression methods. online.stat.psu.edu
2. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning (2nd ed.). Springer. statlearning.com
3. Hernan, M. A., & Robins, J. M. (2020). Causal inference: What if. Chapman & Hall/CRC. miguelhernan.org
4. Hitchcock, C., & Redei, M. (2021). Causal models. Stanford Encyclopedia of Philosophy. plato.stanford.edu
5. Gelman, A., Hill, J., & Vehtari, A. (2020). Regression and other stories. Cambridge University Press. avehtari.github.io
6. NIST/SEMATECH. (2012). What terminology do statisticians use to describe process models? In e-Handbook of statistical methods. itl.nist.gov
7. Bhattacherjee, A. (2012). Social science research: Principles, methods, and practices (2nd ed.). University of South Florida. digitalcommons.usf.edu

A coefficient's raw magnitude depends on the units of its variables, which makes comparing predictors measured on different scales impossible from the raw numbers alone. Standardization addresses this, but it introduces trade-offs that doctoral researchers must weigh deliberately.

Behind the technical question sits a substantive one that reviewers increasingly insist on, namely whether an estimated effect is large enough to matter to anybody. Statistical significance answers only whether an effect is distinguishable from zero at a chosen threshold, so a doctoral report that stops there has declined to answer the question its readers actually asked, and the vocabulary of effect sizes exists precisely to let you answer it.

Key idea: Choosing between raw and standardized coefficients is a choice about which audience you are speaking to, and reporting neither an interval nor a practical benchmark leaves both audiences unserved.

In plain terms

The Brightwater model contains two predictors measured in units that cannot be compared, since one is counted in tutoring hours per week and the other in points on a reading test. Asking whether 2.2 points per hour is bigger than 0.72 points per test point is like asking whether a kilogram is longer than a metre, so the question needs rephrasing before it can be answered at all.

Standardizing rephrases it by converting every variable to a common yardstick, the standard deviation, after which each coefficient reads as "how many standard deviations does the outcome move when this predictor moves one standard deviation." That makes within-model comparison possible, but the yardstick is borrowed from this particular sample, which is why the same standardized coefficient can look quite different in a study drawn from a narrower population.

Unstandardized coefficients (b)

The unstandardized coefficient b is expressed in the original units: dollars per year of education, millimeters of mercury per milligram of drug. Its great virtue is concrete interpretability, and it is the right choice when the units are meaningful and you want a real-world statement of effect. Its limitation is that you cannot tell from b alone whether education or experience "matters more," because a dollar and a year are not comparable.

Standardized coefficients (beta)

The standardized coefficient, often written β or "beta weight," is what you obtain after converting every variable to z-scores (mean 0, standard deviation 1). It expresses the change in Y in standard deviations per one-standard-deviation change in the predictor. Because all predictors are now on a common standard-deviation scale, their standardized coefficients are roughly comparable within a single model, offering a rough ranking of relative importance. In simple regression the standardized slope equals the Pearson correlation.

	Unstandardized (b)	Standardized (beta)
Units	Original units of X and Y	Standard deviations
Best for	Concrete, real-world statements	Comparing predictors within a model
Comparable across studies?	Yes, if same measures/units	Not safely, since SDs differ by sample

Worked example: converting the Brightwater coefficients

Conversion needs only the standard deviations, because the relation between the two scales is beta = b × (SD of the predictor) / (SD of the outcome). In the Brightwater data the spring score has SD = 12, tutoring hours have SD = 1.5, and the fall score has SD = 11, so from the two-predictor model Spring = 18.8 + 2.2(hours) + 0.72(fall):

• Tutoring: beta = 2.2 × (1.5 / 12) = 0.275, so a one-standard-deviation increase in tutoring predicts about a quarter of a standard deviation more reading.
• Fall score: beta = 0.72 × (11 / 12) = 0.66, which is more than twice as large and confirms what the sums of squares in Lesson 3 already implied.
• Simple model check: the one-predictor slope of 4.0 converts to beta = 4.0 × (1.5 / 12) = 0.50, which equals the Pearson correlation between tutoring and spring score, exactly as the theory requires.

Both numbers describe the same fitted model, so nothing has been gained or lost in the translation, only re-expressed. The raw coefficient of 2.2 answers a district administrator's question about points per hour, while the standardized 0.275 answers a reviewer's question about which predictor carries more weight in the equation.

Key idea: beta = b times the ratio of the predictor's standard deviation to the outcome's, so the two coefficients are one estimate viewed through two different rulers.

The pitfalls of standardization

Standardized coefficients are seductive but treacherous. First, a standard deviation is a sample-specific quantity, so a sample with a restricted range on the predictor produces a smaller predictor SD, which shrinks the standardized coefficient even though the true unstandardized relationship is unchanged. This makes betas poorly suited to cross-study comparison, contrary to common belief, and it explains why studies of homogeneous populations routinely report weaker standardized effects than studies of heterogeneous ones. Second, standardizing a binary predictor (say, a 0/1 group indicator) produces a coefficient that is hard to interpret, because a "one standard deviation change" in a dichotomy is not a natural quantity. For categorical predictors, keep the unstandardized coefficient and interpret it as a group difference.

Worked example: range restriction shrinks beta without touching b

Suppose the district repeats the analysis on the subset of students receiving between 2 and 4 tutoring hours, which halves the predictor's standard deviation from 1.5 to 0.75 while leaving the underlying relationship of 4.0 points per hour completely intact. Because the outcome's variance decomposes as b-squared times the predictor variance plus the residual variance, the full-sample figures are 144 = 16(2.25) + 108, so the residual variance is 108 and stays fixed under restriction.

In the restricted subsample the outcome variance becomes 16(0.5625) + 108 = 117, giving SD = 10.82, and therefore:

beta = 4.0 × (0.75 / 10.82) = 0.277, down from 0.50 in the full sample.

The unstandardized coefficient of 4.0 points per hour did not move at all, yet the standardized coefficient nearly halved and the correlation fell with it. Anyone comparing these two studies by their betas would conclude that tutoring "worked better" in the broader sample, which is precisely backwards, since the two samples exhibit identical effects and differ only in who was sampled.

The binary-predictor problem is equally concrete. Free-lunch eligibility in Brightwater is coded 0 or 1 with 40 percent eligible, so its standard deviation is the square root of (0.4 × 0.6) = 0.49. If the coefficient is -5.0 points, the standardized version is -5.0 × (0.49 / 12) = -0.20, a number describing a one-standard-deviation change in a variable that only ever takes two values. Report the 5-point gap instead.

Key idea: Standardized coefficients are hostages to who was sampled, so a difference in betas between two studies may reflect nothing but a difference in variability.

Effect size, not just significance

Recall from prior study that statistical significance answers only whether an effect is distinguishable from zero, not whether it is large. Doctoral reporting therefore always pairs tests with effect sizes. In regression the natural effect sizes are the coefficients themselves (with confidence intervals) and variance-explained measures. For a single predictor's unique contribution there are two distinct quantities, and they must not be conflated. The squared semipartial correlation expresses the predictor's unique contribution as a share of the outcome's total variance, whereas partial eta-squared expresses it as a share of the variance that remains after the other predictors have been credited, which is a smaller denominator and therefore always yields the larger number.

sr² = SS_effect / SS_total    whereas    partial eta² = SS_effect / (SS_effect + SS_residual)

Worked example: two effect sizes for one coefficient

The Brightwater two-predictor model has R-squared = 0.635, and dropping tutoring leaves a fall-score-only model with R-squared = 0.568, so the squared semipartial correlation for tutoring is 0.635 - 0.568 = 0.067. Tutoring therefore uniquely accounts for 6.7 percent of the total variance in spring reading.

Partial eta-squared uses the same numerator with a different denominator. With SS_total = 34,416, the effect sum of squares is 0.067 × 34,416 = 2,306, and the residual sum of squares is 12,562, so partial eta-squared = 2,306 / (2,306 + 12,562) = 0.155. You can confirm this from the t statistic alone, since t-squared / (t-squared + df) = 43.3 / (43.3 + 237) = 0.155.

The two figures, 6.7 percent and 15.5 percent, describe the identical coefficient and differ by more than a factor of two. Reporting the larger one while calling it "variance explained" is a common and material overstatement, so name which index you used. Cohen's f-squared offers a third view, computed as sr-squared divided by (1 - R-squared) = 0.067 / 0.365 = 0.18, which sits near his medium benchmark of 0.15.

None of this settles whether the effect matters educationally, and that judgement requires leaving the statistics behind. A typical Brightwater participant receives 2 hours of tutoring per week, so the adjusted model predicts a gain of 2 × 2.2 = 4.4 points, which is 4.4 / 12 = 0.37 of a standard deviation. If a full year of ordinary fourth-grade growth in this district is about 9 points, the program is buying roughly half a year of additional reading growth, and that sentence, not the p-value, is what a school board can act on.

Cohen's rough benchmarks (small, medium, large) exist for these, but benchmarks are field-dependent conventions, not natural constants; a "small" standardized effect can be practically decisive in a high-stakes context, and a "large" one can be trivial if the outcome barely matters. Report the number, its interval, and its practical meaning, and let readers judge.

Key idea: Squared semipartial and partial eta-squared answer different questions with the same numerator, and reporting one under the other's name inflates the claim by a factor that can exceed two.

Where people get stuck

• Comparing betas across studies. Because the yardstick is the sample's own variability, two studies of identical effects will report different betas whenever their populations differ in spread, and the restricted Brightwater subsample moved from 0.50 to 0.277 without any change in the underlying relationship.
• Standardizing dichotomies. A one-standard-deviation change in free-lunch eligibility is not a thing that can happen to a student, so the group difference in raw units is the only defensible report.
• Confusing partial eta-squared with variance explained. The two indices share a numerator and differ in denominator, and the partial version is systematically larger because it discards other predictors' variance from the base.
• Treating Cohen's benchmarks as thresholds. They were offered as rough conventions for behavioural research in the absence of better information, and a mortality effect of d = 0.10 outranks a mood effect of d = 0.80 in every way that matters.
• Reporting an effect size without an interval. An estimate of 0.275 based on 240 students and one based on 24 carry entirely different warrants, and only the interval reveals which you are reading.

Key idea: An effect size is a claim about magnitude, so it needs a unit, an interval, and a sentence about consequences before it can do any work.

Recap

• Unstandardized coefficients carry meaning in real units and are the right report whenever those units are interpretable.
• Standardized coefficients rescale everything to standard deviations via beta = b times SD-of-predictor divided by SD-of-outcome, which permits within-model comparison.
• Brightwater's tutoring coefficient of 2.2 becomes beta = 0.275, the fall-score coefficient of 0.72 becomes beta = 0.66, and the simple-model slope of 4.0 becomes beta = 0.50, which equals the correlation.
• Range restriction shrinks rather than inflates a standardized coefficient, and halving the predictor's spread moved beta from 0.50 to 0.277 while the raw slope stayed at 4.0.
• Standardizing a binary predictor produces an uninterpretable quantity, so report the group difference in points instead.
• The squared semipartial for tutoring is 0.067, partial eta-squared is 0.155, and Cohen's f-squared is 0.18; these are three different indices, not three names for one.
• Practical significance requires a benchmark from outside the model: 2 hours a week buys about 4.4 points, roughly half a year of ordinary reading growth.

Sources

1. Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. pmc.ncbi.nlm.nih.gov
2. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates. find source ↗
3. Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology. American Psychologist, 73(1), 3-25. pubmed.ncbi.nlm.nih.gov
4. Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337-350. pmc.ncbi.nlm.nih.gov
5. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. amstat.org
6. Penn State Eberly College of Science. (n.d.). STAT 501: Regression methods. online.stat.psu.edu
7. Gelman, A., Hill, J., & Vehtari, A. (2020). Regression and other stories. Cambridge University Press. avehtari.github.io

Every regression result carries an implicit warranty: the inferences are trustworthy if a set of assumptions holds. A skilled analyst does not recite the assumptions; they interrogate them, and they know which violations are fatal to the estimates themselves and which merely corrupt the standard errors. That distinction organizes this lesson.

It also determines what you should do when a violation appears, since a problem that biases the coefficients demands a different model whereas a problem that only distorts the standard errors can often be repaired without changing the estimates at all. Confusing the two categories leads researchers to abandon perfectly serviceable models over cosmetic issues while proceeding confidently with models whose central estimates are simply wrong.

Key idea: Sort every assumption violation into one of two bins - it either corrupts the estimate or it corrupts the uncertainty around the estimate - because the remedies for the two bins have nothing in common.

In plain terms

Think of the residuals as the model's leftovers, and think of diagnostics as the practice of examining those leftovers for structure. A healthy model leaves behind formless noise with no discernible pattern, whereas an unhealthy one leaves a shape, and the shape tells you which assumption failed.

A curve in the leftovers means the relationship was never straight, so the coefficients themselves are describing the wrong thing. A widening fan means the model is more accurate for some cases than others, which does not make the coefficients wrong but does make the reported precision fictional. Clumps of leftovers that resemble each other within schools or within patients mean the observations were never independent, so the model has been counting the same information more than once. Each of these has its own picture, its own consequence, and its own repair.

The assumptions, and what each violation costs

• Linearity. The relationship between predictors and the mean of Y is linear in the parameters. Violation biases the coefficients: a straight line through a curved relationship misestimates the effect everywhere. Detect with a plot of residuals versus fitted values; a systematic curve signals nonlinearity.
• Independence of errors. Residuals are uncorrelated across cases. Violated by clustered data (students in classrooms) or time series (repeated measures). Non-independence usually does not bias the coefficients but badly understates standard errors, producing false precision and inflated significance. This assumption motivates the multilevel models of Module 6.
• Homoscedasticity. The error variance is constant across the range of predictions. Heteroscedasticity (a fan-shaped residual plot) leaves coefficients unbiased but makes standard errors wrong, so confidence intervals and p-values cannot be trusted. Detect with a residuals-versus-fitted plot; remedy with robust (heteroscedasticity-consistent) standard errors or a variance-stabilizing transformation.
• Normality of errors. The residuals are approximately normal. This matters mainly for small-sample inference; by the central limit theorem, coefficient sampling distributions become approximately normal in large samples even when errors are not. Check with a normal quantile-quantile (Q-Q) plot of residuals.
• No perfect multicollinearity. No predictor is an exact linear combination of others (treated in the next lesson).

Two refinements keep this list from being misapplied. First, dependence does not always shrink standard errors, because although a cluster-level predictor such as school funding is almost always reported with standard errors that are far too small, a predictor that varies purely within clusters can have its standard error overstated under the same clustering. The safe generalization is that dependence makes naive standard errors wrong, and only a model of the dependence structure tells you in which direction. Second, the normality assumption concerns the conditional distribution of the outcome, and it governs small-sample inference and individual prediction intervals rather than the large-sample behaviour of the coefficients themselves.

Key idea: Nonlinearity attacks the coefficients, heteroscedasticity and dependence attack the standard errors, and non-normality mostly attacks small samples and individual predictions.

Worked example: what heteroscedasticity costs

Suppose the Brightwater residuals are well behaved on average, with a residual mean square of 53.0 and therefore a typical residual of about 7.3 points, but that their spread is not constant. Among students whose predicted score falls below 65 the residual standard deviation is roughly 4.5, while among students predicted above 78 it is roughly 10.5, so the residual variances stand at 20.3 and 110.3 respectively, a ratio of more than five to one. Plotted, this produces the classic fan.

The coefficients remain unbiased under this pattern, so the tutoring estimate stays at 2.2 points per hour. What changes is the uncertainty attached to it. Recomputing with heteroscedasticity-consistent standard errors moves the standard error from 0.33 to roughly 0.41, so the test statistic falls from t = 2.2 / 0.33 = 6.58 to t = 2.2 / 0.41 = 5.37, and the 95 percent interval widens from roughly 1.54 to 2.86 out to roughly 1.39 to 3.01. The substantive conclusion survives here, but note what happened: the model overstated its own precision by about 24 percent, and a weaker effect could easily have been carried across the significance threshold by that overstatement alone.

The remedy is cheap and should be routine in observational work. Robust standard errors leave the estimates untouched and repair only the inference, which is exactly the right response to a problem located entirely in the second of our two bins.

Key idea: Heteroscedasticity does not move the estimate at all, it moves the error bar, and the fix is a different standard-error formula rather than a different model.

Reading the residuals-versus-fitted plot

The single most informative diagnostic plots residuals on the vertical axis against fitted values on the horizontal. The ideal is a formless horizontal band centered on zero. A curve indicates nonlinearity; a funnel or fan (spread growing with the fitted value) indicates heteroscedasticity; isolated points far from the band flag potential outliers. One plot thus probes three assumptions at once, which is why it is the first thing an experienced analyst examines.

Two cautions accompany the formal tests that accompany these plots. Significance tests for heteroscedasticity or non-normality, such as Breusch-Pagan or Shapiro-Wilk, inherit the general property that power grows with sample size, so in a study of 5,000 cases they will reliably flag departures too small to matter while in a study of 30 they will miss departures large enough to invalidate everything. Judge the magnitude of a violation from the plot and the consequences, not from whether a diagnostic test crossed .05.

Key idea: One residual-versus-fitted plot probes linearity, constant variance, and outliers simultaneously, which makes it the highest-yield thirty seconds in applied regression.

Influence and outliers

Not every unusual point is equally dangerous. A point can be an outlier in Y (large residual), a point of high leverage (extreme in the predictor space), or, most consequentially, an influential point whose removal would substantially change the coefficients. Cook's distance summarizes influence by combining residual size and leverage; conventionally, points with notably large Cook's distance deserve scrutiny. The correct response to an influential point is investigation, not automatic deletion: it may be a data-entry error, or it may be the most informative observation you have. Report what you found and what you did.

Worked example: leverage beats residual size

Cook's distance for case i combines the two ingredients multiplicatively, as D_i = [e_i² / (p × MS_res)] × [h_i / (1 - h_i)²], where p is the number of estimated parameters, e_i is the residual, and h_i is the leverage. The Brightwater two-predictor model has p = 3 and MS_res = 53.0, and average leverage is p / n = 3 / 240 = 0.0125, so anything above about 0.025 counts as high leverage.

Compare two students. The first is a dramatic outlier with a residual of 25 points but an entirely typical tutoring and fall profile, so her leverage is only 0.01:

D = [625 / (3 × 53)] × [0.01 / (0.99)²] = 3.93 × 0.0102 = 0.040

The second student is far less surprising in her score, missing by only 10 points, but she sits at the extreme of the predictor space with leverage 0.35 because she combined the most tutoring in the district with a very low fall score:

D = [100 / (3 × 53)] × [0.35 / (0.65)²] = 0.629 × 0.828 = 0.521

The second student influences the coefficients roughly thirteen times as much as the first, despite having a residual less than half the size, because leverage enters through a term that explodes as h approaches 1. This is the practical reason that scanning a list of large residuals is a poor way to find influential cases, and it is the same phenomenon that let one mistyped score reverse the slope in Lesson 2.

Interpret thresholds loosely. The conventional cutoffs of D greater than 1 or D greater than 4 / n serve only as attention-getters, so with n = 240 the second cutoff is 0.017 and both students would appear on the list while only one of them matters. The defensible workflow is to refit the model without the suspect cases, report how much the coefficients moved, and let the reader judge; deleting a case and reporting only the survivor's model is not a sensitivity analysis but a concealed decision.

Key idea: Influence is residual size multiplied by leverage, so an unremarkable miss at an extreme predictor value can outweigh a spectacular miss in the middle of the data.

Where people get stuck

• Testing the outcome for normality before fitting. The assumption concerns residuals, so the sequence is fit first and inspect afterward; a bimodal outcome is entirely compatible with well-behaved residuals.
• Trusting formal assumption tests over plots. Because their power scales with sample size, these tests flag trivial departures in large samples and miss serious ones in small samples, which is the opposite of what you need.
• Treating heteroscedasticity as a reason to abandon the model. The estimates remain unbiased, so the appropriate response is robust standard errors, not a different set of predictors.
• Equating outliers with influential cases. The Brightwater comparison gave Cook's distances of 0.040 and 0.521 for residuals of 25 and 10 points respectively, which inverts the ranking a residual list would produce.
• Deleting inconvenient observations silently. Removal is a modelling decision that changes the estimand, and it belongs in the methods section alongside the results obtained both ways.
• Reading independence as a formality. Clustering is the assumption most often violated by design in education and health research, and it is the one whose violation Module 6 devotes an entire lesson to repairing.

Key idea: Diagnostics are a reasoning process about which warranty clause failed and what it cost, not a checklist of tests to pass before reporting results.

Recap

• Nonlinearity biases the coefficients, so it requires respecification; heteroscedasticity and dependence corrupt the standard errors, so they require better inference.
• Dependence does not have a fixed direction of distortion, since cluster-level predictors get standard errors that are too small while purely within-cluster predictors can get standard errors that are too large.
• Normality of residuals matters for small-sample inference and for individual prediction intervals, not for large-sample coefficient inference.
• A Brightwater residual spread of 4.5 at low predictions and 10.5 at high ones moved the tutoring standard error from 0.33 to 0.41, shrinking t from 6.58 to 5.37 and widening the interval to roughly 1.39 to 3.01.
• The residuals-versus-fitted plot tests three assumptions at once: a curve means nonlinearity, a fan means heteroscedasticity, and stray points mean potential outliers.
• Cook's distance multiplies squared residual by a leverage term, and the worked cases gave D = 0.040 for a 25-point miss at leverage 0.01 versus D = 0.521 for a 10-point miss at leverage 0.35.
• Influential cases warrant investigation and a reported sensitivity analysis, never silent deletion.

Sources

1. NIST/SEMATECH. (2012). How can I tell if a model fits my data? In e-Handbook of statistical methods. itl.nist.gov
2. NIST/SEMATECH. (2012). Levene test for equality of variances. In e-Handbook of statistical methods. itl.nist.gov
3. Cook, R. D. (1977). Detection of influential observation in linear regression. Technometrics, 19(1), 15-18. find source ↗
4. Anscombe, F. J. (1973). Graphs in statistical analysis. The American Statistician, 27(1), 17-21. find source ↗
5. Penn State Eberly College of Science. (n.d.). STAT 501: Regression methods. online.stat.psu.edu
6. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning (2nd ed.). Springer. statlearning.com
7. Gelman, A., Hill, J., & Vehtari, A. (2020). Regression and other stories. Cambridge University Press. avehtari.github.io

Two structural problems in the predictor set can undermine a regression even when every assumption about the errors holds: predictors that are too intertwined (multicollinearity), and a model chosen by fishing rather than theory. Both are matters of research judgment as much as statistics.

The two problems are related more closely than they first appear, because an analyst confronted with collinear predictors is precisely the analyst most tempted to let an algorithm decide which of them to keep. Understanding why that temptation must be resisted requires seeing what collinearity actually does to a coefficient and what automated selection actually does to a p-value.

Key idea: Collinearity is a problem of insufficient information for a question you asked, whereas stepwise selection is a problem of asking the data to choose the question and then pretending you asked it in advance.

In plain terms

Imagine the Brightwater district decides to include both a fall reading score and a fall vocabulary score in the model, and imagine those two measures agree with each other almost perfectly because they were built from overlapping item pools. The regression must now decide how much credit for spring achievement belongs to reading and how much to vocabulary, but the data contain almost no cases where the two disagree, so there is nothing to base the decision on.

The software does not refuse. It produces two coefficients, but they are enormously uncertain, they swing wildly if you drop a handful of students, and each may fail its own significance test while the pair jointly explains the outcome beautifully. That combination of a strong overall model with weak individual coefficients is collinearity's calling card, and the honest response is usually to admit that these data cannot separate the two constructs.

What multicollinearity is

Multicollinearity is high correlation among predictors. When two predictors carry nearly the same information, the model struggles to allocate credit between them. Perfect multicollinearity (one predictor an exact linear combination of others) makes the coefficients mathematically undefined and must be removed. Severe but imperfect multicollinearity has a characteristic signature: coefficients with large standard errors, hence wide confidence intervals and non-significant t-tests, even while the overall model fits well and the F-test is significant. The estimates also become unstable, swinging wildly with small changes in the data.

Detecting it: VIF and tolerance

The standard diagnostic is the variance inflation factor (VIF). For predictor X_j, regress it on all the other predictors and obtain that regression's R²_j. Then:

VIF_j = 1 / (1 - R²_j)    and    Tolerance_j = 1 - R²_j = 1 / VIF_j

A VIF of 1 means the predictor is uncorrelated with the others; larger values mean its standard error is inflated by that factor's square root. If X_j is 90 percent predictable from the others (R²_j = 0.90), then VIF = 1/(1 minus 0.90) = 10, a common rule-of-thumb threshold for concern.

Remedies include dropping or combining redundant predictors, collecting more data, or, if the collinear set is theoretically a single construct, forming a composite or using the factor-analytic methods of Module 6. Crucially, multicollinearity does not bias the coefficients; it inflates their variance. If your interest is prediction rather than isolating individual effects, moderate collinearity may not matter at all.

Worked example: what a VIF of 10.5 costs

The square-root relationship is what makes VIF interpretable, because a variance inflated by a factor of VIF corresponds to a standard error inflated by the square root of VIF. Reading that table is the fastest way to calibrate your alarm:

R-squared of predictor on the others	VIF	Standard error inflated by
0.00	1.0	1.00 (no penalty)
0.50	2.0	1.41
0.80	5.0	2.24
0.90	10.0	3.16
0.95	20.0	4.47

Now apply it to Brightwater. In the two-predictor model the fall reading coefficient of 0.72 carries a standard error of 0.046, giving t = 0.72 / 0.046 = 15.8, which is about as decisive as applied research ever gets. Suppose the district then adds a fall vocabulary score that correlates 0.95 with fall reading, so that fall reading is now 90.5 percent predictable from the other predictors and VIF = 1 / (1 - 0.905) = 10.5.

The standard error is multiplied by the square root of 10.5, which is 3.24, so it rises from 0.046 to roughly 0.149, and even if the coefficient happened to stay at 0.72 the test statistic would collapse from 15.8 to about 4.8. In practice the coefficient does not stay put either, because the credit is now being split arbitrarily between two near-identical measures, and a coefficient of 0.40 for reading paired with 0.33 for vocabulary would fit the data essentially as well as any other division of the same total.

Notice what has and has not gone wrong. The model still predicts spring scores just as accurately, R-squared barely moves, and the joint F-test for the two baseline measures remains overwhelming. Only the attempt to attribute the effect to one measure rather than the other has failed, which is why collinearity is harmless for prediction and fatal for the interpretation of individual coefficients.

Key idea: Collinearity does not destroy information about the set of predictors, it destroys information about how to divide credit within the set, so it matters exactly when your question is about an individual coefficient.

The trouble with stepwise selection

Faced with many candidate predictors, software offers stepwise selection (forward, backward, or both), which adds or removes predictors according to significance thresholds. It is popular and, for confirmatory inference, largely indefensible. The problems are severe:

• The p-values and confidence intervals it reports are invalid, because the same data were used to choose the variables and to test them, and this selection is not acknowledged in the final inference.
• It capitalizes on chance, selecting predictors that fit sample-specific noise, so results replicate poorly.
• It substitutes an algorithm for theory, producing models that may fit yet make no substantive sense.

Worked example: stepwise selection on pure noise

Put numbers on the second objection, because the magnitude surprises people. Suppose an analyst runs a forward stepwise procedure on the 240 Brightwater students using 20 candidate predictors that are all pure random noise, entering any variable that reaches p < .05. Each individual noise variable has a 5 percent chance of qualifying, so the expected number entering the model is 20 × 0.05 = 1, and the probability that at least one enters is:

1 - (0.95)²⁰ = 1 - 0.358 = 0.642

Nearly two runs in three will therefore produce a "significant" predictor of reading achievement from variables that are, by construction, meaningless. With 50 candidates the figure rises to 1 - (0.95)⁵⁰ = 0.923. Worse, the surviving variable's reported p-value will sit below .05 and the write-up will treat it as a 5 percent risk, when the procedure that generated it carried a 64 percent chance of producing something.

The same arithmetic explains why R-squared cannot police this. Under a pure null, the expected R-squared from fitting k predictors to n cases is k / (n - 1), so 15 random predictors on 240 students deliver an expected R-squared of 15 / 239 = 0.063, which looks like a modest but real finding. Adjusted R-squared handles it correctly, since 1 - (1 - 0.063) × (239 / 224) = 0.000, returning the truth that nothing was explained.

Key idea: A stepwise run on 20 pure-noise predictors returns a significant result 64 percent of the time, so its reported p-values describe a procedure that was never actually followed.

Principled alternatives

For explanatory research, build the model from theory: include predictors because a defensible argument says they belong, and pre-specify the model before seeing the outcomes where possible. When the goal is genuinely predictive and you must select among many variables, use methods designed for it, such as penalized regression (ridge or lasso, which shrink coefficients to control overfitting) evaluated by cross-validation on held-out data rather than in-sample p-values. The governing principle is the same one from Module 1 of the companion methods course: a model chosen by exploring the data cannot then be honestly "confirmed" on that same data.

Notice that penalized regression is also a principled response to collinearity, since ridge deliberately accepts a small bias in exchange for a large reduction in variance, which is exactly the trade a researcher facing VIF = 10.5 should want to make. What it does not deliver is an unbiased estimate of an individual coefficient, so the choice between ridge and simply reporting the collinearity honestly depends once more on whether your question is predictive or explanatory.

Two further disciplines are worth naming. Pre-specify the covariate set in a preregistration where the design permits it, so that the model is a commitment rather than an outcome, and where exploration genuinely is the goal, label it exploratory and validate the resulting model on data that played no part in selecting it. Lesson 17 develops both practices in detail.

Key idea: Selecting variables and testing them on the same data invalidates the test, so honest model building either commits in advance or validates on data held out from the search.

Where people get stuck

• Treating VIF thresholds as laws. The value of 10 is a convention, and a VIF of 6 in a study whose central question is the coefficient of that very predictor is more serious than a VIF of 15 on a nuisance covariate nobody will interpret.
• Believing collinearity biases estimates. It inflates variance and leaves the expected value alone, which is why the correct complaint is imprecision rather than distortion.
• Diagnosing collinearity from a correlation matrix alone. Three predictors can be pairwise uncorrelated and yet nearly linearly dependent as a set, which only a VIF or condition-index computation reveals.
• Dropping a collinear predictor to "fix" the model. Removing a genuine confounder to reduce a VIF trades an imprecision problem for a bias problem, which is a strictly worse position.
• Reporting stepwise p-values at face value. The 64 percent figure from 20 noise predictors shows how far the reported error rate can sit from the real one.
• Confusing prediction goals with explanation goals. Cross-validated predictive accuracy says nothing about whether a coefficient represents a causal effect, and a theory-driven explanatory model may predict less well than a lasso that is scientifically empty.

Key idea: Ask what the model is for before diagnosing it, because collinearity and selection are serious problems for explanation and often minor ones for prediction.

Recap

• Multicollinearity is high correlation among predictors, and its signature is large standard errors and unstable, non-significant coefficients alongside a strong overall model fit.
• VIF = 1 / (1 - R-squared of that predictor on the others), and the standard error is inflated by the square root of the VIF.
• A fall vocabulary measure correlating 0.95 with fall reading pushes VIF to 10.5, multiplying the standard error by 3.24 and collapsing t from 15.8 to about 4.8.
• Collinearity leaves the coefficients unbiased and prediction unharmed; it destroys only the ability to attribute the effect to one member of the collinear set.
• Forward stepwise selection on 20 pure-noise predictors returns at least one significant variable 64 percent of the time, since 1 minus 0.95 to the twentieth power = 0.642.
• Fifteen random predictors on 240 cases give an expected R-squared of 15 / 239 = 0.063, which adjusted R-squared correctly reduces to about zero.
• Principled alternatives are theory-driven specification for explanation, and penalized regression with cross-validation for prediction.

Sources

1. Vatcheva, K. P., Lee, M., McCormick, J. B., & Rahbar, M. H. (2016). Multicollinearity in regression analyses conducted in epidemiologic studies. Epidemiology, 6(2), 227. pmc.ncbi.nlm.nih.gov
2. Kim, J. H. (2019). Multicollinearity and misleading statistical results. Korean Journal of Anesthesiology, 72(6), 558-569. pmc.ncbi.nlm.nih.gov
3. Smith, G. (2018). Step away from stepwise. Journal of Big Data, 5, 32. journalofbigdata.springeropen.com
4. Whittingham, M. J., Stephens, P. A., Bradbury, R. B., & Freckleton, R. P. (2006). Why do we still use stepwise modelling in ecology and behaviour? Journal of Animal Ecology, 75(5), 1182-1189. find source ↗
5. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). Linear model selection and regularization. In An introduction to statistical learning (2nd ed.). Springer. statlearning.com
6. Harrell, F. E. (2015). Regression modeling strategies (2nd ed.). Springer. find source ↗
7. Penn State Eberly College of Science. (n.d.). STAT 501: Regression methods. online.stat.psu.edu

The claim in Module 1 that ANOVA is just regression rests on a technique: representing categories as numbers. And the most substantively interesting regressions involve interactions, where the effect of one variable depends on another. Both live here.

These two devices account for a large share of the misreadings in published regression tables, because a dummy coefficient means nothing until the reader knows the reference category, and a lower-order coefficient in an interaction model means nothing until the reader knows where zero falls on the other variable. Both problems are solvable in a sentence of reporting, and both are routinely left unsolved.

Key idea: Dummy coefficients are comparisons against a category you chose, and interaction models turn every lower-order coefficient into a conditional statement about a specific value of another predictor.

In plain terms

Regression multiplies predictors by numbers, so a variable recording whether a student was in the control group, the small-group programme, or one-to-one tutoring cannot enter the equation directly. The fix is to translate the three categories into a set of yes-or-no columns, one fewer than the number of categories, and let the leftover category serve as the baseline everything else is measured against.

Interactions solve a different problem. An ordinary model insists that tutoring is worth the same number of points to every student, regardless of how well they were already reading, which is an assumption rather than a finding and is frequently false. Multiplying two predictors together and adding the product lets the model say that tutoring is worth more to weak readers than to strong ones, and the coefficient on that product is what measures the difference.

Dummy coding a categorical predictor

A categorical variable with g levels is entered as g minus 1 dummy (indicator) variables, each coded 0 or 1. One level is omitted and becomes the reference category. Consider a predictor "treatment group" with levels Control, Drug A, Drug B. Choosing Control as reference, we create two dummies: D_A = 1 for Drug A else 0, and D_B = 1 for Drug B else 0. The model Y = b₀ + b₁D_A + b₂D_B yields:

• b₀ = the mean of the reference group (Control), since both dummies are 0 there.
• b₁ = the difference between Drug A's mean and Control's mean.
• b₂ = the difference between Drug B's mean and Control's mean.

So each dummy coefficient is a comparison against the reference, and the choice of reference category changes the coefficients' values though not the model's fit. Why g minus 1 dummies rather than g? Including one dummy per level plus an intercept creates perfect multicollinearity (the dummies would sum to a column of ones, identical to the intercept), the so-called dummy-variable trap. Omitting one level resolves it.

Worked example: three Brightwater formats, three parameterizations

The Brightwater district ran three arms with 80 students each, producing mean spring scores of 64 for the control arm, 70 for small-group tutoring, and 76 for one-to-one tutoring, with a grand mean of 70. Coding control as the reference gives the model:

Spring = 64 + 6(Small-group) + 12(One-to-one)

Read it back: control students have both dummies at zero and are predicted 64, small-group students are predicted 64 + 6 = 70, and one-to-one students are predicted 64 + 12 = 76, which reproduces the three means exactly. That exactness is the point, since a saturated dummy model can do nothing else.

Now switch the reference to small-group and refit. The coefficients become b₀ = 70, b_Control = -6, and b_One-to-one = +6, which look like an entirely different result and are not one, because the predicted means, the residuals, the R-squared, and the omnibus F are all untouched. Only the questions being asked have changed, from "how does each arm compare with doing nothing" to "how does each arm compare with small-group tutoring."

A third scheme, effect coding, is worth knowing because it is what classical ANOVA uses under the hood. Code each dummy as +1 for its own group, 0 for other groups, and -1 for the omitted group, and the intercept becomes the unweighted grand mean of 70 while each coefficient becomes that group's deviation from it: 0 for small-group and +6 for one-to-one, with the control deviation recoverable as 0 - (-6) ... more directly, as the negative of the sum, giving -6 and hence a control mean of 64. Again the fit is identical.

Two reporting rules follow. Always name the reference category in the table caption, since a coefficient of 6 is uninterpretable without it, and always test a multi-category predictor with an incremental F on its full set of dummies rather than by eyeballing which individual contrasts reached significance.

Key idea: Changing the reference category changes every dummy coefficient and changes nothing about the model, so the coefficients are answers to a question the analyst chose rather than properties of the data.

Interactions: when effects depend on other variables

A model with only additive terms assumes each predictor's effect is the same at every value of the others. That assumption often fails. An interaction (moderation) term relaxes it. For two predictors X and Z, we add their product:

Y = b₀ + b₁X + b₂Z + b₃(X·Z) + e

The coefficient b₃ on the product is the interaction: it tells you how the slope of X changes for each one-unit increase in Z. Rearranging shows this explicitly, since the effect of X is (b₁ + b₃Z), a conditional slope that depends on Z.

Key idea: Adding a product term converts a single slope into a formula for a slope, so the model now reports a different effect of X for every value of Z.

The interpretation trap you must avoid

When an interaction is in the model, b₁ and b₂ are no longer main effects. They are simple (conditional) effects at the value zero of the other variable. Specifically, b₁ is the effect of X when Z = 0, and b₂ is the effect of Z when X = 0. If zero is not a meaningful or observed value of Z, then b₁ describes a case that may not exist, and reading it as "the effect of X" is a serious error.

This is why analysts center continuous predictors (subtract their mean) before forming an interaction: after centering, zero is the mean, so b₁ becomes the effect of X at the average level of Z, a far more interpretable quantity. Centering also reduces the nonessential collinearity between the product term and its components. To understand a significant interaction, compute and plot simple slopes: the slope of X at low, mean, and high values of Z (often the mean and one standard deviation on either side). A significant b₃ tells you the slopes differ; the simple-slopes analysis tells you how.

Worked example: centering changes the numbers, not the model

Fit the Brightwater interaction with tutoring hours and fall score both centered at their means of 3.0 and 62 respectively, and suppose the result is:

Spring = 70.0 + 2.2(hours_c) + 0.72(fall_c) - 0.10(hours_c × fall_c)

The conditional slope of tutoring is 2.2 - 0.10(fall_c), so tutoring is worth 2.2 + 1.1 = 3.3 points per hour to a student one standard deviation below average in the fall (fall_c = -11, a raw score of 51), 2.2 points at the average, and 2.2 - 1.1 = 1.1 points to a student one standard deviation above (a raw score of 73). Tutoring helps the students who need it most, which is a substantively important finding that no additive model could have expressed.

Check one prediction end to end. A student with 5 tutoring hours (hours_c = +2) and a fall score of 51 is predicted 70.0 + 2.2(2) + 0.72(-11) - 0.10(2)(-11) = 70.0 + 4.4 - 7.92 + 2.2 = 68.68. The same figure arrives from the conditional-slope route, since the intercept at fall_c = -11 is 70.0 - 7.92 = 62.08 and adding 3.3(2) = 6.6 gives 68.68.

Now expand the identical model in raw, uncentered units. Multiplying out (X - 3) and (F - 62) throughout yields:

Spring = 0.16 + 8.4(hours) + 1.02(fall) - 0.10(hours × fall)

Substituting hours = 5 and fall = 51 returns 0.16 + 42 + 52.02 - 25.5 = 68.68, exactly as before, because these are the same model written twice. Yet the tutoring coefficient now reads 8.4 rather than 2.2, nearly four times larger, and a reader who quotes it as "the effect of tutoring" has quoted the effect for a hypothetical student whose fall reading score was zero. No such child exists in the district.

One detail deserves emphasis because it contradicts a widespread belief. The interaction coefficient is -0.10 in both parameterizations, with the same standard error, the same t statistic, and the same p-value, and the two models share an identical R-squared and identical residuals. Centering therefore does not make an interaction easier to detect and does not repair any collinearity that matters; it repairs only the interpretability of the lower-order terms.

Finally, locate where the effect disappears. Setting the conditional slope to zero gives 2.2 - 0.10(fall_c) = 0, so fall_c = 22, a raw fall score of 84 that lies two standard deviations above the mean. Within the observed range of the data, then, the estimated tutoring effect is positive for essentially every student, which is a far more useful statement than "the interaction was significant."

Key idea: Centering changes the lower-order coefficients and nothing else, so it is an interpretability device rather than a statistical remedy.

Where people get stuck

• Reading a lower-order coefficient as a main effect. Once a product term is present, b₁ is the effect of X at Z = 0, which is why the uncentered Brightwater model reported 8.4 for a quantity that is genuinely 2.2 at the average student.
• Believing centering strengthens the interaction test. The product coefficient, its standard error, and its p-value are invariant to centering, and the collinearity that centering removes was never affecting them.
• Dropping a non-significant lower-order term. Removing X while retaining X times Z forces a constrained model that few researchers intend and almost none justify, so the convention is to retain all lower-order terms.
• Testing a multi-category interaction one product at a time. A three-level factor interacting with a continuous predictor generates two product terms, and the correct test is a joint incremental F on 2 degrees of freedom.
• Concluding a difference in significance implies a significant difference. Finding that tutoring predicts reading among weak readers (p = .01) but not among strong ones (p = .12) is not evidence of moderation; only the test of the interaction coefficient is.
• Forgetting to name the reference category. A dummy coefficient of 6 is meaningless in isolation, and the same model reported against a different baseline yields -6.

Key idea: Interaction models are read from the conditional slope formula, not from the coefficient table, and the table alone will mislead almost every reader who tries.

Recap

• A categorical predictor with g levels enters as g minus 1 dummy variables, with the omitted level serving as the reference; including all g plus an intercept produces the dummy-variable trap.
• The Brightwater arms with means 64, 70, and 76 give Spring = 64 + 6(Small-group) + 12(One-to-one) against a control reference, and 70, -6, +6 against a small-group reference, with identical fit.
• Effect coding puts the grand mean in the intercept and each group's deviation in its coefficient, which is the parameterization classical ANOVA uses.
• An interaction term lets one predictor's slope depend on another, since collecting terms gives a conditional slope of b₁ + b₃Z.
• With Spring = 70.0 + 2.2(hours) + 0.72(fall) - 0.10(product) in centered units, tutoring is worth 3.3 points per hour to weak fall readers, 2.2 at the average, and 1.1 to strong readers.
• The identical model in raw units reads 0.16 + 8.4(hours) + 1.02(fall) - 0.10(product), and both versions predict 68.68 for the same student.
• Centering changes the lower-order coefficients but leaves the interaction coefficient, its p-value, the residuals, and R-squared completely unchanged.

Sources

1. Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Sage Publications. find source ↗
2. Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011). Erroneous analyses of interactions in neuroscience: A problem of significance. Nature Neuroscience, 14(9), 1105-1107. nature.com
3. Penn State Eberly College of Science. (n.d.). STAT 501: Regression methods. online.stat.psu.edu
4. NIST/SEMATECH. (2012). Are the means equal? In e-Handbook of statistical methods. itl.nist.gov
5. Gelman, A., Hill, J., & Vehtari, A. (2020). Regression and other stories. Cambridge University Press. avehtari.github.io
6. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning (2nd ed.). Springer. statlearning.com
7. UCLA Office of Advanced Research Computing. (n.d.). Introduction to linear mixed models and coding schemes. Statistical Methods and Data Analytics. stats.oarc.ucla.edu

Analysis of variance is taught as a distinct method, but Module 1 promised it is regression with categorical predictors. Here we cash that promise and add a continuous covariate to reach ANCOVA, keeping the unifying GLM view throughout.

The practical payoff of the unified view is that everything you learned about diagnostics, effect sizes, and interaction interpretation transfers without modification. An ANOVA has residuals to examine, an ANCOVA has a partial coefficient to read, and a factorial design has a product term whose lower-order coefficients are conditional in exactly the way Lesson 8 described.

Key idea: ANOVA is regression on dummy variables and ANCOVA is regression on dummy variables plus a covariate, so nothing new is being assumed and nothing new needs to be learned.

In plain terms

Brightwater ran three tutoring arms and wants to know whether the arms differ in average spring reading score. The logic of the test is a comparison of two kinds of spread: how far apart the three arm averages sit from one another, against how much students scatter within each arm. If the arms are further apart than the internal scatter would lead you to expect, something real is separating them.

ANCOVA adds one refinement to that comparison. Because the arms did not start the year equal, part of any spring gap simply reflects where the students began, so the analysis first removes the influence of the fall score and then compares what is left. That removal buys two things at once, since it both fairer-compares the arms and shrinks the internal scatter, making genuine differences easier to see.

One-way ANOVA

One-way ANOVA compares the means of a continuous outcome across the levels of one categorical factor. It partitions variance exactly as regression does: total variability splits into between-groups variability (differences among the group means) and within-groups variability (spread inside each group). The test statistic is the ratio of the corresponding mean squares:

F = MS_between / MS_within

A large F says the group means are more spread out than within-group noise would predict. This is identical to the overall F-test of a regression whose predictors are the group dummies. The null hypothesis is that all group means are equal. Importantly, a significant omnibus F says only that the means are not all equal; it does not say which differ. That requires post-hoc comparisons (such as Tukey's HSD) that control the familywise error rate inflated by making many pairwise tests, or planned contrasts specified in advance.

Worked example: the Brightwater one-way ANOVA

The three arms held 80 students each with mean spring scores of 64, 70, and 76, so the grand mean is 70 and the between-groups sum of squares is 80 times the summed squared deviations of those means:

SS_between = 80[(64 - 70)² + (70 - 70)² + (76 - 70)²] = 80(36 + 0 + 36) = 5,760, on 2 degrees of freedom.

With a within-arm standard deviation of 10 points, the within-groups mean square is 100 on 240 - 3 = 237 degrees of freedom, giving SS_within = 23,700 and therefore SS_total = 29,460. The test statistic follows immediately as F(2, 237) = 2,880 / 100 = 28.8, which is decisive by any threshold.

Report an effect size alongside it. Eta-squared is 5,760 / 29,460 = 0.196, so the arm assignment accounts for about 20 percent of the variance in spring reading. Eta-squared is known to be optimistically biased upward, so the less biased omega-squared is often preferred, computed as (5,760 - 2 × 100) / (29,460 + 100) = 5,560 / 29,560 = 0.188. For the extreme pair, Cohen's d = (76 - 64) / 10 = 1.2, an unusually large educational effect that should prompt scepticism about the design rather than celebration.

The omnibus F says only that the three means are not all equal, so locating the differences requires follow-up. Tukey's honestly significant difference uses a critical studentized range of about 3.33 for three groups at these degrees of freedom, giving a threshold of 3.33 × the square root of (100 / 80) = 3.33 × 1.118 = 3.72 points. Every pairwise gap here (6, 6, and 12) clears it comfortably.

The reason for using Tukey rather than three separate t-tests is arithmetic. Three independent tests at alpha = .05 carry a familywise error rate of up to 1 - (0.95)³ = 0.143, and a six-arm study with 15 pairwise comparisons reaches 1 - (0.95)¹⁵ = 0.537. Without correction, better than even odds of at least one false claim.

Key idea: The omnibus F answers whether any means differ, the effect size answers by how much, and a multiplicity correction is what keeps the follow-up comparisons honest.

Factorial ANOVA and interaction

With two or more factors, factorial ANOVA estimates each factor's main effect and, crucially, their interaction, the same product-term logic from Module 3. A significant interaction means the effect of one factor depends on the level of the other, and it takes interpretive priority: when an interaction is present, main effects can be misleading because they average over conditions in which the effect genuinely differs. Always interpret the interaction first, then the simple effects within it.

Worked example: a two-by-two Brightwater design

Suppose the district crosses format (small-group versus one-to-one) with dosage (1 hour versus 3 hours per week), placing 60 students in each of the four cells. The observed cell means are 66 and 70 for small-group at 1 and 3 hours, and 68 and 80 for one-to-one at 1 and 3 hours, so the grand mean is 71.

Marginal means give format levels of 68 and 74 and dosage levels of 67 and 75, so with 120 students per marginal level the sums of squares are 120[(68 - 71)² + (74 - 71)²] = 2,160 for format and 120[(67 - 71)² + (75 - 71)²] = 3,840 for dosage. The interaction sum of squares uses the residual of each cell from its two marginals, which is +2, -2, -2, +2 across the four cells, giving 60(4 + 4 + 4 + 4) = 960.

Each of these carries 1 degree of freedom, so against a within-cell mean square of 100 on 236 degrees of freedom the three F statistics are 21.6 for format, 38.4 for dosage, and 9.6 for the interaction, the last comfortably significant.

Now read the interaction before anything else, because it changes what the main effects mean. Going from 1 to 3 hours is worth 70 - 66 = 4 points in the small-group arm but 80 - 68 = 12 points in the one-to-one arm, and the "main effect of dosage" of 8 points is simply the average of those two figures, describing neither arm accurately. A district that read only the main effect would triple tutoring hours in the small-group programme and be disappointed.

Key idea: A main effect in the presence of an interaction is an average over conditions that genuinely differ, so it can describe no actual subgroup in the study.

ANCOVA: adding a covariate

Analysis of covariance (ANCOVA) adds one or more continuous covariates to an ANOVA. In GLM terms it is simply a regression with both group dummies and a continuous predictor. It serves two purposes:

• Increased power. A covariate that predicts the outcome absorbs residual variance, shrinking MS_error and making group differences easier to detect.
• Adjustment. It compares adjusted means: the group means one would expect if every group had the same average value of the covariate. In a pretest-posttest design, adjusting the posttest for the pretest sharpens the comparison.

Worked example: adjusted means and the power gain

The Brightwater arms did not begin the year equal, since the fall reading means were 60 for control, 62 for small-group, and 64 for one-to-one, against a grand covariate mean of 62. With a pooled within-group covariate slope of 0.72, each arm's adjusted mean is its observed mean minus 0.72 times its covariate departure from the grand mean:

• Control: 64 - 0.72(60 - 62) = 64 + 1.44 = 65.44
• Small-group: 70 - 0.72(62 - 62) = 70.00
• One-to-one: 76 - 0.72(64 - 62) = 76 - 1.44 = 74.56

The one-to-one advantage over control shrinks from 12.00 points to 9.12, because roughly a quarter of the raw gap was a head start rather than a treatment effect. That shrinkage is the adjustment purpose doing its work.

The power purpose works simultaneously and in the opposite direction. If the covariate correlates 0.75 with the outcome within arms, the residual variance falls from 100 to 100(1 - 0.75²) = 43.75, and the adjusted between-groups sum of squares becomes 80[(65.44 - 70)² + 0 + (74.56 - 70)²] = 3,327 on 2 degrees of freedom. The resulting statistic is F(2, 236) = 1,663 / 43.75 = 38.0, larger than the unadjusted 28.8 even though the estimated group differences got smaller. The covariate paid for itself by removing noise faster than it removed signal.

Key idea: ANCOVA shrank the estimated one-to-one advantage from 12.0 to 9.12 points while raising F from 28.8 to 38.0, because it removed more error variance than group difference.

The assumption ANCOVA rests on

ANCOVA's validity depends on homogeneity of regression slopes: the relationship between the covariate and the outcome must be the same in every group, meaning the covariate does not interact with the grouping factor. If the slopes differ, the single adjusted comparison is meaningless because the group gap depends on where you are along the covariate, and you have, in fact, discovered an interaction that should be modeled rather than adjusted away. A further caution: in observational data, using ANCOVA to "equate" pre-existing groups on a covariate can mislead, because groups that differ naturally may differ on unmeasured variables too. Adjustment is not randomization, a theme Module 6 develops.

Testing the assumption is straightforward, since you simply add the arm-by-covariate interaction to the model and examine its incremental F on 2 degrees of freedom for a three-arm design. A significant result does not mean the analysis has failed; it means the honest answer is a set of arm differences that vary along the covariate, reported as conditional effects in the manner of Lesson 8.

The deeper warning, argued forcefully by Miller and Chapman, concerns what adjustment means when groups differ systematically on the covariate for reasons bound up with what the groups are. Adjusting Brightwater arms for fall score is defensible if enrolment was essentially arbitrary with respect to baseline ability, but if strong readers deliberately chose one-to-one tutoring then the adjusted comparison describes a population of students who do not exist, namely one-to-one students with a control group's baseline. This is the essence of Lord's paradox, in which two defensible analyses of the same pretest-posttest data yield opposite conclusions because they answer different questions. Neither is a computational error, and no statistical test can adjudicate between them; only an argument about the design can.

Key idea: Adjusting for a covariate that is entangled with group membership produces a number about a hypothetical population, which is why ANCOVA in observational data is a causal claim requiring a causal argument.

Where people get stuck

• Reading a significant omnibus F as "all groups differ." It rejects only the hypothesis that every mean is identical, and the Brightwater Tukey threshold of 3.72 points is what actually licensed the three pairwise claims.
• Running all pairwise t-tests uncorrected. Three comparisons push the familywise error rate to 0.143 and fifteen comparisons push it to 0.537, so the correction is not bureaucratic caution.
• Interpreting main effects before the interaction. The 8-point dosage main effect averaged a 4-point effect and a 12-point effect and described neither arm.
• Assuming ANCOVA equates non-equivalent groups. It equates them on the measured covariate only, and Lord's paradox shows that the resulting comparison may answer a question nobody asked.
• Skipping the homogeneity-of-slopes check. If the covariate slope differs by arm, a single adjusted difference does not exist, and reporting one conceals an interaction.
• Reporting eta-squared as if unbiased. It overstates the population effect, especially in small samples, which is why omega-squared of 0.188 sits below the eta-squared of 0.196 here.

Key idea: Almost every ANOVA error is a claim that outruns the test, whether by naming which groups differ, averaging across an interaction, or treating adjustment as randomization.

Recap

• One-way ANOVA partitions variance into between-groups and within-groups pieces and forms F as the ratio of their mean squares, which is identical to the overall F of a regression on group dummies.
• Brightwater's three arms gave SS_between = 5,760 on 2 df, MS_within = 100 on 237 df, and F(2, 237) = 28.8, with eta-squared 0.196, omega-squared 0.188, and d = 1.2 for the extreme pair.
• Tukey's threshold of 3.72 points identified all three pairwise differences, and uncorrected testing would have carried a familywise error rate of 0.143 for three comparisons or 0.537 for fifteen.
• The two-by-two design gave F values of 21.6, 38.4, and 9.6 for format, dosage, and their interaction, and the significant interaction made the 8-point dosage main effect an average of a 4-point and a 12-point effect.
• ANCOVA adjusted the arm means to 65.44, 70.00, and 74.56, shrinking the largest gap from 12.00 to 9.12 points.
• Because the covariate cut residual variance from 100 to 43.75, the adjusted F rose to 38.0 despite the smaller estimated differences.
• ANCOVA requires homogeneity of regression slopes, and in observational data it makes a causal claim that Lord's paradox shows can be argued either way without an appeal to design.

Sources

1. Miller, G. A., & Chapman, J. P. (2001). Misunderstanding analysis of covariance. Journal of Abnormal Psychology, 110(1), 40-48. pubmed.ncbi.nlm.nih.gov
2. Vickers, A. J., & Altman, D. G. (2001). Statistics notes: Analysing controlled trials with baseline and follow up measurements. BMJ, 323(7321), 1123-1124. pmc.ncbi.nlm.nih.gov
3. Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. pmc.ncbi.nlm.nih.gov
4. NIST/SEMATECH. (2012). Are the means equal? In e-Handbook of statistical methods. itl.nist.gov
5. Penn State Eberly College of Science. (n.d.). STAT 502: Analysis of variance and design of experiments. online.stat.psu.edu
6. Navarro, D. (2019). Comparing several means (one-way ANOVA). In Learning statistics with R. learningstatisticswithr.com
7. Gelman, A., Hill, J., & Vehtari, A. (2020). Regression and other stories. Cambridge University Press. avehtari.github.io

Many outcomes are not continuous: a patient recovers or does not, a student graduates or does not, a loan defaults or does not. Forcing a 0/1 outcome into ordinary regression fails badly, and repairing it introduces the wider framework of the generalized linear model, of which OLS is one member.

Brightwater supplies the natural example, because the district cares less about a student's exact score than about whether she cleared the proficiency benchmark of 75 points. Recoding spring achievement as pass or fail throws away information, which is a real cost, but it produces the outcome that funding decisions actually turn on, and it forces us to confront a model whose coefficients cannot be read the way regression coefficients are read.

Key idea: Logistic regression keeps the familiar linear predictor and changes only what that predictor is a linear function of, which is why every interpretive difficulty in the model concerns the scale rather than the structure.

In plain terms

A probability is trapped between 0 and 1, but a straight line is not, so any attempt to predict a probability with a straight line will eventually predict something impossible. The trick logistic regression uses is to stop modelling the probability directly and start modelling a transformed version of it that has no ceiling and no floor.

The transformation runs in two steps. First convert the probability into odds, which is the chance of the event divided by the chance of no event, so a 50 percent probability becomes odds of 1 and a 90 percent probability becomes odds of 9. Then take the natural logarithm, which stretches the odds scale symmetrically in both directions and lets a straight line run through it forever without ever implying an impossible probability. The cost of this trick is that the coefficients now describe movement on the log-odds scale, which nobody has any intuition about, and translating them back into something a reader can picture is most of the work of this lesson.

Why OLS breaks on a binary outcome

Regressing a 0/1 outcome directly (the "linear probability model") violates the framework in ways that matter: predictions can fall below 0 or above 1, which are impossible probabilities; the errors are necessarily heteroscedastic (their variance depends on the predicted probability); and the linear form poorly captures a relationship that must saturate near 0 and 1. We need a model that keeps predictions in [0, 1] and respects the binary nature of the data.

Both failures are easy to see numerically. Suppose OLS on the Brightwater pass indicator returns Pass = -1.30 + 0.035(fall score). A student with a fall score of 30 is then predicted to pass with probability -0.25, and a student with a fall score of 80 with probability 1.50, neither of which is a number. Meanwhile the variance of a binary outcome is fixed by its own mean at p(1 - p), so it equals 0.25 where the predicted probability is 0.50 but only 0.09 where the predicted probability is 0.10, a spread of nearly three to one that no OLS standard error accounts for.

Key idea: A binary outcome carries its own variance function, p(1 - p), so heteroscedasticity is not an accident of the data but a mathematical certainty of the model.

The logistic solution: link and odds

Logistic regression models not the outcome directly but the log-odds of the outcome, which are unbounded and so can be a linear function of the predictors. Define the odds of an event with probability p as p/(1 minus p). The natural log of the odds is the logit. The model is:

ln( p / (1 - p) ) = b₀ + b₁X₁ + ... + b_kX_k

The right side is the familiar linear predictor; the left side, the logit, is the link function that connects it to a probability. Solving back for p gives the S-shaped logistic (sigmoid) curve that stays neatly between 0 and 1. Because the outcome is not normal, coefficients are estimated by maximum likelihood rather than least squares: we choose the coefficients that make the observed pattern of successes and failures most probable.

Reading the coefficients: log-odds and odds ratios

A raw logistic coefficient b_j is the change in the log-odds of the outcome per one-unit increase in X_j. Log-odds are not intuitive, so we exponentiate: e^b_j is the odds ratio (OR), the multiplicative change in the odds per one-unit increase in the predictor. Reading it:

• OR = 1 (equivalently b = 0): the predictor does not change the odds.
• OR > 1 (b positive): the odds increase; OR = 1.5 means the odds are 1.5 times as large, a 50 percent increase per unit.
• OR < 1 (b negative): the odds decrease; OR = 0.8 means the odds are multiplied by 0.8, a 20 percent reduction per unit.

Two interpretive cautions are essential at the doctoral level. First, an odds ratio is not a risk ratio. Odds and probabilities coincide only when the outcome is rare; for common outcomes an odds ratio can be much larger than the corresponding relative risk, and reporting "1.5 times more likely" for an OR of 1.5 is often wrong.

Second, the effect on the probability is not constant: because the curve is S-shaped, the same one-unit change moves probability a lot in the middle of the curve and little near 0 or 1. Overall model fit is judged not by R² but by likelihood-based measures such as the likelihood-ratio test (comparing nested models) and information criteria, with pseudo-R² measures reported cautiously as approximate analogs.

Worked example: one odds ratio, three different probability jumps

Suppose the Brightwater model for passing the 75-point benchmark is logit(p) = -1.20 + 0.40(tutoring hours), so the odds ratio is e^0.40 = 1.49 and every additional weekly hour multiplies the odds of passing by that same factor. Now convert to probabilities at three places on the curve, remembering that p = odds / (1 + odds).

• 0 to 1 hour. The logit moves from -1.20 to -0.80, so the odds move from 0.301 to 0.449 and the probability moves from 0.232 to 0.310, a rise of 7.8 percentage points.
• 3 to 4 hours. The logit moves from 0.00 to 0.40, so the odds move from 1.000 to 1.492 and the probability moves from 0.500 to 0.599, a rise of 9.9 percentage points.
• 6 to 7 hours. The logit moves from 1.20 to 1.60, so the odds move from 3.320 to 4.953 and the probability moves from 0.769 to 0.832, a rise of 6.3 percentage points.

One coefficient, one odds ratio, three different practical effects. The largest movement occurs where the predicted probability sits nearest 0.50, which is exactly what the S-shaped curve implies, and the calculus confirms it: the marginal effect on probability is b × p(1 - p), which at p = 0.50 equals 0.40 × 0.25 = 0.10, matching the 9.9-point jump computed directly.

This is why reporting an odds ratio alone is an incomplete answer to a policy question. A superintendent asking "how many more children will pass" needs predicted probabilities at realistic covariate values, or a marginal effect, not a multiplicative factor on a scale nobody experiences.

Key idea: The odds ratio is constant across the whole predictor range precisely because the probability effect is not, so the two must both be reported.

Worked example: how far an odds ratio can sit from a risk ratio

Take the 3-to-4-hour comparison above, where the probability of passing rose from 0.500 to 0.599. The risk ratio is simply 0.599 / 0.500 = 1.20, while the odds ratio is 1.49, so describing this result as "students were 1.49 times as likely to pass" overstates the relative increase by roughly a quarter.

Now repeat the calculation for a rare outcome. Suppose the event is failing to graduate at all, with a baseline probability of 0.02 and therefore baseline odds of 0.0204. Applying the same odds ratio of 1.49 gives new odds of 0.0304 and hence a new probability of 0.0295, so the risk ratio is 0.0295 / 0.02 = 1.48, essentially indistinguishable from the odds ratio of 1.49.

That contrast is the whole rule. Odds and probabilities converge when events are rare, so an odds ratio approximates a risk ratio only in that regime, and for common outcomes such as passing a benchmark that half the district clears, the approximation fails badly. Journals in medicine and epidemiology now flag this explicitly, and a doctoral write-up should either report the risk ratio directly or say "odds" and mean it.

Key idea: An odds ratio of 1.49 corresponded to a risk ratio of 1.20 for a 50 percent outcome and 1.48 for a 2 percent outcome, so the phrase "times more likely" is only safe when the event is rare.

Judging fit, and how much data you need

Because there is no sum of squares to partition, fit is assessed through the likelihood. Suppose the Brightwater sample splits evenly, 120 passing and 120 failing, so the null model's deviance is -2 × 240 × ln(0.5) = 332.7, and suppose adding tutoring hours reduces the deviance to 298.4. The likelihood-ratio chi-square is then 332.7 - 298.4 = 34.3 on 1 degree of freedom, which is decisive, and McFadden's pseudo-R-squared is 1 - (298.4 / 332.7) = 0.103.

Calibrate that last number carefully, since McFadden values are not comparable to OLS R-squared values and 0.2 to 0.4 already represents excellent fit for this index. Reporting "the model explained only 10 percent of the variance" would misdescribe a strong result, which is why pseudo-R-squared should always be named rather than presented as R-squared.

Sample size in logistic regression is governed by the number of events, not the number of cases, because a study with 5,000 students and 12 failures carries the information of 12 failures. The traditional guidance from simulation work is at least 10 events per predictor, which with 120 passes and 120 failures would license roughly 12 predictors here; more recent work suggests 5 to 9 events per variable is often adequate, while very sparse designs remain genuinely unreliable. Note also that maximum likelihood fails outright under separation, where some predictor perfectly divides passes from failures, producing coefficients that run off to infinity with enormous standard errors.

Key idea: The information in a logistic regression is carried by the rarer of the two outcomes, so count events rather than cases when you plan the study.

Where people get stuck

• Reporting an odds ratio as "times more likely." The Brightwater odds ratio of 1.49 corresponded to a 20 percent increase in the chance of passing, not a 49 percent one.
• Expecting a constant probability effect. The identical coefficient produced jumps of 7.8, 9.9, and 6.3 percentage points at three points on the same curve.
• Comparing odds ratios across models. Unlike linear coefficients, logistic coefficients rescale when covariates are added even if those covariates are unrelated to the predictor, so a change between nested models is not straightforwardly confounding.
• Reading pseudo-R-squared as variance explained. McFadden's 0.103 here signals a solid model, and treating it on an OLS scale would misreport the finding.
• Planning sample size from n rather than events. With 240 students and only 15 failures, the effective constraint is 15, which permits at most one or two predictors under any rule of thumb.
• Ignoring separation warnings. Infinite coefficients and standard errors in the thousands mean the model has not converged, not that the predictor is extraordinarily powerful.

Key idea: Every recurring logistic error is a scale error, caused by carrying linear-regression intuitions onto the log-odds scale where they do not apply.

Recap

• OLS on a binary outcome predicts impossible probabilities, and the worked model gave -0.25 and 1.50 at fall scores of 30 and 80.
• Binary outcomes are inherently heteroscedastic because their variance is p(1 - p), which falls from 0.25 at p = 0.50 to 0.09 at p = 0.10.
• Logistic regression models the log-odds as a linear function of the predictors and is estimated by maximum likelihood rather than least squares.
• With logit(p) = -1.20 + 0.40(hours), the odds ratio is e to the 0.40 = 1.49, constant everywhere.
• That same coefficient moved probability by 7.8, 9.9, and 6.3 percentage points at the 0-to-1, 3-to-4, and 6-to-7 hour comparisons, matching the marginal-effect formula b times p(1 - p).
• The odds ratio of 1.49 corresponded to a risk ratio of 1.20 for a 50 percent outcome but 1.48 for a 2 percent outcome, which is why the approximation requires rare events.
• Fit uses the likelihood-ratio chi-square, here 332.7 minus 298.4 = 34.3 on 1 df, with McFadden pseudo-R-squared of 0.103 that must not be read on an OLS scale.
• Sample size is governed by the count of events, with roughly 5 to 10 events per predictor as working guidance, and separation signals non-convergence rather than a strong effect.

Sources

1. Sperandei, S. (2014). Understanding logistic regression analysis. Biochemia Medica, 24(1), 12-18. pmc.ncbi.nlm.nih.gov
2. Davies, H. T. O., Crombie, I. K., & Tavakoli, M. (1998). When can odds ratios mislead? BMJ, 316(7136), 989-991. bmj.com
3. Norton, E. C., Dowd, B. E., & Maciejewski, M. L. (2018). Odds ratios: Current best practice and use. JAMA, 320(1), 84-85. pubmed.ncbi.nlm.nih.gov
4. Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996). A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 49(12), 1373-1379. pubmed.ncbi.nlm.nih.gov
5. Vittinghoff, E., & McCulloch, C. E. (2007). Relaxing the rule of ten events per variable in logistic and Cox regression. American Journal of Epidemiology, 165(6), 710-718. pubmed.ncbi.nlm.nih.gov
6. UCLA Office of Advanced Research Computing. (n.d.). Logistic regression: Data analysis examples. Statistical Methods and Data Analytics. stats.oarc.ucla.edu
7. Penn State Eberly College of Science. (n.d.). STAT 504: Analysis of discrete data. online.stat.psu.edu

Module 3 introduced interactions mechanically. Here we treat them as a substantive research question: moderation asks whether the strength or direction of an X-to-Y relationship depends on a third variable, the moderator. The question "does it always work, or only for some people or in some conditions?" is a moderation question, and it is often more scientifically interesting than the average effect.

Moderation questions also carry a hazard that the arithmetic conceals, since interactions are estimated far less precisely than main effects, and a literature full of underpowered moderation tests will be a literature full of exaggerated and irreproducible moderators. This lesson therefore treats estimation, probing, and power as one continuous problem rather than three separate ones.

Key idea: Asking whether an effect holds everywhere is scientifically richer than asking whether it exists on average, but it is also the more expensive question in sample size.

In plain terms

Brightwater's average finding is that tutoring is worth 2.2 points per weekly hour, but averages hide variation, and the district's real question is whether that figure describes every child or only the typical one. Moderation asks whether the payoff differs between students who arrived reading well and students who arrived struggling.

The model answers by letting the tutoring slope itself depend on the fall score, so instead of one number describing everybody, you get a formula that produces a different number for each kind of student. Reading the result then means substituting a few representative fall scores into that formula and seeing what tutoring is worth to each. The picture that emerges is what you report, since a coefficient on a product term communicates almost nothing on its own.

Moderation is an interaction

A moderator M changes the effect of X on Y. Statistically, moderation is the interaction term X·M from Module 3; "moderation" is the conceptual name, "interaction" the statistical mechanism. The regression is:

Y = b₀ + b₁X + b₂M + b₃(X·M) + e

The test of moderation is the test of b₃. If b₃ differs significantly from zero, the slope of X on Y changes across levels of M. If b₃ is estimated near zero with a confidence interval tight enough to exclude anything of interest, the effect of X is the same regardless of M, whatever the theory hoped.

That qualification matters more than it may appear. A non-significant interaction is not evidence of no moderation, because interaction tests are chronically underpowered and a wide interval routinely spans everything from a substantial positive moderation to a substantial negative one. Reporting "the interaction was not significant, so the effect was uniform across participants" mistakes an inconclusive test for a null finding, which is the specific error the ASA statement on p-values warns about. The defensible report gives b₃, its interval, and a sentence saying which moderating effects the data can and cannot rule out.

Key idea: A non-significant interaction licenses "we could not detect moderation," never "there is no moderation," and only the confidence interval distinguishes the two.

Direction of moderation

The sign of b₃ tells you the form. Recall the conditional slope of X is (b₁ + b₃M). A positive b₃ means the effect of X grows stronger (more positive) as M increases; a negative b₃ means it weakens or reverses as M increases. A moderator can even flip an effect's sign: a coping strategy that helps under low stress might hurt under high stress, which no single average effect could reveal.

Probing with simple slopes

A significant interaction says the slopes differ but not what they are, so you probe it. The standard approach, the pick-a-point or simple-slopes method, evaluates the slope of X at chosen values of M, conventionally the mean and one standard deviation above and below (labeled, loosely, low, average, and high M). Each simple slope has its own estimate and significance test.

A common and informative pattern is a significant conditional effect at high M but not at low M, which you would report as "X predicted Y among high-M participants but not among low-M participants." A more complete alternative, the Johnson-Neyman technique, identifies the entire range of M values over which the effect of X is statistically significant, rather than testing only two or three arbitrary points.

Worked example: simple slopes with their own standard errors

Return to the centered Brightwater model, Spring = 70.0 + 2.2(hours) + 0.72(fall) - 0.10(hours × fall), and suppose the interaction coefficient carries a standard error of 0.032, giving t = -3.13 and a 95 percent interval from -0.163 to -0.037. The interaction is real and negative, so tutoring matters more to weaker fall readers.

Each simple slope needs its own standard error, and the formula combines the uncertainty in both coefficients: SE of the slope at M equals the square root of [Var(b₁) + M²Var(b₃) + 2M × Cov(b₁, b₃)]. With centered predictors the covariance term is near zero, so with Var(b₁) = 0.33² = 0.1089 and Var(b₃) = 0.032² = 0.001024 the three conventional probe points give:

• Low fall score (M = -11, raw 51): slope = 2.2 + 1.1 = 3.30; SE = square root of (0.1089 + 121 × 0.001024) = 0.483; t = 6.84; interval 2.35 to 4.25.
• Average fall score (M = 0, raw 62): slope = 2.20; SE = 0.330; t = 6.67; interval 1.55 to 2.85.
• High fall score (M = +11, raw 73): slope = 2.2 - 1.1 = 1.10; SE = 0.483; t = 2.28; interval 0.15 to 2.05.

Three features of this table deserve comment. All three slopes are positive and statistically distinguishable from zero, so nobody is harmed by tutoring, but the high-fall-score interval very nearly touches zero and its width is essentially the same as the low-score interval despite describing a slope one-third the size. Notice also that the standard errors at the probe points are larger than the standard error at the mean, because moving away from the centre compounds the uncertainty in the interaction term. Conditional effects at the extremes of a moderator are always estimated less precisely, which is the reason simple slopes evaluated far from the data should not be reported at all.

Key idea: A simple slope is an estimate with its own standard error that grows as you move away from the moderator's mean, so precision degrades exactly where the substantive interest usually lies.

Worked example: the Johnson-Neyman region of significance

Probing at three arbitrary points leaves an obvious question unanswered: where exactly does the tutoring effect stop being distinguishable from zero? Johnson-Neyman answers it by solving for the moderator values at which the ratio of the conditional slope to its standard error equals the critical value of about 1.97.

Setting (2.2 - 0.10M)² equal to 1.97² × (0.1089 + 0.001024M²) and rearranging produces a quadratic whose relevant root is M = 12.0. In raw units that is a fall reading score of 62 + 12 = 74.

Now the finding can be stated in a sentence a school board can act on: tutoring produced a detectable benefit for every student who entered the year reading below 74 points, and for students above that threshold the data cannot distinguish its effect from zero. Verify the boundary by checking either side, since at M = 11 the ratio is 1.10 / 0.483 = 2.28 and significant, while at M = 13 it is 0.90 / 0.531 = 1.70 and not.

Report the region of significance together with how much of the sample falls inside it, because a threshold at the 84th percentile of fall scores means the effect held for the great majority of students, whereas a threshold at the 20th percentile would mean something entirely different.

Key idea: Johnson-Neyman converts "the interaction was significant" into "the effect held for students below a fall score of 74," which is the form a decision-maker can use.

Why interactions need so much more data

Interaction estimates are intrinsically noisier than main-effect estimates, and the arithmetic of a balanced two-by-two design shows why. The main-effect contrast averages two cells against two cells, whereas the interaction contrast is a difference of differences, so its standard error is roughly twice as large. Because required sample size scales with the square of the standard error, detecting an interaction of the same magnitude as a main effect takes about four times the sample, and detecting one half that size takes about sixteen times the sample.

Two consequences follow, and both are visible throughout the moderation literature. Underpowered studies miss real moderation constantly, which is why a null interaction should be reported as inconclusive rather than negative. And when an underpowered study does reach significance, the estimate it reports must be large to have crossed the threshold at all, so published interaction effects are systematically exaggerated. Add the fact that most datasets contain a dozen plausible moderators, and moderation analysis becomes one of the easiest places in applied statistics to generate an irreproducible finding without any conscious dishonesty.

The protections are the ones this course keeps returning to. Specify the moderator in advance, report the interval rather than the verdict, treat unplanned moderator hunting as explicitly exploratory, and expect a genuine moderation finding to require a sample several times larger than the one that established the main effect.

Key idea: Detecting an interaction half the size of a main effect needs roughly sixteen times the sample, which makes underpowered moderation the single most reliable source of exaggerated findings.

Two disciplines that prevent errors

First, center (or standardize) X and M before forming the product, so the lower-order coefficients are effects at the mean rather than at a possibly meaningless zero, as Module 3 stressed. Second, always plot the interaction: draw the predicted line of Y on X separately for low and high M. The plot converts an abstract product coefficient into a picture a reader can grasp, and it guards against overinterpreting a statistically significant but substantively tiny interaction. Report b₃, its confidence interval, the simple slopes, and the plot together; any one alone can mislead.

Key idea: Centring makes the coefficients readable and plotting makes the finding communicable, and neither substitutes for reporting the interval around the interaction itself.

Where people get stuck

• Treating a null interaction as evidence of uniformity. Given how imprecisely interactions are estimated, a non-significant b₃ usually means the study could not tell, and the interval is what shows which moderating effects remain live.
• Comparing two subgroup p-values instead of testing the interaction. Finding p = .01 in one group and p = .12 in another does not establish that the groups differ, and only the product term addresses that question.
• Probing at moderator values nobody has. A simple slope one standard deviation above the mean is meaningful when data exist there and is pure extrapolation when they do not, and its standard error inflates in either case.
• Hunting for moderators after seeing the results. With a dozen candidate moderators available in most datasets, something will interact, and presenting the survivor as a planned hypothesis is the HARKing that Lesson 17 addresses.
• Reading a significant interaction as a large one. The Brightwater interaction of -0.10 shifts the tutoring slope by only 1.1 points across a two-standard-deviation swing in the moderator, which the plot makes obvious and the p-value conceals.
• Confusing moderation with mediation. Moderation says the effect differs by condition, mediation says it travels through a mechanism, and the next lesson separates them carefully.

Key idea: Moderation claims fail most often on precision and on preregistration, not on arithmetic.

Recap

• Moderation is the substantive name for a statistical interaction, and testing it means testing the coefficient on the product term.
• The conditional slope of X is b₁ + b₃M, so a negative b₃ means the effect weakens as the moderator rises.
• A non-significant interaction is inconclusive rather than negative, and only the interval separates "no moderation" from "we could not detect it."
• With b₃ = -0.10 (SE 0.032), the Brightwater tutoring slope was 3.30 at a fall score of 51, 2.20 at 62, and 1.10 at 73.
• Simple slopes carry their own standard errors, 0.483 at the probe points versus 0.330 at the mean, so precision falls as you move away from the moderator's centre.
• Johnson-Neyman located the boundary at a fall score of 74, above which the tutoring effect was not distinguishable from zero.
• Interactions require roughly four times the sample for an effect the size of a main effect and sixteen times for one half that size, which is why published moderation findings are systematically exaggerated.

Sources

1. Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Sage Publications. find source ↗
2. Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011). Erroneous analyses of interactions in neuroscience: A problem of significance. Nature Neuroscience, 14(9), 1105-1107. nature.com
3. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafo, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376. nature.com
4. Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460-465. sites.stat.columbia.edu
5. Kenny, D. A. (2018). Moderation. davidakenny.net
6. Hayes, A. F. (n.d.). The PROCESS macro for SPSS, SAS, and R. processmacro.org
7. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. amstat.org

Where moderation asks when or for whom an effect holds, mediation asks how or why: it posits a mechanism, an intervening variable through which X transmits its effect to Y. This is among the most used and most abused analyses in the social sciences, so precision here is essential.

The abuse is easy to describe. Three regressions run on cross-sectional data produce a number, the number is labelled an indirect effect, and a mechanism is announced, even though nothing in the arithmetic distinguishes the proposed ordering from two or three rival orderings that would fit the same correlations equally well. The statistics are straightforward; the warrant for calling them a mechanism is not.

Key idea: Mediation analysis estimates a quantity that is only a mechanism if the causal ordering is true, and the ordering is established by design and argument, never by the model.

In plain terms

Brightwater knows that tutoring raises reading scores, and now wants to know why. One plausible story is that tutoring gets children reading more on their own, and it is the extra independent reading, rather than the tutoring session itself, that produces the gain.

Testing that story means splitting the total effect into two channels. One channel runs through independent reading, since tutoring increases reading minutes and reading minutes increase scores, and multiplying those two links gives the size of the channel. The other channel is whatever remains, the part of the tutoring effect that arrives by some route the model has not named. If the first channel is large and the second is small, the story looks good, but "looks good" is doing a great deal of work in that sentence.

The mediation model

A mediator M lies on the causal path X to M to Y. The classic decomposition uses three coefficients: the effect of X on M (path a), the effect of M on Y controlling for X (path b), and the effect of X on Y controlling for M (path c', the direct effect). The indirect effect, the part of X's influence that runs through M, is the product a × b. The total effect of X on Y is:

c = c' + (a × b)    (total = direct + indirect)

When the indirect effect accounts for the entire relationship (c' near zero), the mediation is called full; when a direct effect remains alongside a nonzero indirect effect, it is partial. Modern practice de-emphasizes this labeling and focuses on estimating the indirect effect and its uncertainty.

Worked example: decomposing the Brightwater tutoring effect

Take independent reading as the proposed mediator, measured as minutes per week outside school, with every model adjusted for the fall reading score. Suppose the three regressions return:

• Path a = 12.0, with standard error 2.5: each additional weekly tutoring hour predicts 12 more minutes of independent reading.
• Path b = 0.09, with standard error 0.025: each additional weekly minute of independent reading predicts 0.09 more spring points, holding tutoring constant.
• Path c' = 1.12: the tutoring effect that survives once independent reading is in the model.

The indirect effect is the product, a × b = 12.0 × 0.09 = 1.08 points per tutoring hour, and the total effect reassembles exactly as c = c' + ab = 1.12 + 1.08 = 2.20, which is the adjusted tutoring coefficient from Lesson 4. About 1.08 / 2.20 = 49 percent of the tutoring effect travels through independent reading, at least under the assumed ordering.

Read the two channels substantively, because they imply different policies. If half the benefit arrives because tutored children read more on their own, then a cheaper intervention that increases independent reading directly might capture a large share of the gain, whereas the remaining 1.12 points require something the tutoring session itself provides. That is the kind of question mediation exists to raise, and the kind of answer it cannot deliver on its own.

Key idea: The indirect effect of 1.08 and the direct effect of 1.12 sum to the total effect of 2.20, so the decomposition is exact arithmetic even when the causal story behind it is speculative.

From causal steps to the indirect effect

An older approach (the Baron and Kenny causal-steps method) required a chain of separate significance tests, including a significant total effect before mediation could be claimed. This has fallen out of favor for good reasons: it has low power, and an indirect effect can exist even when the total effect is non-significant (for instance, if a mediated positive path and a direct negative path partly cancel). Contemporary practice tests the indirect effect a×b directly.

Because the product of two normally distributed coefficients is not itself normal, the preferred method is bootstrapping: repeatedly resample the data, recompute a×b in each resample, and use the resulting distribution to form a confidence interval. If that interval excludes zero, the indirect effect is significant. Bootstrapping respects the skewed sampling distribution of the product and generally has more power than the older normal-theory (Sobel) test.

One refinement of that advice has firmed up since the method became standard. Bias-corrected bootstrap intervals were recommended for years on power grounds, but simulation work has repeatedly found that they carry Type I error rates above the nominal level, sometimes substantially so. Current guidance therefore favours the plain percentile bootstrap, which trades a little power for accurate error control, and a doctoral report should name which variant it used rather than saying "bootstrapped" and leaving the reader to guess.

Worked example: three ways to test the same indirect effect

The Sobel test approximates the standard error of the product by the delta method, as the square root of (b² times the variance of a, plus a² times the variance of b). With the Brightwater numbers:

SE = square root of [(0.09)²(2.5)² + (12.0)²(0.025)²] = square root of (0.0506 + 0.0900) = square root of 0.1406 = 0.375

That gives z = 1.08 / 0.375 = 2.88 and a symmetric 95 percent interval of 1.08 plus or minus 1.96(0.375), running from 0.345 to 1.815. The symmetry is the giveaway that something is wrong, since the sampling distribution of a product is genuinely skewed and no symmetric interval can represent it faithfully.

A percentile bootstrap with 5,000 resamples might instead return an interval of roughly 0.42 to 1.84, visibly shifted and asymmetric around the point estimate of 1.08. It excludes zero, so the indirect effect is supported, and its shape carries information the Sobel interval discarded. Note what neither method establishes: both are inferences about a parameter in an assumed causal ordering, and neither tests the ordering.

A third scenario shows why the old causal-steps requirement was abandoned. Suppose a and b are unchanged, so the indirect effect remains +1.08, but tutoring also displaces free-choice activities in a way that costs 1.08 points directly, making c' = -1.08 and the total effect c = 0.00 exactly. Under Baron and Kenny's rule the analysis would stop at step one for want of a significant total effect, and a genuine mechanism operating in both directions would go unreported. This inconsistent mediation pattern is precisely why the modern approach tests a × b directly.

Key idea: Test the product directly with a percentile bootstrap, because the Sobel interval imposes a symmetry the product does not have and the causal-steps gate hides mechanisms that cancel.

The assumptions people forget

Mediation is a causal claim dressed in regression clothing, and the statistics do not supply the causation. Three assumptions are routinely violated and rarely acknowledged:

• Correct causal order. The model assumes X causes M causes Y. If M and Y are measured at the same time, or if Y could influence M, the estimated "mediation" may be an artifact of a misspecified order. Establishing temporal precedence (X before M before Y) is a design task, not a statistical one.
• No unmeasured confounding of the M-to-Y relationship in particular. Even in an experiment that randomizes X, the mediator M is not randomized; any variable that affects both M and Y confounds path b and biases the indirect effect. Randomizing X does not rescue this.
• No measurement error in the mediator, which otherwise biases the paths, typically attenuating b.

The honest reporting standard is therefore to present mediation from cross-sectional data as a model-consistent account, not proof of mechanism, and to note that alternative causal orderings may fit the same data equally well. A significant bootstrapped indirect effect is necessary for the mediation story, but it is far from sufficient to establish it.

Worked example: how much the forgotten assumptions cost

Put magnitudes on two of those three assumptions, because their consequences are usually described qualitatively and then ignored.

First, confounding of the b path. Suppose an unmeasured home literacy environment raises both independent reading and spring achievement, and that its omission inflates the estimated b from a true 0.05 to the observed 0.09. The indirect effect then reports 12.0 × 0.09 = 1.08 when the truth is 12.0 × 0.05 = 0.60, an overstatement of 80 percent. Randomizing tutoring would have done nothing to prevent this, because randomization balanced the determinants of X and left the determinants of M untouched.

Second, measurement error in the mediator. Independent reading minutes are self-reported by nine-year-olds, so suppose the measure has a reliability of 0.70. Random error in a predictor attenuates its coefficient, so an observed b of 0.09 is consistent with a true value nearer 0.09 / 0.70 = 0.129, and the indirect effect is understated by roughly 30 percent while the direct effect c' absorbs the difference and is correspondingly overstated. Notice that the two biases here run in opposite directions and do not cancel in any predictable way.

The lesson is not that mediation should be avoided but that its estimates carry uncertainty far beyond the confidence interval printed beside them. Sensitivity analysis, which asks how strong an unmeasured confounder of the b path would have to be to reduce the indirect effect to zero, is now expected in careful work and is far more informative than another decimal place on the bootstrap interval.

Key idea: A plausible unmeasured confounder inflated the indirect effect by 80 percent and plausible measurement error deflated it by 30 percent, so the bootstrap interval describes the smallest of the uncertainties involved.

Where people get stuck

• Reading a significant indirect effect as evidence of mechanism. The arithmetic is exact and the causal ordering is an assumption, so the same three correlations support several stories the model cannot distinguish.
• Measuring X, M, and Y at one time point. Temporal precedence is a design property, and a cross-sectional mediation model asserts an ordering that the data cannot corroborate.
• Believing randomization of X protects the whole model. It secures path a and leaves path b exposed to every confounder of mediator and outcome, which the 80 percent inflation above illustrates.
• Reporting "full mediation." The label depends on whether c' happened to clear a significance threshold, which is a function of sample size rather than of mechanism, and modern reporting has abandoned it.
• Requiring a significant total effect first. The inconsistent-mediation example produced c = 0.00 with a real indirect effect of +1.08, which the causal-steps gate would have suppressed.
• Using bias-corrected bootstrap intervals by default. Their Type I error rate exceeds the nominal level, and the percentile bootstrap is the safer current default.

Key idea: Mediation is a causal model estimated with correlational tools, so its credibility rests on design, timing, and sensitivity analysis rather than on the interval around a × b.

Recap

• Mediation posits X causes M causes Y, decomposing the total effect as c = c' + ab, where ab is the indirect effect and c' the direct effect.
• The Brightwater paths a = 12.0 and b = 0.09 give an indirect effect of 1.08, which with c' = 1.12 reassembles the total effect of 2.20, so 49 percent of the tutoring effect ran through independent reading.
• The Sobel standard error of 0.375 gives z = 2.88 and a symmetric interval of 0.345 to 1.815, but the product's distribution is skewed and a percentile bootstrap interval of about 0.42 to 1.84 represents it better.
• Bias-corrected bootstrap intervals have inflated Type I error rates, so the percentile bootstrap is the current default and the variant used should be named.
• An inconsistent-mediation case with c' = -1.08 and ab = +1.08 gives a total effect of exactly zero, which is why the causal-steps requirement of a significant total effect was abandoned.
• An unmeasured confounder of the mediator-outcome path inflated the indirect effect from a true 0.60 to an observed 1.08, and randomizing X would not have prevented it.
• Mediator reliability of 0.70 attenuates path b, so an observed 0.09 is consistent with a true 0.129, understating the indirect effect and overstating the direct one.

Sources

1. Preacher, K. J., & Hayes, A. F. (2008). Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behavior Research Methods, 40(3), 879-891. link.springer.com
2. Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research. Journal of Personality and Social Psychology, 51(6), 1173-1182. find source ↗
3. Kenny, D. A. (2021). Mediation. davidakenny.net
4. Hayes, A. F. (n.d.). The PROCESS macro for SPSS, SAS, and R. processmacro.org
5. Hernan, M. A., & Robins, J. M. (2020). Causal inference: What if. Chapman & Hall/CRC. miguelhernan.org
6. Hitchcock, C., & Redei, M. (2021). Causal models. Stanford Encyclopedia of Philosophy. plato.stanford.edu
7. Pearl, J. (2009). Causal inference in statistics: An overview. Statistics Surveys, 3, 96-146. projecteuclid.org

Module 3 flagged that clustered data violate the independence-of-errors assumption. This lesson develops the remedy: multilevel models (also called hierarchical linear models, mixed-effects models, or random-effects models), which are indispensable whenever observations are nested.

The Brightwater design has been hiding this problem since Lesson 1, because the 240 students sit inside 12 schools, and children in the same school share a principal, a curriculum, a neighbourhood, and a staffroom culture. Every standard error quoted so far has quietly assumed that a student in one school tells you nothing about a student in another beyond what the predictors say, and that assumption is false.

Key idea: Clustered data contain less independent information than the case count suggests, so the first task is to measure how much less and the second is to model it.

In plain terms

Suppose you want to know the average reading level of a district and you can afford to test 240 children. Testing 240 children drawn from 240 different schools would tell you far more than testing 20 children from each of 12 schools, because in the second design the children keep repeating the same schools back at you.

A multilevel model handles this by giving each school its own baseline and treating the collection of school baselines as a distribution to be estimated rather than as twelve unrelated facts. That single change accomplishes three things at once: standard errors stop pretending to information the design never collected, school-level predictors become askable, and estimates for small or noisy schools get pulled toward the district average instead of being trusted at face value.

Nested data and why independence fails

Data are nested (hierarchical) when lower-level units cluster within higher-level units: students within classrooms within schools, patients within hospitals, repeated measurements within persons. Units in the same cluster tend to resemble one another, so their residuals are correlated, violating independence. Ignoring the nesting and running ordinary regression produces standard errors that are too small, and hence false confidence and inflated Type I error, because the analysis counts correlated observations as if each contributed full independent information. Treating the cluster mean as the unit instead (aggregating) discards within-cluster variation and can produce the ecological fallacy, inferring individual relationships from group-level associations.

The intraclass correlation

The degree of clustering is quantified by the intraclass correlation coefficient (ICC), the proportion of total variance that lies between clusters:

ICC = σ²_between / (σ²_between + σ²_within)

An ICC near zero means clusters barely differ and ordinary regression is roughly acceptable; a substantial ICC signals meaningful clustering that must be modeled. Even a modest ICC, combined with large clusters, can drastically inflate naive significance, so the ICC is not a formality but a decision input.

Worked example: the design effect in Brightwater

Fit an empty multilevel model to the 240 spring scores and suppose it partitions the total variance of 144 into a between-school component of 18 and a within-school component of 126. The intraclass correlation is therefore:

ICC = 18 / (18 + 126) = 18 / 144 = 0.125

Twelve and a half percent of the variation in reading lies between schools rather than between students, which sounds unremarkable until you compute what it does to precision. The design effect converts an ICC and a cluster size into a multiplier on the sampling variance:

Design effect = 1 + (m - 1) × ICC = 1 + (20 - 1)(0.125) = 3.375

The effective sample size is therefore 240 / 3.375 = 71 students, not 240. Standard errors from an ordinary regression are too small by a factor of the square root of 3.375, which is 1.84, so the tutoring standard error of 0.33 quoted throughout this course should be nearer 0.61 once clustering is respected. The t statistic falls from 6.58 to about 3.6, which still supports the finding but represents a very different quantity of evidence.

Two comparisons calibrate the intuition. Halving the ICC to 0.05 gives a design effect of 1 + 19(0.05) = 1.95 and an effective sample of 123, so small ICCs still matter. Holding the ICC at 0.05 but raising the cluster size to 100 gives a design effect of 1 + 99(0.05) = 5.95, which is worse than the original scenario despite a much smaller ICC. Cluster size and ICC multiply, which is why studies of pupils within large schools or patients within busy hospitals cannot ignore nesting on the grounds that the correlation looks small.

Key idea: An ICC of 0.125 with 20 students per school cut 240 observations down to an effective 71, and doubling cluster size hurts more than doubling the ICC.

Random intercepts and random slopes

A multilevel model partitions the coefficients into fixed and random parts.

• A random-intercept model lets each cluster have its own baseline level of the outcome, drawn from a distribution of cluster intercepts. It says: schools differ in average achievement, and we model that spread rather than pretending it away.
• A random-slope model additionally lets the effect of a predictor vary across clusters. It says: the relationship between, say, study time and achievement may be stronger in some schools than others, and that variation is itself of interest.

The fixed effects are the average coefficients across clusters; the random effects are the cluster-specific departures, summarized by their variances (the variance components). A great strength of the approach is partial pooling: estimates for small or noisy clusters are pulled (shrunk) toward the overall mean, borrowing strength across clusters and yielding more stable, less overfit cluster estimates than analyzing each cluster separately.

Worked example: shrinkage in action

Partial pooling is not a vague metaphor but a specific weighted average, and the weight is computable. A cluster's estimate is pulled toward the grand mean by the factor lambda = tau-squared / (tau-squared + sigma-squared / n_j), where tau-squared is the between-cluster variance, sigma-squared the within-cluster variance, and n_j the number of cases in that cluster.

Take two Brightwater schools that both recorded a mean spring score of 76 against a district mean of 70. The first tested all 20 of its fourth-graders:

lambda = 18 / (18 + 126/20) = 18 / 24.3 = 0.741, so the estimate becomes 70 + 0.741(6) = 74.4.

The second is a small rural school with only 5 fourth-graders:

lambda = 18 / (18 + 126/5) = 18 / 43.2 = 0.417, so the estimate becomes 70 + 0.417(6) = 72.5.

Identical raw averages, different model estimates, because the second school's average rests on five children and is therefore mostly noise. This is exactly the behaviour a thoughtful analyst would want, and it is the reason league tables built from raw cluster means systematically place tiny schools at both the top and the bottom. As n_j grows, lambda approaches 1 and shrinkage vanishes, so the model penalizes only the estimates that deserve penalizing.

Variance components also serve as effect sizes. If adding tutoring hours reduces the within-school variance from 126 to 110, the level-1 variance explained is (126 - 110) / 126 = 0.127, and if adding school funding reduces the between-school variance from 18 to 9, the level-2 variance explained is (18 - 9) / 18 = 0.50. Reporting the two separately is far more informative than a single pooled R-squared, which multilevel models do not cleanly provide.

Key idea: Shrinkage weights each cluster by its own precision, so a mean of 76 from 20 students becomes 74.4 while the same mean from 5 students becomes 72.5.

Cross-level questions

Multilevel models also let predictors at different levels coexist: a student-level predictor (hours studied) and a school-level predictor (school funding) in one model. A cross-level interaction then asks whether a school-level variable moderates a student-level effect, for example whether funding strengthens the study-time-to-achievement slope. These questions are simply unaskable in single-level regression, which is why nested designs demand the multilevel framework rather than treating clustering as a nuisance to be patched over.

The catch is sample size at the upper level, and it is a serious one. Brightwater has 240 students but only 12 schools, so any school-level coefficient is effectively estimated from twelve observations however many children were tested. Simulation work generally recommends around 30 clusters before trusting level-2 fixed effects and considerably more, often 50 or above, before trusting estimates of variance components; below those thresholds, restricted maximum likelihood combined with a small-sample degrees-of-freedom correction such as Kenward-Roger is the standard safeguard. A cross-level interaction is more demanding still, for the same reason interactions were more demanding in Lesson 11.

One vocabulary warning prevents a persistent confusion. In multilevel modelling, a "fixed effect" is an average coefficient across clusters and a "random effect" is a cluster-specific departure drawn from an estimated distribution. In econometrics, a "fixed-effects model" means something different, namely one that includes a separate dummy for every cluster and thereby discards all between-cluster information, which also makes it impossible to estimate any cluster-level predictor. Both usages are correct within their own literatures, and a reader who imports one into the other will misread the model entirely.

Key idea: The number of clusters, not the number of cases, governs what a multilevel model can say about cluster-level predictors, and twelve schools is a thin foundation.

Where people get stuck

• Treating a small ICC as permission to ignore clustering. An ICC of 0.05 with 100 cases per cluster gives a design effect of 5.95, which is worse than an ICC of 0.125 with clusters of 20.
• Aggregating to cluster means instead. Averaging within schools discards all within-school variation and invites the ecological fallacy, in which a school-level association is read as a student-level one.
• Counting cases when the relevant number is clusters. A school funding coefficient in Brightwater rests on 12 data points, and no amount of student-level data changes that.
• Confusing the two meanings of "fixed effects." The multilevel sense is an average coefficient; the econometric sense is a cluster dummy specification that cannot estimate cluster-level predictors at all.
• Adding random slopes reflexively. Each random slope adds variance and covariance parameters, and with few clusters the model will often fail to converge or will converge to a boundary estimate that should not be interpreted.
• Expecting a single R-squared. Variance is explained separately at each level, and the Brightwater figures of 0.127 within schools and 0.50 between them convey more than any pooled summary could.

Key idea: Multilevel modelling replaces one number for precision with a structure, and the discipline it demands is counting information at the level where each question lives.

Recap

• Nested data violate independence because units in a cluster resemble one another, and ordinary regression responds by understating standard errors for cluster-level predictors.
• The intraclass correlation is between-cluster variance over total variance, and Brightwater's 18 / 144 gives ICC = 0.125.
• The design effect of 1 + (20 - 1)(0.125) = 3.375 cut the effective sample from 240 to 71 and inflated the correct standard error from 0.33 to about 0.61.
• Cluster size and ICC multiply, so an ICC of 0.05 with 100 per cluster is more damaging than an ICC of 0.125 with 20.
• Random intercepts let each cluster have its own baseline; random slopes let a predictor's effect vary by cluster and are themselves a substantive question.
• Shrinkage weighted by lambda = tau-squared / (tau-squared + sigma-squared / n) turned an observed school mean of 76 into 74.4 with 20 students but 72.5 with 5.
• Variance explained is reported by level, here 0.127 within schools and 0.50 between them, and cluster-level inference is limited by the 12 schools rather than the 240 students.

Sources

1. Austin, P. C., & Merlo, J. (2017). Intermediate and advanced topics in multilevel logistic regression analysis. Statistics in Medicine, 36(20), 3257-3277. pmc.ncbi.nlm.nih.gov
2. UCLA Office of Advanced Research Computing. (n.d.). Introduction to linear mixed models. Statistical Methods and Data Analytics. stats.oarc.ucla.edu
3. Centre for Multilevel Modelling. (n.d.). Multilevel models. University of Bristol. bristol.ac.uk
4. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage Publications. find source ↗
5. Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). Sage Publications. find source ↗
6. Gelman, A., Hill, J., & Vehtari, A. (2020). Regression and other stories. Cambridge University Press. avehtari.github.io
7. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. jstatsoft.org

Many constructs in the human sciences, depression, motivation, political trust, cannot be measured by a single item without error. The response is to use several items and model the latent variable they share. Factor analysis is the family of methods for doing so, and it links the measurement concerns of the companion methods course to formal statistical modeling.

Brightwater brings the problem into view through a twelve-item Reading Motivation Scale administered to all 240 students. Nobody believes that any single item measures motivation, and nobody believes the twelve items measure twelve different things, so the analytic question is how many underlying dimensions the responses reflect and which items belong to which.

Key idea: Factor analysis asks what small number of unobserved variables could have produced the observed pattern of correlations among many items, and answering that question is prerequisite to every regression built on the resulting scores.

In plain terms

Twelve questions about reading motivation will not be answered independently, because a child who says she enjoys reading will probably also say she looks forward to library visits. Those two answers rise and fall together, and the reason they do is presumably something about the child that neither question measures directly.

Factor analysis works backwards from that observation. It examines which items move together and infers how many hidden dimensions would be needed to account for the whole pattern of agreement. If items 1 through 6 agree with one another but not much with items 7 through 12, and those in turn agree among themselves, the evidence points to two dimensions rather than one, and naming them is the researcher's job rather than the software's.

The latent-variable idea

A latent variable (factor) is an unobserved construct inferred from the pattern of correlations among observed indicators. The organizing assumption is that the latent factor causes the responses: people high in the trait tend to respond high across all its indicators, which is why those indicators correlate. Factor analysis reverses this logic, using the observed correlations to estimate the underlying factors. A crucial conceptual distinction is between reflective indicators, which are effects of the latent variable (the usual case in psychometrics), and formative indicators, which combine to define a composite; the modeling differs, and treating one as the other is a common error.

Exploratory versus confirmatory

• Exploratory factor analysis (EFA) is used when you do not have a firm hypothesis about how many factors underlie a set of items or which items load on which factor. It lets the data suggest a structure. Decisions include how many factors to retain (guided by eigenvalues, the scree plot, and parallel analysis) and how to rotate the solution to make it interpretable. Rotation can be orthogonal (factors kept uncorrelated, as in varimax) or oblique (factors allowed to correlate, usually more realistic for psychological constructs).
• Confirmatory factor analysis (CFA) is used when theory specifies the structure in advance. You fix which items load on which factors and test whether that hypothesized measurement model fits the data, using fit indices (such as the comparative fit index and the root mean square error of approximation). CFA is a special case of structural equation modeling and is the appropriate tool for validating an established scale.

Worked example: deciding how many factors to keep

Extracting factors from the twelve Brightwater items produces eigenvalues of 4.80, 2.10, 0.95, 0.82, 0.71, and smaller values thereafter. Because the twelve standardized items contribute twelve units of variance in total, the first factor accounts for 4.80 / 12 = 40.0 percent and the second for 2.10 / 12 = 17.5 percent, giving a cumulative 57.5 percent for a two-factor solution.

Three retention criteria can be applied to those numbers, and they do not always agree. Kaiser's rule keeps every factor with an eigenvalue above 1.00, which here retains exactly two, and the scree plot shows an obvious elbow after the second value. Parallel analysis, the most defensible of the three, generates eigenvalues from random data of the same size and keeps only real factors that beat their random counterparts; for twelve variables and 240 cases the random values run at roughly 1.35, 1.24, and 1.16, so only 4.80 and 2.10 survive while the third real eigenvalue of 0.95 falls short of its random benchmark of 1.16.

Notice that the third eigenvalue failed parallel analysis even though it sits close to Kaiser's threshold, and that pattern generalizes: the eigenvalue-greater-than-one rule is known to over-extract, sometimes badly, because sampling error alone lifts several eigenvalues above 1.00 in any real dataset. Parallel analysis and the scree plot should carry the decision, with substantive interpretability as the final check, since a statistically retained factor that nobody can name is not a finding.

Key idea: Kaiser's eigenvalue rule routinely retains factors that random data would also produce, so parallel analysis is the defensible default and interpretability is the final test.

Reading the output

A factor loading is the correlation (in the standardized, orthogonal case) between an indicator and the factor; loadings near plus or minus 1 mean the item strongly reflects the factor, and loadings near zero mean it does not. The communality of an item is the proportion of its variance explained by the retained factors, the sum of its squared loadings; its complement is the item's uniqueness (specific variance plus error).

Items with low communality are poorly explained by the factor solution and are candidates for revision or removal. A clean solution shows simple structure: each item loads strongly on one factor and weakly on the others, so the factors are distinct and interpretable.

Worked example: loadings, communalities, and a bad item

Suppose the two Brightwater factors are Enjoyment and Self-efficacy, correlated at 0.55, so an oblique rotation is appropriate. Three items from the pattern matrix illustrate the range of quality a real scale contains:

Item	Enjoyment	Self-efficacy	Communality (orthogonal formula)	Communality (oblique, correct)
3. "I read for fun at home"	0.82	0.05	0.67	0.72
9. "I can work out hard words"	0.11	0.80	0.65	0.70
12. "Reading class goes quickly"	0.34	0.41	0.28	0.44

The orthogonal column is the familiar sum of squared loadings, so item 3 gives 0.82² + 0.05² = 0.6724 + 0.0025 = 0.67 and item 12 gives 0.34² + 0.41² = 0.28. That formula is only correct when factors are uncorrelated, which they are not here. With correlated factors the communality must include the cross-term, as p₁² + p₂² + 2p₁p₂(factor correlation), so item 3 becomes 0.6749 + 2(0.82)(0.05)(0.55) = 0.72 and item 12 becomes 0.2837 + 2(0.34)(0.41)(0.55) = 0.44. Reporting the orthogonal figure after an oblique rotation understates every communality, and the understatement grows with the size of the cross-loadings.

Item 12 fails on its own terms regardless of which formula is used. It loads moderately on both factors and strongly on neither, so it violates simple structure, and its uniqueness of roughly 0.56 means over half its variance belongs to nothing the model has identified. The item is a candidate for rewriting or deletion, and the decision should be made and reported before any factor scores are computed, not after seeing how the resulting scale performs in the outcome analysis.

Confirmatory analysis then tests a pre-specified version of this structure. Suppose the CFA of the two-factor model returns a chi-square of 89.2 on 53 degrees of freedom with p = .002, alongside a comparative fit index of 0.972, an RMSEA of 0.053, and an SRMR of 0.045. The chi-square rejects, yet the descriptive indices all sit within conventional bounds, and the resolution is that the chi-square test becomes hypersensitive as sample size grows and will reject essentially any model at n = 240 or above. Treat the conventional cutoffs of CFI at or above 0.95, RMSEA at or below 0.06, and SRMR at or below 0.08 as rough guides derived from one simulation literature, not as pass marks.

Key idea: After an oblique rotation, communality includes a cross-term involving the factor correlation, and a cross-loading item like item 12 breaks simple structure whichever formula you use.

Why this matters for inference

Factor analysis is not a decorative preliminary; it is how you establish that your measured variables mean what you claim, the construct validity at the heart of measurement. Feeding a poorly validated composite into the regression and multilevel models of this course propagates measurement error into every downstream estimate, attenuating relationships and biasing conclusions. Establishing a sound measurement model first is therefore a prerequisite for trustworthy structural analysis, not an optional flourish.

Worked example: what unreliable measurement costs downstream

The damage is calculable. When two variables are each measured with error, the correlation you observe is the true correlation multiplied by the square root of the product of their reliabilities:

r_observed = r_true × square root of (reliability of X × reliability of Y)

Suppose the Brightwater Enjoyment scale has a reliability of 0.70, which is common for a six-item self-report from nine-year-olds, and the spring reading test has a reliability of 0.95. The attenuation factor is the square root of (0.70 × 0.95) = the square root of 0.665 = 0.816. A true correlation of 0.50 between reading enjoyment and achievement would therefore appear in your data as 0.50 × 0.816 = 0.41, and working backwards, an observed 0.41 is consistent with a true 0.41 / 0.816 = 0.50.

Two consequences follow directly. Your study is systematically underpowered relative to what the true relationship would warrant, because you are chasing an effect nearly 20 percent smaller than the real one. And in a multiple regression, unreliability in one predictor does not merely shrink its own coefficient; it also biases the coefficients of the predictors correlated with it, since adjustment for a noisily measured covariate is incomplete adjustment. Improving a scale from a reliability of 0.70 to 0.90 raises the attenuation factor from 0.816 to 0.925, which is often a cheaper route to power than recruiting more participants.

Key idea: A reliability of 0.70 shrank a true correlation of 0.50 to an observed 0.41, so measurement quality buys statistical power as surely as sample size does.

Where people get stuck

• Using the eigenvalue-greater-than-one rule alone. It over-extracts, and the Brightwater third factor at 0.95 would have looked borderline defensible while failing parallel analysis against a random benchmark of 1.16.
• Applying the orthogonal communality formula after an oblique rotation. The cross-term matters, and item 3 moved from 0.67 to 0.72 while item 12 moved from 0.28 to 0.44.
• Reading a significant CFA chi-square as fatal. The test rejects nearly every model at large n, which is why CFI, RMSEA, and SRMR are reported alongside it.
• Treating fit cutoffs as pass marks. The familiar 0.95 and 0.06 thresholds came from a particular simulation study and were never intended as universal criteria.
• Running EFA and then calling the result confirmed. Discovering a structure and testing it require different samples, exactly as variable selection and hypothesis testing did in Lesson 7.
• Modelling formative indicators reflectively. A composite defined by its components, such as socioeconomic status built from income, education, and occupation, does not cause its indicators, so a reflective measurement model misstates the causal direction.

Key idea: Measurement decisions are modelling decisions, so factor retention, rotation, and item deletion all belong in the methods section rather than in an unreported preprocessing step.

Recap

• A latent variable is an unobserved construct inferred from the pattern of correlations among observed indicators, and reflective indicators are effects of it while formative indicators define a composite.
• EFA discovers structure and CFA tests a pre-specified one, and the two require different samples.
• Brightwater eigenvalues of 4.80 and 2.10 explained 40.0 and 17.5 percent of the twelve-item variance, and parallel analysis retained exactly two factors while rejecting a third at 0.95 against a random benchmark of 1.16.
• Oblique rotation is appropriate when factors correlate, as the Brightwater factors did at 0.55.
• Communality after an oblique rotation adds a cross-term, giving 0.72 rather than 0.67 for item 3 and 0.44 rather than 0.28 for item 12.
• Item 12 loaded 0.34 and 0.41 across both factors, violating simple structure and marking it for revision.
• A CFA chi-square of 89.2 on 53 df rejected at p = .002 while CFI 0.972, RMSEA 0.053, and SRMR 0.045 all passed, because chi-square is hypersensitive at large n.
• Reliabilities of 0.70 and 0.95 attenuate a true correlation of 0.50 to an observed 0.41, so measurement quality is a power decision.

Sources

1. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272-299. find source ↗
2. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis. Structural Equation Modeling, 6(1), 1-55. find source ↗
3. Stefana, A., Damiani, S., Granziol, U., Provenzani, U., Solmi, M., Youngstrom, E. A., & Fusar-Poli, P. (2024). Psychological, psychiatric, and behavioral sciences measurement scales: Best practice guidelines for their development and validation. Frontiers in Psychology, 15, 1494261. pmc.ncbi.nlm.nih.gov
4. UCLA Office of Advanced Research Computing. (n.d.). A practical introduction to factor analysis. Statistical Methods and Data Analytics. stats.oarc.ucla.edu
5. Price, P. C., Jhangiani, R., Chiang, I.-C. A., Leighton, D. C., & Cuttler, C. (2017). Reliability and validity of measurement. In Research methods in psychology (2nd Canadian ed.). BCcampus. opentextbc.ca
6. Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology. American Psychologist, 73(1), 3-25. pubmed.ncbi.nlm.nih.gov
7. Bhattacherjee, A. (2012). Social science research: Principles, methods, and practices (2nd ed.). University of South Florida. digitalcommons.usf.edu

This course has repeatedly warned that a regression coefficient is not automatically a causal effect. Now we make the conditions explicit. Modern causal inference gives a precise language, the directed acyclic graph, for deciding what to control, and a toolkit of quasi-experimental designs for approximating randomization when it is impossible.

Key idea: Causal inference is a design problem dressed as an estimation problem, and the credibility of any causal claim rests on assumptions that the data themselves can never verify.

In plain terms

To know whether tutoring helped a particular Brightwater child, you would need to see both her actual score with tutoring and the score she would have received without it. The second number does not exist, and no statistical technique will conjure it, so every causal method in existence is a strategy for finding a substitute.

Randomization is the best substitute because it makes the two groups alike in every respect, measured and unmeasured, before anything happens. When randomization is impossible, the alternatives all take the same shape: find a comparison group, argue that it resembles what the treated group would have looked like untreated, and be explicit about what could make that argument wrong. The methods differ in which comparison they use and which assumption they lean on, but none of them escapes the need for the argument.

The counterfactual core

A causal effect is a comparison of counterfactuals: the outcome for a unit if treated versus if untreated. Since we never observe both for the same unit (the "fundamental problem of causal inference"), we rely on a comparison group meant to stand in for the unobserved counterfactual. Randomization makes the treated and control groups exchangeable in expectation, which is why it licenses causal claims. Without it, we must argue that adjustment achieves the same balance, and DAGs tell us what adjustment is needed.

Worked example: decomposing a naive comparison

Suppose 120 Brightwater students enrolled in tutoring and 120 did not, and their spring means were 74 and 66 respectively, a naive difference of 8 points. Now imagine we could see the counterfactual, and that the tutored group would have averaged 70 had they not been tutored, because they were stronger readers to begin with. The naive difference then splits exactly:

74 - 66 = (74 - 70) + (70 - 66) = 4 + 4

The first term, 4 points, is the average treatment effect on the treated, the genuine causal quantity. The second term, also 4 points, is selection bias, the difference between the two groups that would have existed with no programme at all. Half the headline was real and half was self-selection, and no amount of statistical sophistication can separate them without an assumption about that unobservable 70.

Distinguish three estimands while we are here, because papers conflate them constantly. The average treatment effect (ATE) is the effect if everybody were treated, the average treatment effect on the treated (ATT) is the effect among those who actually enrolled, and the local average treatment effect (LATE) is the effect among a specific subgroup whose treatment status responds to some particular nudge. These differ whenever the effect varies across people, which it did in Lesson 11, so a well-written results section names which one it estimated.

Key idea: An observed difference of 8 points decomposed into a causal effect of 4 and a selection bias of 4, and only an assumption about the unobserved counterfactual could have told you so.

Directed acyclic graphs

A DAG encodes assumed causal relationships as arrows among variables. Three configurations govern what you should and should not control:

• A confounder is a common cause of both treatment and outcome (X ← C → Y). It opens a spurious "backdoor" path; you must adjust for it (or a variable that blocks the path) to obtain the causal effect. This is the classic reason to include control variables.
• A mediator lies on the causal path from treatment to outcome (X → M → Y). Adjusting for it removes part of the very effect you want to estimate, so for the total effect you should not control it. This is overcontrol bias.
• A collider is a common effect of two variables (X → K ← Y). It is naturally closed; adjusting for it (or selecting on it) opens a spurious association between its causes. Conditioning on a collider is collider bias, a subtle and underappreciated way that adding a "control" variable can create bias rather than remove it.

The lesson is stark: "controlling for more variables" is not automatically safer. The DAG tells you which variables to include (confounders, to block backdoor paths) and which to leave alone (mediators and colliders). The set of variables that blocks all backdoor paths without opening new ones is a valid adjustment set.

Worked example: a collider manufactures a correlation

Collider bias sounds abstract until you count cases. Take 1,000 Brightwater children, each of whom independently either has strong decoding skill or does not (500 each) and independently either has strong vocabulary or does not (500 each). Independence means the four combinations contain 250 children apiece and the correlation between decoding and vocabulary in the full population is exactly zero.

Now suppose the district assigns a child to the advanced reading group if she is strong in either skill, which places 750 children in that group and excludes only the 250 who are weak in both. Analyse the advanced group alone, as any researcher studying that group would, and the counts become 250 with both skills, 250 with decoding only, 250 with vocabulary only, and zero with neither.

Within this group, a child with strong decoding has vocabulary 250 / 500 = 50 percent of the time, while a child with weak decoding has vocabulary 250 / 250 = 100 percent of the time. The phi coefficient works out to -0.50. Two traits that are genuinely independent now appear strongly and negatively related, and the only thing that happened was conditioning on their common effect.

The mechanism generalizes wherever selection depends on multiple causes. It explains why hospital-based samples show spurious associations between diseases, why studies of admitted students find that test scores and essays trade off, and why "controlling for" a variable that both the exposure and the outcome influence makes an analysis worse rather than better. A control variable is not a free good, and the DAG is what tells you which kind you are adding.

Key idea: Conditioning on a common effect turned a correlation of exactly zero into minus 0.50, which is why "adjusting for more variables" can create bias rather than remove it.

Quasi-experimental designs

When randomization is impossible, several designs exploit features of the world to approximate it:

• Difference-in-differences (DiD) compares the before-to-after change in a treated group with the change in an untreated comparison group. By differencing out fixed group differences and common time trends, it isolates the treatment effect, provided the groups would have followed parallel trends absent treatment, an assumption that must be argued, not assumed.
• Regression discontinuity (RDD) applies when treatment is assigned by whether a continuous running variable crosses a cutoff (a test score threshold for a scholarship, say). Units just above and just below the cutoff are nearly identical except for treatment, so comparing outcomes at the threshold estimates a local causal effect with a credibility approaching that of an experiment.
• Instrumental variables (IV) address unmeasured confounding by finding an instrument that affects the treatment but influences the outcome only through the treatment (the exclusion restriction) and is unrelated to the confounders. The instrument induces variation in treatment that is "as good as random," recovering a causal effect for those whose treatment responds to it.

Worked example: three designs, three sets of numbers

Difference-in-differences. Brightwater launched its programme in 2023 while neighbouring Eastvale did not. Brightwater's district mean rose from 66 in 2022 to 74 in 2023, a gain of +8, while Eastvale's rose from 68 to 71, a gain of +3. The three estimates available from these four numbers differ sharply:

• Before-and-after in Brightwater alone: +8 points, which credits the programme with the region-wide trend as well.
• Cross-sectional comparison in 2023: 74 - 71 = +3 points, which penalizes Brightwater for having started lower.
• Difference-in-differences: 8 - 3 = +5 points, which removes both the fixed district gap and the common time trend.

The whole estimate rests on parallel trends, meaning Brightwater would have gained the same +3 as Eastvale had it done nothing. That assumption cannot be tested for the treatment year, but its plausibility can be assessed by plotting several pre-treatment years and confirming the two districts moved together before 2023.

Regression discontinuity. Suppose eligibility for free tutoring required a fall score below 55. Fitting a separate line on each side of the cutoff and reading their values at exactly 55 gives a fitted value of 63.0 on the eligible side and 58.5 on the ineligible side, so the estimated effect is a jump of 4.5 points. Two health checks are obligatory. Examine the density of fall scores around 55 for bunching just below the threshold, which would indicate that scores were manipulated to secure eligibility, and confirm that pre-treatment covariates such as free-lunch status are continuous at the cutoff. Even when both checks pass, the estimate is a local effect for students near a score of 55 and says nothing about a child scoring 35.

Instrumental variables. Suppose distance from home to the tutoring centre is used as an instrument, on the argument that it strongly affects attendance but influences reading only through attendance. If each additional kilometre reduces weekly tutoring by 0.30 hours (the first stage) and reduces spring scores by 0.66 points (the reduced form), the Wald estimate is:

IV estimate = -0.66 / -0.30 = 2.2 points per tutoring hour

Note two things. Instrument strength matters enormously, because if the first stage were only -0.05 with an F statistic of 3 rather than 28, the same reduced form would yield -0.66 / -0.05 = 13.2, an absurd and wildly unstable estimate; a first-stage F above roughly 10 is the conventional minimum. And the estimate is a LATE for compliers, meaning students whose attendance actually responds to distance, so it need not equal the effect for a family that would enrol regardless.

Key idea: The same four district means yielded +8, +3, and +5 depending on the comparison chosen, so the design, not the arithmetic, determines what the number means.

None of these is a statistical trick that manufactures causation from nothing; each rests on an assumption (parallel trends, no manipulation of the cutoff, a valid instrument) that is itself untestable in full and must be defended substantively. That is the recurring moral of causal inference: the credibility of a causal claim rests on the credibility of its design and assumptions, not on the sophistication of the estimator.

Where people get stuck

• Adding every available covariate. Confounders must be adjusted for, mediators must not be if you want the total effect, and colliders turned a zero correlation into minus 0.50 when conditioned upon.
• Confusing the estimands. ATE, ATT, and LATE coincide only when the effect is constant across people, and the Brightwater decomposition estimated an ATT of 4 rather than an ATE.
• Treating parallel trends as testable. Pre-period parallelism is evidence about the assumption, not proof of it, since the assumption concerns a counterfactual in the treatment year.
• Generalizing a discontinuity estimate. The 4.5-point jump describes students near a fall score of 55, and extrapolating it to the whole district misreads the design.
• Using a weak instrument. A first-stage F of 3 turned a plausible 2.2 into 13.2, and weak-instrument estimates are biased toward the very OLS estimate they were meant to correct.
• Believing an untestable assumption is therefore unfalsifiable. Density tests, covariate balance checks, placebo outcomes, and pre-trend plots all bear on these assumptions, and a paper that reports none of them has not argued its case.

Key idea: Every quasi-experimental estimate is a conditional claim, so reporting it responsibly means naming the assumption, testing what can be tested, and stating who the estimate applies to.

Recap

• A causal effect compares counterfactual outcomes for the same unit, which are never both observable, so every method substitutes a comparison group and defends the substitution.
• A naive Brightwater difference of 8 points decomposed into an ATT of 4 and selection bias of 4, and ATE, ATT, and LATE are different quantities whenever effects vary.
• Confounders open backdoor paths and must be adjusted for; mediators transmit the effect and must not be, for a total effect; colliders are closed until you condition on them.
• Selecting 750 of 1,000 children on either of two independent skills produced a phi of minus 0.50 between skills that were genuinely uncorrelated.
• Four district means gave +8 before-and-after, +3 cross-sectionally, and a difference-in-differences estimate of +5, which rests on parallel trends.
• A regression discontinuity at a fall score of 55 estimated a 4.5-point local jump, requiring density and covariate-balance checks and generalizing only near the cutoff.
• An instrumental-variables Wald ratio of minus 0.66 over minus 0.30 gave 2.2 points per hour, but a first stage of minus 0.05 would have given 13.2, which is why first-stage F above about 10 matters.

Sources

1. Hernan, M. A., & Robins, J. M. (2020). Causal inference: What if. Chapman & Hall/CRC. miguelhernan.org
2. Pearl, J. (2009). Causal inference in statistics: An overview. Statistics Surveys, 3, 96-146. projecteuclid.org
3. Textor, J., van der Zander, B., Gilthorpe, M. S., Liskiewicz, M., & Ellison, G. T. H. (2016). DAGitty: Drawing and analyzing causal diagrams. dagitty.net
4. Cunningham, S. (2021). Causal inference: The mixtape. Yale University Press. mixtape.scunning.com
5. Huntington-Klein, N. (2022). The effect: An introduction to research design and causality. Chapman & Hall/CRC. theeffectbook.net
6. Angrist, J. D., & Krueger, A. B. (2001). Instrumental variables and the search for identification. Journal of Economic Perspectives, 15(4), 69-85. aeaweb.org
7. Columbia Mailman School of Public Health. (n.d.). Difference-in-difference estimation. Population Health Methods. publichealth.columbia.edu

A study too small to detect a real effect wastes resources and, worse, litters the literature with false negatives and unstable estimates. Power analysis is the discipline of planning a study large enough to answer its question, and it is expected in doctoral research and grant proposals alike.

The stakes go beyond wasted effort. An underpowered literature does not merely miss real effects; it also publishes exaggerated ones, because a small study can only reach significance by overestimating the effect it found. Power analysis is therefore a matter of research integrity as much as research economics, and this lesson treats it as both.

Key idea: Power is the probability that a study can see what it is looking for, and a study that cannot see clearly will report only the mirages.

In plain terms

Suppose Brightwater wants to run a proper randomized trial next year and asks how many children it needs. The answer depends on how small an improvement would still be worth paying for, since detecting a large improvement takes few children and detecting a small one takes many.

Four quantities are locked together in one equation, so fixing any three determines the fourth. Those are the effect you want to detect, the sample size, the significance threshold, and the probability of detection. Researchers normally fix the last two by convention, name the smallest effect worth finding, and solve for how many participants that requires. The arithmetic is straightforward; the honesty lies in naming an effect size you actually believe rather than the one that yields a convenient sample.

The power quartet

Statistical power is the probability of correctly rejecting a false null, that is, of detecting an effect that truly exists. It equals 1 minus β, the complement of the Type II (false-negative) error rate. Power is determined by four interlocking quantities, any three of which fix the fourth:

• Significance level (α). The Type I error rate, conventionally 0.05. A stricter α (say 0.01) makes rejection harder and thereby lowers power, all else equal.
• Effect size. The magnitude of the true effect. Larger true effects are easier to detect, so power rises with effect size. This is usually the hardest input to specify and the one that most influences the answer.
• Sample size (n). More data means less sampling error and more power. Sample size is the quantity researchers most often solve for.
• Power (1 minus β). The target probability of detection, conventionally set to 0.80, meaning a 20 percent chance of missing a real effect. Higher targets (0.90, 0.95) demand larger samples.

Because these four are locked together, an a priori power analysis fixes α, the desired power, and a plausible effect size, and solves for the required sample size. This is the calculation that belongs in a preregistration or a grant methods section.

Worked example: sizing the next Brightwater trial

For a two-group comparison at alpha = .05 two-tailed with 80 percent power, the required sample per group is well approximated by a formula you can evaluate by hand, where 1.96 and 0.8416 are the normal quantiles for the two error rates:

n per group = 2(1.96 + 0.8416)² / d² = 15.70 / d²

Substituting three conventional effect sizes shows how brutally the requirement scales:

Cohen's d	Points, with SD = 12	n per group	Total n
0.80 (large)	9.6	25	50
0.50 (medium)	6.0	63	126
0.25	3.0	251	502
0.20 (small)	2.4	393	786

Halving the effect size quadruples the sample, because d enters squared. That single fact explains most of what goes wrong in study planning, since a researcher who assumes d = 0.50 and encounters a true d = 0.25 has built a study with roughly a quarter of the intended sensitivity.

Now evaluate the existing Brightwater design of 120 students per arm. Against a true d = 0.50 the noncentrality is 0.50 × the square root of (120/2) = 3.87, giving power of about 97 percent, which is ample. Against a true d = 0.25, however, the noncentrality is only 0.25 × 7.75 = 1.94, giving power of roughly 49 percent. The same study is excellent for a 6-point effect and a coin flip for a 3-point one, and the district would need 502 students to bring the second case up to 80 percent.

Two adjustments belong in any real proposal. Clustering inflates the requirement by the design effect from Lesson 13, so 502 students in schools of 20 with an ICC of 0.125 becomes 502 × 3.375 = 1,694 students across roughly 85 schools, which may end the conversation. Anticipated attrition inflates it again, so an expected 15 percent loss requires dividing by 0.85, a further 18 percent increase.

Key idea: A design with 97 percent power for a 6-point effect had 49 percent power for a 3-point one, and clustering multiplied the corrected requirement by a further 3.375.

Where the effect size comes from

The credibility of a power analysis rests entirely on the effect-size estimate feeding it, and there is no purely statistical way to conjure one. Legitimate sources include meta-analytic averages for the phenomenon, the effect reported by a well-conducted prior study (discounted, since published effects are often inflated by publication bias), or the smallest effect size of interest, the smallest effect that would be practically or theoretically meaningful.

The last is often the most defensible: rather than guessing the true effect, you power the study to detect anything large enough to matter, and treat smaller effects as acceptable to miss. Overoptimistic effect sizes are the commonest way power analyses go wrong, yielding underpowered studies dressed in the appearance of planning.

The post-hoc power fallacy

After a study returns a non-significant result, researchers are sometimes asked to compute "observed" or post-hoc power using the effect size actually estimated in that sample. This is uninformative and misleading. Observed power is a deterministic transformation of the p-value: a non-significant result always corresponds to low observed power, so reporting it adds nothing beyond the p-value and cannot explain the null.

If you want to interpret a non-significant finding, the right tools are the confidence interval (does it exclude effects large enough to matter?) or an equivalence test (is the effect statistically smaller than a bound you care about?). Power analysis is a design tool to be used before data collection; used afterward on the observed effect, it answers no useful question.

Worked example: why observed power tells you nothing new

The claim that observed power merely restates the p-value can be checked arithmetically. Take a two-sided test where the observed effect lands exactly on the critical value, so p = .05 precisely. The noncentrality implied by that observed effect is exactly 1.96, so observed power equals the probability that a normal variable exceeds 1.96 - 1.96 = 0, which is 0.50. Every study with p = .05 has observed power of about 50 percent, regardless of its design, sample size, or subject matter.

Repeat with p = .20, where the observed test statistic is 1.28. Observed power is the probability that a normal variable exceeds 1.96 - 1.28 = 0.68, which is 0.25. So a null result always yields low observed power, by construction, and reporting it as an explanation for the null is circular reasoning dressed as analysis.

Key idea: p = .05 always implies observed power near 0.50 and p = .20 always implies about 0.25, so observed power is the p-value wearing a different hat.

Worked example: the exaggeration tax on small studies

Low power does more than hide real effects; it distorts the ones it finds, a phenomenon Gelman and Carlin call a Type M (magnitude) error. Suppose the true effect is a modest d = 0.20 and a researcher runs 30 participants per group. The noncentrality is 0.20 × the square root of 15 = 0.77, so power is only about 12 percent.

Now ask what a significant result from that study would have to look like. Significance requires the observed d to exceed 1.96 divided by the square root of 15, which is 1.96 / 3.87 = 0.51. Every publishable result from this design therefore reports an effect at least two and a half times the truth, and the ones that reach the most prestigious journals will report more. Nothing dishonest has occurred; the filter does the work.

A related danger is the Type S (sign) error, in which an underpowered study reaches significance in the wrong direction. With very low power, an appreciable share of significant findings have the wrong sign entirely, so the literature accumulates confident claims that point backwards. Together these two effects explain how a field can produce many significant findings and few replicable ones, which is the subject of the final lesson.

Key idea: A 12-percent-powered study can only publish effects at least 2.5 times the truth, so underpowered literatures are biased upward even when every individual analysis is honest.

Where people get stuck

• Choosing the effect size to justify the sample you can afford. Working backwards from a feasible n to a "detectable effect" and then declaring that effect plausible is power analysis performed in reverse.
• Taking effect sizes straight from a published pilot. Published effects are inflated by the filter described above, so an unadjusted borrowed estimate reliably produces an underpowered study.
• Computing observed power after a null result. It is a deterministic function of the p-value, so p = .05 always gives about 0.50 and p = .20 about 0.25, and it explains nothing.
• Ignoring clustering and attrition. The Brightwater requirement of 502 became 1,694 once the design effect was applied, and expected dropout inflates it further.
• Powering for the main effect and then testing moderators. Lesson 11 showed interactions need roughly four to sixteen times the sample, so a well-powered main analysis is an underpowered moderation analysis.
• Reading a non-significant result as no effect. The right instruments are the confidence interval and, when the question really is equivalence, a formal equivalence test against a bound you specified in advance.

Key idea: Power analysis fails almost always at the effect-size input rather than the arithmetic, which is why the smallest effect of interest is the most defensible starting point.

Recap

• Power is the probability of rejecting a false null and equals 1 minus the Type II error rate, jointly determined with alpha, effect size, and sample size.
• For a two-group comparison at alpha = .05 with 80 percent power, n per group is about 15.70 divided by d-squared.
• That gives 25 per group at d = 0.80, 63 at d = 0.50, 251 at d = 0.25, and 393 at d = 0.20, so halving the effect quadruples the sample.
• Brightwater's 120 per arm has about 97 percent power for a 6-point difference but only 49 percent for a 3-point one.
• A design effect of 3.375 turned a required 502 students into 1,694, and anticipated attrition inflates the figure further.
• The smallest effect size of interest is usually the most defensible input, since published effects are systematically inflated.
• Observed power is a deterministic function of the p-value, giving about 0.50 at p = .05 and 0.25 at p = .20, so it can never explain a null result.
• A study with 12 percent power can only publish effects at least 2.5 times the true size, which is the Type M error that inflates underpowered literatures.

Sources

1. Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. find source ↗
2. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafo, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376. nature.com
3. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175-191. link.springer.com
4. Heinrich Heine University Dusseldorf. (n.d.). G*Power: Statistical power analyses. gpower.hhu.de
5. Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1), 19-24. find source ↗
6. Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355-362. pmc.ncbi.nlm.nih.gov
7. Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337-350. pmc.ncbi.nlm.nih.gov

The credibility of the entire quantitative enterprise rests on whether results can be trusted and reproduced. A decade of replication difficulties across fields has made reproducibility a central methodological concern, and doctoral researchers are now expected to build transparency into their workflow from the start rather than bolt it on at publication.

Key idea: Every model in this course produces a number, and whether that number deserves belief depends on choices made before and around the analysis rather than inside it.

In plain terms

Imagine analysing the Brightwater data with no bad intentions at all. You must decide whether to use raw scores or the pass-fail benchmark, which covariates to include, what to do with three unusually low scores, and whether to look at boys and girls separately. Each decision seems reasonable, and any competent analyst would defend whichever one they made.

The trouble is that you make those decisions after seeing the data, and each one quietly multiplies the number of results you could have reported. If you then present the one that came out cleanly, the p-value beside it describes a single test that was never performed in isolation. Nobody lied, no rule was broken, and the finding is still unlikely to survive a fresh dataset. Preventing that outcome is what this lesson is about.

Two kinds of reproduction

Precision matters here. Computational reproducibility means that another analyst, given your data and code, obtains the identical results you reported. It is a minimum, and it is entirely under your control. Replicability is more demanding: a new study, with new data, finds a consistent result. A finding can be perfectly reproducible (your code runs and matches your tables) yet fail to replicate (the effect vanishes in fresh data). This course's models cannot deliver replicability by themselves; they can, and must, be reported reproducibly.

What the replication evidence actually shows

The scale of the problem is documented rather than asserted. The Open Science Collaboration repeated 100 studies drawn from three prominent psychology journals using original materials and larger samples. Ninety-seven percent of the original studies had reported statistically significant results; only 36 percent of the replications did. Mean effect sizes fell by roughly half, from about r = 0.40 in the originals to about r = 0.20 in the replications, and comparable exercises have since been run in economics, cancer biology, and social science with broadly similar findings.

Three mechanisms from earlier lessons combine to produce this. Underpowered designs mean that published effects have passed a filter that only inflated estimates can clear, which is the Type M error of Lesson 16. Analytic flexibility means that the reported analysis is one of many that could have been run. And publication bias means the null results never entered the literature at all, so a field that ran twenty studies of a nonexistent effect can present a record in which every published study found it.

Ioannidis's much-cited argument assembles the same ingredients formally, showing that when statistical power is low, the pre-study probability of a true effect is small, and researcher flexibility is high, the proportion of published claims that are true can fall below half. None of this requires misconduct, and treating it as a problem of dishonest individuals rather than of incentives and defaults is the fastest way to fail to fix it.

Key idea: Ninety-seven percent of a set of published psychology findings were significant and 36 percent replicated, with effect sizes halving, which is what low power plus flexibility plus publication bias predicts.

The confirmatory-exploratory distinction

The single most important discipline is to keep confirmatory and exploratory analyses distinct and honestly labeled. Confirmatory analysis tests hypotheses specified before seeing the data; its p-values mean what they claim. Exploratory analysis searches the data for patterns; it is valuable and generative, but its p-values are not valid confirmatory evidence, because searching inflates the chance of finding something. Two practices corrupt this line and inflate false positives:

• P-hacking: trying many analyses, subsets, covariates, or outcome definitions and reporting only those that cross the significance threshold. Each undisclosed choice is a researcher degree of freedom, and exploiting many of them all but guarantees a spurious "finding."
• HARKing (Hypothesizing After the Results are Known): presenting a hypothesis discovered by exploration as though it had been predicted in advance, converting an exploratory result into a fake confirmatory one.

Worked example: counting the garden of forking paths

Gelman and Loken's phrase for the subtler version of this problem is the garden of forking paths, and its point is that a researcher need not run many analyses to be affected. It is enough that the analysis actually run was chosen after seeing the data, because the p-value describes a procedure rather than a computation.

Count the paths in the Brightwater analysis. Two defensible outcome definitions, raw spring score or the pass-fail benchmark; three defensible covariate sets, none, fall score alone, or fall score plus free-lunch status; two defensible outlier rules, keep everything or trim cases beyond three standard deviations; and two defensible subgroup framings, whole sample or split by sex. That is 2 × 3 × 2 × 2 = 24 analyses, every one of which a reviewer would accept.

If tutoring had no effect whatsoever and the 24 analyses were independent, the probability that at least one returns p < .05 is:

1 - (0.95)²⁴ = 1 - 0.292 = 0.708

Roughly 71 percent of the time, a completely null dataset yields a publishable Brightwater finding. The analyses here are correlated rather than independent, so the true figure is lower, but the direction and the order of magnitude are right, and even a modest garden of twenty paths gives 1 - (0.95)²⁰ = 64 percent.

Simmons, Nelson, and Simonsohn demonstrated the same point by simulation with four flexibilities that no reviewer would question individually: measuring two outcome variables, adding ten more observations if the first result was not significant, controlling for participant sex, and dropping one of three experimental conditions. Used together, they raise the false-positive rate from the nominal 5 percent to roughly 61 percent. That number, not any story about fraud, is the core of the replication crisis.

Key idea: Twenty-four individually defensible analytic choices give a null dataset about a 71 percent chance of producing a significant result, which is why the analysis plan must precede the data.

Preregistration and registered reports

The corrective is to commit in advance. Preregistration time-stamps the hypotheses, design, sample-size rationale, and analysis plan in a public registry before data collection, so the confirmatory-exploratory boundary cannot be blurred after the fact. It does not forbid exploration; it requires that exploration be labeled as such. A registered report goes further: the introduction and methods are peer-reviewed and provisionally accepted before data collection, so publication depends on the quality of the question and design rather than on whether the results turn out significant, directly countering publication bias.

The transparent workflow

Concrete habits make an analysis reproducible and let others check it:

• Analyze with scripted code rather than point-and-click steps, so every transformation is recorded and re-runnable; avoid manual edits that leave no trace.
• Share data and code (or a documented synthetic dataset when data are sensitive) in a public repository, with enough documentation for a stranger to reproduce the tables and figures.
• Report all measures, conditions, exclusions, and analytic decisions, including the ones that did not work, so the reader sees the full garden of forking paths rather than the one path that reached significance.
• Report effect sizes and confidence intervals, not p-values alone, so results can be cumulated across studies in future meta-analyses.

Add three more that quantitative work specifically requires. Report the sample-size rationale from Lesson 16, including the effect size assumed and its source, so readers can judge whether a null result was informative. Report a sensitivity or multiverse analysis when several analytic paths were defensible, showing the tutoring coefficient across all twenty-four Brightwater specifications rather than the one that read best. And follow the reporting standard for your design, whether that is the APA Journal Article Reporting Standards, CONSORT for randomized trials, or STROBE for observational studies, each of which exists precisely to make the forking paths visible.

These practices are not bureaucratic overhead; they are what separates a durable contribution from a result that evaporates on scrutiny. The sophistication of your models, everything in this course, buys credibility only if the analysis behind them is transparent enough to be believed and reproduced.

Key idea: A multiverse table showing an effect across every defensible specification is worth more to a reader than a single well-chosen estimate, because it reports the garden rather than one path through it.

Where people get stuck

• Believing the problem is dishonesty. The 71 percent figure came from twenty-four choices any reviewer would accept, so good intentions provide no protection and only advance commitment does.
• Treating preregistration as a straitjacket. It does not forbid exploration; it requires labelling, and a preregistered study that reports both planned confirmatory tests and clearly flagged exploratory findings is stronger than one that reports either alone.
• Confusing reproducibility with replicability. Perfectly reproducible code can produce a finding that vanishes in new data, and only the second failure tells you the effect was not real.
• Reporting only the analyses that worked. Undisclosed exclusions, measures, and conditions are what convert a 5 percent error rate into a 61 percent one.
• Assuming a large sample fixes it. Sample size addresses power, not analytic flexibility, and a garden of forking paths in 10,000 cases is still a garden.
• Sharing data without documentation. A spreadsheet with unlabelled columns and no codebook is not open data, and computational reproducibility requires that a stranger regenerate every table without asking you a question.

Key idea: Transparency is a set of habits adopted before the data arrive, because almost nothing in this lesson can be retrofitted after the analysis is done.

Recap

• Computational reproducibility means your data and code regenerate your results; replicability means new data reach a consistent conclusion, and the second is much harder.
• Of 100 replicated psychology studies, 97 percent of originals were significant but only 36 percent of replications were, with mean effect sizes falling from about r = 0.40 to about r = 0.20.
• Low power, analytic flexibility, and publication bias jointly explain that pattern without requiring anyone to behave dishonestly.
• Confirmatory analysis tests hypotheses fixed before the data; exploratory analysis searches the data, and its p-values are not confirmatory evidence.
• Twenty-four defensible Brightwater specifications give a null dataset roughly a 71 percent chance of yielding p < .05, since 1 minus 0.95 to the twenty-fourth power = 0.708.
• Four ordinary researcher degrees of freedom raise the false-positive rate from 5 percent to about 61 percent in simulation.
• Preregistration timestamps the hypotheses and analysis plan, and registered reports secure acceptance on design before results are known, countering publication bias directly.
• A quantitative report should include scripted code, shared data, a sample-size rationale, effect sizes with intervals, a multiverse or sensitivity analysis, and the reporting standard appropriate to the design.

Sources

1. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. pubmed.ncbi.nlm.nih.gov
2. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366. journals.sagepub.com
3. Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460-465. sites.stat.columbia.edu
4. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLOS Medicine, 2(8), e124. journals.plos.org
5. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600-2606. pmc.ncbi.nlm.nih.gov
6. Munafo, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., ... Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. nature.com
7. Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832. pmc.ncbi.nlm.nih.gov
8. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196-217. pubmed.ncbi.nlm.nih.gov