Research Methods & Design

Before you choose a survey over an interview, or a regression over a thematic analysis, you have already made philosophical commitments - usually without noticing. The purpose of this lesson is to make those commitments explicit, because a defensible research design is one whose methods follow coherently from its assumptions about reality and knowledge.

Three nested levels

A research paradigm is a shared set of beliefs about how the world works and how it can be studied. It is usefully decomposed into three levels, each constraining the next.

• Ontology asks what exists and what the nature of reality is. Is there a single reality independent of observers (a realist ontology), or is reality constructed through human meaning-making and therefore multiple and local (a relativist ontology)?
• Epistemology asks what can be known and what the relationship is between the knower and the known. Can the researcher stand apart from the object of study and observe it objectively, or is knowledge unavoidably shaped by the researcher's position and values?
• Methodology asks how, given those answers, we should proceed to generate knowledge. Methodology is the logic of inquiry; methods are the specific techniques (a t-test, a coding scheme) that operationalize it.

The direction of constraint matters. Ontology conditions epistemology, which conditions methodology, which selects methods. A researcher who believes social categories such as "race" or "class" are constructed rather than natural kinds will not treat them as fixed independent variables in the same way a strict realist would. The methods are not wrong in the abstract; they are wrong when detached from the assumptions that would justify them.

Why this is not merely academic

Reviewers and dissertation committees routinely reject work not because the statistics are miscalculated but because the design is internally incoherent: it claims an interpretivist interest in lived meaning yet forces experience into pre-set Likert categories, or it claims positivist generalization from a purposively chosen sample of six. Naming your paradigm forces you to check that the pieces fit.

Axiology and reflexivity

A fourth level, axiology, concerns the role of values in inquiry. Positivist traditions aspire to value-free observation; interpretivist and critical traditions hold that values are inescapable and should instead be disclosed. This is why qualitative writing so often includes a reflexivity statement in which the researcher accounts for how their own background may have shaped data collection and interpretation. Reflexivity is not a confession of bias to be apologized for; it is a methodological control appropriate to a paradigm that denies the possibility of a view from nowhere.

As you progress through this course, return to these three questions for every design you evaluate: What is assumed to exist? What can be known about it, and by whom? And does the chosen method actually deliver that kind of knowledge? A study that cannot answer these coherently has a problem no sample size can fix.

The two families of assumption you meet most often are positivism and interpretivism, with post-positivism as the dominant modern refinement of the first. Knowing what each claims - and what it does not - lets you read a methods section and immediately see the criteria against which the work wants to be judged.

Positivism

Classical positivism holds that there is a single objective reality governed by regularities, and that the proper aim of science is to discover those regularities through systematic observation. Its hallmarks are a realist ontology, an objective epistemology in which the researcher is separate from the studied, and a preference for deductive reasoning: derive hypotheses from theory, then test them against data. Explanation takes a causal, often law-like form, and quantification is prized because it disciplines observation and supports replication.

Post-positivism

Twentieth-century philosophy of science made naive positivism untenable, and post-positivism absorbed the lessons rather than abandoning the goal of objective knowledge. Three revisions matter most:

• Fallibilism. All knowledge is provisional. We do not prove theories true; at best we fail to refute them. This reframes science as the elimination of error rather than the accumulation of certainty.
• Falsifiability. Karl Popper argued that what demarcates science is that its claims forbid something observable - they are falsifiable. A theory compatible with every conceivable outcome explains nothing. This is why a well-posed hypothesis specifies what result would count against it.
• Theory-ladenness of observation. There are no pure, uninterpreted facts; what we notice is shaped by the concepts and instruments we bring. Objectivity is therefore reconceived as an achievement of the community - through peer review, replication, and critical scrutiny - rather than a property of a lone observer.

Interpretivism

Interpretivism (with roots in hermeneutics and Weber's notion of Verstehen, or interpretive understanding) begins from a different ontology: the social world is not a set of external objects but a web of meanings that people actively construct. You cannot understand a strike, a diagnosis, or a ritual by measuring its surface behavior alone; you must grasp what it means to the actors. The knower and the known are entangled, so knowledge is co-constructed and context-bound. The aim shifts from causal explanation and generalization toward rich, contextual understanding. Reasoning tends to be inductive, building concepts up from data rather than testing them from above.

Dimension	Positivism / Post-positivism	Interpretivism
Ontology	Single (fallibly knowable) reality	Multiple, socially constructed realities
Aim	Causal explanation, prediction	Understanding of meaning in context
Logic	Primarily deductive, hypothesis-testing	Primarily inductive, concept-building
Researcher role	Detached, minimizing influence	Instrument, reflexively engaged
Quality criteria	Internal/external validity, reliability	Credibility, transferability, dependability

Neither family is superior in the abstract. The right question is which set of assumptions matches your object of study and your inferential goal. A pharmacologist testing a dose-response curve and an ethnographer studying how nurses improvise care are answering different kinds of questions, and each borrows its standards of rigor from a different tradition.

Every empirical study moves between theory and evidence, and it does so using one or more of three forms of inference. Being precise about which you are using clarifies what your conclusion can legitimately claim.

Deduction

Deductive inference moves from general premises to a specific conclusion that follows necessarily. If the premises are true and the argument is valid, the conclusion cannot be false. Classic form: All humans are mortal; Socrates is human; therefore Socrates is mortal. Deduction is truth-preserving but not ampliative - it never tells you more than was already contained in the premises. In research, deduction dominates the confirmatory phase: you derive a testable prediction from a theory and confront it with data.

Induction

Inductive inference generalizes from specific observations to a broader claim. Observing many white swans, one infers that swans are white. Induction is ampliative - the conclusion says more than the premises - but for that very reason it is not truth-preserving: the conclusion can be false even when every premise is true (the black swan). This is Hume's problem of induction: no finite run of observations guarantees the next case. Induction dominates the exploratory, pattern-finding phase and is central to grounded-theory approaches that build concepts from data.

Abduction

Abductive inference, articulated by C. S. Peirce, is inference to the best explanation. Faced with a surprising observation, you ask what hypothesis, if true, would render it a matter of course, and you provisionally adopt the most plausible such hypothesis. A physician reasoning from symptoms to the most likely diagnosis is reasoning abductively. Abduction is the logic of discovery - it generates candidate explanations - which are then developed deductively into predictions and tested. Like induction, abduction is fallible; unlike enumerative induction, it can introduce genuinely new theoretical terms.

Mode	Direction	Truth-preserving?	Typical role
Deduction	General to specific	Yes	Testing predictions
Induction	Specific to general	No (ampliative)	Generalizing patterns
Abduction	Observation to best explanation	No (ampliative)	Generating hypotheses

The cycle in practice

Mature research programs cycle through all three. A surprising anomaly prompts an abductive leap to a candidate explanation; the explanation is elaborated deductively into specific, falsifiable predictions; data collection and inductive generalization assess how far the pattern holds; and residual anomalies restart the cycle. Recognizing where you are in this loop keeps you honest: exploratory findings arrived at inductively should not be dressed up as confirmatory tests, and a hypothesis generated by abduction from a dataset cannot then be "confirmed" on that same dataset without circularity.

A dissertation lives or dies on the quality of its question. A vague topic ("social media and mental health") is not a research question; it is a field. Your task is to carve out of that field one question that is narrow enough to answer with available resources and important enough to be worth answering.

Three families of question

• Descriptive questions ask what is happening: how prevalent, how distributed, how it varies. "What proportion of first-year graduate students report clinical-range anxiety?" A description needs a defensible sample and sound measurement but no comparison group.
• Relational questions ask how two or more variables covary: "Is screen time associated with sleep quality?" These require variation on both variables and appropriate association statistics, but association alone does not license a causal claim.
• Causal questions ask whether changing one variable produces a change in another: "Does reducing evening screen time improve sleep quality?" Causal claims demand a design that can rule out alternative explanations - ideally random assignment, or a strong quasi-experimental substitute.

The type of question dictates the design. A common error is to ask a causal question and answer it with a cross-sectional correlational study, then hedge with the word "impact" as if the hedge conferred causal warrant. It does not.

The FINER criteria

A widely used checklist holds that a good question is Feasible, Interesting, Novel, Ethical, and Relevant. Feasibility - access to participants, adequate sample, time, and skills - is where most ambitious projects fail. Novelty does not require a wholly new topic; replication in a new population or a resolution of a contradiction in the literature is a legitimate contribution.

Sharpening with PICO

In applied and health fields the PICO frame disciplines a comparative question by forcing four elements to the surface: Population, Intervention (or exposure), Comparison, and Outcome. "Among first-year doctoral students (P), does a four-week sleep-hygiene program (I) versus a waitlist (C) reduce self-reported insomnia severity (O)?" Notice how much this specifies: the sample frame, the manipulation, the counterfactual, and the measured endpoint. Each element is now a design decision you can defend or critique.

Scope and the funnel

Think of question formation as a funnel: a broad area narrows to a specific problem, then to a gap identified in the literature, then to a precise question, and finally to hypotheses or aims. If you cannot state, in one sentence, exactly what observation would answer your question, it is not yet sharp enough. Write that sentence before you design anything else; every later choice - sample, measures, analysis - exists to serve it.

A hypothesis is a specific, testable statement about the relationship between variables, ideally deduced from a theory so that testing the hypothesis also tests the theory. This lesson ties hypothesis writing to the inferential machinery of null-hypothesis significance testing (NHST), and corrects the misinterpretations that plague reported results.

From construct to prediction

A theory posits relationships among abstract constructs. To test it you derive a hypothesis stated in terms of measured variables. The research (alternative) hypothesis, denoted H₁, asserts that an effect or relationship exists. It may be directional ("group A scores higher than group B") when theory predicts a direction, or non-directional ("the groups differ") when it does not. A directional hypothesis is a stronger, more falsifiable claim and, correspondingly, warrants a one-tailed test only when the direction is specified in advance for principled reasons.

The null hypothesis

The null hypothesis H₀ typically states that there is no effect - no difference between groups, or no association. NHST does not attempt to prove H₁ directly. Instead it asks: if H₀ were true, how surprising would data at least this extreme be? That probability is the p-value. If it is small (below a pre-set threshold alpha, conventionally 0.05), we reject H₀ as an implausible account of the data. This indirect logic mirrors the falsificationist stance of the previous module: we do not confirm theories, we fail to reject nulls.

What a p-value is not

The p-value is among the most misreported quantities in science. It is not the probability that the null hypothesis is true, and it is not the probability that the results occurred "by chance." Formally, it is the probability of obtaining a test statistic at least as extreme as the one observed, assuming H₀ is true. Consequently:

• A non-significant result (p greater than alpha) does not prove the null; absence of evidence is not evidence of absence.
• Statistical significance is not the same as practical importance; with a large enough sample, a trivial effect can be "significant."
• A significant result does not report the size of the effect. Always accompany p-values with an effect size and a confidence interval.

Two kinds of error

	H0 actually true	H0 actually false
Reject H0	Type I error (alpha)	Correct decision (power)
Fail to reject H0	Correct decision	Type II error (beta)

A Type I error is a false positive: rejecting a true null, its long-run rate set by alpha. A Type II error is a false negative: failing to detect a real effect, its rate denoted beta. Statistical power equals 1 minus beta, the probability of detecting an effect that truly exists. Power rises with larger samples, larger true effects, and lower measurement error. Planning the sample size to achieve adequate power (commonly 0.80) before data collection is a hallmark of rigorous confirmatory design - and it must be done in advance, because power calculated after a null result is uninformative.

A literature review is not an annotated bibliography and not a chronological march through everything ever published. It is an argument: a synthesis of prior work organized to demonstrate that your question is unanswered, answerable, and worth answering. This lesson distinguishes review types and lays out how to build the argument rigorously.

Types of review

• A narrative (traditional) review surveys a body of work to characterize its themes and debates. It is flexible and interpretive but vulnerable to selection bias, because the author chooses what to include without a pre-registered protocol.
• A systematic review answers a specific question using an explicit, reproducible protocol: pre-specified search strings across named databases, transparent inclusion and exclusion criteria, and often a formal quality appraisal of each study. Reporting standards such as PRISMA require documenting how many records were identified, screened, and excluded, and why. When studies are statistically pooled, the systematic review becomes a meta-analysis.
• A scoping review maps the extent, range, and nature of evidence on a broad topic - useful when a field is too heterogeneous or immature for a narrow systematic question. It charts what exists rather than adjudicating an effect.

Synthesis, not summary

The defining move of a strong review is synthesis: grouping studies by construct, method, or finding and speaking to what the body of work collectively shows, where it agrees, where it conflicts, and why. A telltale sign of a weak review is the "study X found... study Y found... study Z found..." paragraph, which summarizes serially without integrating. Organize thematically or conceptually, not one-source-per-paragraph.

Identifying a defensible gap

The review must land on a gap - but "no one has studied exactly this" is the weakest kind of gap and often signals that the question is unimportant. Stronger gaps include: a contradiction in the evidence that your study can resolve; a population or context to which an established finding has never been extended; a methodological limitation shared across prior studies that a better design would overcome; or a theoretical tension that competing frameworks predict differently. State the gap explicitly and show that answering it advances knowledge, not merely that it fills a blank.

Managing bias and the corpus

Two hazards deserve attention. First, publication bias: studies with significant, positive results are more likely to be published, so a review that samples only the published literature can overstate an effect. Systematic reviewers actively search grey literature and trial registries to counter this. Second, the temptation to cite only work that supports your thesis; a credible review engages seriously with contrary findings and explains them. Keep a transparent record of search terms, dates, and decisions so a reader could, in principle, reconstruct how you arrived at your corpus. That transparency is what separates a scholarly review from an opinion essay with footnotes.

The rules that govern research with human participants were written in response to real harm. Understanding that history is not ceremonial; it explains why the requirements exist and guards against treating them as bureaucratic friction to be minimized.

The historical record

The Nuremberg Code (1947) emerged from the trials of physicians who conducted lethal experiments on concentration-camp prisoners. Its first and central principle is that voluntary consent of the human subject is absolutely essential. The Declaration of Helsinki (first adopted by the World Medical Association in 1964 and revised repeatedly since) extended ethical guidance for medical research, emphasizing that the well-being of the individual takes precedence over the interests of science and society.

In the United States, the pivotal domestic scandal was the Tuskegee syphilis study, in which the U.S. Public Health Service observed the untreated progression of syphilis in Black men for four decades, from 1932 to 1972, withholding penicillin even after it became the standard cure in the 1940s and deceiving participants about their condition. Public revelation of Tuskegee led to the National Research Act of 1974 and the commission that produced the Belmont Report (1979), the ethical cornerstone of U.S. human-subjects regulation.

The three Belmont principles

• Respect for persons. Individuals are autonomous agents entitled to make their own decisions, and persons with diminished autonomy are entitled to protection. In practice this principle grounds informed consent and special safeguards for vulnerable populations.
• Beneficence. Researchers must maximize possible benefits and minimize possible harms. This principle grounds the risk-benefit analysis that sits at the center of ethical review; a study is justifiable only when its risks are reasonable in relation to its anticipated benefits.
• Justice. The benefits and burdens of research must be distributed fairly. No group should bear the risks while another reaps the rewards. This principle grounds equitable selection of participants and speaks directly to why Tuskegee, which concentrated harm on an already-marginalized population, was so grievous a violation.

Principles to applications

Principle	Core commitment	Primary application
Respect for persons	Autonomy and protection of the vulnerable	Informed consent
Beneficence	Maximize benefit, minimize harm	Risk-benefit assessment
Justice	Fair distribution of benefits and burdens	Equitable participant selection

Notice that the principles can pull against one another. A study that would yield large social benefit (beneficence) might recruit a captive or desperate population most easily (violating justice), or might work better scientifically without full disclosure (straining respect for persons). Ethical review is the structured negotiation of these tensions, not the mechanical application of a single rule. Keep the three principles in view as you design; they are the criteria an ethics board will apply to your protocol.

This lesson turns the Belmont principles into the operational machinery every researcher must navigate: the elements of valid consent, the tiers of Institutional Review Board (IRB) review, and the extra protections owed to vulnerable groups. The U.S. framework is codified in federal regulation known as the Common Rule (45 CFR 46), which many countries mirror in substance.

Valid informed consent

Consent is a process, not merely a signed form. To be valid it requires three components working together:

• Disclosure of the information a reasonable person would need: the study's purpose, procedures, foreseeable risks and discomforts, anticipated benefits, alternatives, confidentiality protections, and whom to contact with questions.
• Comprehension. Information must be presented in language and at a reading level the participant can understand. Disclosure that cannot be understood is not disclosure.
• Voluntariness. Agreement must be free of coercion (a threat) and of undue influence (an excessive inducement). Critically, consent must be revocable: participants may withdraw at any time without penalty.

Tiers of IRB review

Not all research carries equal risk, and review is calibrated accordingly. Under the Common Rule the categories are, in ascending order of scrutiny:

1. Exempt. Certain low-risk categories - for example, some anonymous survey research or research in established educational settings - may qualify as exempt from ongoing review. The crucial point: the researcher does not decide exemption unilaterally; the IRB or its designee makes that determination.
2. Expedited. Research posing no more than minimal risk and fitting defined categories may be reviewed by the chair or an experienced member rather than the full board. Minimal risk means the probability and magnitude of harm are no greater than those ordinarily encountered in daily life or routine examinations.
3. Full board. Research exceeding minimal risk requires review at a convened meeting of the full committee, with a majority present.

Vulnerable populations

Some groups have compromised autonomy or heightened susceptibility to coercion and therefore receive additional safeguards: children (who give assent while a parent or guardian gives permission), prisoners (whose confinement raises coercion concerns), pregnant women and fetuses, and persons with cognitive impairments. For participants who cannot consent for themselves, a legally authorized representative may consent on their behalf, subject to stricter risk limits.

Confidentiality, deception, and debriefing

Beyond consent, researchers must protect data through confidentiality (limiting who can link data to identities) and, where feasible, anonymity (collecting no identifiers at all). Some designs require deception - withholding or misstating the true purpose to prevent reactive behavior. Deception is permissible only when the study could not be done otherwise, the risks are minimal, and participants are debriefed afterward: told the true purpose, given the chance to withdraw their data, and helped to a proper understanding. The bar is high precisely because deception is in tension with respect for persons; it must be justified to the IRB, not assumed.

Research tests relationships among abstractions - motivation, poverty, trust - but data can only be collected on concrete observables. Operationalization is the bridge: the specification of exactly how an abstract construct will be measured in a given study. The quality of that bridge determines whether your findings mean anything.

Constructs and their operational definitions

A construct is an abstract concept, not directly observable, that a theory invokes - for example, "socioeconomic status." An operational definition states the procedure that will stand in for it: perhaps a composite of income, education, and occupational prestige. Any single operationalization is a partial, contestable rendering of the construct. Different reasonable operationalizations can yield different results, which is why you must justify yours and acknowledge what it omits. The gap between construct and measure is where construct validity, treated in the next lesson, will live.

Variables by role

• An independent variable (IV) is the presumed cause or predictor; in an experiment it is the manipulated factor.
• A dependent variable (DV) is the presumed effect or outcome, the thing measured for change.
• A confounding variable is a third variable associated with both IV and DV that can produce a spurious association; uncontrolled confounds are the chief enemy of causal inference.
• A mediator lies on the causal path between IV and DV and explains how the effect occurs; a moderator changes the strength or direction of the IV-DV relationship, specifying when or for whom it holds.

Levels of measurement

Stevens' taxonomy classifies variables by the information their numbers carry, and this classification constrains which statistics are legitimate.

Level	Property	Example	Permissible center
Nominal	Categories, no order	Blood type	Mode
Ordinal	Ordered, unequal intervals	Likert agreement	Median
Interval	Equal intervals, no true zero	Celsius temperature	Mean
Ratio	Equal intervals, true zero	Reaction time	Mean, geometric mean

The distinction between interval and ratio hinges on a true (absolute) zero that denotes the absence of the quantity. Celsius has no true zero - 0 degrees is not the absence of temperature - so ratios are meaningless (20 degrees C is not "twice as hot" as 10 degrees C). Reaction time has a true zero, so ratio statements are valid. Treating an ordinal variable as if it were interval - a common shortcut with Likert data - is a modeling assumption that must be defended, not a fact.

Why this discipline matters

Measurement level is not pedantry: computing a mean of nominal categories is nonsense, and applying a technique that assumes interval data to genuinely ordinal data can distort conclusions. Fix the construct, choose an operational definition you can defend, classify each variable's role and level, and only then select an analysis. Doing these in the wrong order is how researchers end up with sophisticated statistics applied to meaningless numbers.

Reliability is the consistency or reproducibility of a measurement. A reliable bathroom scale gives the same reading when you step on it twice in a row. Reliability is a precondition for validity but not a guarantee of it: a scale miscalibrated by five kilograms is perfectly reliable and perfectly wrong. This lesson defines the forms of reliability and shows how to read the coefficients that quantify them.

Classical test theory

The organizing idea is simple: any observed score is the sum of a true score and error, written X = T + E. Reliability is the proportion of observed-score variance that reflects true-score variance rather than random error. Perfectly reliable measurement (all true score, no error) has a reliability of 1; pure noise has a reliability of 0. The aim is to shrink the error term relative to the signal.

Forms of reliability

• Test-retest reliability assesses stability over time: administer the same measure to the same people on two occasions and correlate the scores. It presumes the construct itself is stable across the interval - inappropriate for genuinely changeable states such as mood.
• Internal consistency assesses whether items intended to tap one construct hang together. The most reported index is Cronbach's alpha, which increases with both the average inter-item correlation and the number of items. Values around 0.70 to 0.80 are commonly treated as acceptable for research scales, though a very high alpha (above roughly 0.95) can signal redundant, near-duplicate items rather than a virtue.
• Inter-rater reliability assesses agreement between independent observers coding the same events. Because raw percent agreement is inflated by chance, use a chance-corrected index such as Cohen's kappa for two raters, which subtracts the agreement expected by chance.
• Parallel-forms reliability correlates two equivalent versions of an instrument built from the same content domain, useful when repeated testing risks memory effects.

Reading a coefficient

Reliability coefficients generally range from 0 to 1, higher being better, and the acceptable threshold depends on stakes: a screening instrument used for research tolerates lower reliability than a test used to make high-stakes decisions about individuals, where 0.90 or above is often demanded. Kappa is interpreted on its own conventions, with values above about 0.60 frequently described as substantial agreement, though such labels are heuristics rather than laws.

Reliability and error

Because reliability caps validity, unreliable measurement attenuates observed relationships: random error in the variables biases correlations toward zero, so a real effect can be masked by a noisy instrument. Improving reliability - by adding good items, training raters, standardizing administration, or clarifying wording - therefore increases your ability to detect true effects. The practical lesson is that measurement quality is not a formality to report and forget; it directly governs statistical power and the credibility of every relationship you estimate.

Where reliability asks whether a measure is consistent, validity asks whether it measures what it claims to measure. A measure can be reliable without being valid, but it cannot be valid without being reliable, so validity is the more demanding and ultimately more important criterion. Note that this lesson concerns the validity of measurement; the validity of a study's causal and generalizing inferences (internal and external validity) is a separate topic taken up in Module 6.

Content validity

Content validity asks whether the items adequately sample the full domain of the construct. A statistics exam that tests only descriptive methods but claims to assess "statistical competence" under-represents the domain and thus lacks content validity. Content validity is judged largely by expert review against a clear specification of the domain; a closely related, more superficial notion is face validity, whether the measure merely appears relevant to lay observers.

Criterion validity

Criterion validity asks whether the measure relates as expected to an external criterion. It comes in two temporal flavors:

• Predictive validity: the measure predicts a future criterion (an admissions test predicting later grades).
• Concurrent validity: the measure agrees with a criterion assessed at the same time (a new depression screener correlating with an established diagnostic interview).

Construct validity

Construct validity is the overarching question of whether the measure truly captures the theoretical construct, and modern measurement theory treats content and criterion evidence as contributing to it. Two complementary lines of evidence are central:

• Convergent validity: the measure correlates strongly with other measures of the same or theoretically related constructs.
• Discriminant (divergent) validity: the measure correlates weakly with measures of different, theoretically unrelated constructs. A self-esteem scale should relate to related well-being measures (convergent) but not to, say, verbal IQ (discriminant).

The classic tool for examining both at once is the multitrait-multimethod matrix, which arranges correlations among several traits each measured by several methods, so that convergent evidence (same trait, different methods) can be compared against discriminant evidence and against the confound of shared method.

Threats to construct validity

Two systematic threats deserve special vigilance. Construct under-representation occurs when the measure taps too narrow a slice of the construct (the descriptive-only statistics exam). Construct-irrelevant variance occurs when the measure is contaminated by something extraneous - for example, a "math ability" test whose word problems are so linguistically dense that it inadvertently measures reading ability. Both distort the meaning of scores. Establishing construct validity is therefore not a single coefficient but a cumulative, theory-guided argument assembled from multiple sources of evidence - which is exactly why a serious methods section devotes real space to defending its measures rather than assuming them.

Sampling is the logic by which a manageable subset licenses claims about a larger whole. Whether that license is valid depends first on clean definitions and then on how cases are selected. Probability sampling - in which every element has a known, non-zero chance of selection - is what makes formal statistical inference to a population defensible.

Three populations, not one

• The target population is the entire group you wish to describe (all registered nurses in a country).
• The sampling frame is the operational list from which you actually draw (a professional registry). The frame is almost never a perfect image of the target.
• The sample is the set of elements selected from the frame.

The mismatch between target and frame is coverage error. If the registry omits recently licensed or unregistered nurses, no sampling technique applied to it can recover them; the flaw is baked in before selection begins. Identifying frame limitations is a required part of any honest sampling account.

Probability designs

• Simple random sampling (SRS): every element and every combination of elements is equally likely to be chosen. It is the theoretical baseline for inference but requires an enumerable frame.
• Systematic sampling: choose a random start and take every k-th element. Efficient, but dangerous if the frame has a hidden periodicity that aligns with the interval k.
• Stratified sampling: divide the frame into homogeneous strata (say, by region) and sample within each. This guarantees representation of every stratum and, when strata are internally homogeneous, yields more precise estimates than SRS for the same sample size. Proportionate stratification samples each stratum in proportion to its size; disproportionate stratification deliberately over-samples small but important strata (later corrected with weights).
• Cluster sampling: divide the population into naturally occurring clusters (schools, city blocks), randomly select whole clusters, and study all or a random subset of elements within them. It is economical when a full frame is unavailable or fieldwork is geographically dispersed, but because members of a cluster resemble one another, cluster samples are less precise than SRS of the same size.

Stratified versus cluster

These two are easily confused but have opposite structure. In stratified sampling you divide into groups and sample from every group, seeking within-stratum homogeneity to gain precision. In cluster sampling you divide into groups and sample only some groups, accepting a precision penalty in exchange for logistical feasibility. A useful heuristic: stratify to reduce sampling error; cluster to reduce cost.

Design	All groups sampled?	Effect on precision vs SRS	Main motive
Stratified	Yes, every stratum	Can improve precision	Representation, precision
Cluster	No, selected clusters	Typically reduces precision	Cost, feasibility

Because probability designs support quantified uncertainty, they let you attach a margin of error and confidence interval to estimates. That formal bridge from sample to population is the payoff for the discipline of a known selection probability - and it is precisely what the non-probability methods in the next lesson cannot provide.

Not all research can or should use probability sampling. Qualitative studies, hard-to-reach populations, and early exploratory work routinely rely on non-probability sampling, in which selection probabilities are unknown. The methods are legitimate for their purposes, but they do not license formal statistical generalization to a population, and pretending otherwise is a serious error.

Common non-probability methods

• Convenience sampling: selecting whoever is easiest to reach (students in your class, shoppers at one mall). Cheap and fast, but its representativeness is unknown and usually poor.
• Purposive (judgment) sampling: deliberately selecting cases that fit criteria relevant to the research question - information-rich cases in qualitative work, or extreme/typical/deviant cases chosen on principled grounds. Here selectivity is a feature, aimed at insight rather than generalization.
• Quota sampling: setting targets for subgroup counts (for example, equal numbers by age band) and filling them non-randomly. It mimics stratification's structure without random selection within cells, so bias can enter through how the quotas are filled.
• Snowball sampling: existing participants refer others, useful for hidden or stigmatized populations reachable only through social networks. Because referrals follow network ties, the sample over-represents the well-connected.

Two fundamentally different problems

It is essential to separate two sources of error that behave in opposite ways as sample size grows.

• Sampling error is the random discrepancy between a sample estimate and the population value that arises purely because you observed a subset. It is not a mistake; it is inherent to sampling, it is quantifiable in probability samples (via the standard error), and it shrinks as sample size increases.
• Bias is a systematic tendency for estimates to be off in a particular direction, arising from flawed selection, coverage, nonresponse, or measurement. Bias does not shrink with sample size; a larger biased sample simply estimates the wrong value more precisely.

The large-sample fallacy

The most famous illustration is the 1936 Literary Digest poll, which mailed millions of ballots and collected about 2.4 million responses, yet wrongly predicted the U.S. presidential election. Its frame - drawn heavily from telephone and automobile owners during the Depression - was skewed toward the affluent, and those who returned ballots were self-selected. Enormous size could not rescue a biased design; a much smaller but better-sampled contemporaneous poll got it right. The lesson is permanent: quantity does not cure bias. When you read a study boasting a huge sample, ask first how cases were selected, not how many there were.

Nonresponse

Even a well-drawn probability sample degrades if those who respond differ systematically from those who do not - nonresponse bias. A 15 percent response rate from a perfect frame can be more misleading than a modest but complete one, because the missing 85 percent may share a relevant characteristic (say, disengagement) correlated with the outcome. Reporting response rates and probing who is missing is therefore part of a defensible sampling account, not an optional courtesy.

The experiment is the strongest single design for causal inference because it is built to satisfy the logical requirements of a causal claim. Those requirements, stated by John Stuart Mill and refined since, are three: the cause must covary with the effect, the cause must precede the effect in time, and alternative explanations must be ruled out. Correlational designs can establish the first; only a strong design secures all three.

The counterfactual and its problem

A causal effect is defined by a counterfactual: the difference between what happened to a unit under treatment and what would have happened to that same unit without treatment. The difficulty - the fundamental problem of causal inference - is that we never observe both outcomes for the same unit; a person is either treated or not. Experiments solve this at the group level: if two groups are equivalent in expectation before treatment, the untreated group's outcome estimates the counterfactual for the treated group.

Random assignment as the engine

Random assignment allocates participants to conditions by a chance mechanism. Its power is that, in expectation, it balances the groups on all characteristics - measured and unmeasured, known and unknown - before treatment. This is what neutralizes confounding: any pre-existing difference is distributed by chance rather than tied to condition. No other technique controls unmeasured confounders, which is why the randomized controlled trial (RCT) is the benchmark for causal claims. Randomization does not guarantee balance in any single small study, but it guarantees no systematic bias and makes the probability of imbalance calculable.

A critical distinction

Two uses of "random" are routinely conflated but do entirely different jobs:

	Random sampling	Random assignment
Question addressed	To whom can results generalize?	Did the treatment cause the effect?
Supports	External validity	Internal validity
Mechanism	Selecting cases from a population	Allocating selected cases to conditions

A study can have one without the other. A tightly controlled lab experiment on undergraduate volunteers uses random assignment (strong internal validity) but not random sampling (limited external validity). A survey may use random sampling (generalizable description) but no assignment (no causal warrant). Rigorous design requires knowing which you have.

Core experimental elements

Beyond assignment, experiments deploy several controls. A control group provides the counterfactual baseline. Blinding keeps participants (single-blind) and often experimenters (double-blind) unaware of condition, preventing expectancy effects from contaminating outcomes. A placebo control isolates the specific effect of a treatment from the effect of merely receiving something. And a manipulation check verifies that the independent variable was actually delivered as intended. These are not embellishments; each closes a specific route by which an apparent effect could be an artifact.

Internal validity is the degree to which a study establishes that the independent variable, and not something else, caused the change in the dependent variable. Campbell and Stanley catalogued the recurring threats, and knowing them by name lets you diagnose a weak design instantly. Many threaten single-group or pre-post designs; randomization neutralizes several but not all.

Threats especially dangerous to single-group pre-post designs

• History: an external event occurring between pretest and posttest that affects the outcome (a policy change during a study of an economics intervention). The observed change may be due to the event, not the treatment.
• Maturation: natural changes within participants over time (growing older, more tired, more practiced) that mimic a treatment effect.
• Testing: the effect of taking a pretest on posttest scores (participants improve simply from prior exposure to the test).
• Instrumentation: a change in the measuring instrument or observers over time (a rater grows more lenient), so the change reflects the instrument, not the participants.
• Regression to the mean: when participants are selected for extreme scores, their scores tend to move toward the average on retest for purely statistical reasons, independent of any treatment.

Threats especially dangerous to multi-group designs

• Selection: pre-existing differences between groups when assignment is not random, so a posttest difference may reflect who was in each group rather than the treatment. This is the threat random assignment is designed to remove.
• Attrition (mortality): differential dropout between groups. If the treatment group loses its least-motivated members while the control retains everyone, the groups are no longer comparable even if assignment began random.
• Selection interactions: selection can combine with maturation or history so that non-equivalent groups also change at different natural rates.

A worked diagnosis

Suppose a reading clinic enrolls the lowest-scoring 5 percent of students, delivers tutoring, and reports large gains at retest. Before crediting the tutoring, a careful reader flags at least three threats. Regression to the mean predicts that an extreme-low group will rise on retest regardless. Maturation predicts reading improves with time in school anyway. Testing predicts familiarity with the assessment inflates the second score. Without a comparable control group that experiences the same passage of time, testing, and statistical regression, the design cannot separate the tutoring's effect from these artifacts. The remedy is structural: add a randomized control group, which subtracts history, maturation, testing, and regression because both groups experience them equally.

Why the catalogue matters

Each named threat corresponds to a design feature that removes it. A pretest-posttest control-group design handles most single-group threats through the control group; random assignment handles selection; tracking and reporting dropout addresses attrition; using stable, blinded instruments handles instrumentation. Design is, in large part, the systematic anticipation and closure of these specific threats before data collection - never a hopeful patch afterward.

A study can be internally airtight yet still tell us little about the world if its result does not travel. External validity is the extent to which a causal finding generalizes beyond the specific participants, settings, treatments, and times of the study. And because randomization is often impossible for ethical or practical reasons, this lesson also introduces quasi-experimental designs, which pursue causal inference without random assignment by other means of approximating the counterfactual.

Threats to external validity

• Interaction of selection and treatment: the treatment works for the studied sample (often WEIRD - Western, educated, industrialized, rich, democratic - convenience samples) but not for other populations. A finding on undergraduates may not hold for older, less-educated, or non-Western groups.
• Interaction of setting and treatment: an effect obtained in a controlled laboratory may vanish in a messy field setting, or vice versa.
• Interaction of history and treatment: an effect tied to a particular historical moment may not replicate at another time.

There is often a real tension between internal and external validity: tightening experimental control (a sterile lab, an artificial task) can strengthen causal inference while making the situation less like the real world it is meant to inform. Neither should be sacrificed reflexively; the balance depends on the study's purpose, and generalization is ultimately established through replication across samples and settings, not asserted from a single study.

Quasi-experimental designs

When you cannot assign at random, you can still design to rule out alternatives.

• Nonequivalent groups design: compare a treatment and a comparison group that were not randomly formed (say, two schools), ideally with pretests so you can gauge and adjust for initial differences. The central worry is selection, so the pretest and careful matching are the defenses.
• Interrupted time-series design: take many measurements before and after an intervention on the same unit; a change in the level or slope of the series at the point of intervention, against the established trend, supports a causal reading and helps rule out simple maturation.
• Regression-discontinuity design: when treatment is assigned by a cutoff on a continuous measure (for example, a scholarship for those scoring above a threshold), compare units just above and just below the cutoff, which are nearly equivalent by chance. Under its assumptions this design can approach the credibility of a randomized experiment near the threshold.
• Difference-in-differences: compare the before-after change in a treated group to the before-after change in an untreated comparison group, differencing out stable group differences and common time trends under a parallel-trends assumption.

The unifying idea

Every quasi-experimental design is an attempt to construct a credible counterfactual without the randomizer's guarantee. Each rests on assumptions - no differential selection trend, parallel trends, continuity at the cutoff - that must be argued, probed, and, where possible, tested. Quasi-experiments are not second-rate experiments to apologize for; they are the responsible way to pursue causal questions where randomization is unethical or infeasible, provided their identifying assumptions are made explicit and defended with the same rigor an experimenter brings to randomization.

Surveys are the workhorse of social measurement, but their apparent simplicity is deceptive: small wording choices can swing results by double-digit margins. The total survey error framework organizes everything that can go wrong into errors of representation (who answers) and errors of measurement (what their answers mean), and good design attacks both.

Errors of representation

These concern the gap between the population and the people whose answers you obtain: coverage error (the frame misses part of the population), sampling error (random subset variation), and nonresponse error (those who answer differ from those who do not). These were treated in Module 5; here the point is that a beautifully worded questionnaire cannot rescue a study whose respondents are unrepresentative.

Errors of measurement

Even with the right people, the instrument can distort. Watch for these classic questionnaire defects:

• Double-barreled questions ask two things at once ("Do you support increased funding for schools and hospitals?"), so a single answer cannot be interpreted. Split them.
• Leading questions embed a preferred answer ("Don't you agree that the mayor has failed?"), pushing responses toward it.
• Loaded language uses emotionally charged terms that shift responses regardless of the substantive content.
• Ambiguous or vague terms ("often," "regularly") mean different things to different respondents, injecting noise.
• Response-set biases. Acquiescence is the tendency to agree with statements regardless of content, countered by balancing positively and negatively worded items. Social desirability bias is the tendency to answer in a way that looks good, strongest for sensitive topics, mitigated by anonymity, neutral wording, and self-administration.

Order and format effects

Question order matters: an earlier item can prime a frame that colors later answers (asking about a specific worry before overall happiness can lower the happiness rating). For closed items, offering an explicit "don't know" option, deciding whether to include a neutral midpoint, and controlling the number of scale points all shape the distribution of responses. None of these choices is neutral, so each should be deliberate.

Open versus closed and the value of piloting

Closed-ended items are easy to analyze and compare but constrain answers to the researcher's categories, risking missing what matters to respondents. Open-ended items capture unanticipated content but demand coding. Because respondents interpret items in ways designers rarely foresee, the single most valuable safeguard is a pilot test - a small-scale trial run, ideally with cognitive interviewing in which respondents think aloud as they answer, revealing where wording is misread. A survey deployed without piloting is an instrument of unknown validity, and its polished appearance offers no assurance that respondents understood the questions as intended.

Qualitative research seeks to understand meaning, process, and context in depth rather than to quantify variation across many cases. Its logic, established in Module 1, is interpretivist: the researcher is the primary instrument, and analysis is typically inductive. This lesson surveys the main traditions and data-collection methods so you can match approach to question.

Five traditions

Tradition	Central question	Typical product
Phenomenology	What is the lived experience of a phenomenon?	The essence of an experience
Grounded theory	What theory is grounded in these data?	A theory built from data
Ethnography	How does this culture-sharing group work?	A cultural portrait
Case study	What can be learned from this bounded case?	An in-depth case analysis
Narrative inquiry	How do people story their experience?	A restoried account

These are not interchangeable. Grounded theory aims to build theory from data through systematic coding and constant comparison; phenomenology aims to distill the essence of a lived experience; ethnography studies a culture-sharing group in its natural setting, classically through extended immersion. Choosing a tradition commits you to its data-collection and analytic conventions.

Data-collection methods

• Interviews range from structured (fixed questions, near-survey) through semi-structured (a guide of topics with freedom to probe, the qualitative workhorse) to unstructured (open, conversational). Good interviewing centers on open questions, active listening, and probing without leading.
• Participant observation, the backbone of ethnography, places the researcher in the setting along a continuum from complete observer to complete participant, recorded in detailed field notes.
• Focus groups use group interaction itself as data, surfacing shared understandings and points of contention that individual interviews might miss.
• Documents and artifacts - records, images, material objects - provide unobtrusive evidence not shaped by the researcher's presence.

Sampling and saturation

Qualitative sampling is almost always purposive: cases are chosen because they are information-rich for the question, not to represent a population statistically. A common stopping rule is data saturation - the point at which additional data yield no new codes, themes, or insight. Saturation is a judgment about informational redundancy, not a fixed number, though it should be reasoned about and reported rather than left implicit.

The researcher as instrument

Because a human being, not a standardized scale, collects and interprets the data, subjectivity is not a contaminant to be eliminated but a resource to be managed through reflexivity. The qualitative researcher documents their assumptions, keeps an audit trail of decisions, and sometimes brackets (in phenomenology) prior beliefs to see the phenomenon afresh. This is the disciplined counterpart to the experimenter's controls: different machinery for a different kind of knowledge, held to standards appropriate to its own tradition rather than borrowed wholesale from the quantitative one.

Qualitative analysis transforms unstructured text - transcripts, field notes, documents - into a defensible interpretive account. It is systematic, not impressionistic, and it has its own standards of rigor. This lesson covers the mechanics of coding and the trustworthiness framework that plays the role validity and reliability play in quantitative work.

Coding

Coding is the assignment of short labels to segments of data that capture their content or meaning, enabling the researcher to retrieve, compare, and organize related material. Analysts often distinguish two logics of code creation:

• Inductive (open) coding derives codes from the data themselves, letting categories emerge - the default in grounded theory.
• Deductive (a priori) coding applies a predetermined codebook drawn from theory or prior research.

In grounded theory the sequence typically moves from open coding (fracturing data into concepts), to axial coding (relating categories to subcategories), to selective coding (integrating around a core category). Throughout, constant comparison - continually comparing new segments against existing codes - keeps categories grounded, and memo-writing records the analyst's developing interpretations. Codes cluster into broader themes, patterns of meaning that answer the research question. A theme is not merely a frequent word; it is an interpretive construct the analyst argues for from the evidence.

Trustworthiness

Because positivist validity and reliability presuppose a single objective reality, Lincoln and Guba proposed four parallel criteria for judging qualitative rigor, each with associated techniques.

Criterion	Quantitative parallel	Illustrative techniques
Credibility	Internal validity	Triangulation, member checking, prolonged engagement
Transferability	External validity	Thick description enabling readers to judge fit
Dependability	Reliability	Audit trail of decisions and changes
Confirmability	Objectivity	Reflexivity, documentation linking findings to data

• Credibility asks whether the findings are believable representations of participants' realities. Triangulation - convergence across multiple data sources, methods, investigators, or theories - and member checking - returning interpretations to participants for their reaction - are its main supports.
• Transferability concerns whether findings can inform other contexts. The researcher's job is not to claim generalization but to provide thick description - richly detailed context - so readers can judge applicability to their own settings.
• Dependability concerns whether the process is consistent and traceable, evidenced by an audit trail.
• Confirmability concerns whether findings are grounded in data rather than the researcher's bias, supported by reflexivity and clear documentation linking claims to evidence.

Rigor as argument

Note that responsibility for transferability is deliberately shared: the researcher supplies thick description, and the reader decides fit - a very different stance from statistical generalization. Across all four criteria, qualitative rigor is an argument that the interpretation is well grounded, transparently produced, and defensible against alternatives, assembled from triangulation, thick description, audit trails, and reflexivity rather than from a single coefficient.

Mixed-methods research intentionally combines quantitative and qualitative data within a single study or program, on the premise that the two together can answer questions neither could answer alone. Numbers can establish that an effect exists and how large it is; words can explain why it occurs and what it means to those involved. The value lies not in doing both side by side but in integration - deliberately connecting the strands so the whole exceeds the sum.

The pragmatist rationale

Mixing methods raises a philosophical worry: positivism and interpretivism rest on opposed assumptions, so can they be combined coherently? The most common answer is pragmatism, which treats the research question as paramount and methods as tools chosen for their usefulness in answering it, declining to make the paradigm war a precondition for empirical work. On this view, the "paradigm incompatibility" objection is set aside in favor of what actually illuminates the problem. Pragmatism is not an evasion; it is an explicit stance with its own philosophical lineage.

Core designs

Mixed-methods designs are distinguished by two features: the timing of the strands (sequential or concurrent) and the priority given to each. Three prototypes recur:

• Convergent (parallel) design: collect quantitative and qualitative data at roughly the same time, analyze each separately, then merge the results to compare or corroborate. Used when you want a fuller picture by triangulating findings from both strands.
• Explanatory sequential design: quantitative first, then qualitative. You obtain numerical results and then use qualitative follow-up to explain them - especially surprising, non-significant, or outlier findings. Numbers point to what needs explaining; words supply the explanation.
• Exploratory sequential design: qualitative first, then quantitative. You explore a poorly understood phenomenon qualitatively, then use those insights to build something testable - for instance, developing a survey instrument or hypotheses grounded in the qualitative phase, and then testing them on a larger sample.

Design	Timing	Core purpose
Convergent	Concurrent	Merge to corroborate or complete
Explanatory sequential	Quant then qual	Explain quantitative results
Exploratory sequential	Qual then quant	Build and test from qualitative insight

Integration and its challenges

The intellectual core - and the hardest part - is integration: joining the strands through merging, connecting (one phase informing the sampling or instruments of the next), or embedding. A joint display, a table or figure arraying quantitative and qualitative results side by side against common dimensions, is a standard integration device. Integration also creates distinctive difficulties: reconciling divergent findings when the strands disagree (which can be revealing rather than embarrassing), balancing the different sample-size logics of the two traditions, and the practical reality that competent mixed-methods work demands fluency in both toolkits. Done well, however, integration is exactly what makes a mixed-methods study more than a quantitative and a qualitative study stapled together.

Research that is not communicated does not exist as knowledge, and research that cannot be checked does not deserve trust. This final lesson covers the conventions of scholarly reporting and the transparency practices that have become central to credible science.

The structure of a report

Most empirical papers follow the IMRaD structure - Introduction, Methods, Results, and Discussion - which maps onto the logic of inquiry:

• Introduction: the problem, the gap, and the research question or hypotheses (the "why").
• Methods: participants, materials, procedure, and analysis in enough detail that another competent researcher could reproduce the study (the "how"). This is the section on which reproducibility depends.
• Results: what was found, reported without interpretation, with appropriate statistics, effect sizes, and uncertainty.
• Discussion: what the findings mean, how they relate to prior work, their limitations, and their implications (the "so what").

Scholarly work is vetted through peer review, in which independent experts evaluate a manuscript before publication. Peer review is a quality filter, not a guarantee of truth: it can miss errors and fraud, and it operates on the work as described rather than as actually performed.

The replication crisis

Large-scale replication efforts in the 2010s, most prominently in psychology, found that a substantial fraction of published findings did not replicate when studies were repeated. Several practices were implicated:

• P-hacking: trying many analyses, subsets, or exclusions and reporting only those that cross the significance threshold, which inflates false positives.
• HARKing (Hypothesizing After the Results are Known): presenting a hypothesis discovered in the data as though it had been predicted in advance, erasing the distinction between exploratory and confirmatory work.
• Underpowered studies: small samples that, when they do reach significance, tend to overestimate effect sizes and replicate poorly.
• Publication bias: journals' preference for positive results, which distorts the cumulative record.

It is worth distinguishing two related ideals. Reproducibility often refers to obtaining the same results from the same data and code; replicability refers to obtaining consistent results from new data collected under the same procedure. Both are needed, and both were found wanting.

Open-science responses

The reform movement offers concrete remedies. Pre-registration - publicly time-stamping hypotheses and the analysis plan before data collection - hardens the line between confirmatory and exploratory analysis and blocks HARKing and much p-hacking. The registered report goes further: peer review and in-principle acceptance occur before results are known, so publication no longer hinges on the outcome. Open data and open materials let others reproduce analyses and reuse instruments. And a priori power analysis guards against underpowered designs. None of these makes fraud impossible or replaces good judgment, but together they shift the incentives toward transparency - the direction in which credible, cumulative science must move. The through-line of this entire course holds here too: rigorous method is the disciplined anticipation of the ways an inference can go wrong, and honest reporting is what lets the community check that the anticipation succeeded.