Probability Theory

Probability theory is the branch of mathematics that measures uncertainty. It starts by describing a random experiment, any process whose outcome cannot be predicted with certainty, such as flipping a coin, rolling a die, or drawing a card. Before we can attach a number to any chance, we need something more basic: a complete list of everything that could happen. Writing that list is the skill this lesson builds, and it is the quiet first step behind every probability calculation you will ever do.

The big picture

You already reason about "what could happen" all the time. When you wonder whether it will rain, you are weighing two outcomes, rain and no rain. When you roll a die in a board game, you know the result will be one of six numbers, no more and no fewer. Probability theory takes that everyday habit and makes it exact.

Think of the sample space as the menu of a restaurant. Before anyone can talk about the chance of ordering soup, the full menu has to be on the table. The menu does not say what anyone will order; it says what the options are. Numbers come in the next lesson, when we attach probabilities to the menu. Today we build menus, and we learn the set language, union, intersection, and complement, that groups menu items into events.

Key idea: every probability problem begins with the same move. List what can happen, then describe what you care about as a subset of that list.

The sample space

The sample space, written S (some books use the Greek letter Omega), is the set of all possible outcomes of the experiment. Read that aloud as "S is the set of everything that could happen." Each individual outcome is called a sample point. A few examples:

• One coin flip: S = {H, T}, two outcomes, heads or tails.
• One roll of a six-sided die: S = {1, 2, 3, 4, 5, 6}, six outcomes.
• Two coin flips: S = {HH, HT, TH, TT}, four outcomes.

Look closely at the two-flip space, because it hides the most common early mistake in the subject. The outcome HT means "heads on the first flip, then tails on the second." The outcome TH means the reverse order. They feel similar, but they are different complete results, the same way "soup then salad" and "salad then soup" are different dinners. An outcome must record the result of the whole experiment, one entry per flip, in order.

The same idea scales up. Roll two dice, one red and one blue. One outcome is the ordered pair (3, 5), meaning red shows 3 and blue shows 5. The red die has 6 faces, and for each of those the blue die has 6 faces, so the sample space holds 6 × 6 = 36 ordered pairs, from (1, 1) up to (6, 6). We will meet this 36-outcome space again and again, so it is worth picturing it as a 6 by 6 grid.

Sample spaces come in three sizes. A space can be finite, like the die's six outcomes. It can be countably infinite, meaning the outcomes go on forever but can still be listed in order: the number of flips until the first head is one of 1, 2, 3, and so on, with no largest possible value. Or it can be continuous, like the exact time a bus arrives, which could be any real number in an interval; between any two times there are infinitely many more times, so no list can capture them.

Try it. Flip a coin, then roll a die. Write the sample space. Work it out before reading on.

Worked answer. Each outcome must record both results in order: the flip, then the roll. Pair each of the 2 flip results with each of the 6 roll results: S = {H1, H2, H3, H4, H5, H6, T1, T2, T3, T4, T5, T6}. That is 2 × 6 = 12 outcomes. What we just did: we listed one complete record per outcome, and we counted by pairing stages rather than by writing outcomes at random.

Key idea: an outcome is one complete story of the experiment. If the experiment has stages, the outcome must name the result of every stage, in order.

Events as subsets

An event is any subset of the sample space, that is, any collection of outcomes we care about. We say an event occurs if the actual outcome is one of its members. For the die, "rolling an even number" is the event A = {2, 4, 6}. If the die lands 4, event A occurred, because 4 is on A's member list. If it lands 3, A did not occur.

An event with a single outcome, like {5}, is a simple event; an event with several outcomes is a compound event. Two extreme cases are allowed and useful. The whole space S is the certain event, since whatever happens is somewhere in S. The empty set, written with empty braces, is the impossible event, since it has no members for the outcome to match.

Most of the work in a probability problem is translating English into a subset. The phrase picks out members; you list them. On one die roll:

• "At least 5" means 5 or more: the event {5, 6}.
• "At most 2" means 2 or fewer: the event {1, 2}.
• "More than 4" means strictly above 4: the event {5, 6} again, since a die cannot show 4.5.

On the 36-outcome two-dice space, the event "the sum is 7" collects every ordered pair whose entries add to 7: {(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)}, six members. Notice that (3, 4) and (4, 3) both appear, because red-3-with-blue-4 and red-4-with-blue-3 are different outcomes that happen to share a sum.

Key idea: an event is a membership list. To decide whether it occurred, check whether the actual outcome is on the list, nothing more.

Combining events

Because events are sets, we combine them with set operations, and each operation matches an everyday word:

• The union "A or B" (A union B) contains every outcome in A, in B, or in both. The mathematical "or" is inclusive: both counts.
• The intersection "A and B" (A intersect B) contains only outcomes in both A and B.
• The complement "not A" (written A with a bar, or A-complement) contains every outcome in S that is not in A.

A reliable way to build any of these is a one-outcome-at-a-time membership check. Take the die with A = {2, 4, 6} (even) and B = {1, 2, 3} (small). To form "A or B," walk through S asking each number "are you in A, or in B, or both?" The yes answers are 1, 2, 3, 4, 6, so A or B = {1, 2, 3, 4, 6}. To form "A and B," ask "are you in both?" Only 2 says yes, so A and B = {2}. And not A = {1, 3, 5}, the leftovers of S once A is removed.

Two events are mutually exclusive (or disjoint) if they share no outcomes, so their intersection is empty and they cannot both happen at once. For the die, A = {2, 4, 6} and B = {1, 3, 5} are mutually exclusive: no number is both even and odd.

One more pattern is worth meeting early because it saves effort later. The complement of a union obeys De Morgan's law: "not (A or B)" is the same event as "(not A) and (not B)." In words, the outcome avoids both A and B exactly when it misses A and also misses B. Check it concretely with A even and B small: A or B = {1, 2, 3, 4, 6}, so not (A or B) = {5}. Meanwhile not A = {1, 3, 5} and not B = {4, 5, 6}, whose intersection is also {5}. The two routes agree, and they always do.

Key idea: or means union, and means intersection, not means complement. When a compound event looks tangled, run a membership check outcome by outcome.

Worked example

Roll one die. Let A = {2, 4, 6} (even) and C = {5, 6} (greater than 4). We will build three events, one step at a time.

• Step 1, the union. Collect every number in A or in C, listing each member once: A or C = {2, 4, 5, 6}. The outcome 6 is in both sets, but it appears once, because a set lists members, not votes.
• Step 2, the intersection. Keep only numbers in both: 2 and 4 are even but not above 4, and 5 is above 4 but odd, so A and C = {6}.
• Step 3, the complement of A. Remove A's members from S: not A = {1, 3, 5}.

What we just did: we treated each event as a member list and combined lists with or, and, not. A Venn diagram makes these overlaps easy to see.

In the picture, the rectangle is the whole sample space S, each circle is an event, and the overlap region is the intersection. The union is everything covered by at least one circle, and the complement of A is everything outside A's circle, including the part of C that does not overlap A and the empty corners of the rectangle.

Here is the same skill on a richer experiment. Draw one card from a standard 52-card deck, so S has 52 outcomes. Let H be "the card is a heart" (13 members) and F be "the card is a face card," meaning jack, queen, or king (12 members, three per suit).

• The intersection "H and F" holds the face cards that are hearts: {jack of hearts, queen of hearts, king of hearts}, 3 members.
• The union "H or F" holds every heart plus the face cards from the other three suits. Counting without double-listing: 13 + 12 - 3 = 22 members. We subtracted 3 because those three cards would otherwise be listed twice, once as hearts and once as face cards.
• The complement "not H" holds the 52 - 13 = 39 cards that are clubs, diamonds, or spades.

What we just did: the count 13 + 12 - 3 is our first taste of the inclusion-exclusion idea, which returns in the next lesson as the addition rule for probabilities. The set work you do here becomes arithmetic there.

Try it

Flip a coin twice, so S = {HH, HT, TH, TT}. Let A be "the first flip is heads" and B be "at least one tail." Write A and B as subsets, then find "A and B" and "not B." Work it out before reading the answer.

Worked answer. Step 1: A collects outcomes whose first letter is H, so A = {HH, HT}. Step 2: B collects outcomes containing at least one T, so B = {HT, TH, TT}; only HH misses the list. Step 3: the intersection keeps outcomes on both lists, so A and B = {HT}. Step 4: the complement of B is everything else in S, so not B = {HH}. Notice that "not (at least one tail)" came out as "no tails at all," which is exactly what "zero tails" means: HH.

Where people get stuck

The classic wrong turn has a famous name attached. The mathematician Jean d'Alembert argued in 1754 that two coin flips have three outcomes, "two heads, one head, no heads," so each should have probability 1/3. The flaw is that his three descriptions are not equally detailed: "one head" bundles together the two distinct outcomes HT and TH. The honest sample space has four equally likely outcomes, and "one head" covers two of them. Whenever outcomes are secretly bundles, split them apart until each one is a single complete result.

A second trap: choosing a legal sample space that is useless for the question. For two dice you could take the sums {2, 3, ..., 12} as your sample space, and that is a valid list of possibilities. But those 11 outcomes are not equally likely (a sum of 7 can happen six ways, a sum of 2 only one way), so counting members would mislead you later. The 36 ordered pairs are the space in which every outcome carries equal weight. Choose the space that makes outcomes interchangeable.

Third, complements of "at least" statements. The complement of "at least one head" in two flips is not "at least one tail." Run the membership check: "at least one head" is {HH, HT, TH}, so its complement is {TT}, the event "no heads." The opposite of "one or more" is "zero," a fact that will power many fast calculations in later lessons.

Common misconceptions

• "An event is one outcome." An event is any subset: it may hold one outcome, several, all of them, or none. Both extremes, S and the empty set, are genuine events.
• "Mutually exclusive means unrelated." It means the events cannot both happen; their member lists share nothing. Whether two events are "unrelated" in the informational sense is a different idea, independence, which gets its own lesson.
• "The complement is the opposite extreme." The complement of "rolling a 6" is not "rolling a 1"; it is {1, 2, 3, 4, 5}, everything other than 6. Complement means "everything else," not "the reverse."
• "HT and TH are the same thing." They describe different histories. Sets ignore repetition, but outcomes must record order when the experiment has ordered stages.

Recap

• A random experiment is described by its sample space S, the set of all possible complete outcomes.
• Sample spaces can be finite, countably infinite, or continuous.
• An event is a subset of S; it occurs when the actual outcome is one of its members.
• Or, and, not translate to union, intersection, complement; De Morgan's law links them.
• Mutually exclusive events have empty intersection and cannot happen together.
• Pick the sample space whose outcomes are equally detailed, and let membership checks do the work.

Every probability question begins here: name the experiment, write the sample space, and translate the words of the problem into events built with union, intersection, and complement. Getting this translation right is half the battle in the rest of the course.

Sources

1. OpenStax. (2023). 3.1 Terminology. In Introductory statistics 2e. Rice University. openstax.org
2. OpenStax. (2023). 3.5 Tree and Venn diagrams. In Introductory statistics 2e. Rice University. openstax.org
3. OpenStax. (2023). 7.5 Basic concepts of probability. In Contemporary mathematics. Rice University. openstax.org
4. Siegrist, K. (n.d.). 02: Probability spaces. In Probability, mathematical statistics, and stochastic processes. LibreTexts. stats.libretexts.org
5. Encyclopedia of Mathematics. (n.d.). Probability space. European Mathematical Society. encyclopediaofmath.org
6. Weisstein, E. W. (n.d.). Sample space. MathWorld: A Wolfram web resource. mathworld.wolfram.com
7. Grinstead, C. M., & Snell, J. L. (n.d.). Introductory probability. LibreTexts. stats.libretexts.org

To turn events into numbers we assign each event A a probability P(A), read aloud as "the probability of A." Rather than define probability by a formula, modern mathematics builds it on three short rules, the axioms of probability, first set out cleanly by Andrey Kolmogorov in 1933. Everything else in this course, every rule, every named distribution, follows from these three statements. This lesson introduces them, derives the first consequences, and puts them to work on dice and cards.

The big picture

A probability is a number between 0 and 1 that measures how likely an event is. Probability 0 marks an event that cannot happen, probability 1 an event that is certain, and 0.5 an even split. A weather report of "30% chance of rain" is the statement P(rain) = 0.3 in everyday clothing.

Here is an analogy that makes the axioms feel natural. Imagine you have exactly 1 kilogram of sand, and you spread all of it over the sample space, giving each outcome its share. The probability of an event is the total weight of sand sitting on that event's outcomes. Three facts about sand are then automatic: no pile weighs less than zero, the whole space carries the full 1 kilogram, and the weight on two regions that do not overlap is the sum of their separate weights. Those three facts, translated into symbols, are exactly the axioms.

One more reading is useful. If you repeat an experiment many times, the fraction of repeats on which A occurs, its relative frequency, settles down in the long run, and those fractions obey the same three rules. A fraction of repeats cannot be negative, the outcome always lands somewhere in S, and counts of non-overlapping events add. The axioms are the shared grammar of both readings.

Key idea: probability behaves like weight. Each event carries an amount between 0 and 1, the whole space carries exactly 1, and non-overlapping events combine by addition.

The three axioms

1. Nonnegativity: for every event A, P(A) is greater than or equal to 0. Probabilities are never negative. There is no such thing as a chance below zero, in the same way a pile of sand cannot have negative weight.
2. Normalization: the whole sample space has probability P(S) = 1. Something in S is certain to happen: when you roll the die, some face comes up. All the sand is on the table.
3. Additivity: if A and B are mutually exclusive, then P(A or B) = P(A) + P(B). For disjoint events, probabilities add. (This extends to any number of pairwise disjoint events.) The weight on two separate regions is the sum of the two weights, provided the regions share nothing.

Read axiom 3 aloud slowly, because it carries a condition that is easy to miss: "if A and B cannot both happen, then the probability that one of them happens is the sum of their probabilities." The condition is the mutually exclusive part. When events overlap, plain addition counts the overlap twice, and we will need a corrected rule. That corrected rule is coming two paragraphs down, and it is one of the most used formulas in the course.

Notice what the axioms do not say. They never tell you that a die is fair or that heads has probability 1/2. Those are modeling choices you supply based on symmetry or data. The axioms are the accounting rules that any honest assignment of probabilities must respect, the way bookkeeping rules constrain a ledger without telling you the prices.

Key idea: the axioms are the constitution of probability. They do not hand you numbers; they guarantee that whatever numbers you assign fit together consistently.

Consequences of the axioms

From these three rules alone, several everyday facts can be proven. Here is the list, then the reasoning behind the two most useful entries.

• The impossible event: P(empty set) = 0.
• Bounds: every probability satisfies 0 is less than or equal to P(A) is less than or equal to 1.
• The complement rule: since A and "not A" are disjoint and together fill S, we get P(A) + P(not A) = 1, so P(not A) = 1 - P(A).
• The general addition rule: for any two events, disjoint or not, P(A or B) = P(A) + P(B) - P(A and B). We subtract the overlap so it is not counted twice.

Where does the complement rule come from? Walk through it in small steps.

• Step 1. A and "not A" are mutually exclusive: no outcome is both in A and outside A. (Definition of complement.)
• Step 2. Together they cover all of S: every outcome is either in A or not. (Definition again.)
• Step 3. By axiom 3, P(A or not A) = P(A) + P(not A). (The two pieces are disjoint.)
• Step 4. But "A or not A" is S itself, and axiom 2 says P(S) = 1. So P(A) + P(not A) = 1.
• Step 5. Subtract P(A) from both sides: P(not A) = 1 - P(A).

What we just did: we split the certain event into two non-overlapping pieces and let additivity do the arithmetic. Every derivation in this lesson has that same shape: cut an event into disjoint pieces, add the pieces.

Now the general addition rule, with numbers before symbols. Roll one die, and let A = {2, 4, 6} and C = {5, 6}. Adding the raw sizes counts 3 + 2 = 5 outcomes, but the union {2, 4, 5, 6} has only 4 members, because the outcome 6 sits in both events and got counted twice. Subtracting the overlap once repairs the count: 3 + 2 - 1 = 4.

Probabilities inherit the same repair: P(A or C) = 3/6 + 2/6 - 1/6 = 4/6. In general, P(A or B) = P(A) + P(B) - P(A and B), and when A and B are disjoint the overlap term is 0, which hands back axiom 3.

Key idea: the complement rule and the addition rule are not extra laws to memorize separately. They are the axioms, rearranged.

Equally likely outcomes

When a sample space has N outcomes that are all equally likely, the 1 unit of probability is shared evenly, so each outcome has probability 1/N, and for any event A,

P(A) = (number of outcomes in A) / (total number of outcomes) = |A| / N.

Read that aloud: "the probability of A is the count of outcomes in A divided by the count of all outcomes." This formula is why the set work of the last lesson matters: once events are member lists, probability is counting.

Worked example. Draw one card from a standard 52-card deck, so N = 52 and each card has probability 1/52. The event "a heart" has 13 outcomes, so P(heart) = 13/52 = 1/4 = 0.25. The event "a king" has 4 outcomes, so P(king) = 4/52 = 1/13, about 0.077.

Using the addition rule. Find P(king or heart). Step 1: P(king) = 4/52. Step 2: P(heart) = 13/52. Step 3: the overlap is the single king of hearts, so P(king and heart) = 1/52. Step 4: assemble the rule:

P(king or heart) = 4/52 + 13/52 - 1/52 = 16/52 = 4/13, about 0.31.

What we just did: we counted each pile, spotted the double-counted card, and removed it once. If you instead listed the union directly, you would find 16 cards, the 13 hearts plus the kings of spades, clubs, and diamonds, and 16/52 again. Two routes, one answer, which is a good habit of self-checking.

Using the complement rule. The probability of "not a heart" is 1 - 13/52 = 39/52 = 3/4. The complement rule is the fastest route whenever a problem contains the words "at least one." To see it on familiar ground, flip two coins and ask for P(at least one head). The complement of "at least one head" is "no heads," which is the single outcome TT with probability 1/4. So P(at least one head) = 1 - 1/4 = 3/4. Compare that with listing {HH, HT, TH} and adding 3/4 directly: same answer, but the complement route stays quick even when "at least one" ranges over many trials.

One more equally-likely example, using the 36-outcome two-dice space from the last lesson. The event "the sum is 7" has the six members (1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1), so P(sum is 7) = 6/36 = 1/6. The event "the sum is 12" has the single member (6, 6), so its probability is 1/36. The sums are not equally likely, and that is fine: the underlying 36 ordered pairs are, and all probabilities flow from counting them.

Try it

Roll two dice. Find P(the sum is at least 10). Work it out with the 36-outcome space before reading on.

Worked answer. Step 1: "at least 10" means a sum of 10, 11, or 12. Step 2: list members by sum. Sum 10: (4, 6), (5, 5), (6, 4), three outcomes. Sum 11: (5, 6), (6, 5), two outcomes. Sum 12: (6, 6), one outcome. Step 3: the three sub-events are disjoint (a roll has only one sum), so additivity applies: 3 + 2 + 1 = 6 favorable outcomes. Step 4: divide by the total: P(sum at least 10) = 6/36 = 1/6, about 0.167. What we just did: translate words to outcomes, cut into disjoint pieces, count, add, divide.

Where people get stuck

The most common error is adding probabilities of events that overlap. P(king) + P(heart) = 4/52 + 13/52 = 17/52 is wrong as a union probability, because the king of hearts is counted twice. A reliable alarm: if adding probabilities ever gives you a number above 1, overlap has been double counted somewhere, since no probability can exceed 1.

A second stumble is applying the equally-likely formula to outcomes that are not equally likely. The two-dice sums 2 through 12 give 11 outcomes, but P(sum is 7) is not 1/11; it is 6/36. The formula |A| / N is only licensed when every outcome carries the same weight, which is why choosing the right sample space in lesson 1 was worth the care.

Third, sign slips with the complement rule. P(not A) = 1 - P(A) requires that "not A" be the full complement, everything else in S. The complement of "at least one head" is "no heads," not "at least one tail." If your complement and your event do not together cover S exactly once, step 2 of the derivation breaks and the rule does not apply.

Finally, a subtle point about probability 0. The impossible event always has probability 0, and in a finite sample space the converse holds too: probability 0 means the event contains no outcomes. In continuous settings, which arrive in the lesson on density curves, an event can have probability 0 yet still be possible, the way a dart can land on a single exact point. File that away; it will make sense when probability becomes area.

Common misconceptions

• "Either it happens or it does not, so the probability is 50-50." Two outcomes need not be equally likely. A lottery ticket wins or loses, but P(win) is nowhere near 0.5. Equal probabilities must be earned by symmetry or evidence, not assumed from a two-item list.
• "The axioms determine every probability." They only enforce consistency. Fairness of a die is a modeling assumption; the axioms then convert that assumption into the value 1/6 per face.
• "Probabilities near 0 mean the event will not happen." Rare events happen constantly, because there are so many of them. A specific bridge hand is astronomically unlikely, yet every deal produces one.
• "Addition works for any two events." Plain addition is only for mutually exclusive events. Overlapping events need the subtraction term P(A and B).

Recap

• P(A) is a number from 0 to 1 measuring how likely A is; think of it as the weight sitting on A's outcomes.
• The three axioms: probabilities are nonnegative, P(S) = 1, and disjoint events add.
• Complement rule: P(not A) = 1 - P(A), the workhorse for "at least one" questions.
• General addition rule: P(A or B) = P(A) + P(B) - P(A and B), subtracting the double-counted overlap.
• With equally likely outcomes, P(A) = |A| / N, so probability reduces to counting.

The next lesson sharpens that last point: when sample spaces get large, we need tools that count outcomes without listing them one by one.

Sources

1. OpenStax. (2023). 3.1 Terminology. In Introductory statistics 2e. Rice University. openstax.org
2. OpenStax. (2023). 3.3 Two basic rules of probability. In Introductory statistics 2e. Rice University. openstax.org
3. Encyclopedia of Mathematics. (n.d.). Probability. European Mathematical Society. encyclopediaofmath.org
4. Siegrist, K. (n.d.). 02: Probability spaces. In Probability, mathematical statistics, and stochastic processes. LibreTexts. stats.libretexts.org
5. MacTutor History of Mathematics Archive. (n.d.). Andrey Kolmogorov. University of St Andrews. mathshistory.st-andrews.ac.uk
6. Shafer, G., & Vovk, V. (2006). The sources of Kolmogorov's Grundbegriffe. Statistical Science, 21(1). doi.org/10.1214/088342305000000467
7. Grinstead, C. M., & Snell, J. L. (n.d.). Introductory probability. LibreTexts. stats.libretexts.org

When outcomes are equally likely, a probability is a ratio of two counts: favorable outcomes over total outcomes. So we need efficient ways to count outcomes without listing them all, because listing breaks down fast. Nobody wants to write out the 2,598,960 possible poker hands. The mathematics of counting is called combinatorics, and this lesson builds its three essential tools.

The big picture

Every counting method in this lesson is a shortcut for a list you could, in principle, write out by hand. That is a comforting way to hold the subject: if a formula ever feels doubtful, shrink the problem until the list is small, write the list, and watch the formula agree with it. We will do exactly that several times.

There are really only two questions to ask about any counting problem. First, is the selection built in stages? If so, multiply the number of choices at each stage. Second, does the order of the selection matter? If yes, you are counting permutations; if no, you are counting combinations. Almost every problem in this lesson, and in the exercises, is settled by those two questions.

Key idea: multiply across stages, then decide whether order matters. Order matters: permutation. Order does not: combination.

The multiplication principle

If a task is done in stages, with n1 choices at the first stage, n2 at the second, and so on, then the total number of ways to complete the whole task is the product n1 x n2 x .... Rolling a die then flipping a coin gives 6 x 2 = 12 outcomes. A 4-digit PIN allows 10 x 10 x 10 x 10 = 10,000 codes.

Why multiply and not add? Picture the die-then-coin experiment as a tree. The trunk splits into 6 branches, one per die face. Each of those branches splits again into 2 twigs, heads or tails. Twigs at the end: 6 groups of 2, which is 6 × 2 = 12. Addition would count 6 + 2 = 8, as if you performed one stage or the other, but the experiment performs both, one after another. "And then" means multiply; "or instead" means add.

A wardrobe example makes it concrete. With 3 shirts and 2 pairs of pants, each shirt pairs with each pair of pants, so there are 3 × 2 = 6 outfits. Add a choice of 4 pairs of shoes and the count becomes 3 × 2 × 4 = 24 complete outfits.

One care point: the principle needs the number of choices at each stage to be the same no matter what was picked earlier, even if the specific options differ. A 4-digit code with no repeated digit has 10 choices for the first digit, then 9 remaining, then 8, then 7, so 10 x 9 x 8 x 7 = 5,040 codes. Which nine digits remain at stage two depends on stage one, but there are always nine of them, so the multiplication stands.

Key idea: stages chain by multiplication because every choice at one stage pairs with every choice at the next.

Factorials

The number of ways to arrange n distinct objects in a row is n factorial, written n!, defined as n! = n x (n-1) x ... x 2 x 1. It is the multiplication principle applied to seats: n choices for the first position, n - 1 for the second (one object is used up), and so on down to 1. For example 5! = 5 x 4 x 3 x 2 x 1 = 120: five books can fill a shelf in 120 different orders.

Factorials grow at a startling pace: 6! = 720, 10! = 3,628,800, and 52!, the number of ways to shuffle a deck, has 68 digits. This growth is why we count with formulas instead of lists.

By convention 0! = 1. Read that as "there is exactly one way to arrange nothing," the empty arrangement. The convention is not a whim; it is the value that makes the permutation and combination formulas below work smoothly when k equals n or 0.

Permutations: order matters

A permutation counts arrangements where order matters. The number of ordered arrangements of k objects chosen from n distinct objects is

P(n, k) = n! / (n - k)!.

Worked example. How many ways can gold, silver, and bronze medals be awarded among 10 runners? Order matters (first is not the same as third), so build it in stages: 10 possible gold winners, then 9 left for silver, then 8 for bronze, giving P(10, 3) = 10 x 9 x 8 = 720.

Now watch the formula say the same thing. 10! / (10 - 3)! = 10! / 7!, and writing both out, 10! = 10 x 9 x 8 x 7!, so the 7! cancels top and bottom, leaving 10 × 9 × 8. The formula is not new machinery; it is the staged product with the unused tail of the factorial divided away. In practice you rarely compute big factorials: cancel first, then multiply the short product that survives.

Key idea: a permutation is the multiplication principle with a shrinking pool: n, then n - 1, then n - 2, for k stages.

Combinations: order does not matter

A combination counts selections where order does not matter. The number of ways to choose k objects from n, ignoring order, is the binomial coefficient

C(n, k) = n! / (k! (n - k)!), read "n choose k".

Worked example. A committee of 2 is chosen from 5 people. Order does not matter, so C(5, 2) = 5! / (2! 3!) = (5 x 4) / (2 x 1) = 10.

Here is where the formula comes from, in two steps. Step 1: count ordered picks: P(5, 2) = 5 x 4 = 20 ways to pick a first person then a second. Step 2: notice each committee got counted twice: picking Ana then Ben and picking Ben then Ana build the same committee. Every group of 2 can be ordered in 2! = 2 ways, so the ordered count overstates by a factor of 2. Divide it away: 20 / 2 = 10. In general C(n, k) = P(n, k) / k!: count ordered, then divide by the k! orders that describe the same group.

With 5 people named A, B, C, D, E, you can check by listing: AB, AC, AD, AE, BC, BD, BE, CD, CE, DE. Ten pairs, matching the formula, which is the shrink-and-list habit from the start of the lesson.

A symmetry worth knowing: C(n, k) = C(n, n - k). Choosing 2 people to serve is the same act as choosing the 3 people who sit out, so C(5, 2) = C(5, 3) = 10. When k is large, count the leftovers instead.

Key idea: combinations are permutations with the order divided out: C(n, k) = P(n, k) / k!.

Putting counting to work in probability

How many 5-card poker hands are there from a 52-card deck? A hand is a handful, not a sequence, so order does not matter: C(52, 5) = 2,598,960. What is the probability that all 5 cards are hearts? There are 13 hearts, and the number of 5-heart hands is C(13, 5) = 1,287. Since every hand is equally likely,

P(all hearts) = C(13, 5) / C(52, 5) = 1,287 / 2,598,960 = 0.000495, about 1 in 2,020.

What we just did: we used the same species of count, combinations, for both the favorable outcomes and the total, so the "does order matter" decision was made once and applied consistently. Mixing a permutation numerator with a combination denominator is a classic way to get nonsense.

Counting also answers the question that started probability theory as a subject. In 1654 the gambler Antoine Gombaud, known as the Chevalier de Méré, asked why betting on at least one six in 4 rolls of a die won money, while betting on at least one double-six in 24 rolls of two dice lost. The exchange of letters between Blaise Pascal and Pierre de Fermat that settled it is usually counted as the birth of the subject.

Set up the first bet by counting. Four rolls have 6^4 = 1,296 equally likely outcomes. The rolls with no six at all leave 5 faces per roll: 5^4 = 625 outcomes. So, by the complement rule,

P(at least one six) = 1 - 625/1,296 = 671/1,296 = 0.518.

The second bet: each of the 24 double-rolls has 36 outcomes, 35 of which miss double-six, so the no-double-six fraction is (35/36)^24 = 0.509, and P(at least one double-six) = 1 - 0.509 = 0.491. The first game favors the bettor at 51.8%; the second quietly loses at 49.1%. De Méré could feel the difference across many evenings of play but could not compute it; a page of counting explains his bankroll.

The birthday problem

Here is the most famous counting surprise. In a room of 23 people, what is the probability that at least two share a birthday? Assume 365 equally likely birthdays and ignore leap years. Intuition says the chance is small, since 23 is tiny next to 365. The count says otherwise.

Work the complement: P(at least one shared birthday) = 1 - P(all 23 birthdays different). Try 3 people first. All different means: the first person takes any of 365 days, the second must miss 1 taken day (364 choices), the third must miss 2 (363 choices). So P(all different) = (365 x 364 x 363) / 365^3 = 0.9918, and P(some match) = 0.0082, still small.

For 23 people the same staged product runs from 365 down to 343, which is the permutation P(365, 23), divided by 365^23 total outcomes. Multiplying the 23 factors gives P(all different) = 0.4927. Therefore

P(at least one shared birthday) = 1 - 0.4927 = 0.5073, better than a coin flip.

Why does intuition misfire? Because the question is not about your birthday matching someone else's; it is about any pair matching. A room of 23 people contains C(23, 2) = 253 pairs, and 253 chances at a coincidence add up. The counting keeps honest track of all the pairs the intuition forgets.

Try it

A club has 6 women and 4 men. A committee of 3 is drawn at random. Find the probability the committee has exactly 2 women. Work it out before reading on.

Worked answer. Step 1, total outcomes: C(10, 3) = 120 equally likely committees. Step 2, favorable outcomes in stages: choose the 2 women, C(6, 2) = 15 ways, and then choose the 1 man, C(4, 1) = 4 ways, so 15 × 4 = 60 committees. Step 3, divide: P(exactly 2 women) = 60/120 = 0.5. What we just did: combinations inside each group, the multiplication principle across groups, and one division at the end.

Where people get stuck

The first trap is answering the order question wrongly. A "combination lock" actually wants a permutation: 5-31-17 and 17-31-5 do not both open it. Committees, hands, and teams ignore order; codes, rankings, and race results respect it. Ask "if I shuffle my selection, is it a different result?" If shuffling changes nothing, divide the order out.

The second trap is double counting in "at least" problems. From 6 women and 4 men, count committees of 3 with at least 2 women. Tempting but wrong: choose 2 women, C(6, 2) = 15, then anyone from the remaining 8 people, giving 15 × 8 = 120.

The correct count splits into disjoint cases: exactly 2 women, 15 × 4 = 60, plus exactly 3 women, C(6, 3) = 20, total 80. The wrong method counts each all-women committee three times, once for each pair of its members chosen at the first step: 60 + 3 × 20 = 120. When cases can overlap, cut them into exact, disjoint cases first.

The third trap is factorial arithmetic. Never compute 52! to find C(52, 5); cancel first. C(52, 5) = (52 x 51 x 50 x 49 x 48) / 5!, five factors over 120, which a calculator handles happily.

Common misconceptions

• "Permutations and combinations are unrelated formulas." They are one idea apart: P(n, k) = k! × C(n, k). Ordered counts are unordered counts times the number of orders.
• "With replacement and without replacement count the same." Four digits with repetition allowed: 10,000. Without: 5,040. The pool shrinks only when used items are not returned.
• "Counting is only for games." The same tools count passwords, DNA sequences, committee assignments, lottery designs, and the test statistics of modern experiments.
• "If the numbers are equally likely, my intuition about them is reliable." The birthday problem says otherwise. Trust the count, and use intuition to choose which count to set up.

Recap

• Multiplication principle: staged tasks multiply their per-stage choice counts.
• n! counts full arrangements of n distinct objects; 0! = 1 by useful convention.
• P(n, k) = n!/(n - k)! counts ordered selections; think shrinking pool.
• C(n, k) = n!/(k!(n - k)!) counts unordered selections; divide the orders out.
• Probabilities with equally likely outcomes are favorable count over total count, with both counts of the same kind.
• The complement rule turns "at least one" into "none," as in de Méré's bets and the birthday problem.

The key decision in every counting problem is a single question: does order matter? If yes, use a permutation; if no, use a combination. Answer that first, and the rest is arithmetic.

Sources

1. OpenStax. (2023). 7.1 The multiplication rule for counting. In Contemporary mathematics. Rice University. openstax.org
2. OpenStax. (2023). 7.2 Permutations. In Contemporary mathematics. Rice University. openstax.org
3. OpenStax. (2023). 7.3 Combinations. In Contemporary mathematics. Rice University. openstax.org
4. Weisstein, E. W. (n.d.). Birthday problem. MathWorld: A Wolfram web resource. mathworld.wolfram.com
5. Encyclopedia of Mathematics. (n.d.). Combinatorial analysis. European Mathematical Society. encyclopediaofmath.org
6. MacTutor History of Mathematics Archive. (n.d.). Blaise Pascal. University of St Andrews. mathshistory.st-andrews.ac.uk
7. MacTutor History of Mathematics Archive. (n.d.). Pierre Fermat. University of St Andrews. mathshistory.st-andrews.ac.uk

New information changes probabilities. If you learn that a rolled die shows an even number, the chance it is a 2 is no longer 1/6. If a card drawn from the deck is known to be red, the chance it is the ace of spades has dropped to 0. Conditional probability is the tool that revises a probability once we know some event has occurred, and it is the single most used idea in the rest of this course.

The big picture

You update probabilities informally all day. Seeing dark clouds raises your probability of rain. Hearing a cough raises a doctor's probability of a chest infection. Learning that a friend owns a truck changes your guess about where they live. In each case the pattern is the same: some event B is known to have happened, and the probability of another event A gets recomputed in that light. The notation for the updated number is P(A | B), read aloud as "the probability of A given B." The vertical bar is the word "given."

The mental picture to keep is a shrinking world. Before the information arrives, the whole sample space S is in play. Learning that B occurred throws away every outcome outside B. What remains is a smaller world, and inside it we re-ask our question about A. Conditional probability is ordinary probability, computed on the survivors.

Key idea: P(A | B) answers "of the times B happens, what fraction also have A?" New information shrinks the world; the bar records what the world has shrunk to.

The definition

The probability of A given B is defined as

P(A | B) = P(A and B) / P(B), provided P(B) is greater than 0.

Why this formula? Conditioning on B shrinks the world to just the outcomes in B. Within that world, the outcomes where A also happens are exactly the outcomes of "A and B." Dividing by P(B) rescales the shrunken world so its total probability is 1 again, the way a pie chart of the survivors must still total 100%. Numerator: the part of A living inside B. Denominator: the size of the new world.

Worked example. Roll a fair die. Let A = {2} and B = {2, 4, 6} (even). Work in steps.

• Step 1. The overlap: A and B is {2}, so P(A and B) = 1/6. (A sits entirely inside B.)
• Step 2. The new world: P(B) = 3/6 = 1/2. (Three outcomes survive the news.)
• Step 3. Divide: P(A | B) = (1/6) / (1/2) = 2/6 = 1/3.

What we just did: knowing the roll is even shrank six outcomes down to three equally likely survivors {2, 4, 6}, and the outcome 2 is one of those three. The update raised the probability from 1/6 to 1/3. Notice that conditioning can raise a probability, lower it, or leave it unchanged; the direction depends on how A overlaps B.

One warning before we go on: P(A | B) and P(B | A) are different questions with different answers. The fraction of even rolls that are 2 is 1/3, while the fraction of rolls of 2 that are even is 1. Keeping the two directions straight is exactly what Bayes' theorem, two lessons from now, is for.

Reading a two-way table

Conditional probabilities are easy to read from a table of counts. Suppose 200 students are classified by whether they study music and whether they play a sport:

	Sport	No sport	Total
Music	30	20	50
No music	90	60	150
Total	120	80	200

The chance a random student plays a sport given they study music is P(sport | music) = 30 / 50 = 0.60: restrict to the 50 music students, then find the fraction who play a sport. The conditioning event tells you which row or column to live in; its total becomes your denominator.

Run a few more reads to build the habit.

• P(music | sport) = 30/120 = 0.25: among the 120 athletes, 30 study music. Same overlap cell, different denominator, different answer: 0.25 versus 0.60. The two directions of conditioning really are different questions.
• P(sport) = 120/200 = 0.60: no conditioning, so the denominator is everyone.
• P(music and sport) = 30/200 = 0.15: the joint probability uses the grand total. Compare it with P(sport | music) = 30/50. Same 30 students on top, different world below.

What we just did: every one of these numbers is "a cell divided by a total," and the choice of total is the entire game. Joint probabilities divide by 200. Conditional probabilities divide by the total of the group you were told about.

Key idea: in a table, "given X" means "divide by X's total." If your denominator is still the grand total, you have not conditioned yet.

The multiplication rule

Rearranging the definition gives the multiplication rule for a joint probability:

P(A and B) = P(B) x P(A | B) = P(A) x P(B | A).

Read the middle version aloud: "the probability that both happen equals the probability the first happens, times the probability the second happens given the first did." This is the natural way to compute the probability of a sequence of events, one after another: each new factor is conditioned on everything so far.

Worked example. Draw two cards without replacement. What is the probability both are aces? Step 1: P(first is an ace) = 4/52. Step 2: given the first was an ace, 3 aces remain among 51 cards, so P(second is an ace | first was) = 3/51. Step 3: multiply: (4/52) x (3/51) = 12/2,652 = 1/221, about 0.0045. What we just did: the second factor is a conditional probability because the deck the second draw sees depends on the first draw.

The rule chains to any length. For three events, P(A and B and C) = P(A) x P(B | A) x P(C | A and B). Three hearts in a row without replacement: (13/52) x (12/51) x (11/50) = 0.0129, about 1.3%. Each numerator and each denominator drops by one as cards leave the deck, which is the bookkeeping the conditioning bar enforces.

With and without replacement

Worked example. An urn has 5 red and 3 blue balls (8 total). Draw two without replacement. Find the probability both are red. The first draw is red with probability 5/8. Given the first was red, only 4 red remain out of 7, so the second is red with probability 4/7. By the multiplication rule,

P(both red) = (5/8) x (4/7) = 20/56 = 5/14, about 0.357.

If we had drawn with replacement (returning the first ball), the draws would not affect each other and the answer would be (5/8) x (5/8) = 25/64 = 0.391. The difference is exactly the difference between conditional and unconditional second-draw probabilities: 4/7 versus 5/8. Removing a red ball made the second red slightly harder to get.

Notice when the distinction matters. With 8 balls, removing one changes the mix noticeably. If the urn held 5 million red and 3 million blue, removing one ball would barely move the fractions, and the two answers would nearly agree. This is why opinion polls, which sample people without replacement from a huge population, are usually analyzed as if draws were independent: the pool is too large to feel each removal.

A classic puzzle: the two-children problem

Conditional probability has famous puzzles, and this one repays careful setup. A family has two children, each independently equally likely to be a boy or a girl. Writing births in order, the sample space is {BB, BG, GB, GG}, four equally likely outcomes.

Question 1: given that the older child is a boy, what is the probability both are boys? The conditioning event is {BB, BG}, two outcomes. The favorable overlap is {BB}. So the answer is 1/2.

Question 2: given that at least one child is a boy, what is the probability both are boys? Now the conditioning event is {BB, BG, GB}, three outcomes, and the overlap is still {BB}. The answer is 1/3, not 1/2. The information "at least one boy" is weaker than "the older is a boy," so it rules out less and leaves a bigger world of survivors. Same question about A, different B, different answer. Writing out the shrunken world resolves what verbal intuition muddles.

Try it

A drawer has 4 black and 6 white socks. You pull two socks without replacement. Find the probability both are white. Work it out before reading on.

Worked answer. Step 1: P(first white) = 6/10. Step 2: given a white sock is gone, 5 whites remain among 9 socks, so P(second white | first white) = 5/9. Step 3: multiply: (6/10) x (5/9) = 30/90 = 1/3. What we just did: both the numerator and the denominator of the second factor dropped by one, because one white sock left the drawer and the drawer got smaller.

Where people get stuck

The most damaging confusion is swapping the two directions. P(positive test | healthy) and P(healthy | positive test) can differ enormously, and mistaking one for the other in courtrooms has a name, the prosecutor's fallacy. The table exercise is the antidote: the two directions use different denominators, 50 versus 120 in our example, so they are different numbers.

A second stumble is mixing up joint and conditional probability. "The probability a student does music and sport" is 30/200 = 0.15. "The probability a music student does sport" is 30/50 = 0.60. Same overlap, different worlds. When a sentence contains "given," "among," "of those," or "if we know," the denominator shrinks to that group.

Third, updating only half of the second-draw fraction. After removing a red ball, the second draw is 4/7: one fewer red on top, one fewer ball below. Writing 4/8 or 5/7 updates one count and forgets the other. Say the story aloud as you write each factor: "4 reds left, 7 balls left."

Common misconceptions

• "Conditioning always lowers a probability." It can raise one: learning the roll was even lifted P(rolled a 2) from 1/6 to 1/3. Information moves probabilities toward the truth, in either direction.
• "You can only condition on earlier events." The bar records information, not time order. Given that the second card was an ace, the probability the first was an ace is a legitimate question, and it also equals 3/51 by symmetry.
• "P(A | B) is a strange new kind of probability." It is ordinary probability on a restricted sample space; all the axioms and rules from earlier lessons still hold inside the shrunken world.
• "Without replacement is a minor technicality." In small pools it changes answers a lot: 5/14 versus 25/64 in the urn. Only in very large pools do the two blur together.

Recap

• P(A | B) = P(A and B) / P(B): the fraction of B's world where A also happens.
• In tables, "given" picks the row or column whose total becomes the denominator.
• Multiplication rule: P(A and B) = P(B) P(A | B) = P(A) P(B | A); chains extend factor by factor.
• Without replacement, each draw conditions the next; with replacement, draws do not interact.
• P(A | B) and P(B | A) are different questions; the next lessons build the machinery, independence and Bayes' theorem, that connects them.

Sources

1. OpenStax. (2023). 3.1 Terminology. In Introductory statistics 2e. Rice University. openstax.org
2. OpenStax. (2023). 3.4 Contingency tables. In Introductory statistics 2e. Rice University. openstax.org
3. OpenStax. (2023). 7.9 Conditional probability and the multiplication rule. In Contemporary mathematics. Rice University. openstax.org
4. Encyclopedia of Mathematics. (n.d.). Conditional probability. European Mathematical Society. encyclopediaofmath.org
5. Weisstein, E. W. (n.d.). Conditional probability. MathWorld: A Wolfram web resource. mathworld.wolfram.com
6. Siegrist, K. (n.d.). 02: Probability spaces. In Probability, mathematical statistics, and stochastic processes. LibreTexts. stats.libretexts.org
7. Grinstead, C. M., & Snell, J. L. (n.d.). Introductory probability. LibreTexts. stats.libretexts.org

Sometimes knowing that B happened tells you nothing about A. The die you roll today does not care what yesterday's roll showed; a coin has no memory of its last landing. Then A and B are independent. Independence is one of the most useful ideas in probability because it lets us multiply probabilities freely, and one of the most dangerous, because assuming it where it fails produces confident nonsense. This lesson builds the definition, shows how to test it with numbers, and walks the places it breaks.

The big picture

The last lesson was about information that changes probabilities: learning B shrinks the world and moves P(A) to P(A | B). Independence is the special case where the move goes nowhere. The news arrives, the world shrinks, and the fraction devoted to A is exactly what it was before. B happened, and A's prospects did not budge.

That is worth restating as the working definition: A and B are independent when P(A | B) = P(A). The conditional equals the unconditional; the information is worthless for predicting A. From there, one line of algebra produces the multiplication form that does all the practical work. Start with the multiplication rule, P(A and B) = P(B) x P(A | B), and replace P(A | B) with P(A): the joint probability becomes a plain product.

Key idea: independence means the conditioning bar changes nothing. Learning B leaves P(A) exactly where it stood, so joint probabilities factor into products.

The definition

Two events A and B are independent if

P(A and B) = P(A) x P(B).

Equivalently, and often more intuitively, A and B are independent when P(A | B) = P(A): conditioning on B does not change the probability of A. The two versions are the same statement in different clothes: divide both sides of the product form by P(B) and the conditional form appears. If the equation fails, the events are dependent.

Two remarks make the definition easier to use. First, it is symmetric: if B tells you nothing about A, then A tells you nothing about B; you never need to check both directions. Second, it is a numerical condition, not a feeling. Events that sound unrelated can be dependent, and events that sound linked can be independent, so the honest move is always to compute the three probabilities and compare.

Worked example: coins are independent

Flip a fair coin twice. Let A be "heads on the first flip" and B be "heads on the second flip." Test the definition in steps.

• Step 1. P(A) = 1/2: two of the four outcomes {HH, HT, TH, TT} have a first H, namely HH and HT.
• Step 2. P(B) = 1/2: the outcomes HH and TH.
• Step 3. P(A and B) = P({HH}) = 1/4.
• Step 4. Compare: P(A) x P(B) = (1/2)(1/2) = 1/4. Equal, so the flips are independent.

What we just did: we verified with counts what intuition claims, that the coin has no memory. So P(two heads) = 1/2 x 1/2 = 1/4, and ten heads in a row has probability (1/2)^10 = 1/1,024. Rare, but each new flip still lands heads with probability 1/2 no matter the streak, a point we return to below.

Worked example: testing for independence

Return to the 200-student table (30 music-and-sport, 50 music, 120 sport). We have P(music) = 50/200 = 0.25, P(sport) = 120/200 = 0.60, and P(music and sport) = 30/200 = 0.15. The product P(music) x P(sport) = 0.25 x 0.60 = 0.15 equals the joint probability, so in this dataset studying music and playing a sport are independent. Equivalently, P(sport | music) = 30/50 = 0.60 = P(sport): music students play sports at exactly the school-wide rate. The music information is useless for predicting sport, which is precisely what independence means.

Now a dependent pair, from one card drawn out of 52. Let A be "a king" and B be "a face card" (jack, queen, king, 12 cards). Then P(A) = 4/52, P(B) = 12/52, and since every king is a face card, P(A and B) = 4/52. Compare with the product: (4/52) x (12/52) = 48/2,704 = 0.018, while 4/52 = 0.077. Not equal, so the events are dependent. Sensibly so: learning the card is a face card lifts the chance of a king from 4/52 to P(A | B) = 4/12 = 1/3. Information that changes the odds is exactly what dependence is.

Key idea: to test independence, compute P(A), P(B), and P(A and B), then check the product. The arithmetic settles what wording cannot.

Independence is not mutual exclusivity

These two ideas are often confused, but they are nearly opposites:

• Mutually exclusive events cannot both happen: P(A and B) = 0.
• Independent events have P(A and B) = P(A) P(B).

If A and B each have positive probability and are mutually exclusive, then learning B occurred tells you A definitely did not, which is a huge amount of information, so they are strongly dependent, not independent. Take one coin flip with A = "heads" and B = "tails." They are disjoint, and P(A | B) = 0, a long way from P(A) = 1/2. In short, two events with nonzero probability cannot be both mutually exclusive and independent: disjointness makes the product P(A)P(B) positive while the joint probability is 0, and those can never match.

A picture helps. Mutually exclusive events are circles that do not touch. Independent events are circles that overlap by exactly the "expected" amount, the product of their sizes. No overlap is itself a very special, very informative amount of overlap.

Independence of several events and the complement trick

Events are mutually independent if the multiplication rule holds for every subset of them, and in modeling we usually justify this by the physical separateness of the trials: different coins, different customers, different components. Then joint probabilities are long products.

A common use is "at least one." Suppose a component works with probability 0.9, and three components fail or work independently. The probability all three work is 0.9 x 0.9 x 0.9 = 0.729. The probability at least one fails is easiest by complement: 1 - 0.729 = 0.271. Trying to add failure probabilities directly, 0.1 + 0.1 + 0.1 = 0.3, overcounts the cases where two or three fail together; the complement route sidesteps the overlap bookkeeping entirely.

The same trick powers redundancy engineering. If each of three independent backup systems fails with probability 0.1, the chance all three fail is 0.1^3 = 0.001, so P(at least one backup works) = 1 - 0.001 = 0.999. Three unreliable parts, arranged in parallel, behave like one very reliable system. This is why aircraft carry duplicate instruments and data centers keep multiple copies of your files: independence turns multiplication into safety.

One fine point for careful readers: pairwise independence does not guarantee mutual independence. Flip two fair coins and let A = "first is heads," B = "second is heads," C = "both flips match." Each pair multiplies correctly: every intersection of two of them is the single outcome HH with probability 1/4, matching the product 1/2 × 1/2. But P(A and B and C) = P(HH) = 1/4, while the triple product is 1/8. Knowing any two of these events pins down the third, so the trio is not mutually independent even though every pair is.

The gambler's fallacy

On 18 August 1913, the roulette wheel at Monte Carlo landed on black 26 times in a row. As the streak grew, gamblers piled ever larger bets on red, reasoning that red was "due." The wheel disagreed, and the casino collected a fortune. Each spin is independent of the last: given 25 blacks already, the chance the next spin is black is unchanged, about 18/37 on a European wheel. The streak was astonishing in advance, and irrelevant for the next spin.

The gambler's fallacy is treating independent trials as if they were sampling without replacement, as if the universe kept an urn of outcomes that must balance out. Contrast the two settings honestly. Drawing cards without replacement, dependence is real: seeing many red cards go by genuinely raises the chance the next is black, because the deck is depleting. Coin flips, die rolls, and roulette spins deplete nothing. Ask "does the mechanism remember?" A deck remembers; a wheel does not.

Key idea: independence is a property of the mechanism, not of your run of luck. Streaks change beliefs about fairness, perhaps, but for a fair mechanism they change nothing about the next trial.

Try it

Roll a fair die twice, using the 36-outcome sample space of ordered pairs. Let A = "the first roll is a 6" and B = "the two rolls sum to 7." Are A and B independent? Then answer the same question for B' = "the two rolls sum to 12." Work both before reading on.

Worked answer. Step 1: P(A) = 6/36 = 1/6 (pairs (6, 1) through (6, 6)). Step 2: P(B) = 6/36 = 1/6 (the six pairs summing to 7). Step 3: A and B is the single pair (6, 1), so P(A and B) = 1/36, and P(A)P(B) = (1/6)(1/6) = 1/36. Equal, so A and B are independent: whatever the first roll shows, exactly one second-roll value completes a 7.

Step 4: for B', P(B') = 1/36 (only (6, 6)), and P(A and B') = 1/36, but P(A)P(B') = (1/6)(1/36) = 1/216. Not equal, so A and B' are dependent: a sum of 12 forces the first roll to be a 6.

Where people get stuck

The costliest error is assuming independence to license multiplication. Rainy days cluster, exam questions share topics, machine parts share power supplies: multiplying their probabilities as if separate understates joint risk, sometimes wildly. A famous legal example: in the 1999 trial of Sally Clark, the probability of two natural infant deaths in one family was presented as 1 in 73 million by squaring 1 in 8,543, treating the deaths as independent. Deaths in the same family share genetics and environment, so the squaring was invalid, and the conviction built on it was eventually overturned. Independence must be argued from the mechanism, never assumed for convenience.

A second stumble is reading dependence as causation. Independence is about information, not influence. Ice cream sales and drowning deaths are dependent (both rise in summer) with no causal link between them. Dependence says "knowing one tells you about the other," and nothing more.

Third, misreading streaks. "Five heads in a row, tails is due" and "five heads in a row, heads is hot" are both wrong for a fair coin; the next flip is 1/2 either way. What a long streak can legitimately do is make you question whether the coin is fair, which is a question about the mechanism, taken up by statistics.

Common misconceptions

• "Independent means the events cannot happen together." That is mutual exclusivity, and it is nearly the opposite: disjoint events with positive probability are always dependent.
• "Independence is obvious from the story." Stories mislead in both directions; the product test on actual numbers is the only referee. The first-roll-6 and sum-7 example fools most people.
• "Checking all pairs is enough for a group of events." Pairwise independence can hold while the collection is not mutually independent, as the two-coin matching example shows.
• "After several failures, a success is due." Only in depleting pools, like decks. Memoryless mechanisms owe you nothing.

Recap

• A and B are independent when P(A and B) = P(A)P(B), equivalently P(A | B) = P(A).
• Test independence by computing; do not trust verbal plausibility.
• Disjoint and independent are incompatible for events of positive probability.
• For mutually independent events, joint probabilities are products, and "at least one" is handled by 1 minus P(none).
• Independence belongs to the mechanism; streaks of past outcomes do not change a memoryless trial.

With independence and conditioning both in hand, the next lesson combines them into the two most consequential formulas of the course: the law of total probability and Bayes' theorem.

Sources

1. OpenStax. (2023). 3.2 Independent and mutually exclusive events. In Introductory statistics 2e. Rice University. openstax.org
2. OpenStax. (2023). 3.3 Two basic rules of probability. In Introductory statistics 2e. Rice University. openstax.org
3. Encyclopedia of Mathematics. (n.d.). Independence. European Mathematical Society. encyclopediaofmath.org
4. Siegrist, K. (n.d.). 02: Probability spaces. In Probability, mathematical statistics, and stochastic processes. LibreTexts. stats.libretexts.org
5. Grinstead, C. M., & Snell, J. L. (n.d.). Introductory probability. LibreTexts. stats.libretexts.org
6. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124-1131. doi.org ↗
7. Tversky, A., & Kahneman, D. (1983). Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological Review, 90(4), 293-315. doi.org/10.1037/0033-295X.90.4.293

Two closely linked results let us combine and then reverse conditional probabilities. The law of total probability assembles an overall probability from case-by-case pieces; Bayes' theorem turns P(evidence | hypothesis) into P(hypothesis | evidence), the direction we actually care about. Together they are the backbone of statistical reasoning, machine learning, spam filtering, and everyday diagnosis, and they power the most famous surprises in probability.

The big picture

Real questions rarely hand you probabilities in the direction you need. Medicine measures how often a test is positive among people known to have a disease; a patient wants the reverse, the chance of disease given a positive. A spam filter knows how often spam contains the word "free"; it needs the chance an email containing "free" is spam. In both cases the known conditional points one way and the needed conditional points the other.

The reversal takes two steps, and they are this lesson. First, compute the overall probability of the evidence by splitting the world into cases, which is the law of total probability. Second, ask what fraction of that evidence-probability came from the hypothesis you care about, which is Bayes' theorem. Every calculation ahead, however famous, is those two steps in costume.

Key idea: total probability averages over cases to get P(evidence); Bayes' theorem divides one case's share by that total to reverse the conditional.

The law of total probability

Suppose the sample space is split into disjoint cases that cover everything, for example B and "not B." Then the probability of any event A is the weighted average of its conditional probabilities across those cases:

P(A) = P(A | B) P(B) + P(A | not B) P(not B).

Read it aloud: "the chance of A is the chance of A in the first case, weighted by how likely that case is, plus the same for the other case." The weights are the case probabilities and must sum to 1. With more than two cases the sum grows a term per case, and the requirement is always the same: the cases must be disjoint (no overlap) and exhaustive (nothing missed), a setup called a partition.

Worked example. Suppose it rains on 30% of days. When it rains, your bus is late with probability 0.6; when it does not, with probability 0.1. What fraction of days is the bus late?

• Step 1. The cases: rain (probability 0.3) and no rain (probability 0.7). Disjoint, exhaustive.
• Step 2. The pieces: P(late | rain) = 0.6 and P(late | no rain) = 0.1.
• Step 3. Weight and add: P(late) = (0.6)(0.3) + (0.1)(0.7) = 0.18 + 0.07 = 0.25.

What we just did: the overall rate 0.25 landed between 0.1 and 0.6, and closer to 0.1, because the no-rain case carries more weight. A total-probability answer is always a compromise between the case rates, pulled toward the likelier case. If your "average" drifts outside the range of the case values, a weight has gone missing.

Bayes' theorem

Bayes' theorem reverses a conditional probability, turning P(A | B) into P(B | A). It falls out of the multiplication rule in three short steps.

• Step 1. The joint probability has two factorizations: P(A and B) = P(A | B) P(B) and also P(A and B) = P(B | A) P(A). (Both are the multiplication rule from two directions.)
• Step 2. Set them equal: P(B | A) P(A) = P(A | B) P(B).
• Step 3. Divide both sides by P(A):

P(B | A) = P(A | B) P(B) / P(A),

where the denominator P(A) is usually computed with the law of total probability. Here P(B) is the prior (what we believed before the evidence A), and P(B | A) is the posterior (our updated belief after seeing A). The theorem is bookkeeping, not magic: of all the probability that flows into the evidence, it asks what share flowed through the hypothesis.

Worked example. Continue the bus story. Given that the bus is late today, what is the probability it is raining? We want P(rain | late). Numerator: P(late | rain) P(rain) = (0.6)(0.3) = 0.18. Denominator: P(late) = 0.25 from before. So P(rain | late) = 0.18 / 0.25 = 0.72. Lateness raised the probability of rain from the prior 0.30 to the posterior 0.72, because lateness is six times more common on rainy days.

Worked example: a medical test

A disease affects 1% of a population, so P(D) = 0.01. A test has sensitivity 99% (it is positive for 99% of people who have the disease, so P(+ | D) = 0.99) and specificity 95% (it is negative for 95% of healthy people, so P(+ | not D) = 0.05). A random person tests positive. What is the probability they actually have the disease, P(D | +)?

Step 1, total probability of a positive test.

P(+) = P(+ | D) P(D) + P(+ | not D) P(not D) = (0.99)(0.01) + (0.05)(0.99) = 0.0099 + 0.0495 = 0.0594.

Step 2, Bayes' theorem.

P(D | +) = P(+ | D) P(D) / P(+) = 0.0099 / 0.0594 = 0.1667, that is about 1/6, only 16.7%.

This famous result surprises almost everyone: even after a positive result from a fairly accurate test, the person probably does not have the disease. The reason is the low base rate. Because 99% of people are healthy, the many false positives from that large group (about 4.95% of everyone) swamp the relatively few true positives (0.99% of everyone).

If the symbols feel slippery, recount the story with 10,000 concrete people. About 100 of them have the disease (1%), and the test catches 99. The other 9,900 are healthy, and 5% of them, about 495 people, receive false positives anyway. Line up everyone who tested positive: 99 + 495 = 594 people, of whom only 99 are actually ill. The fraction is 99/594 = 1/6, the same 16.7%, no formula required. Natural frequencies like these are how the calculation is taught to physicians, and they are a fine way to check any Bayes answer. A tree diagram makes the two positive paths visible.

The two positive branches total 0.0594, and the true-positive branch is 0.0099 of that, giving the 16.7% posterior. Bayes' theorem is simply careful bookkeeping of these branches. Notice what would repair the number: screen a higher-risk group (raise the prior) or use a more specific test (shrink the false-positive branch). Both moves show up directly in the arithmetic, which is why follow-up tests after a positive screen are standard practice.

The Monty Hall problem

The most argued-about probability puzzle in history is a Bayes problem in disguise. On a game show, a car sits behind one of three doors, goats behind the other two. You pick door 1. The host, who knows where the car is and always opens a goat door from the two you did not pick, opens door 3. He offers a switch to door 2. Should you take it?

Most people reason "two doors remain, so it is 50-50." The cases say otherwise. Your first pick is right with probability 1/3 and wrong with probability 2/3, and the host's reveal never changes those facts, because he can always show a goat.

• Case 1, probability 1/3: your door hides the car. Switching loses.
• Case 2, probability 2/3: your door hides a goat. The host's hands are tied: he must open the other goat door, so the remaining door hides the car. Switching wins.

So switching wins with probability 2/3. The Bayes version confirms it. Priors: car behind 1, 2, or 3, each 1/3. Likelihood of the observation "host opens door 3": if the car is behind door 1, he picks between doors 2 and 3, so 1/2; if behind door 2, he is forced to open door 3, so 1; if behind door 3, impossible, 0. Total probability of the observation: (1/3)(1/2) + (1/3)(1) + (1/3)(0) = 1/2. Posterior for door 2: (1/3)(1) / (1/2) = 2/3.

The host's knowledge is the entire trick. A host who opened a random unpicked door and merely happened to reveal a goat would leave the doors at 1/2 each; a forced reveal is stronger evidence than a lucky one. When columnist Marilyn vos Savant published the 2/3 answer in 1990, thousands of readers, some with mathematics doctorates, wrote in to insist on 50-50. The readers had the wrong likelihoods, and Bayes had it right.

Try it

At an email provider, 20% of incoming mail is spam. The word "free" appears in 60% of spam messages and in 4% of legitimate messages. An email arrives containing "free." What is the probability it is spam? Work both steps before reading on.

Worked answer. Step 1, total probability of seeing "free": P(free) = (0.60)(0.20) + (0.04)(0.80) = 0.12 + 0.032 = 0.152. Step 2, Bayes: P(spam | free) = 0.12 / 0.152 = 0.789, about 79%. What we just did: the word is strong but not conclusive evidence, because legitimate mail is four times more common than spam and occasionally says "free" too. Real filters multiply evidence from many words in exactly this fashion.

Where people get stuck

The central hazard is swapping P(A | B) for P(B | A). Sensitivity 99% does not mean a positive person is 99% likely to be ill; those are different denominators, test-takers with the disease versus test-takers with a positive. Courtrooms meet the same swap as the prosecutor's fallacy: "the match probability for an innocent person is 1 in a million" is not "the defendant is innocent with probability 1 in a million." Whenever a conditional arrives, say its denominator out loud before using it.

The second hazard is dropping a path from the denominator. P(evidence) must include every route to the evidence: true positives and false positives; spam saying "free" and ham saying "free." Omitting the route you find unlikely does not make it disappear; it makes your posterior too confident. The tree diagram exists precisely to keep all branches on the page.

Third, neglecting the base rate. Psychologists Daniel Kahneman and Amos Tversky showed that people presented with vivid case evidence, a confident witness, a positive test, tend to ignore the prior almost entirely. The remedy that works in practice is the 10,000-people recount: base rates become headcounts, and headcounts are hard to ignore.

Common misconceptions

• "A 99% accurate test means a positive is 99% certain." Not when the condition is rare: at 1% prevalence the posterior was 1/6. Accuracy alone never determines the posterior; the prior always matters.
• "Priors are subjective guesswork." In these problems the prior is a measured base rate: prevalence, spam share, rain frequency. Bayes' theorem runs on data.
• "Two options left means 50-50." Monty Hall's doors and the boy-girl puzzles say otherwise: how the options survived the evidence sets their probabilities, not their count.
• "Bayes' theorem is advanced mathematics." It is one division applied to the multiplication rule; the 10,000-person table computes it with grade-school arithmetic.

Recap

• Partition the world into disjoint, exhaustive cases; total probability weights each case's conditional by the case probability.
• Bayes' theorem: P(B | A) = P(A | B) P(B) / P(A), prior in, posterior out.
• The denominator P(A) collects every path to the evidence, most often via total probability.
• Low base rates drag posteriors down; false positives from the large healthy group swamp true positives from the small ill one.
• Natural frequencies (recount with 10,000 people) and tree diagrams are reliable ways to keep the bookkeeping honest.

Sources

1. OpenStax. (2023). 7.9 Conditional probability and the multiplication rule. In Contemporary mathematics. Rice University. openstax.org
2. Encyclopedia of Mathematics. (n.d.). Bayes formula. European Mathematical Society. encyclopediaofmath.org
3. Weisstein, E. W. (n.d.). Total probability theorem. MathWorld: A Wolfram web resource. mathworld.wolfram.com
4. Weisstein, E. W. (n.d.). Monty Hall problem. MathWorld: A Wolfram web resource. mathworld.wolfram.com
5. Bayes, T. (1763). An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53, 370-418. doi.org ↗
6. Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684-704. doi.org/10.1037/0033-295X.102.4.684
7. Morgan, J. P., Chaganty, N. R., Dahiya, R. C., & Doviak, M. J. (1991). Let's make a deal: The player's dilemma. The American Statistician, 45(4), 284-287. doi.org/10.1080/00031305.1991.10475821

A random variable assigns a number to each outcome of an experiment, so we can do arithmetic with chance. We write random variables with capital letters like X, and particular values with lowercase x, so "X = x" reads "the random variable X came out equal to the value x." This lesson introduces the three numbers that summarize a random variable: its distribution, its center, and its spread.

The big picture

Outcomes are often not numbers. "HH," "the customer bought the blue jacket," "the die shows a five and a two" are events, not quantities. But almost every practical question is numeric: how many heads, how much revenue, what total? A random variable is the bridge, a labeling rule that attaches a number to every outcome. Flip two coins and let X count the heads: the outcome HT gets the label 1, HH gets 2, TT gets 0.

Once outcomes wear numeric labels, two summary questions make sense. Where is the center of the distribution, the long-run average if the experiment repeats forever? That is the expected value. And how far do results wander from that center, are they huddled close or scattered wide? That is the variance and its square root, the standard deviation. Center and spread, a two-number portrait of randomness, carry most of this course from here on.

Key idea: a random variable turns outcomes into numbers; expectation locates the balance point of those numbers, and variance measures the scatter around it.

Discrete random variables and the pmf

A discrete random variable takes separate, countable values (like a count). Its behavior is captured by a probability mass function (pmf), p(x) = P(X = x), which must satisfy two conditions: every p(x) is between 0 and 1, and the values sum to 1. The pmf is the complete inventory of what X can be and how likely each value is.

For a single fair die, X = the number rolled, and p(x) = 1/6 for each x in 1 through 6.

For the two-coin count of heads, read the pmf straight off the sample space {HH, HT, TH, TT}: the value 0 comes only from TT, so p(0) = 1/4; the value 1 comes from HT and TH, so p(1) = 2/4 = 1/2; the value 2 comes only from HH, so p(2) = 1/4. Check the total: 1/4 + 1/2 + 1/4 = 1. That final check is worth the five seconds every time; a pmf that does not sum to 1 has lost or double-counted an outcome somewhere.

Key idea: the pmf is a complete list of values and weights. Nonnegative weights, total weight 1, no exceptions.

Expected value

The expected value (or mean) of X, written E[X] or the Greek letter mu, is the long-run average value, computed as a probability-weighted sum:

E[X] = sum of x times p(x), over all values x.

Read it aloud: "each value, times how often it happens, added up." Values that occur often count more; rare values count less. That is exactly how you would average a semester grade with weighted assignments, and it is the same arithmetic.

Worked example. For the fair die, E[X] = 1(1/6) + 2(1/6) + ... + 6(1/6) = (1+2+3+4+5+6)/6 = 21/6 = 3.5. The expected value need not be a possible outcome; no face shows 3.5. It is the balance point of the distribution, the spot where the pmf would balance on a fingertip, and the average you would see per roll over thousands of rolls.

Worked example. For the two-coin head count: E[X] = 0(1/4) + 1(1/2) + 2(1/4) = 0 + 0.5 + 0.5 = 1. On average, two flips deliver one head, which squares with common sense.

Expectation is how gambling and insurance are priced. Bet 1 dollar on red at American roulette: the wheel has 38 pockets, 18 red, so you win 1 dollar with probability 18/38 and lose 1 dollar with probability 20/38. Then

E[winnings] = (+1)(18/38) + (-1)(20/38) = -2/38 = -0.0526 dollars.

You lose about 5.3 cents per dollar bet, on average. Any single spin wins or loses a whole dollar; the expectation only reveals itself across many spins, which is why casinos, who play thousands of times a night, care about it more than any single gambler does. An insurance premium is the same computation from the other side: expected payout plus costs.

Key idea: E[X] is a weighted average and a long-run per-play rate. It predicts totals over many repetitions, not the next single result.

Variance and standard deviation

Two distributions can share a center yet differ wildly in personality: a savings account and a volatile stock can both average 4% growth. The missing summary is spread. The variance measures spread around the mean as the expected squared distance from it:

Var(X) = E[(X - mu)^2] = E[X^2] - (E[X])^2.

Why square the distances? Raw deviations average to zero by the definition of the mean, positives canceling negatives, so we square first to make every deviation count positively. The second form (the "computing formula") is usually easier: find the mean of the squares, then subtract the square of the mean. The standard deviation is the square root of the variance and carries the same units as X, which makes it the number you quote.

Worked example. For the fair die, first find E[X^2] = (1 + 4 + 9 + 16 + 25 + 36)/6 = 91/6 = 15.1667. Then Var(X) = 15.1667 - (3.5)^2 = 15.1667 - 12.25 = 2.9167 (exactly 35/12). The standard deviation is square root of 2.9167 = 1.708. Loosely, a die roll typically lands about 1.7 away from 3.5, which matches a glance at the faces.

Worked example. For the two-coin head count: E[X^2] = 0(1/4) + 1(1/2) + 4(1/4) = 1.5, so Var(X) = 1.5 - 1^2 = 0.5 and the standard deviation is about 0.707. Small numbers, small spread: with only two flips the count cannot wander far from 1.

Run the arithmetic checks as you go: variance can never be negative, since it averages squares. A negative variance always means the subtraction was done backwards or E[X^2] was computed with unsquared values.

Linearity and scaling rules

Two rules save enormous effort. For constants a and b:

• Expectation is linear: E[aX + b] = a E[X] + b. (This holds for sums too: E[X + Y] = E[X] + E[Y] always, even if X and Y are dependent.)
• Variance scaling: Var(aX + b) = a^2 Var(X). Adding a constant b shifts the data but does not change its spread, so b drops out; multiplying by a stretches spread by a factor of a-squared.

Worked example. Suppose X has mean 5 and variance 4, and we define Y = 2X + 3. Then E[Y] = 2(5) + 3 = 13, and Var(Y) = 2^2 (4) = 16, so the standard deviation of Y is 4. These rules will let us read off the mean and variance of the named distributions in the next lessons without re-deriving them each time.

A physical version makes the rules memorable. Convert daily temperatures from Celsius to Fahrenheit, F = 1.8C + 32. The average converts the same way any temperature does: if the Celsius mean is 20, the Fahrenheit mean is 1.8(20) + 32 = 68. But the +32 does nothing to day-to-day variability; only the stretch factor 1.8 does. A Celsius standard deviation of 5 becomes 1.8 × 5 = 9 Fahrenheit, and the variance multiplies by 1.8^2 = 3.24. Shifts move the center; only rescaling changes the spread.

Worked example: pricing a game. A carnival booth charges 2 dollars to roll one die: you win the face value in dollars if the roll is even, nothing if odd. Let W be the payout. Its pmf: P(W = 2) = P(W = 4) = P(W = 6) = 1/6, and P(W = 0) = 1/2. So E[W] = 2(1/6) + 4(1/6) + 6(1/6) = 12/6 = 2 dollars.

Expected net result: 2 - 2 = 0. This game is exactly fair; a real carnival would charge 2.50 dollars, pocketing an expected 50 cents per play. What we just did: build the pmf, weight, add, compare with the price.

Try it

A raffle sells 100 tickets at 1 dollar each; one ticket wins a 60-dollar prize. Let X be your net gain from buying one ticket. Write the pmf of X, then compute E[X]. Work it out before reading on.

Worked answer. Step 1, the values: if you win, you gain 60 - 1 = 59 dollars; if you lose, you are out 1 dollar. Step 2, the pmf: P(X = 59) = 1/100 = 0.01 and P(X = -1) = 99/100 = 0.99. Check: 0.01 + 0.99 = 1. Step 3, weight and add: E[X] = 59(0.01) + (-1)(0.99) = 0.59 - 0.99 = -0.40. On average a ticket costs you 40 cents. What we just did: the same weighted-average recipe, applied to money, with the check that the pmf sums to 1.

Where people get stuck

The name "expected value" causes the most trouble, because E[X] is often a value you never expect on any single trial. No die shows 3.5, and no raffle ticket loses exactly 40 cents. Read E[X] as "long-run average per play," never as "the most likely result." The most likely single value is a different summary (the mode), and the two can disagree sharply.

Next, the squared-versus-unsquared slip in variance. E[X^2] means "square the values first, then weight and add." For the die, that is 91/6. Computing (E[X])^2 = 12.25 instead, or forgetting to subtract it, gives nonsense. Keep the two quantities on separate lines and subtract at the end, and remember the order: mean of squares minus square of the mean, never the reverse, which would go negative.

Third, units. Variance is in squared units, dollars-squared, degrees-squared, which is why we report the standard deviation instead. When an answer needs interpreting, take the square root and say the units aloud: "typically about 1.7 pips away from the mean."

Finally, resist inventing conditions for linearity. E[X + Y] = E[X] + E[Y] requires nothing at all, not independence, not identical distributions. (Variance is the one that needs care with sums, and that story comes two lessons before the finale.)

Common misconceptions

• "E[X] is what will happen next." It is the average over many repetitions. Single trials routinely land far from it, and for lopsided bets like raffles, most trials land below it.
• "Standard deviation is the average distance from the mean." Close but not exact: it is the square root of the average squared distance, which weights big deviations more heavily. It is a usable ruler for "typical distance," not a literal one.
• "Adding 10 to every value adds 10 to the spread." Shifts move every value together, so distances between values, and hence variance, do not change: Var(X + 10) = Var(X).
• "A fair game is worth playing." Fair means E = 0, but variance still matters: a fair coin flip for your life savings has expectation zero and enormous spread. Center and spread are separate judgments.

Recap

• A random variable labels each outcome with a number; its pmf lists values and probabilities, summing to 1.
• E[X] = sum of x times p(x): the probability-weighted balance point and long-run per-trial average.
• Var(X) = E[X^2] - (E[X])^2 measures spread; its square root, the standard deviation, shares X's units.
• E[aX + b] = aE[X] + b; Var(aX + b) = a^2 Var(X): shifts move the mean only, stretches scale the spread.
• Check yourself constantly: pmf sums to 1, variance is never negative, units make sense.

Next we meet the first named distributions, Bernoulli and binomial, where these formulas stop being abstract and start being read off a label.

Sources

1. OpenStax. (2023). 4.1 Probability distribution function (PDF) for a discrete random variable. In Introductory statistics 2e. Rice University. openstax.org
2. OpenStax. (2023). 4.2 Mean or expected value and standard deviation. In Introductory statistics 2e. Rice University. openstax.org
3. Encyclopedia of Mathematics. (n.d.). Random variable. European Mathematical Society. encyclopediaofmath.org
4. Encyclopedia of Mathematics. (n.d.). Mathematical expectation. European Mathematical Society. encyclopediaofmath.org
5. Encyclopedia of Mathematics. (n.d.). Variance. European Mathematical Society. encyclopediaofmath.org
6. Siegrist, K. (n.d.). 04: Expected value. In Probability, mathematical statistics, and stochastic processes. LibreTexts. stats.libretexts.org
7. Khan Academy. (n.d.). Random variables. khanacademy.org

Many experiments reduce to a single yes-or-no question. Did the coin land heads? Did the customer click? Did the part pass inspection? Repeating that question a fixed number of times and counting the yeses leads to the most important discrete distribution in statistics, the binomial. This lesson builds it from its one-trial ingredient, the Bernoulli distribution, and shows how to compute with it accurately.

The big picture

From here on, the course works with named distributions: standard random variables whose pmf, mean, and variance have been worked out once and for all. A named distribution is a pre-solved template. When a real situation matches the template's conditions, you inherit every formula instantly, no new derivation required. The skill being trained is therefore twofold: recognize when the template applies, and then use its formulas correctly.

The binomial template fits any count of successes in a fixed number of identical, independent yes-no trials: heads in 10 flips, defective parts in a sample of 50, voters favoring a candidate in a random sample of 1,000, made free throws in 20 attempts. One template, thousands of applications.

Key idea: a distribution is a reusable answer. Match the setting to the template, then read off probabilities, mean, and variance from stock formulas.

The Bernoulli distribution

A Bernoulli trial is a single experiment with exactly two outcomes, called success (X = 1) and failure (X = 0), where success has probability p. "Success" is a technical label, not praise: if we are counting defective parts, finding a defect counts as a success. The pmf is P(X = 1) = p and P(X = 0) = 1 - p. A quick calculation gives its mean and variance:

• E[X] = 1(p) + 0(1 - p) = p.
• Var(X) = E[X^2] - p^2 = p - p^2 = p(1 - p).

The variance line uses a small trick worth seeing: since X is only ever 0 or 1, squaring changes nothing, 0 squared is 0 and 1 squared is 1, so E[X^2] = E[X] = p. Then the computing formula gives p - p², which factors as p(1 - p).

For a fair coin (success = heads, p = 0.5), the mean is 0.5 and the variance is 0.25. For p = 0.6, the mean is 0.6 and the variance is 0.6 x 0.4 = 0.24. For p = 0.9 the variance falls to 0.9 x 0.1 = 0.09. Notice the pattern: variance is largest at p = 0.5, where the trial is most unpredictable, and shrinks toward 0 as p approaches 0 or 1, where the outcome is nearly a foregone conclusion. The formula p(1 - p) is quietly measuring how uncertain a yes-no question is.

The binomial setting

A binomial random variable X counts the number of successes in n Bernoulli trials. Four conditions must hold (the "BINS" checklist):

1. Binary: each trial is a success or a failure.
2. Independent: trials do not affect one another.
3. Number fixed: the number of trials n is set in advance.
4. Same probability: the success probability p is the same on every trial.

Practice the checklist on near misses, because they are common. Drawing 5 cards without replacement and counting hearts fails the I: each draw changes the deck, so trials are dependent (that count has its own distribution, the hypergeometric). Flipping until the first head fails the N: the number of flips is not fixed in advance (that is the geometric distribution, next lesson). Free throws in a game where fatigue grows fails the S if p drifts. When all four letters check out, and only then, the binomial formulas below apply.

The binomial formula

The probability of exactly k successes in n trials is

P(X = k) = C(n, k) p^k (1 - p)^(n - k).

The formula has two working parts, and each is worth hearing separately. The tail p^k (1 - p)^(n-k) is the probability of one specific ordering of k successes and n - k failures: by independence, the per-trial probabilities multiply. The front C(n, k) counts how many orderings exist, using last module's combinations, because the successes can sit in any k of the n trial slots.

See it concretely with n = 3, k = 2, p = 0.5. The orderings with two successes are SSF, SFS, and FSS, and each has probability (0.5)^2 (0.5)^1 = 1/8. Three disjoint orderings at 1/8 apiece: P(X = 2) = 3/8. The formula packages that count-times-probability argument: C(3, 2) = 3 orderings, each at p^2(1-p).

Worked example. Flip a fair coin 5 times. What is the probability of exactly 3 heads? Here n = 5, k = 3, p = 0.5:

P(X = 3) = C(5, 3) (0.5)^3 (0.5)^2 = 10 x (0.5)^5 = 10/32 = 0.3125.

Worked example with p not equal to 0.5. A quiz has 10 questions, each answered correctly with probability 0.3, independently. The probability of exactly 2 correct is

P(X = 2) = C(10, 2) (0.3)^2 (0.7)^8 = 45 x 0.09 x 0.05765 = 0.2335.

Follow the arithmetic order: the combination first (45), then each power separately (0.3 squared is 0.09; 0.7 to the eighth is 0.05765), then multiply the three pieces. Powers before products prevents most calculator slips.

Questions about ranges add the relevant exact probabilities, since different values of k are disjoint events. For the 5-flip coin, P(at least 4 heads) = P(X = 4) + P(X = 5) = 5/32 + 1/32 = 6/32 = 0.1875. And "at most" questions often go fastest by complement: P(at most 3 heads) = 1 - 0.1875 = 0.8125.

Mean and variance of a binomial

Because a binomial X is the sum of n independent Bernoulli variables, one per trial, its mean and variance are n times the Bernoulli values:

E[X] = np and Var(X) = np(1 - p).

The mean needs no memorizing once you see it through linearity of expectation: each trial contributes p on average, and n trials contribute np. For the 10-question quiz, E[X] = 10 x 0.3 = 3 correct answers on average, with variance 10 x 0.3 x 0.7 = 2.1 and standard deviation about 1.449. The mean np is the single most useful fact about a binomial: it is your best guess for the number of successes.

A basketball player who makes 75% of free throws and attempts 20 has E[X] = 20 x 0.75 = 15 makes, variance 20 x 0.75 x 0.25 = 3.75, standard deviation about 1.94. So a typical game lands within about 2 makes of 15; a night with 9 makes is a three-standard-deviation surprise, worth a second look, while 13 is routine variation. Mean plus-or-minus a couple of standard deviations is a quick, powerful reality check.

Shape is worth a mental picture too. When p = 0.5 the pmf is symmetric around n/2. When p is small the histogram leans right: most mass sits at low counts, with a tail stretching toward larger ones. When p is large the lean reverses. In every case the tallest bars sit at or next to np, and the standard deviation sets how quickly the bars die away on either side.

Worked example: quality control. A production line runs 2% defective, and an inspector samples 10 items (independence is reasonable because the line's output is vast). Let X count defects, binomial with n = 10, p = 0.02. Then P(X = 0) = (0.98)^10 = 0.817, so P(at least one defect) = 1 - 0.817 = 0.183. And P(X = 1) = C(10, 1)(0.02)(0.98)^9 = 10 x 0.02 x 0.834 = 0.167. What we just did: the complement rule handled "at least one," and the exact formula handled "exactly one." Small samples usually miss rare defects, which is why acceptance sampling plans choose n carefully.

Try it

A 4-question multiple-choice quiz has 4 options per question, and you guess every answer blindly, so p = 0.25 per question, independently. Find the probability of exactly 2 correct. Work it out before reading on.

Worked answer. Step 1, check BINS: binary (right or wrong), independent guesses, n = 4 fixed, same p = 0.25. Step 2, the pieces: C(4, 2) = 6; (0.25)^2 = 0.0625; (0.75)^2 = 0.5625. Step 3, multiply: P(X = 2) = 6 x 0.0625 x 0.5625 = 0.211, about 21%. What we just did: combination, success power, failure power, product, in that order every time.

Where people get stuck

The most frequent slip is dropping the C(n, k) factor. The bare product p^k(1-p)^(n-k) is the probability of one particular arrangement, and unless the question names an exact arrangement ("the first three flips are heads, then two tails"), you must count all the arrangements that deliver k successes. If your binomial answer looks suspiciously small, the missing combination is the first suspect.

Second, exponent swaps. The exponent on p is the success count k; the exponent on 1 - p is the failure count n - k. In the quiz example, writing (0.3)^8(0.7)^2 instead of (0.3)^2(0.7)^8 changes the answer by orders of magnitude. Label each factor in words, "two successes, eight failures," before raising anything to a power.

Third, answering "at least" with "exactly." P(at least one defect) is not P(exactly one); it is 1 - P(none), or a sum of exact terms. Translate the words into a set of k values first, then decide whether direct addition or the complement is shorter.

Finally, using the binomial when BINS fails, most often the I. Sampling without replacement from a small population makes trials dependent; the binomial is then only an approximation, decent when the sample is under about 10% of the population and shaky beyond that.

Common misconceptions

• "The most likely count is n/2." The distribution centers at np, not n/2. For n = 10, p = 0.3, counts near 3 are most likely, and 5 heads-style intuition misleads.
• "np must be a whole number." The mean is an average, not a possible value: n = 5, p = 0.3 gives E[X] = 1.5 successes, a perfectly sensible long-run rate.
• "Binomial requires p = 0.5." Any fixed p in (0, 1) works; p = 0.5 is merely the symmetric special case.
• "With many trials, anything can happen, so the histogram flattens." The opposite: for large n the probability piles up near np in a bell shape, a preview of the Central Limit Theorem waiting at the end of the course.

Recap

• Bernoulli: one yes-no trial; mean p, variance p(1 - p), maximal uncertainty at p = 0.5.
• Binomial: the count of successes in n independent same-p trials; check all four BINS conditions.
• P(X = k) = C(n, k) p^k (1 - p)^(n-k): orderings counted by the combination, each ordering priced by independence.
• E[X] = np and Var(X) = np(1 - p); mean plus-or-minus two standard deviations flags surprising results.
• "At least" and "at most" questions are sums of exact terms, often shortest through the complement.

Next come two more discrete templates, the Poisson for counting rare events in an interval, and the geometric for waiting until the first success.

Sources

1. OpenStax. (2023). 4.3 Binomial distribution. In Introductory statistics 2e. Rice University. openstax.org
2. OpenStax. (2023). 7.10 The binomial distribution. In Contemporary mathematics. Rice University. openstax.org
3. NIST/SEMATECH. (n.d.). 1.3.6.6.18 Binomial distribution. In e-Handbook of statistical methods. National Institute of Standards and Technology. itl.nist.gov
4. Encyclopedia of Mathematics. (n.d.). Binomial distribution. European Mathematical Society. encyclopediaofmath.org
5. Weisstein, E. W. (n.d.). Bernoulli distribution. MathWorld: A Wolfram web resource. mathworld.wolfram.com
6. Weisstein, E. W. (n.d.). Binomial distribution. MathWorld: A Wolfram web resource. mathworld.wolfram.com
7. MacTutor History of Mathematics Archive. (n.d.). Jacob Bernoulli. University of St Andrews. mathshistory.st-andrews.ac.uk

Two more discrete distributions round out the essential toolkit: the Poisson, for counting events in a span of time or space, and the geometric, for waiting until the first success. Both are one-parameter templates, both come with clean formulas, and both describe an enormous range of real processes, from help-desk calls and typos to lottery streaks and equipment failures.

The big picture

The binomial needed a fixed number of trials n. But many counting questions have no visible trials at all. How many calls reach a help desk in an hour? How many typos sit on a page? How many meteors cross the sky tonight? There is no n to point to, only a window of time or space and an average rate. The Poisson distribution is the binomial's limit for exactly this situation.

The second template flips the question. Instead of "how many successes in n trials?" it asks "how many trials until the first success?" Rolling a die until the first 6, pitching sales calls until the first yes, testing parts until the first defect: that waiting count follows the geometric distribution. Counts in a window, waits until a hit: learn to hear which question a problem is asking, and the formulas follow.

Key idea: Poisson counts events in an interval at a known average rate; geometric counts the trials spent waiting for the first success.

The Poisson distribution

The Poisson distribution models the number of events that occur in a fixed interval when events happen independently at a constant average rate lambda (the Greek letter, its mean number of events). Examples include calls arriving at a help desk per hour, typos per page, or decays per second. Its pmf, for k = 0, 1, 2, ..., is

P(X = k) = e^(-lambda) lambda^k / k!,

where e is the constant 2.71828..., the base of natural logarithms. Read the formula aloud: "e to the minus lambda, times lambda to the k, divided by k factorial." A defining feature is that the Poisson mean and variance are equal:

E[X] = lambda and Var(X) = lambda.

Worked example. A call center receives on average 3 calls per minute, so lambda = 3. Find the probability of exactly 2 calls in a given minute:

P(X = 2) = e^(-3) 3^2 / 2! = e^(-3) x 9 / 2 = 0.049787 x 4.5 = 0.2240.

The probability of no calls is P(X = 0) = e^(-3) = 0.0498, using lambda^0 = 1 and 0! = 1. The probability of at most one call is P(X = 0) + P(X = 1) = e^(-3)(1 + 3) = 0.0498 x 4 = 0.1991, and so, by the complement, P(at least 2 calls) = 1 - 0.1991 = 0.8009.

Mind the window. Lambda is a count per interval, so changing the interval rescales it: at 3 calls per minute, a 2-minute window has lambda = 6, and a 20-second window has lambda = 1. Matching lambda's units to the question's window is the first thing to check in any Poisson problem.

Where does this formula come from? The Poisson is the binomial pushed to a limit: many tiny opportunities, each rarely taken, with a steady expected total np. Picture a 1,000-word page where each word independently carries a typo with probability 0.002, so lambda = np = 2 typos on average. The binomial gives P(no typos) = (0.998)^1000 = 0.135, and the Poisson gives e^(-2) = 0.135, agreeing to three decimals. As n grows and p shrinks with np fixed, the agreement becomes exact. That is why the Poisson fits situations with countless small chances: each moment is a micro-trial, and lambda is the budget of expected events.

The classic demonstration is real data. In 1898, Ladislaus von Bortkiewicz tallied Prussian cavalry soldiers kicked to death by horses: 200 corps-years of records, averaging 0.61 deaths per corps per year. The Poisson with lambda = 0.61 predicts P(X = 0) = e^(-0.61) = 0.5434, or about 108.7 of the 200 corps-years with zero deaths. The records show 109. Rare, independent misfortunes at a steady rate really do follow the formula.

Key idea: Poisson = counts from many small independent chances at a steady rate. Set lambda to the average for your window, and mean equals variance equals lambda.

The geometric distribution

The geometric distribution models the number of independent Bernoulli trials (each with success probability p) needed to get the first success. The random variable X takes values 1, 2, 3, ... (the trial on which the first success lands). To get the first success on trial k, the first k - 1 trials must fail and the kth must succeed. Independence multiplies those requirements:

P(X = k) = (1 - p)^(k - 1) p.

Read it as a story: "fail, fail, ..., fail (k minus 1 times), then succeed." Its mean and variance are

E[X] = 1 / p and Var(X) = (1 - p) / p^2.

The mean 1/p is intuitive: if each trial succeeds with probability p, you expect to wait about 1/p trials for the first success. A die's 6 shows up in about 6 rolls; a 1-in-100 scratch ticket pays in about 100 tickets. Notice the variance grows fast as p shrinks: rare successes make waits not only long on average but wildly variable.

Worked example. Each attempt to start an old lawnmower succeeds with probability p = 0.2, independently. On average it takes 1/0.2 = 5 pulls. The probability the first success is on the third pull is

P(X = 3) = (0.8)^2 (0.2) = 0.64 x 0.2 = 0.128.

The probability it takes more than 3 pulls is the probability the first 3 all fail: P(X greater than 3) = (0.8)^3 = 0.512. This clean "all failures so far" form, P(X greater than k) = (1 - p)^k, is the geometric's survival function and its fastest computational route.

It also exposes the distribution's signature: memorylessness. Given no success yet, the future looks the same as a fresh start. Ten failed pulls in, the chance the next 3 pulls also fail is still (0.8)^3 = 0.512, exactly as it was at pull one. The mower does not remember your effort. This is the flip side of the gambler's fallacy from the independence lesson: no success is "due," ever, because each trial resets the clock.

Worked example: the coupon collector

Geometric waits stack beautifully. Suppose a cereal brand packs one of 6 toys, chosen uniformly, in each box, and you want the full set. How many boxes on average? Split the quest into 6 stages, one per new toy, and use linearity of expectation from the random-variables lesson.

• Stage 1: any toy is new, p = 6/6, expected wait 1 box.
• Stage 2: 5 of 6 toys are new, p = 5/6, expected wait 6/5 = 1.2 boxes.
• Stage 3: p = 4/6, expected wait 1.5 boxes. Then 6/3 = 2, then 6/2 = 3, and finally p = 1/6, expected wait 6 boxes.

Total: 1 + 1.2 + 1.5 + 2 + 3 + 6 = 14.7 boxes on average. Each stage is a geometric wait, expectation 1/p, and expectations add even though the stages are random. Notice the shape of the answer: the last missing toy costs more than the first five combined, which every collector of anything has felt.

Choosing the right model

• Binomial: fixed number of trials n, count the successes.
• Poisson: no fixed n, count events over an interval at a known rate.
• Geometric: keep trying until the first success, count the trials.

Practice the sorting on three quick scenarios. "Out of 50 seeds planted, how many sprout?" Fixed n = 50: binomial. "How many customers walk in between noon and 1 pm?" A window and a rate, no n: Poisson. "How many auditions until the first callback?" Waiting for the first hit: geometric. The distributions answer different questions before they answer them with different formulas, and reading the question type is most of the solution.

Try it

Two quick ones. (a) A server logs errors at an average rate of 1.5 per hour. What is the probability of a completely error-free hour? (b) You roll a fair die until the first 6. What is the probability the first 6 arrives exactly on roll 4? Work both before reading on.

Worked answer. (a) Poisson with lambda = 1.5 for the one-hour window: P(X = 0) = e^(-1.5) = 0.223, about a 22% chance of a quiet hour. (b) Geometric with p = 1/6: three failures then a success, P(X = 4) = (5/6)^3 (1/6) = (125/216)(1/6) = 125/1296 = 0.096, just under 10%. What we just did: (a) asked for a count in a window, (b) asked for a wait until a hit, and each question picked its own distribution.

Where people get stuck

The top Poisson error is a units mismatch: using a per-hour lambda on a half-hour question. Always restate the rate for the exact window first. At 3 calls per minute, a 90-second window uses lambda = 4.5, and only then does the pmf come out.

Second, the Poisson needs its assumptions: events independent, arriving one at a time, at a steady rate. Bus arrivals bunch (buses cluster behind delays), so bus counts often violate independence; insurance claims after one hurricane arrive together. When events cluster, the Poisson understates the variance, and a glance at data showing variance well above the mean is the standard warning sign.

Third, conventions. Our geometric counts all trials including the successful one, so X starts at 1 and E[X] = 1/p. Some textbooks count only the failures before the first success, so their variable starts at 0 and has mean (1 - p)/p. Both are fine; mixing them mid-problem is not. Check which count a formula expects before borrowing it.

Fourth, small algebra traps in P(X = 0): lambda^0 = 1 and 0! = 1, so P(X = 0) = e^(-lambda), nothing more. Writing 0 for either piece zeroes out a perfectly good probability.

Common misconceptions

• "The Poisson is only for exotic rare events." It fits any count built from many small independent chances at a steady rate: emails, raindrops on a tile, mutations per genome, goals per match.
• "If lambda = 3, then 3 is far likelier than its neighbors." The pmf is fairly flat near its center: P(X = 2) and P(X = 3) both equal 4.5e^(-3) = 0.224. The mean is a center of mass, not a spike.
• "After many failures, the first success is overdue." Memorylessness says the wait ahead is distributed as if you had never started; only depleting pools (cards, raffle tickets without replacement) owe you anything.
• "Mean 1/p means success is guaranteed within 1/p trials." With p = 0.2, the chance of needing more than 5 pulls is (0.8)^5 = 0.328, roughly a third. The mean is a long-run average, not a deadline.

Recap

• Poisson: counts in a window at rate lambda; pmf e^(-lambda) lambda^k / k!; mean and variance both lambda; rescale lambda to the window.
• It arises as the many-trials, tiny-p limit of the binomial with np = lambda, verified famously by the horse-kick data.
• Geometric: trials until the first success; pmf (1 - p)^(k-1) p; mean 1/p; survival (1 - p)^k; memoryless.
• Coupon collector: a sum of geometric waits, settled by linearity of expectation, 14.7 boxes for 6 toys.
• Choose the model by the question: fixed-n count (binomial), windowed count (Poisson), wait for first hit (geometric).

Counts and waits cover the discrete world. The next module lets random variables take any value in an interval, where probability becomes area under a curve.

Sources

1. OpenStax. (2023). 4.6 Poisson distribution. In Introductory statistics 2e. Rice University. openstax.org
2. OpenStax. (2023). 4.4 Geometric distribution. In Introductory statistics 2e. Rice University. openstax.org
3. NIST/SEMATECH. (n.d.). 1.3.6.6.19 Poisson distribution. In e-Handbook of statistical methods. National Institute of Standards and Technology. itl.nist.gov
4. Encyclopedia of Mathematics. (n.d.). Poisson distribution. European Mathematical Society. encyclopediaofmath.org
5. Weisstein, E. W. (n.d.). Coupon collector's problem. MathWorld: A Wolfram web resource. mathworld.wolfram.com
6. MacTutor History of Mathematics Archive. (n.d.). Siméon-Denis Poisson. University of St Andrews. mathshistory.st-andrews.ac.uk
7. Quine, M. P., & Seneta, E. (1987). Bortkiewicz's data and the law of small numbers. International Statistical Review, 55(2), 173. doi.org/10.2307/1403193

Some quantities, like waiting times, heights, or temperatures, vary continuously and can take any value in an interval. A bus can arrive after 4 minutes, 4.3 minutes, or 4.31779 minutes; between any two possible values sit infinitely many more. For these we replace the probability mass function with a probability density function, and probability itself changes costume: it becomes area under a curve.

The big picture

Here is the bridge from the discrete world. Imagine measuring the waiting times of thousands of bus riders and drawing a histogram. With wide bins, you get a chunky bar chart, and the fraction of riders in any bin is that bar's area. Narrow the bins and the staircase smooths out; in the limit of infinitely fine bins, the outline becomes a curve. That curve is the density, and the fraction of riders between any two times is the area under it between those times, exactly as it was for the bars.

One habit has to update. In the discrete world we asked "what is the probability X equals this value?" In the continuous world that question always answers zero, and the useful questions are about intervals: the probability the wait is under 5 minutes, or between 4 and 7. Areas answer interval questions, and the whole apparatus of this module, uniform now, exponential and normal next, is machinery for computing such areas.

Key idea: continuous probability is area under a density curve. Individual points carry no area; intervals do.

Density and area

A continuous random variable X is described by a probability density function (pdf) f(x). Unlike a pmf, f(x) is not itself a probability; instead, probability is the area under the curve. The probability that X falls between a and b equals the area beneath f(x) from a to b. Two rules make a valid density:

• f(x) is greater than or equal to 0 everywhere (no negative density).
• The total area under f(x) equals 1.

Because f(x) is a height, not a probability, it can legally exceed 1. A uniform density on the interval from 0 to 0.5 has height 2, and nothing is wrong: the rectangle's area is 0.5 × 2 = 1, exactly as required. Densities report probability per unit of x, the way a population-density map reports people per square mile; only when height is multiplied by width does a probability appear.

Because a single point has zero width, it encloses zero area, so P(X = c) = 0 for any exact value c. Only intervals have positive probability. This is less strange than it first sounds. Throw a dart at a line: the chance of hitting any prespecified exact point, to infinitely many decimal places, is zero, yet the dart certainly lands somewhere. In continuous settings, probability zero does not mean impossible; it means infinitely fine-grained.

A useful consequence: for continuous variables, P(X is less than c) and P(X is less than or equal to c) are equal, since the endpoint contributes nothing. Endpoint fussiness, which genuinely matters for discrete counts, evaporates here.

One more piece of vocabulary earns its keep now and pays off for the rest of the course. The cumulative distribution function (cdf) is F(x) = P(X less than or equal to x), the total area to the left of x. It climbs from 0 to 1 as x sweeps across the possible values, and any interval probability is a difference of two cdf values: P(a less than X less than or equal to b) = F(b) - F(a). Left-area in, subtraction out.

The uniform distribution

The uniform distribution on an interval from a to b spreads probability evenly, so its density is a flat, constant height. To make the total area (a rectangle of width b - a) equal 1, the height must be

f(x) = 1 / (b - a) for x between a and b, and 0 otherwise.

Read that aloud: "one over the length of the interval." A longer interval spreads the same 1 unit of probability thinner, so the flat height drops. The uniform is the continuous version of "equally likely": every sub-interval of the same width captures the same probability, no matter where it sits.

Its mean sits at the midpoint and its variance follows a standard formula:

E[X] = (a + b) / 2 and Var(X) = (b - a)^2 / 12.

The mean is forced by symmetry: the rectangle balances at its center. The variance formula (derived with the integrals of a calculus course, and worth accepting on faith here) says spread depends only on the interval's length, never its location: uniform on [0, 8] and uniform on [100, 108] have identical variance, 64/12 = 5.333. Sliding a distribution along the line moves its center and leaves its spread alone, the same shift rule we met with discrete variables.

Worked example

A bus arrives at a uniformly random time between 2 and 10 minutes after you reach the stop, so a = 2 and b = 10. The density height is 1 / (10 - 2) = 1/8 = 0.125. Probabilities are just rectangle areas (width times height):

• P(X is less than 5) = (5 - 2) x (1/8) = 3/8 = 0.375.
• P(4 is less than X is less than 7) = (7 - 4) x (1/8) = 3/8 = 0.375.

Work each one the same way, in three small steps. Step 1: find the piece of the interval the event occupies (from 2 to 5, width 3). Step 2: multiply width by the flat height (3 × 1/8). Step 3: sanity-check that the answer sits between 0 and 1 and matches intuition (5 is toward the low end of [2, 10], so a value below 3/8 of the way is plausible). Note the two different windows above have the same width, 3 minutes, so they capture identical probability, the uniform's signature.

The mean wait is (2 + 10)/2 = 6 minutes, and the variance is (10 - 2)^2 / 12 = 64/12 = 5.333, so the standard deviation is about 2.31 minutes.

Two extensions squeeze more out of the same rectangle. First, tails: P(X greater than 8) = (10 - 8) x (1/8) = 2/8 = 0.25; a quarter of days the bus makes you wait more than 8 minutes. Second, percentiles run the machine backwards.

The 25th percentile is the x with area 0.25 to its left, so solve (x - 2)(1/8) = 0.25 to get x = 4 minutes. The 75th percentile solves (x - 2)(1/8) = 0.75, giving x = 8. The middle half of your waits lands between 4 and 8 minutes. What we just did: forward questions multiply width by height; backward questions divide area by height to find the width.

Why the uniform matters beyond buses

The uniform on [0, 1] is the workhorse of computer simulation. When software calls a random-number generator, it receives a value meant to behave as uniform between 0 and 1, and every other random quantity a program needs, coin flips, normal measurements, simulated customers, is manufactured from those uniform draws by clever transformations. Estimating a complicated probability by simulating thousands of trials, the Monte Carlo method, therefore rests on this flat little rectangle. Learn its arithmetic well and you have learned the atom from which simulated randomness is built.

Try it

Let X be uniform on the interval from 1 to 7. Find the density height, P(2 < X < 5), the mean, and the standard deviation. Work all four before reading on.

Worked answer. Step 1: height = 1/(7 - 1) = 1/6 = 0.167. Step 2: the window from 2 to 5 has width 3, so P = 3 x (1/6) = 1/2 = 0.5. Step 3: mean = (1 + 7)/2 = 4. Step 4: variance = (7 - 1)^2 / 12 = 36/12 = 3, so the standard deviation is square root of 3 = 1.73. What we just did: every answer came from the same three numbers, a, b, and the width of the window in question.

Where people get stuck

The stubbornest habit to unlearn is reading the density's height as a probability. Height 1/8 does not mean "each minute has probability 1/8"; it means probability accumulates at 1/8 per minute of width. The moment you catch yourself quoting f(x) as a chance, multiply by a width and the units come right. And when a density is taller than 1, as on short intervals, nothing has gone wrong; the area is what must not exceed 1.

Second, forgetting the support, the interval where the density lives. For the bus, P(X less than 12) = 1, not (12 - 2)/8 = 1.25: the rectangle ends at 10, and area beyond it does not exist. Clip every window to [a, b] before multiplying, and a probability above 1 is the alarm that you forgot.

Third, over-carrying discrete habits. In the discrete world, P(X < 3) and P(X less than or equal to 3) can differ by a whole lump of probability; continuous variables have no lumps, so the two match. Meanwhile the truly continuous novelty, that possible values have probability zero, runs the other direction and takes a while to settle. Both are consequences of one fact: points have no width.

Common misconceptions

• "Probability zero means impossible." In continuous settings, every exact outcome has probability zero, yet one of them occurs. Impossibility implies probability zero, but not the reverse.
• "The uniform makes every value equally likely, so some value is likely." Every equal-width interval is equally likely; every single value still has probability zero. The uniform has no most likely value at all.
• "Densities are capped at 1." Only areas are capped at 1. Heights can be any nonnegative size, provided the total area is exactly 1.
• "Continuous models are exotic." Rounding aside, most measured quantities, times, weights, voltages, percentages, are modeled continuously, and the next two lessons cover the two most used continuous families after the uniform.

Recap

• Continuous variables are described by a density f(x): nonnegative, total area 1, probability = area over an interval.
• P(X = c) = 0 for every exact c; interval questions are the meaningful ones, and endpoints do not matter.
• The cdf F(x) = P(X less than or equal to x) accumulates left-area; interval probabilities are cdf differences.
• Uniform on [a, b]: flat height 1/(b - a), mean (a + b)/2, variance (b - a)^2/12.
• Forward questions: width × height. Backward (percentile) questions: solve area equations for x.

The uniform distribution is the continuous cousin of "equally likely" outcomes: every equal-width slice of the interval is equally probable, and probability is always found the same way, as area. Next, a density that is anything but flat: the exponential, where short waits are common and long waits fade away.

Sources

1. OpenStax. (2023). 5.1 Continuous probability functions. In Introductory statistics 2e. Rice University. openstax.org
2. OpenStax. (2023). 5.2 The uniform distribution. In Introductory statistics 2e. Rice University. openstax.org
3. NIST/SEMATECH. (n.d.). 1.3.6.6.2 Uniform distribution. In e-Handbook of statistical methods. National Institute of Standards and Technology. itl.nist.gov
4. Encyclopedia of Mathematics. (n.d.). Uniform distribution. European Mathematical Society. encyclopediaofmath.org
5. Weisstein, E. W. (n.d.). Uniform distribution. MathWorld: A Wolfram web resource. mathworld.wolfram.com
6. Siegrist, K. (n.d.). 03: Distributions. In Probability, mathematical statistics, and stochastic processes. LibreTexts. stats.libretexts.org
7. Grinstead, C. M., & Snell, J. L. (n.d.). Introductory probability. LibreTexts. stats.libretexts.org

The exponential distribution models the waiting time until an event when events occur at a constant average rate. It is the continuous partner of the Poisson: if events per hour follow a Poisson with rate lambda, the time between consecutive events is exponential with the same rate lambda. Where the Poisson counts arrivals in a window, the exponential times the gap until the next one.

The big picture

Picture the arrivals themselves: calls reaching a help desk, clicks of a Geiger counter, customers walking through a door. The last lesson's uniform density was flat; waiting times are not. Short gaps between arrivals are common, and long silences grow steadily rarer, so the density should start high and taper off. The exponential curve does exactly that, dropping by the same fraction for every additional unit of time, the same pattern as radioactive decay.

The distribution is run by one number, the rate lambda. Everything else, the mean wait, the spread, every tail probability, is a one-line formula in lambda. That economy plus one strange, powerful property, memorylessness, is the whole lesson.

Key idea: the exponential describes gaps between steady-rate arrivals: frequent short waits, exponentially fading long ones, all controlled by the single rate lambda.

Density and rate

For a rate lambda greater than 0, the exponential density for x greater than or equal to 0 is

f(x) = lambda e^(-lambda x).

Read it aloud: "lambda, times e to the minus lambda x." The density starts at its highest value lambda at x = 0 and decays toward 0, so short waits are more likely than long ones. Its mean and variance are

E[X] = 1 / lambda and Var(X) = 1 / lambda^2.

The mean 1/lambda is the average time between events: if a bus comes at rate 2 per hour (lambda = 2), you wait 1/2 hour on average. The reciprocal works both directions. Rate 4 calls per hour means a mean gap of 1/4 hour, 15 minutes; a mean gap of 4 minutes means a rate of 1/4 per minute. Keeping "rate" and "mean" straight is mostly a matter of watching units: lambda carries units of events per time, the mean carries time.

Notice one more signature: the standard deviation is the square root of 1/lambda^2, which is 1/lambda, identical to the mean. An exponential wait is as variable as it is long, which is why arrival gaps feel so irregular even at a steady average rate.

The survival function

Tail probabilities have a beautifully simple closed form. The probability of waiting longer than a time t is the survival function

P(X greater than t) = e^(-lambda t),

and therefore the probability of waiting at most t is P(X less than or equal to t) = 1 - e^(-lambda t), which is the cumulative distribution function from the last lesson. Between the two of them, every exponential question reduces to plugging a time into one exponential expression.

The survival form also hands us the median wait, the time that half of all waits beat. Set e^(-lambda t) = 0.5 and solve: t = ln(2)/lambda = 0.693/lambda. This is precisely the "half-life" of radioactive decay: the half-life is the median lifetime of an atom. Note the median 0.693/lambda sits below the mean 1/lambda, a telltale of the long right tail: a minority of very long waits drags the average above the typical experience.

Worked half-life. A component fails at constant rate lambda = 0.5 per year. Its median lifetime is 0.693 / 0.5 = 1.386 years: half of such components are gone within about 17 months, even though the mean lifetime is 2 years. The computation runs backwards too. Carbon-14 has a half-life of 5,730 years, so its decay rate is lambda = 0.693 / 5,730 = 0.000121 per year, the number radiocarbon dating is built on. Median to rate or rate to median, the bridge is always the constant 0.693, which is ln(2).

Worked example

Suppose the time (in hours) until the next customer arrives is exponential with rate lambda = 0.5 per hour, so the mean wait is 1 / 0.5 = 2 hours. Then:

• P(wait more than 3 hours) = e^(-0.5 x 3) = e^(-1.5) = 0.2231.
• P(wait at most 2 hours) = 1 - e^(-0.5 x 2) = 1 - e^(-1) = 1 - 0.3679 = 0.6321.

Walk the first one in steps. Step 1: identify the question as a tail, "more than," so use the survival function. Step 2: multiply rate times time, lambda t = 0.5 × 3 = 1.5, watching that hours multiply hours-to-the-minus-one. Step 3: evaluate e^(-1.5) = 0.2231. A between-times question stacks two of these: P(1 < X < 3) = e^(-0.5 x 1) - e^(-0.5 x 3) = 0.6065 - 0.2231 = 0.3834, the survival at the near edge minus the survival at the far edge.

In the sketch, the shaded region is the tail area beyond t, and its size is the single exponential term e^(-lambda t). Every survival probability is one shaded tail like this one, which is why exponential arithmetic stays so short.

The memoryless property

Like the geometric distribution, the exponential is memoryless: P(X greater than s + t given X greater than s) = P(X greater than t). If a machine has already run 5 hours without failing, the chance it lasts 3 more hours is the same as a brand-new machine lasting 3 hours. The exponential does not "age," which makes it the standard model for the lifetime of components with a constant failure rate and for the gaps between random arrivals.

The proof is one division. With lambda = 0.5, ask for P(X > 8 | X > 5): by the conditional-probability definition it is P(X > 8) / P(X > 5) = e^(-4) / e^(-2.5) = e^(-1.5) = 0.2231, exactly P(X > 3). The exponent subtraction wipes out the 5 hours already survived; only the additional 3 hours matter. What we just did: conditioning divided two survival terms, and the exponential's multiplication rule for exponents erased the past.

Memorylessness is a modeling claim, not a law of nature, so check it against the mechanism. Electronic components in their steady mid-life, arrival gaps in stable traffic, and radioactive decay fit well. Anything that wears out or ages, brake pads, batteries, living things, does not: an 80-year-old and a newborn do not face the same distribution of remaining years. Reliability engineers use the exponential for the flat middle of the "bathtub curve" of failure rates and switch to other models (like the Weibull) for break-in and wear-out phases.

The bridge to the Poisson process

The exponential and the Poisson are two views of one process, and the link is a single line of algebra. Suppose events arrive as a Poisson process at rate lambda per hour. When is the next event more than t hours away? Exactly when the window of length t contains zero events. The Poisson count in that window has mean lambda t, so

P(wait > t) = P(0 events in the window) = e^(-lambda t),

which is precisely the exponential survival function. Counting arrivals in windows gives Poisson; timing the gap to the next arrival gives exponential; same lambda, same process, two questions. This is why call centers, web servers, and emergency rooms model call counts with one distribution and inter-arrival times with the other, and why the two always share a rate.

Try it

Support tickets arrive at a steady rate of 4 per hour, exponentially spaced. Find (a) the probability the next ticket arrives within 15 minutes, and (b) the probability of more than 30 quiet minutes. Mind the units, then work both before reading on.

Worked answer. Convert first: 15 minutes = 0.25 hour, 30 minutes = 0.5 hour. (a) P(X less than or equal to 0.25) = 1 - e^(-4 x 0.25) = 1 - e^(-1) = 1 - 0.368 = 0.632. (b) P(X > 0.5) = e^(-4 x 0.5) = e^(-2) = 0.135. What we just did: matched the time units to the rate's units before multiplying, then used the cdf for "within" and the survival function for "more than."

Where people get stuck

The most common slip is feeding the mean where the rate belongs. If the mean gap is 4 minutes, lambda is 1/4 per minute, not 4. Catch it with units: lambda t must come out dimensionless, so a rate in per-minutes needs t in minutes. If your exponent has units left over, the inputs are mismatched.

Second, sign errors. The survival function is e^(-lambda t), decreasing in t: longer waits must be rarer. If your formula's probabilities grow with t, or exceed 1, the minus sign fell off. A quick self-check on any answer: at t = 0 survival is 1, and it should shrink from there.

Third, mixing the two Poisson-process views. "How many events in 2 hours?" is a Poisson count question with mean lambda times 2. "How long until the next event?" is an exponential time question. The nouns give it away, a count versus a duration, and the units of the answer (events versus hours) must match the noun.

Fourth, over-applying memorylessness. It is a property of this model, valid when the failure rate is genuinely constant. Quoting "the machine is as good as new" about a worn bearing is a modeling error, not a probability theorem.

Common misconceptions

• "With a mean of 2 hours, waits near 2 hours are typical." The distribution is skewed: the median is only 1.39 hours, and P(X > 2) = e^(-1) = 0.368. Most waits fall short of the mean; a long-tail minority stretches far beyond it.
• "Exponential means growth." The name refers to the exponential function in the formula; here it governs decay of the survival probability as t grows.
• "The sum of two exponential waits is exponential." The gap to the second-next arrival follows a different (gamma) distribution; only the single gap is exponential. Sums of random variables get their own lesson shortly.
• "Memorylessness contradicts common sense about aging." It contradicts aging, which is why it applies only to constant-rate mechanisms, and models of living or wearing systems use other distributions.

Recap

• Exponential: waiting time until the next event at constant rate lambda; density lambda e^(-lambda x) for x at least 0.
• Mean 1/lambda, variance 1/lambda^2, standard deviation equal to the mean; median ln(2)/lambda, below the mean.
• Survival P(X > t) = e^(-lambda t); cdf 1 - e^(-lambda t); between-probabilities subtract two survivals.
• Memoryless: time already survived never changes the distribution of time remaining.
• Poisson counts in windows and exponential gaps between events are the same process seen two ways, sharing one lambda.

Next comes the most celebrated curve in statistics, the normal distribution, and the standardization trick that makes one table answer questions about every bell curve at once.

Sources

1. OpenStax. (2023). 5.3 The exponential distribution. In Introductory statistics 2e. Rice University. openstax.org
2. NIST/SEMATECH. (n.d.). 1.3.6.6.7 Exponential distribution. In e-Handbook of statistical methods. National Institute of Standards and Technology. itl.nist.gov
3. Encyclopedia of Mathematics. (n.d.). Exponential distribution. European Mathematical Society. encyclopediaofmath.org
4. Encyclopedia of Mathematics. (n.d.). Poisson process. European Mathematical Society. encyclopediaofmath.org
5. Weisstein, E. W. (n.d.). Exponential distribution. MathWorld: A Wolfram web resource. mathworld.wolfram.com
6. Siegrist, K. (n.d.). 03: Distributions. In Probability, mathematical statistics, and stochastic processes. LibreTexts. stats.libretexts.org
7. Grinstead, C. M., & Snell, J. L. (n.d.). Introductory probability. LibreTexts. stats.libretexts.org

The normal (Gaussian) distribution is the famous bell curve and the most important continuous distribution in all of statistics. It models heights, measurement errors, test scores, and, crucially, the behavior of sums and averages (as the next module shows). This lesson covers its shape, the standardization trick that reduces every normal question to one reference curve, and the 68-95-99.7 rule that answers most questions before a table is opened.

The big picture

Why does one curve appear in so many unrelated places? A hint sits in what those places share. An adult's height is the sum of many small influences, genes, nutrition, chance; a measurement error is the sum of many tiny disturbances. Whenever a quantity is assembled from many small independent contributions, its distribution is pulled toward the same bell shape, a fact proved in the final lesson as the Central Limit Theorem. The normal is not a coincidence; it is an attractor.

Working with it takes only two ideas. First, the family has exactly two knobs: the mean mu, saying where the bell sits, and the standard deviation sigma, saying how wide it spreads. Second, every member of the family is the same curve wearing different axis labels, so one conversion, the z-score, translates any normal question into a single standard reference. Learn one curve, answer them all.

Key idea: normals form one two-knob family. Standardizing converts any of them to the standard normal, so a single set of reference values serves every bell curve.

Shape and parameters

A normal distribution is symmetric and bell-shaped, fully described by two numbers: its mean mu (the center, where the peak sits) and its standard deviation sigma (the spread). The mean, median, and mode all coincide at mu. Changing mu slides the curve left or right; changing sigma makes it wider and flatter or narrower and taller, but the total area always stays 1.

The standard deviation is visible on the curve itself: the bell's "shoulders," the points where it switches from curving downward to flaring outward, sit exactly one sigma either side of the mean. IQ scores (mu = 100, sigma = 15) and SAT scores (mu = 500, sigma = 100) draw the identical bell; only the number line beneath it changes. That is the family resemblance the z-score will exploit.

The standard normal and z-scores

The standard normal distribution Z has mean 0 and standard deviation 1. Any normal value x can be converted to a z-score that says how many standard deviations it lies from the mean:

z = (x - mu) / sigma.

Read the recipe aloud: "distance from the mean, measured in standard deviations." Step 1: subtract the mean, so the answer is a distance with a sign. Step 2: divide by sigma, so the distance is counted in spread-units rather than raw units. A positive z is above the mean, a negative z below, and z = 0 is the mean itself.

Worked example. Heights are normal with mu = 170 cm and sigma = 10 cm. A height of 190 cm has z = (190 - 170)/10 = 2.0, two standard deviations above average. A height of 155 cm has z = (155 - 170)/10 = -1.5, one and a half below. The formula also runs backwards: the height with z = 1 is x = mu + z sigma = 170 + 10 = 180 cm.

Because z-scores erase units, they compare across scales. Which is stronger: 650 on the SAT (mu = 500, sigma = 100) or 28 on the ACT (mu = 21, sigma = 5)? Compute both: z = 1.5 versus z = (28 - 21)/5 = 1.4. The SAT score sits slightly farther above its crowd, so it is the marginally stronger result. Standardizing lets a single table (or the empirical rule) handle every normal distribution at once.

Negative z-scores carry no stigma; they are positions, not grades. A golfer whose round has z = -2 relative to the field is doing wonderfully, since low scores win golf. The sign reports which side of the mean a value sits on, and the size reports how far; what counts as good depends entirely on the variable.

The 68-95-99.7 empirical rule

For any normal distribution:

• About 68% of values lie within 1 standard deviation of the mean.
• About 95% lie within 2 standard deviations.
• About 99.7% lie within 3 standard deviations.

Symmetry lets you slice these three facts into halves and tails, which is how they get used. If 68% lies within 1 sigma, then 32% lies outside, split into 16% per tail, and 34% sits between the mean and one sigma on either side. Likewise 95% within 2 sigma leaves 2.5% in each tail, and 99.7% within 3 leaves just 0.15% per tail. Values beyond three sigma are rare enough that factories treat them as alarms rather than luck.

Worked example: the empirical rule

IQ scores are modeled as normal with mu = 100 and sigma = 15. About 68% of people score between 100 plus or minus 15, that is 85 to 115. About 95% score between 100 plus or minus 30, that is 70 to 130. A score of 115 has z = (115 - 100)/15 = 1.0, exactly one standard deviation above the mean.

Worked example: a tail probability

What fraction of people score above 115 (z = 1)? Since 68% lie within one standard deviation, 32% lie outside it, split evenly between the two tails by symmetry, so about 16% lie above 115. The exact standard-normal value is P(Z greater than 1) = 0.1587, close to the empirical-rule estimate of 16%. To find the probability of scoring between 85 and 115 (within 1 SD), the empirical rule gives about 68%; the exact value is 0.6827.

For z-values off the rule's grid, a short table of left-areas covers nearly everything: P(Z < 1) = 0.8413, P(Z < 1.5) = 0.9332, P(Z < 2) = 0.9772. Three moves convert any question into these numbers. "Below" reads the table directly. "Above" subtracts from 1: P(Z > 1.5) = 1 - 0.9332 = 0.0668. "Between" subtracts two left-areas: for IQs from 85 to 130, z runs from -1 to 2, and by symmetry P(Z < -1) = P(Z > 1) = 0.1587, so the answer is 0.9772 - 0.1587 = 0.8185, about 82%.

Two more landmarks are worth memorizing because statistics leans on them constantly: P(Z < 1.645) = 0.95, so z = 1.645 marks the 95th percentile, and the central 95% of the curve sits between z = -1.96 and z = +1.96, the origin of the "plus or minus about 2 standard deviations" habit.

Backwards: from percentile to value

Many practical questions arrive reversed: not "what fraction is above 119?" but "what score marks the top 10%?" Run the machine in the other direction. Step 1: translate the percentile into a z-score; the 90th percentile is z = 1.28, because P(Z < 1.28) = 0.90. Step 2: convert z back to raw units with x = mu + z sigma. For IQ: x = 100 + 1.28 x 15 = 119.2, so about 119 marks the top tenth.

A cutoff for the top 2.5% uses z = 1.96: 100 + 1.96 x 15 = 129.4. What we just did: percentile to z, then z to x; the forward problems ran x to z to percentile. Reversed questions are everywhere in practice: admissions offices set cutoffs, manufacturers set tolerance limits, and hospitals set reference ranges, each by choosing a percentile first and converting it to a raw value second. The z-scale is the shared middle step in every one of those conversions, which is why it earns the name "standard."

Try it

A machine cuts bolts with lengths normal, mu = 50 mm, sigma = 0.2 mm. Find (a) the fraction of bolts longer than 50.4 mm, and (b) the fraction between 49.8 and 50.2 mm. Work both before reading on.

Worked answer. (a) Step 1: z = (50.4 - 50)/0.2 = 2.0. Step 2: within 2 sigma sits 95%, so beyond the two 2-sigma marks lies 5%, and the upper tail alone holds 2.5% (exact: 0.0228). (b) The window from 49.8 to 50.2 is mu plus or minus one sigma, so about 68% of bolts qualify. What we just did: standardized once, then let the empirical rule and symmetry do the area work, no integration anywhere.

Where people get stuck

Order of operations in the z-score causes the first wave of slips: subtract the mean first, then divide the whole difference by sigma. Computing 115/15 and then subtracting something, or dividing only part of the difference, gives garbage. Say the recipe as a sentence, "distance from the mean, in SDs," and the order takes care of itself.

Second, dividing by the variance instead of the standard deviation. If a problem reports variance 225, take its square root, sigma = 15, before standardizing. Units are the tell: z must be a pure number, and a distance in points divided by points-squared is not.

Third, misreading table direction. Standard tables report left-areas, P(Z < z). For "above" questions subtract from 1; for negative z use symmetry, P(Z < -1) = P(Z > 1). Sketch the bell, shade the region you want, and the correct subtraction becomes visible before any arithmetic.

Fourth, applying the empirical rule to data that is not roughly normal. Incomes, house prices, and waiting times are strongly skewed; for them "within 2 sigma" can cover a very different fraction. The rule is a property of the bell shape, not of data in general.

Common misconceptions

• "Most data is normal." Many measured quantities are approximately normal, especially sums and averages, but plenty are not. Normality is a modeling assumption to check, often with a quick histogram.
• "Standardizing changes the data." A z-score relabels a value in new units (SDs from the mean); it moves no one and reorders nothing.
• "z-scores beyond 3 are impossible." Merely rare: about 0.3% of values fall outside 3 sigma, roughly 1 in 370. In large datasets such values are expected visitors.
• "68-95-99.7 are exact." They round the true values 68.27%, 95.45%, 99.73%, plenty accurate for estimates, while tables or software provide the exact areas.

Recap

• A normal distribution is fixed by mu (center) and sigma (spread); the bell's shoulders sit one sigma from the mean.
• z = (x - mu)/sigma counts signed standard deviations from the mean; x = mu + z sigma undoes it.
• Empirical rule: about 68%, 95%, and 99.7% within 1, 2, and 3 sigma; halve the leftovers for tails.
• Table technique: below reads left-area, above subtracts from 1, between subtracts two left-areas, symmetry handles negatives.
• Reversed questions go percentile to z to x, with landmarks z = 1.28, 1.645, and 1.96 doing most of the work.

Standardizing plus symmetry lets you answer any normal probability question with a small set of known values. The final module explains why this one curve rules them all: averages of almost anything become normal, which is the Central Limit Theorem.

Sources

1. OpenStax. (2023). 6.1 The standard normal distribution. In Introductory statistics 2e. Rice University. openstax.org
2. OpenStax. (2023). 6.2 Using the normal distribution. In Introductory statistics 2e. Rice University. openstax.org
3. NIST/SEMATECH. (n.d.). 1.3.6.6.1 Normal distribution. In e-Handbook of statistical methods. National Institute of Standards and Technology. itl.nist.gov
4. Encyclopedia of Mathematics. (n.d.). Normal distribution. European Mathematical Society. encyclopediaofmath.org
5. Weisstein, E. W. (n.d.). Standard normal distribution. MathWorld: A Wolfram web resource. mathworld.wolfram.com
6. MacTutor History of Mathematics Archive. (n.d.). Carl Friedrich Gauss. University of St Andrews. mathshistory.st-andrews.ac.uk
7. Khan Academy. (n.d.). Modeling data distributions. khanacademy.org

Every distribution so far has described one random variable at a time. Real questions usually arrive in pairs: a person's height and weight, hours studied and exam score, temperature and ice-cream sales. To reason about a pair we need the joint distribution, which records the probability of every combination of values. From it flow the three tools this lesson builds: marginal distributions, covariance, and correlation.

The big picture

Think of a joint distribution as a city map for two variables. A single variable's distribution is a list along one street, each value with its probability. A pair of variables needs a full grid, one cell for every combination, because the interesting information usually lives in how the two coordinates move together. Do tall people tend to weigh more? Does studying tend to raise scores? Those are questions about the grid, not about either street alone.

This lesson teaches three ways to read the grid. Summing rows or columns recovers each variable's solo distribution, the marginals. Multiplying and averaging in the right pattern produces covariance, a single signed number saying whether the variables rise and fall together. Rescaling covariance gives correlation, a unit-free strength score between -1 and +1. The lesson ends with a warning built into the mathematics: independence forces covariance to zero, but zero covariance does not force independence.

Key idea: the joint table is the complete truth about a pair of random variables. Everything else, marginals, conditionals, covariance, correlation, is computed from its cells.

Joint and marginal distributions

For two discrete variables X and Y, the joint probability mass function p(x, y) = P(X = x and Y = y) gives the probability of each pair of values, and all the entries sum to 1. Each cell is an "and" statement. Consider this joint table, our running example:

	Y = 0	Y = 1	P(X)
X = 0	0.20	0.10	0.30
X = 1	0.10	0.30	0.40
X = 2	0.10	0.20	0.30
P(Y)	0.40	0.60	1.00

Read a few cells aloud to fix the meaning. The top-left inner entry says P(X = 0 and Y = 0) = 0.20: in 20% of trials both variables come up 0. The 0.30 in the middle row says P(X = 1 and Y = 1) = 0.30. First sanity check, the six inner cells must sum to 1: 0.20 + 0.10 + 0.10 + 0.30 + 0.10 + 0.20 = 1.00. They do, so this is a legitimate probability assignment.

The marginal distribution of X is the distribution of X alone, ignoring Y. To find it, sum each row. If X = 1, then Y was either 0 or 1, two disjoint cases, so P(X = 1) = 0.10 + 0.30 = 0.40. Likewise P(X = 0) = 0.20 + 0.10 = 0.30 and P(X = 2) = 0.10 + 0.20 = 0.30. The marginal of Y comes from column sums: P(Y = 0) = 0.20 + 0.10 + 0.10 = 0.40 and P(Y = 1) = 0.10 + 0.30 + 0.20 = 0.60.

The name is literal: these totals sit in the margins of the table, exactly where the P(X) column and P(Y) row appear above. Each marginal is a complete one-variable distribution and must itself sum to 1. Check both: 0.30 + 0.40 + 0.30 = 1.00 for X, and 0.40 + 0.60 = 1.00 for Y.

The table also answers compound and conditional questions by membership check, the same skill as lesson one. For P(X ≥ 1 and Y = 1), collect the qualifying cells: 0.30 + 0.20 = 0.50. For a conditional, use the definition from module two, cell over margin: P(Y = 1 | X = 2) = 0.20 / 0.30 ≈ 0.667. Compare that with the marginal P(Y = 1) = 0.60: learning that X = 2 nudged the probability of Y = 1 upward, a first hint of dependence.

Key idea: cells are "and" probabilities. Row and column sums give each variable's solo distribution, and cell-over-margin ratios give conditionals.

Covariance: do they move together?

Marginals throw away the pairing pattern, and the pattern is the point. We want one number summarizing its direction. The idea: measure each variable's deviation from its own mean, multiply the two deviations, and average over the joint distribution. When X and Y tend to sit above average together and below average together, the product of deviations is usually positive. When one runs high while the other runs low, the product is usually negative. That average product is the covariance:

Cov(X, Y) = E[(X - mu_X)(Y - mu_Y)],

where mu_X and mu_Y are the two means. Expanding the product and applying linearity of expectation gives the shortcut used in practice:

Cov(X, Y) = E[XY] - E[X] E[Y].

Read the shortcut aloud: average the product, then subtract the product of the averages. If pairing carried no information, the two would agree and the covariance would be zero. Covariance measures exactly how far E[XY] drifts from E[X]E[Y].

Worked example, every cell shown. First the means, straight from the margins. E[X] = 0(0.30) + 1(0.40) + 2(0.30) = 0 + 0.40 + 0.60 = 1.0, and E[Y] = 0(0.40) + 1(0.60) = 0.60.

Next E[XY]. The rule for the expectation of a function of the pair: multiply x × y × p(x, y) for every cell, then add. Walk the grid row by row:

• Cell (X = 0, Y = 0): 0 × 0 × 0.20 = 0.
• Cell (X = 0, Y = 1): 0 × 1 × 0.10 = 0.
• Cell (X = 1, Y = 0): 1 × 0 × 0.10 = 0.
• Cell (X = 1, Y = 1): 1 × 1 × 0.30 = 0.30.
• Cell (X = 2, Y = 0): 2 × 0 × 0.10 = 0.
• Cell (X = 2, Y = 1): 2 × 1 × 0.20 = 0.40.

Four of the six terms vanish because a zero value of either variable kills the product; only cells where both variables are nonzero contribute. The sum is E[XY] = 0.30 + 0.40 = 0.70. Now the shortcut:

Cov(X, Y) = 0.70 - (1.0)(0.60) = 0.70 - 0.60 = 0.10.

What we just did: means from the margins, E[XY] from a cell walk, subtract. The positive result says above-average X tends to accompany above-average Y. The table agrees if you look: its two heaviest cells, 0.30 at (1, 1) and 0.20 at (2, 1), sit where both variables are large together.

Key idea: covariance is the average of deviation times deviation. Positive means the variables lean the same way, negative means opposite ways, and E[XY] - E[X]E[Y] computes it fast.

Correlation: covariance with the units removed

Covariance has a defect: its size depends on units. Measure height in centimeters instead of meters and every height deviation becomes 100 times larger, so the covariance with weight becomes 100 times larger, though the relationship itself has not changed at all. The raw number 0.10 therefore tells us the direction of association but not, by itself, the strength.

The fix is the same standardizing move as the z-score lesson: measure each deviation in standard-deviation units. Dividing covariance by both standard deviations produces the correlation coefficient:

rho = Cov(X, Y) / (sigma_X sigma_Y).

Correlation is a pure number, always between -1 and +1. At the extremes, rho = +1 means Y is exactly an increasing straight-line function of X, and rho = -1 an exactly decreasing one. Values near 0 mean little linear association. The sign carries direction, the magnitude carries strength, and the units are gone.

Worked example, continued. We need both standard deviations, via the variance shortcut Var = E[X²] - (E[X])².

• For X: E[X²] = 0²(0.30) + 1²(0.40) + 2²(0.30) = 0 + 0.40 + 1.20 = 1.60, so Var(X) = 1.60 - 1.0² = 0.60 and sigma_X = √0.60 ≈ 0.775.
• For Y: since Y takes only the values 0 and 1, Y² = Y, so E[Y²] = E[Y] = 0.60 and Var(Y) = 0.60 - 0.60² = 0.60 - 0.36 = 0.24, giving sigma_Y = √0.24 ≈ 0.490.

Now divide: rho = 0.10 / (0.775 × 0.490) ≈ 0.264.

A correlation of about 0.26 is a weak positive association: the variables lean the same way, but knowing one leaves plenty of uncertainty about the other. For calibration, correlations near plus or minus 0.9 look like tight straight-line clouds, near 0.5 like loose but visible trends, and near 0.1 like nearly patternless scatter.

Key idea: correlation is covariance after standardizing both variables. It is unit-free, lives in [-1, 1], and measures straight-line association only.

Independence and covariance: a one-way street

Two random variables are independent when every joint cell factors into its margins: p(x, y) = P(X = x) P(Y = y) for every pair of values. That is the random-variable version of event independence from module two, and a single failed cell spoils it. Test our table's top-left cell: independence would demand P(X = 0 and Y = 0) = 0.30 × 0.40 = 0.12, but the table says 0.20. So X and Y are dependent, matching the conditional shift we found earlier.

Independence has a clean consequence. When the joint factors, E[XY] factors too, so E[XY] = E[X]E[Y] and Cov(X, Y) = 0. Independent variables always have zero covariance and zero correlation, which is why a nonzero covariance is proof of dependence.

The converse fails, and one small example shows it. Let X take the values -1, 0, and 1, each with probability 1/3, and let Y = X², so Y is completely determined by X. Compute the ingredients:

• E[X] = (-1)(1/3) + 0(1/3) + 1(1/3) = 0.
• XY = X × X² = X³, and for the values -1, 0, and 1 the cube equals the number itself, so E[XY] = E[X] = 0.
• Therefore Cov(X, Y) = E[XY] - E[X]E[Y] = 0 - 0 × E[Y] = 0.

Zero covariance, yet total dependence: knowing X tells you Y exactly. The factoring test fails concretely, since Y = 0 happens only when X = 0, so P(X = 0 and Y = 0) = 1/3, while P(X = 0) P(Y = 0) = (1/3)(1/3) = 1/9. What went wrong for covariance? The relationship is a symmetric U shape: Y is high when X is very low and when X is very high. The products from the negative side and the positive side cancel exactly. Covariance is blind to curved relationships; it detects straight-line leaning only.

Key idea: independence implies zero covariance, never the reverse. Zero covariance rules out a linear trend, not a relationship.

Try it

A fresh 2 × 2 joint distribution: P(X = 0 and Y = 0) = 0.40, P(X = 0 and Y = 1) = 0.10, P(X = 1 and Y = 0) = 0.10, and P(X = 1 and Y = 1) = 0.40. Find the marginals, E[X], E[Y], E[XY], Cov(X, Y), and rho. Work it out before reading on.

Worked answer. Step 1, marginals: P(X = 0) = 0.40 + 0.10 = 0.50, so P(X = 1) = 0.50, and by the same sums P(Y = 0) = P(Y = 1) = 0.50. Step 2, means: E[X] = 0(0.50) + 1(0.50) = 0.50, and E[Y] = 0.50 the same way. Step 3, the cell walk for E[XY]: only the (1, 1) cell has both values nonzero, contributing 1 × 1 × 0.40 = 0.40. Step 4, the shortcut: Cov = 0.40 - (0.50)(0.50) = 0.40 - 0.25 = 0.15.

Step 5, correlation. Each variable is a Bernoulli with p = 0.5, whose variance is 0.5 × 0.5 = 0.25 and standard deviation 0.50. So rho = 0.15 / (0.50 × 0.50) = 0.15 / 0.25 = 0.60, a moderately strong positive association. You can see it in the table: 80% of the probability sits on the matching diagonal, the cells (0, 0) and (1, 1). What we just did: margins, means, cell walk, shortcut, standardize.

Where people get stuck

The most common error is manufacturing joint probabilities by multiplying marginals. P(X = 2) = 0.30 and P(Y = 1) = 0.60 do not entitle you to claim P(X = 2 and Y = 1) = 0.18; the table says 0.20. Multiplying margins is legal only after independence is known or assumed. In every other case the joint table is the ground truth, and the margins are summaries computed from it, not building blocks that recreate it.

Second, the E[XY] step. It is tempting to compute E[X] × E[Y] and call that E[XY], but the entire content of covariance is that these can differ. E[XY] must be earned cell by cell, value times value times probability, summed over the whole grid. A related slip is forgetting that cells with either value zero contribute nothing.

Third, comparing covariances across problems. A covariance of 7 in one dataset is not stronger evidence of association than 0.10 in another; the two numbers live in different units, X-units times Y-units, and are not comparable. Convert to correlation first.

Fourth, reading rho = 0 as "unrelated." The U-shaped counterexample is the antidote: perfect dependence, zero correlation. Correlation tests one specific pattern, the straight line.

Common misconceptions

• "The marginals determine the joint." They do not. Our Try-it table and an independent table with the same 0.50 margins assign different probabilities to the same cells; the pairing pattern is extra information.
• "Zero covariance means independence." It means no linear trend. Y = X² with symmetric X has covariance 0 and complete dependence.
• "A strong correlation means X causes Y." Correlation measures co-movement, not causation. Ice-cream sales and drowning deaths correlate because summer drives both; a lurking third variable is the standard alternative story.
• "rho = -0.8 is weaker than rho = +0.8." They are equally strong linear associations in opposite directions. The sign is direction; the absolute value is strength.

Recap

• The joint pmf p(x, y) = P(X = x and Y = y) assigns a probability to every pair of values, all summing to 1.
• Marginals come from row and column sums; conditionals come from cell divided by margin.
• Cov(X, Y) = E[XY] - E[X]E[Y], with E[XY] computed cell by cell; the sign gives the direction of co-movement.
• Correlation rho = Cov / (sigma_X sigma_Y) is unit-free and lives in [-1, 1]; its magnitude measures linear strength.
• Independence means every cell factors into its margins, and it forces covariance to 0.
• Zero covariance does not force independence; curved relationships hide from it.

Covariance is more than a descriptive statistic; it is the correction term in the variance of a sum. The next lesson shows that Var(X + Y) picks up an extra 2 Cov(X, Y), so everything computed here feeds directly into the algebra of sums and averages, the road that leads to the Central Limit Theorem.

Sources

1. Encyclopedia of Mathematics. (n.d.). Joint distribution. European Mathematical Society. encyclopediaofmath.org
2. Encyclopedia of Mathematics. (n.d.). Covariance. European Mathematical Society. encyclopediaofmath.org
3. Encyclopedia of Mathematics. (n.d.). Correlation coefficient. European Mathematical Society. encyclopediaofmath.org
4. Weisstein, E. W. (n.d.). Covariance. MathWorld: A Wolfram web resource. mathworld.wolfram.com
5. Weisstein, E. W. (n.d.). Correlation coefficient. MathWorld: A Wolfram web resource. mathworld.wolfram.com
6. Siegrist, K. (n.d.). 04: Expected value. In Probability, mathematical statistics, and stochastic processes. LibreTexts. stats.libretexts.org
7. Grinstead, C. M., & Snell, J. L. (n.d.). Introductory probability. LibreTexts. stats.libretexts.org

Totals and averages are everywhere: the total of two dice, a week of daily sales added up, the average of five lab measurements. Each is a new random variable built by adding old ones, so we need rules for how means, variances, and whole distributions behave under addition. Those rules are this lesson's business, and they are the engine that powers the Central Limit Theorem in the finale.

The big picture

Two rules run the show, and they have very different personalities. The first, linearity of expectation, is as generous as theorems get: the mean of a sum is the sum of the means, always, with no conditions, even when the variables influence each other. The second, the variance rule, is more careful: variances add only after a correction term involving covariance, and the correction vanishes exactly when the covariance is zero, which independence guarantees.

The payoff arrives when we add many copies of the same experiment. A total of n independent draws has mean n times mu, and variance n times sigma², so its standard deviation grows like the square root of n. Divide by n to form the average and the spread shrinks like 1 over the square root of n. That one algebraic fact explains why polls of a thousand people work, why labs repeat measurements, and why casinos survive: averaging cancels noise at a predictable rate.

Key idea: means add unconditionally; variances add under independence (more precisely, whenever covariances are zero). Everything about averages follows from these two rules.

Expectation of a sum: always

The statement first, in symbols and words:

E[X + Y] = E[X] + E[Y], and in general E[X1 + X2 + ... + Xn] = E[X1] + ... + E[Xn].

The expected value of a sum is the sum of the expected values, whatever the variables are, dependent or not. Here is why, for discrete variables, using the joint distribution from last lesson. The sum X + Y is a function of the pair, so its expectation walks the joint table: E[X + Y] = ∑ (x + y) p(x, y), summed over every cell.

Each term splits in two, since (x + y) p = x p + y p. Group the x-parts: summing x p(x, y) over all cells collapses, row by row, to ∑ x P(X = x), which is E[X]. The y-parts collapse the same way to E[Y]. No independence was used anywhere in that argument.

Constants ride along in the natural way: E[aX + bY + c] = a E[X] + b E[Y] + c.

Worked example. Roll two dice. Each die has mean (1 + 2 + 3 + 4 + 5 + 6)/6 = 3.5, so the expected total is 3.5 + 3.5 = 7. Now stress-test the "always" claim with an extreme dependence: let Y be a literal copy of X, the same die read twice. Then X + Y = 2X, and E[2X] = 2 × 3.5 = 7 again. Perfect dependence did not bend linearity at all.

Linearity is also why the binomial mean from module three is np. The number of heads in 10 fair flips is a sum of 10 indicator variables, one per flip, each worth 1 with probability 0.5. Each indicator has mean 0.5, so the count has mean 10 × 0.5 = 5. No formula grinding, only addition.

Key idea: expectation distributes over addition with no fine print. When a variable can be written as a sum of simpler pieces, its mean is the sum of the pieces' means.

Variance of a sum: the covariance correction

Variance is where dependence makes itself felt. Write the deviations a = X - mu_X and b = Y - mu_Y, so that Var(X) = E[a²], Var(Y) = E[b²], and, from last lesson, Cov(X, Y) = E[ab]. The sum X + Y deviates from its own mean by exactly a + b, so:

Var(X + Y) = E[(a + b)²] = E[a² + 2ab + b²] = E[a²] + 2 E[ab] + E[b²],

using the school identity (a + b)² = a² + 2ab + b² and then linearity of expectation on the three terms. Translating back:

Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X, Y).

When X and Y are independent, the covariance is 0 and the correction disappears:

Var(X + Y) = Var(X) + Var(Y).

Worked example. Let X and Y be independent with Var(X) = 4 and Var(Y) = 9. Then Var(X + Y) = 4 + 9 = 13, so the standard deviation of the sum is √13 ≈ 3.61. It is not 2 + 3 = 5. Variances add; standard deviations do not. Standard deviations combine like the sides of a right triangle: the answer √(2² + 3²) is the hypotenuse, and a hypotenuse is shorter than the two legs laid end to end, because independent noises partly cancel.

A dependent example, using last lesson's table: there Var(X) = 0.60, Var(Y) = 0.24, and Cov(X, Y) = 0.10, so Var(X + Y) = 0.60 + 0.24 + 2(0.10) = 1.04, more than the 0.84 that independence would give. Positive covariance inflates a sum's variance because the variables surge and slump together. Negative covariance deflates it, which is the entire logic of hedging: pair an asset with something that moves opposite to it and the total swings less than either piece.

Differences behave the same way, not the opposite way. Since X - Y = X + (-Y), and flipping a variable's sign flips the sign of its covariances but leaves its variance alone, Var(X - Y) = Var(X) + Var(Y) - 2 Cov(X, Y), and for independent variables Var(X - Y) = Var(X) + Var(Y). Subtracting a noisy quantity adds noise. The minus sign squares away.

Key idea: Var(X + Y) = Var(X) + Var(Y) + 2Cov(X, Y). Independence kills the correction; positive covariance inflates the sum's spread; negative covariance dampens it; differences still add variance.

The distribution of a sum: dice totals

Mean and variance summarize a sum, but sometimes we want its full distribution. For independent discrete variables there is a recipe: to find P(T = t) where T = X + Y, list every way to split t between the two variables and add the probabilities, P(T = t) = ∑ P(X = x) P(Y = t - x). The pattern is called a convolution, and the two-dice total shows it at work.

Take T = the total of two fair dice, each outcome pair having probability 1/36. Work a few totals:

• T = 2 needs the split (1, 1) only: P(T = 2) = 1/36.
• T = 3 needs (1, 2) or (2, 1): P(T = 3) = 2/36.
• T = 7: first die 1 pairs with 6, 2 with 5, 3 with 4, 4 with 3, 5 with 2, 6 with 1, six splits in all, so P(T = 7) = 6/36 = 1/6.

Counting all totals from 2 to 12 gives the tally 1, 2, 3, 4, 5, 6, 5, 4, 3, 2, 1 out of 36, a triangle peaking at 7. Something remarkable happened: each die alone is flat, every face equally likely, yet the sum of two flat distributions is not flat. Middle totals win because they can be assembled in many ways, while extremes require both dice to cooperate. Add a third die and the shape smooths further into a mound; keep adding and it approaches a bell. That drift toward the bell is exactly what the Central Limit Theorem formalizes next lesson.

The summary numbers check out too. Each die has variance 35/12 ≈ 2.92 (computed in module three), so the independent total has mean 7 and variance 35/12 + 35/12 = 35/6 ≈ 5.83, standard deviation about 2.42.

iid versus merely uncorrelated

Two vocabulary items organize everything that follows. A collection X1, ..., Xn is iid, independent and identically distributed, when the variables are mutually independent and all share one distribution, hence one mean mu and one standard deviation sigma. Repeated coin flips, repeated die rolls, and random samples from a large population are the standard examples. The iid setting is the Central Limit Theorem's home ground.

A weaker condition is being uncorrelated: every pairwise covariance is zero. Independence implies uncorrelated, but last lesson's U-shaped counterexample shows the reverse fails. The distinction earns its keep here because the two headline rules need different strengths. For n variables,

Var(X1 + ... + Xn) = Var(X1) + ... + Var(Xn) + 2 × (sum of all pairwise covariances),

so variances add whenever the variables are merely pairwise uncorrelated; full independence is not required. The convolution recipe, by contrast, and the distribution-level claims built on it, genuinely need independence. Means, of course, add with no assumption at all.

Key idea: iid means independent plus same distribution. Uncorrelated is weaker, yet already enough for variances to add; independence is needed for statements about the sum's whole distribution.

The sample mean and the square-root-n law

Now the star application. Take n iid observations X1, ..., Xn, each with mean mu and standard deviation sigma, and form the sample mean X-bar = (X1 + ... + Xn)/n. Two short computations give its behavior.

Center: by linearity, the total T = X1 + ... + Xn has E[T] = n mu, and dividing by the constant n gives E[X-bar] = mu. The sample mean is aimed at the true mean; statisticians say it is unbiased.

Spread: by independence, Var(T) = n sigma². Constants come out of variance squared, Var(cX) = c² Var(X), so dividing T by n divides its variance by n²: Var(X-bar) = n sigma² / n² = sigma² / n. Taking square roots, the standard deviation of the sample mean, called the standard error, is sigma / √n.

Worked example. A population has sigma = 15. For samples of n = 25, the standard error is 15 / √25 = 15 / 5 = 3. For n = 100 it is 15 / 10 = 1.5. Notice the exchange rate: quadrupling the sample size only halved the error, because n sits under a square root. Precision is bought at an ever-steeper price.

Why does averaging calm the randomness? In a large sample, unusually high draws and unusually low draws arrive in roughly equal numbers, and adding them lets the surpluses fill the deficits. The cancellation is not perfect, and the leftover wobble is exactly sigma/√n worth. The same law is why a casino's average win per bet becomes nearly certain over millions of bets, and why pooled independent insurance risks have a predictable average.

Key idea: X-bar has mean mu and standard error sigma/√n. Averaging n independent measurements shrinks noise by the factor √n, no faster.

Try it

Four iid measurements X1, X2, X3, X4 each have mean 10 and variance 8. Find the mean and standard deviation of (a) the total T, (b) the average X-bar, and (c) the shortcut 4X1, meaning one measurement multiplied by four. Work all three before reading on.

Worked answer. (a) E[T] = 4 × 10 = 40; by independence Var(T) = 8 + 8 + 8 + 8 = 32, so sd = √32 ≈ 5.66. (b) E[X-bar] = 10; Var(X-bar) = 8/4 = 2, so the standard error is √2 ≈ 1.41.

(c) E[4X1] = 40, matching the total, but Var(4X1) = 4² × 8 = 128, sd ≈ 11.31, twice the total's 5.66. What we just did: adding four independent copies pools four separate noises that partly cancel; multiplying one copy by 4 amplifies a single noise with no cancellation. Sums and multiples share a mean but not a spread.

Where people get stuck

The classic slip is adding standard deviations. With Var(X) = 4 and Var(Y) = 9, the sum's standard deviation is √13 ≈ 3.61, not 5. The rule lives at the variance level, one floor down from standard deviations, so the safe habit is: convert to variances, add, convert back at the very end.

Second, the difference trap. Asked about X - Y, many subtract variances, getting 9 - 4 = 5 or even a negative number, which is impossible, since variance is an average of squares. For independent variables, Var(X - Y) = Var(X) + Var(Y) = 13, the same as the sum. Uncertainty accumulates whether you add or subtract a noisy quantity.

Third, confusing 2X with X1 + X2. Doubling one die roll gives Var(2X) = 4 × 35/12 ≈ 11.67, while rolling twice and adding gives 35/6 ≈ 5.83. The first repeats one random outcome twice, so its noise is fully synchronized with itself; the second draws two fresh outcomes that can offset. Betting 20 dollars on one hand is genuinely riskier than 10 dollars on each of two independent hands.

Fourth, dropping the covariance term for dependent variables. Add ten stocks that all sink in the same recessions and the portfolio variance far exceeds the sum of the ten variances, because all the pairwise covariances pile on. The independent-sum formula is a special case, not the general law; check independence before using it.

Common misconceptions

• "Linearity of expectation requires independence." It requires nothing. Even for a variable added to a copy of itself, means add; only the variance rule cares about dependence.
• "Standard deviations add." Variances add (when covariances vanish). Standard deviations combine like perpendicular legs of a triangle, √(a² + b²), which is less than a + b.
• "Every two-dice total from 2 to 12 is equally likely." The convolution triangle says otherwise: 7 occurs six ways (probability 1/6), while 2 and 12 occur one way each (1/36).
• "Doubling the sample size halves the standard error." The error scales as 1/√n, so doubling n divides the error by √2 ≈ 1.41; it takes four times the data to halve it.

Recap

• E[X + Y] = E[X] + E[Y] always; sums of many variables and constants follow by the same linearity.
• Var(X + Y) = Var(X) + Var(Y) + 2Cov(X, Y); independence (or mere uncorrelatedness) reduces it to plain addition.
• Var(X - Y) also adds the variances for independent variables; subtraction never subtracts noise.
• The distribution of an independent sum comes from the convolution recipe; two flat dice already produce a triangular total peaked at 7.
• iid = independent and identically distributed; uncorrelated is weaker but suffices for variance addition.
• The sample mean has E[X-bar] = mu and standard error sigma/√n, the square-root-n law of averaging.

You now hold the complete bookkeeping for sums: means add, variances add with a covariance correction, and repeated convolution pushes the shape of a total toward a bell. The final lesson names that push. The Central Limit Theorem says the bell is not a coincidence of dice but the universal destination of averages.

Sources

1. OpenStax. (2023). 7.1 The central limit theorem for sample means (averages). In Introductory statistics 2e. Rice University. openstax.org
2. OpenStax. (2023). 7.2 The central limit theorem for sums. In Introductory statistics 2e. Rice University. openstax.org
3. Encyclopedia of Mathematics. (n.d.). Mathematical expectation. European Mathematical Society. encyclopediaofmath.org
4. Encyclopedia of Mathematics. (n.d.). Variance. European Mathematical Society. encyclopediaofmath.org
5. Weisstein, E. W. (n.d.). Variance. MathWorld: A Wolfram web resource. mathworld.wolfram.com
6. Siegrist, K. (n.d.). 04: Expected value. In Probability, mathematical statistics, and stochastic processes. LibreTexts. stats.libretexts.org
7. Khan Academy. (n.d.). Sampling distributions. khanacademy.org

The Central Limit Theorem (CLT) is the crown jewel of probability theory and the reason the normal curve appears everywhere. The last two lessons built the parts: the sample mean X-bar is centered at mu with standard error sigma/√n, and sums of flat dice already bulge in the middle. The CLT supplies the missing piece, the shape, and the shape is always the bell.

The big picture

Here is the puzzle the theorem resolves. Heights, measurement errors, exam averages, and poll results keep producing the same bell-shaped histogram, even though the underlying ingredients, genes, vibrations, quiz questions, individual voters, follow no common pattern. What could force so many unrelated processes into one mold? The answer is not about the ingredients at all. It is about the act of adding them up. Averaging is a machine that manufactures bells, whatever raw material it is fed.

That is the CLT's claim: take many independent draws from essentially any population, form their sum or average, and the result's distribution approaches a normal curve. The population can be skewed, lumpy, or two-humped; the average forgets all of it except the mean and the spread. This lesson states the theorem precisely, shows why the bell emerges, works three standardization examples with real uses, and, equally important, fences off what the theorem does not promise.

Key idea: the normal distribution is the universal destination of averages. The population's shape washes out; only mu and sigma survive, as the center and the sigma/√n spread.

The statement, in words and in symbols

In words: draw n independent observations from one population with mean mu and finite standard deviation sigma. If n is large, the sample mean X-bar behaves like a normal random variable centered at mu with standard deviation sigma/√n, no matter what shape the population has.

In symbols: for X1, ..., Xn iid with mean mu and finite sigma,

X-bar is approximately Normal(mu, sigma²/n), equivalently z = (X-bar - mu) / (sigma / √n) is approximately standard normal,

with the approximation improving as n grows. The same holds for the total T = X1 + ... + Xn, which is approximately Normal with mean n mu and variance n sigma². Every ingredient is familiar: the center mu and standard error sigma/√n came from last lesson's algebra. The theorem's new gift is the license to use the z-table on averages.

Read the picture carefully, because it compares two different histograms. The left shows the population, single values, strongly skewed. The right shows what you would get by repeatedly taking a sample, averaging it, and histogramming those averages. The individuals stay skewed forever; it is the averages that arrange themselves into a bell.

Why the bell appears

The dice from last lesson already told the story in miniature. One die is flat. The total of two dice is a triangle, because middle totals like 7 can be assembled six ways while a 2 needs both dice to cooperate on snake eyes. Middling results have many routes; extreme results require every ingredient to lean the same way at once, and independent ingredients rarely coordinate. Each additional die convolves the shape once more, rounding the corners, and by four or five dice the total's histogram is visibly bell-like.

The same logic runs for any population. An average lands far above mu only if most of the draws conspire high together, which independence makes exponentially unlikely; an average near mu can happen in astronomically many ways. The normal curve is also self-reinforcing: sums of normal variables are exactly normal, so once repeated convolution gets near the bell, further adding keeps it there. In that sense the normal is a fixed point that addition flows toward, which is why so many "sum of many small independent effects" quantities, measurement error above all, look Gaussian in practice.

Key idea: extremes need coordination, middles have many routes. Repeated convolution smooths any starting shape toward the one shape addition preserves, the normal.

How large must n be?

A standard rule of thumb says n at least 30 makes the approximation adequate for most populations. Treat it as a guideline with sliding scale, not a law. If the population is symmetric and short-tailed, averages look normal by n = 10 or sooner. If it is strongly skewed, like incomes or insurance claims, n may need to be in the hundreds before the tails behave. If the population is itself normal, X-bar is exactly normal at every n, including n = 1. For yes/no data there is a sharper check coming below: np and n(1 - p) should both be at least about 10.

Worked example: an average

A population has mean mu = 100 and standard deviation sigma = 15; the shape is unknown. Take a random sample of n = 25. What is the probability the sample mean exceeds 106?

Step 1, standard error: sigma / √n = 15 / √25 = 15 / 5 = 3.

Step 2, standardize the average: z = (106 - 100) / 3 = 6 / 3 = 2.0.

Step 3, normal tail: P(Z > 2) = 0.0228, about 2.3%. (The empirical rule's estimate is 2.5%, since 95% lies within 2 standard deviations.)

Now the instructive contrast. What is the probability one single observation exceeds 106? That z-score uses sigma itself: z = (106 - 100)/15 = 0.4, and P(Z > 0.4) ≈ 0.345. One value beats 106 about a third of the time; an average of 25 values almost never does. Averages are tamer than individuals, by exactly the factor √n = 5, and choosing the right denominator, sigma for one draw, sigma/√n for an average, is the entire skill of CLT problems. What we just did: standard error, z, tail, then compared against the single-draw calculation.

Worked example: 100 coin flips

Flip a fair coin 100 times. How likely is it to land 60 or more heads? The count H is binomial with n = 100 and p = 0.5, a sum of 100 independent 0-or-1 indicators, so the CLT applies to it directly. Its mean is np = 50 and its standard deviation is √(np(1-p)) = √(100 × 0.5 × 0.5) = √25 = 5. Standardize 60: z = (60 - 50)/5 = 2.0, giving a tail of about 0.0228.

The exact binomial answer is 0.0284, so the bell is close and gets closer with a small refinement: since H is a whole number, "60 or more" begins at 59.5 on the continuous curve, and z = (59.5 - 50)/5 = 1.9 gives 0.0287, nearly exact. This whole-number adjustment is called the continuity correction. The success-failure check np = 50 and n(1 - p) = 50, both far above 10, is what certified the approximation in the first place. Sixty heads in a hundred sounds unremarkable, yet it is a two-sigma event; intuition usually underestimates how steady proportions become at this scale.

Worked example: the polling margin of error

A poll asks n = 1000 random voters a yes/no question, and the true population share of yes is p = 0.5. The sample proportion p-hat is an average of 1000 indicator variables, so the CLT makes it approximately normal with mean 0.5 and standard error √(p(1 - p)/n) = √(0.25/1000) = √0.00025 ≈ 0.0158.

From the normal lesson, 95% of a bell sits within 1.96 standard deviations of center. Here that window is 1.96 × 0.0158 ≈ 0.031, about 3 percentage points either way. That is the famous "margin of error plus or minus 3 points" attached to 1000-person polls: it is nothing more than the CLT plus the 1.96 landmark. What we just did: recognized a proportion as an average, computed its standard error, and wrapped a 95% normal window around it. Quadrupling the poll to 4000 people would only halve the margin, the square-root-n price tag again.

Try it

A filling machine pours bags with weight mean mu = 500 grams and standard deviation sigma = 20 grams, shape unknown. (a) For cartons of n = 64 bags, find the probability that the carton's average bag weight is below 495 grams. (b) A single bag is below 495 grams with what approximate probability, if bag weights happened to be normal? Work both before reading on.

Worked answer. (a) Standard error: 20 / √64 = 20 / 8 = 2.5. Standardize: z = (495 - 500) / 2.5 = -2.0. Left tail: P(Z < -2) = 0.0228, about 2.3%, and the CLT licenses this even without knowing the population's shape, because 64 bags are being averaged. (b) For one bag, z = (495 - 500)/20 = -0.25, and P(Z < -0.25) ≈ 0.401, about 40%, but this needed the normality assumption, since no averaging protects a single observation. What we just did: same subtraction both times, different denominators, and only the average gets the CLT's shape guarantee.

What the CLT does not say

Misreadings of the theorem are common enough to deserve their own section, so here are the boundaries, stated bluntly.

• It does not say the population becomes normal. Incomes stay skewed no matter how many people you survey; only the distribution of the sample average normalizes.
• It does not say individual observations settle down. Each new draw is as variable as the first; the steadiness belongs to the average alone.
• It does not repair bias. Averaging a miscalibrated scale gives a very precise wrong answer: the standard error shrinks toward zero while the systematic error stays put.
• It does not work without finite variance. A few heavy-tailed distributions, the Cauchy being the classic case, have so much tail that averages never tame; the average of Cauchy draws is Cauchy again at every n.
• It does not make n = 30 a magic threshold, and it does not apply automatically to dependent draws; correlated observations can need very different treatment.

A little history

The theorem was born inside a coin-flipping calculation. In 1733 Abraham de Moivre, working out odds for large numbers of fair coin tosses, discovered that the unwieldy binomial probabilities were beautifully approximated by the bell-shaped curve, the first appearance of the normal distribution in mathematics. Pierre-Simon Laplace generalized the discovery around 1810, showing sums of many kinds of variables obey it, and the binomial case still carries both names as the de Moivre-Laplace theorem, exactly the coin-flip example above.

Rigorous general proofs came only in the early twentieth century, from Aleksandr Lyapunov in 1901 and later Lindeberg, who pinned down precisely which conditions are needed. The modern name arrived in 1920, when George Pólya called it the central limit theorem, central meaning "of central importance." Francis Galton, writing in 1889, marveled that so orderly a law could govern the apparent chaos of large random samples, and the marvel has not worn off: the CLT remains the working justification for treating messy averages as bell curves.

Where people get stuck

The dominant error is the denominator. Standardizing an average with sigma instead of sigma/√n makes the event look far less surprising than it is; in the worked example above, that slip turns z = 2.0 into z = 0.4. Before computing, ask one question: is this about a single draw or about an average of n draws? Single draws use sigma, averages use the standard error, and only the average carries the CLT's guarantee.

Second, applying normal-shape reasoning to raw data because n is large. A histogram of 10000 incomes is still skewed; largeness of the sample never reshapes individuals. The bell belongs to X-bar, a quantity you only see by imagining many repeated samples.

Third, forgetting the fine print: independence and finite variance. Averages of strongly correlated measurements, like temperatures on consecutive minutes, do not enjoy the sigma/√n shrinkage, because the covariance terms from last lesson refuse to vanish. The theorem earns its power from assumptions, and checking them is part of using it.

Fourth, treating the empirical rule and table landmarks as exact for small n. For n = 5 draws from a skewed population, the "2.5% beyond two sigma" figure can be badly off in one tail. When n is small and the population is far from normal, the honest answers come from the population itself, not the bell.

Common misconceptions

• "The CLT makes everything normal." It speaks only about sums and averages of many independent pieces. Quantities built by multiplication or by a single dominant cause have no bell guarantee.
• "n at least 30 guarantees accuracy." Thirty is a serviceable default; symmetric populations need fewer, heavily skewed ones need far more, and normal populations need none.
• "More data makes each measurement more reliable." Individual variability never changes; sigma is a property of the population. Data volume shrinks only the average's standard error.
• "The sample mean equals mu for large n." X-bar still wobbles around mu; the CLT describes the wobble as normal with width sigma/√n, shrinking but never zero.

Recap

• CLT: for iid draws with mean mu and finite sigma, X-bar is approximately Normal(mu, sigma²/n) for large n, whatever the population's shape.
• Standardize averages with the standard error: z = (X-bar - mu)/(sigma/√n); standardize single draws with sigma.
• The bell emerges because extremes need coordination and repeated convolution smooths every shape toward the normal, which addition preserves.
• Coin counts and poll proportions are sums of indicators, so the CLT covers them; the 1.96 window gives the plus-or-minus-3-point margin for n = 1000.
• The CLT does not normalize populations or individuals, does not fix bias, and needs independence with finite variance.
• History: de Moivre 1733, Laplace circa 1810, Lyapunov 1901, named central by Pólya in 1920.

This closes the course's long arc. Sample spaces gave us events, axioms gave events numbers, random variables condensed outcomes into quantities, distributions organized the quantities, and the algebra of sums explained averages. The CLT crowns it all by handing every well-behaved average to the normal curve, which is the doorway from probability into statistics: confidence intervals and hypothesis tests are the CLT worn as everyday clothing.

Sources

1. OpenStax. (2023). 7.1 The central limit theorem for sample means (averages). In Introductory statistics 2e. Rice University. openstax.org
2. OpenStax. (2023). 7.3 Using the central limit theorem. In Introductory statistics 2e. Rice University. openstax.org
3. Encyclopedia of Mathematics. (n.d.). Central limit theorem. European Mathematical Society. encyclopediaofmath.org
4. MacTutor History of Mathematics Archive. (n.d.). Abraham de Moivre. University of St Andrews. mathshistory.st-andrews.ac.uk
5. Le Cam, L. (1986). The central limit theorem around 1935. Statistical Science, 1(1). doi.org/10.1214/ss/1177013818
6. Lindeberg, J. W. (1922). Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung. Mathematische Zeitschrift, 15(1), 211-225. doi.org/10.1007/BF01494395
7. Pólya, G. (1920). Über den zentralen Grenzwertsatz der Wahrscheinlichkeitsrechnung und das Momentenproblem. Mathematische Zeitschrift, 8(3-4), 171-181. doi.org/10.1007/BF01206525