Probability Theory

Probability theory is the branch of mathematics that measures uncertainty. It starts by describing a random experiment, any process whose outcome cannot be predicted with certainty, such as flipping a coin, rolling a die, or drawing a card.

The sample space

The sample space, written S (some books use the Greek letter Omega), is the set of all possible outcomes of the experiment. Each individual outcome is called a sample point. A few examples:

• One coin flip: S = {H, T}.
• One roll of a six-sided die: S = {1, 2, 3, 4, 5, 6}.
• Two coin flips: S = {HH, HT, TH, TT}, four outcomes.

A sample space can be finite (the die), countably infinite (the number of flips until the first head: 1, 2, 3, and so on), or continuous (the exact time a bus arrives, any real number in an interval).

Events as subsets

An event is any subset of the sample space, that is, any collection of outcomes we care about. We say an event occurs if the actual outcome is one of its members. For the die, "rolling an even number" is the event A = {2, 4, 6}. An event with a single outcome, like {5}, is a simple event; an event with several outcomes is a compound event. The whole space S is the certain event (it always occurs), and the empty set, written as the empty braces, is the impossible event (it never occurs).

Combining events

Because events are sets, we combine them with set operations:

• The union "A or B" (A union B) contains every outcome in A, in B, or in both.
• The intersection "A and B" (A intersect B) contains only outcomes in both A and B.
• The complement "not A" (written A with a bar, or A-complement) contains every outcome in S that is not in A.

Two events are mutually exclusive (or disjoint) if they share no outcomes, so their intersection is empty and they cannot both happen at once. For the die, A = {2, 4, 6} and B = {1, 3, 5} are mutually exclusive.

Worked example

Roll one die. Let A = {2, 4, 6} (even) and C = {5, 6} (greater than 4). Then the union is A or C = {2, 4, 5, 6}, the intersection is A and C = {6}, and the complement of A is not A = {1, 3, 5}. A Venn diagram makes these overlaps easy to see.

Every probability question begins here: name the experiment, write the sample space, and translate the words of the problem into events built with union, intersection, and complement. Getting this translation right is half the battle in the rest of the course.

To turn events into numbers we assign each event A a probability P(A). Rather than define probability by a formula, modern mathematics builds it on three simple rules, the axioms of probability, first set out cleanly by Andrey Kolmogorov. Everything else follows from them.

The three axioms

1. Nonnegativity: for every event A, P(A) is greater than or equal to 0. Probabilities are never negative.
2. Normalization: the whole sample space has probability P(S) = 1. Something in S is certain to happen.
3. Additivity: if A and B are mutually exclusive, then P(A or B) = P(A) + P(B). For disjoint events, probabilities simply add. (This extends to any number of pairwise disjoint events.)

From these three rules alone, several everyday facts can be proven.

Consequences of the axioms

• The impossible event: P(empty set) = 0.
• Bounds: every probability satisfies 0 is less than or equal to P(A) is less than or equal to 1.
• The complement rule: since A and "not A" are disjoint and together fill S, we get P(A) + P(not A) = 1, so P(not A) = 1 - P(A).
• The general addition rule: for any two events, disjoint or not, P(A or B) = P(A) + P(B) - P(A and B). We subtract the overlap so it is not counted twice.

Equally likely outcomes

When a sample space has N outcomes that are all equally likely, each outcome has probability 1/N, and for any event A,

P(A) = (number of outcomes in A) / (total number of outcomes) = |A| / N.

Worked example. Draw one card from a standard 52-card deck, so N = 52. The event "a heart" has 13 outcomes, so P(heart) = 13/52 = 1/4 = 0.25. The event "a king" has 4 outcomes, so P(king) = 4/52 = 1/13.

Using the addition rule. Find P(king or heart). There are 4 kings and 13 hearts, but the king of hearts belongs to both, so the overlap has probability 1/52. Then

P(king or heart) = 4/52 + 13/52 - 1/52 = 16/52 = 4/13, about 0.31.

Using the complement rule. The probability of "not a heart" is 1 - 13/52 = 39/52 = 3/4. The complement rule is the fastest route whenever a problem contains the words "at least one," a pattern you will use constantly.

When outcomes are equally likely, a probability is just a ratio of two counts. So we need efficient ways to count outcomes without listing them all. That is the job of combinatorics.

The multiplication principle

If a task is done in stages, with n1 choices at the first stage, n2 at the second, and so on, then the total number of ways to complete the whole task is the product n1 x n2 x .... Rolling a die then flipping a coin gives 6 x 2 = 12 outcomes. A 4-digit PIN allows 10 x 10 x 10 x 10 = 10,000 codes.

Factorials

The number of ways to arrange n distinct objects in a row is n factorial, written n!, defined as n! = n x (n-1) x ... x 2 x 1. For example 5! = 5 x 4 x 3 x 2 x 1 = 120. By convention 0! = 1.

Permutations: order matters

A permutation counts arrangements where order matters. The number of ordered arrangements of k objects chosen from n distinct objects is

P(n, k) = n! / (n - k)!.

Worked example. How many ways can gold, silver, and bronze medals be awarded among 10 runners? Order matters (first is not the same as third), so P(10, 3) = 10 x 9 x 8 = 720.

Combinations: order does not matter

A combination counts selections where order does not matter. The number of ways to choose k objects from n, ignoring order, is the binomial coefficient

C(n, k) = n! / (k! (n - k)!), read "n choose k".

Worked example. A committee of 2 is chosen from 5 people. Order does not matter, so C(5, 2) = 5! / (2! 3!) = (5 x 4) / (2 x 1) = 10. Notice how permutations and combinations relate: there are more permutations than combinations because each unordered group of 2 can be ordered in 2! = 2 ways, and indeed P(5,2) = 20 = 2! x C(5,2).

Putting counting to work in probability

How many 5-card poker hands are there from a 52-card deck? Order does not matter, so C(52, 5) = 2,598,960. What is the probability that all 5 cards are hearts? There are 13 hearts, and the number of 5-heart hands is C(13, 5) = 1,287. Therefore

P(all hearts) = C(13, 5) / C(52, 5) = 1,287 / 2,598,960 = 0.000495, about 1 in 2,020.

The key decision in every counting problem is a single question: does order matter? If yes, use a permutation; if no, use a combination. Answer that first, and the rest is arithmetic.

New information changes probabilities. If you learn that a rolled die shows an even number, the chance it is a 2 is no longer 1/6. Conditional probability is the tool that revises a probability once we know some event has occurred.

The definition

The probability of A given B, written P(A | B), is the probability that A occurs when we already know B has occurred. It is defined as

P(A | B) = P(A and B) / P(B), provided P(B) is greater than 0.

Conditioning on B shrinks the world to just the outcomes in B; we then ask what fraction of that smaller world also lies in A.

Worked example. Roll a fair die. Let A = {2} and B = {2, 4, 6} (even). Then P(A and B) = P({2}) = 1/6 and P(B) = 3/6 = 1/2, so P(A | B) = (1/6) / (1/2) = 2/6 = 1/3. Knowing the roll is even raises the chance it is a 2 from 1/6 to 1/3.

Reading a two-way table

Conditional probabilities are easy to read from a table of counts. Suppose 200 students are classified by whether they study music and whether they play a sport:

The chance a random student plays a sport given they study music is P(sport | music) = 30 / 50 = 0.60: restrict to the 50 music students, then find the fraction who play a sport.

The multiplication rule

Rearranging the definition gives the multiplication rule for a joint probability:

P(A and B) = P(B) x P(A | B) = P(A) x P(B | A).

This is the natural way to compute the probability of a sequence of events, one after another.

With and without replacement

Worked example. An urn has 5 red and 3 blue balls (8 total). Draw two without replacement. Find the probability both are red. The first draw is red with probability 5/8. Given the first was red, only 4 red remain out of 7, so the second is red with probability 4/7. By the multiplication rule,

P(both red) = (5/8) x (4/7) = 20/56 = 5/14, about 0.357.

If we had drawn with replacement (returning the first ball), the draws would not affect each other and the answer would be (5/8) x (5/8) = 25/64 = 0.391. The difference is exactly the difference between conditional and unconditional second-draw probabilities.

Sometimes knowing that B happened tells you nothing about A. Then A and B are independent. Independence is one of the most useful ideas in probability because it lets us multiply probabilities freely.

The definition

Two events A and B are independent if

P(A and B) = P(A) x P(B).

Equivalently, and often more intuitively, A and B are independent when P(A | B) = P(A): conditioning on B does not change the probability of A. If this equation fails, the events are dependent.

Worked example: coins are independent

Flip a fair coin twice. Let A be "heads on the first flip" and B be "heads on the second flip." Each has probability 1/2, and the outcome HH has probability 1/4. Since P(A and B) = 1/4 = (1/2)(1/2) = P(A)P(B), the flips are independent, matching our intuition that the coin has no memory. So P(two heads) = 1/2 x 1/2 = 1/4.

Worked example: testing for independence

Return to the 200-student table (30 music-and-sport, 50 music, 120 sport). We have P(music) = 50/200 = 0.25, P(sport) = 120/200 = 0.60, and P(music and sport) = 30/200 = 0.15. The product P(music) x P(sport) = 0.25 x 0.60 = 0.15 equals the joint probability, so in this dataset studying music and playing a sport are independent. Equivalently, P(sport | music) = 0.60 = P(sport), confirming it.

Independence is not mutual exclusivity

These two ideas are often confused, but they are nearly opposites:

• Mutually exclusive events cannot both happen: P(A and B) = 0.
• Independent events have P(A and B) = P(A) P(B).

If A and B each have positive probability and are mutually exclusive, then learning B occurred tells you A definitely did not, which is a huge amount of information, so they are strongly dependent, not independent. In short, two events with nonzero probability cannot be both mutually exclusive and independent.

Independence of several events and the complement trick

Events are mutually independent if the multiplication rule holds for every subset of them. A common use is "at least one." Suppose a component works with probability 0.9, and three components fail or work independently. The probability all three work is 0.9 x 0.9 x 0.9 = 0.729. The probability at least one fails is easiest by complement: 1 - 0.729 = 0.271.

Two closely linked results let us combine and then reverse conditional probabilities. They are the backbone of statistical reasoning, machine learning, and everyday diagnosis.

The law of total probability

Suppose the sample space is split into disjoint cases that cover everything, for example B and "not B." Then the probability of any event A is the weighted average of its conditional probabilities across those cases:

P(A) = P(A | B) P(B) + P(A | not B) P(not B).

This is how you find an overall probability when it is easier to reason case by case.

Bayes' theorem

Bayes' theorem reverses a conditional probability, turning P(A | B) into P(B | A). Starting from the two forms of the multiplication rule, P(A and B) = P(A | B) P(B) = P(B | A) P(A), solve for the one you want:

P(B | A) = P(A | B) P(B) / P(A),

where the denominator P(A) is usually computed with the law of total probability. Here P(B) is the prior (what we believed before the evidence A), and P(B | A) is the posterior (our updated belief after seeing A).

Worked example: a medical test

A disease affects 1% of a population, so P(D) = 0.01. A test has sensitivity 99% (it is positive for 99% of people who have the disease, so P(+ | D) = 0.99) and specificity 95% (it is negative for 95% of healthy people, so P(+ | not D) = 0.05). A random person tests positive. What is the probability they actually have the disease, P(D | +)?

Step 1, total probability of a positive test.

P(+) = P(+ | D) P(D) + P(+ | not D) P(not D) = (0.99)(0.01) + (0.05)(0.99) = 0.0099 + 0.0495 = 0.0594.

Step 2, Bayes' theorem.

P(D | +) = P(+ | D) P(D) / P(+) = 0.0099 / 0.0594 = 0.1667, that is about 1/6, only 16.7%.

This famous result surprises almost everyone: even after a positive result from a fairly accurate test, the person probably does not have the disease. The reason is the low base rate. Because 99% of people are healthy, the many false positives from that large group (about 4.95% of everyone) swamp the relatively few true positives (0.99% of everyone). A tree diagram makes the two positive paths visible.

The two positive branches total 0.0594, and the true-positive branch is 0.0099 of that, giving the 16.7% posterior. Bayes' theorem is simply careful bookkeeping of these branches.

A random variable assigns a number to each outcome of an experiment, so we can do arithmetic with chance. We write random variables with capital letters like X, and particular values with lowercase x.

Discrete random variables and the pmf

A discrete random variable takes separate, countable values (like a count). Its behavior is captured by a probability mass function (pmf), p(x) = P(X = x), which must satisfy two conditions: every p(x) is between 0 and 1, and the values sum to 1.

For a single fair die, X = the number rolled, and p(x) = 1/6 for each x in 1 through 6.

Expected value

The expected value (or mean) of X, written E[X] or the Greek letter mu, is the long-run average value, computed as a probability-weighted sum:

E[X] = sum of x times p(x), over all values x.

Worked example. For the fair die, E[X] = 1(1/6) + 2(1/6) + ... + 6(1/6) = (1+2+3+4+5+6)/6 = 21/6 = 3.5. The expected value need not be a possible outcome; it is the balance point of the distribution.

Variance and standard deviation

The variance measures spread around the mean. It is the expected squared distance from the mean:

Var(X) = E[(X - mu)^2] = E[X^2] - (E[X])^2.

The second form (the "computing formula") is usually easier. The standard deviation is the square root of the variance and carries the same units as X.

Worked example. For the fair die, first find E[X^2] = (1 + 4 + 9 + 16 + 25 + 36)/6 = 91/6 = 15.1667. Then Var(X) = 15.1667 - (3.5)^2 = 15.1667 - 12.25 = 2.9167 (exactly 35/12). The standard deviation is square root of 2.9167 = 1.708.

Linearity and scaling rules

Two rules save enormous effort. For constants a and b:

• Expectation is linear: E[aX + b] = a E[X] + b. (This holds for sums too: E[X + Y] = E[X] + E[Y] always, even if X and Y are dependent.)
• Variance scaling: Var(aX + b) = a^2 Var(X). Adding a constant b shifts the data but does not change its spread, so b drops out; multiplying by a stretches spread by a factor of a-squared.

Worked example. Suppose X has mean 5 and variance 4, and we define Y = 2X + 3. Then E[Y] = 2(5) + 3 = 13, and Var(Y) = 2^2 (4) = 16, so the standard deviation of Y is 4. These rules will let us read off the mean and variance of the named distributions in the next lessons without re-deriving them each time.

Many experiments reduce to a single yes-or-no question. Repeating that question a fixed number of times leads to the most important discrete distribution in statistics, the binomial.

The Bernoulli distribution

A Bernoulli trial is a single experiment with exactly two outcomes, called success (X = 1) and failure (X = 0), where success has probability p. Its pmf is P(X = 1) = p and P(X = 0) = 1 - p. A quick calculation gives its mean and variance:

• E[X] = 1(p) + 0(1 - p) = p.
• Var(X) = E[X^2] - p^2 = p - p^2 = p(1 - p).

For a fair coin (success = heads, p = 0.5), the mean is 0.5 and the variance is 0.25. For p = 0.6, the mean is 0.6 and the variance is 0.6 x 0.4 = 0.24.

The binomial setting

A binomial random variable X counts the number of successes in n Bernoulli trials. Four conditions must hold (the "BINS" checklist):

1. Binary: each trial is a success or a failure.
2. Independent: trials do not affect one another.
3. Number fixed: the number of trials n is set in advance.
4. Same probability: the success probability p is the same on every trial.

The binomial formula

The probability of exactly k successes in n trials is

P(X = k) = C(n, k) p^k (1 - p)^(n - k).

The combination C(n, k) counts the number of orderings of k successes among n trials, while p^k (1 - p)^(n-k) is the probability of any one such ordering.

Worked example. Flip a fair coin 5 times. What is the probability of exactly 3 heads? Here n = 5, k = 3, p = 0.5:

P(X = 3) = C(5, 3) (0.5)^3 (0.5)^2 = 10 x (0.5)^5 = 10/32 = 0.3125.

Worked example with p not equal to 0.5. A quiz has 10 questions, each answered correctly with probability 0.3, independently. The probability of exactly 2 correct is

P(X = 2) = C(10, 2) (0.3)^2 (0.7)^8 = 45 x 0.09 x 0.05765 = 0.2335.

Mean and variance of a binomial

Because a binomial X is the sum of n independent Bernoulli variables, its mean and variance are n times the Bernoulli values:

E[X] = np and Var(X) = np(1 - p).

For the 10-question quiz, E[X] = 10 x 0.3 = 3 correct answers on average, with variance 10 x 0.3 x 0.7 = 2.1 and standard deviation about 1.449. The mean np is the single most useful fact about a binomial: it is your best guess for the number of successes.

Two more discrete distributions round out the essential toolkit: the Poisson, for counting events in a span of time or space, and the geometric, for waiting until the first success.

The Poisson distribution

The Poisson distribution models the number of events that occur in a fixed interval when events happen independently at a constant average rate lambda (the Greek letter, its mean number of events). Examples include calls arriving at a help desk per hour, typos per page, or decays per second. Its pmf, for k = 0, 1, 2, ..., is

P(X = k) = e^(-lambda) lambda^k / k!,

where e is about 2.71828. A defining feature is that the Poisson mean and variance are equal:

E[X] = lambda and Var(X) = lambda.

Worked example. A call center receives on average 3 calls per minute, so lambda = 3. Find the probability of exactly 2 calls in a given minute:

P(X = 2) = e^(-3) 3^2 / 2! = e^(-3) x 9 / 2 = 0.049787 x 4.5 = 0.2240.

The probability of no calls is P(X = 0) = e^(-3) = 0.0498, and the probability of at most one call is P(X = 0) + P(X = 1) = e^(-3)(1 + 3) = 0.0498 x 4 = 0.1991.

The geometric distribution

The geometric distribution models the number of independent Bernoulli trials (each with success probability p) needed to get the first success. The random variable X takes values 1, 2, 3, ... (the trial on which the first success lands). To get the first success on trial k, the first k - 1 trials must fail and the kth must succeed, so

P(X = k) = (1 - p)^(k - 1) p.

Its mean and variance are

E[X] = 1 / p and Var(X) = (1 - p) / p^2.

The mean 1/p is intuitive: if each trial succeeds with probability p, you expect to wait about 1/p trials for the first success.

Worked example. Each attempt to start an old lawnmower succeeds with probability p = 0.2, independently. On average it takes 1/0.2 = 5 pulls. The probability the first success is on the third pull is

P(X = 3) = (0.8)^2 (0.2) = 0.64 x 0.2 = 0.128.

The probability it takes more than 3 pulls is the probability the first 3 all fail: P(X greater than 3) = (0.8)^3 = 0.512. This clean "all failures so far" form is the geometric distribution's memoryless property: given no success yet, the future looks the same as a fresh start.

Choosing the right model

• Binomial: fixed number of trials n, count the successes.
• Poisson: no fixed n, count events over an interval at a known rate.
• Geometric: keep trying until the first success, count the trials.

Some quantities, like waiting times, heights, or temperatures, vary continuously and can take any value in an interval. For these we replace the probability mass function with a probability density function.

Density and area

A continuous random variable X is described by a probability density function (pdf) f(x). Unlike a pmf, f(x) is not itself a probability; instead, probability is the area under the curve. The probability that X falls between a and b equals the area beneath f(x) from a to b. Two rules make a valid density:

• f(x) is greater than or equal to 0 everywhere (no negative density).
• The total area under f(x) equals 1.

Because a single point has zero width, it encloses zero area, so P(X = c) = 0 for any exact value c. Only intervals have positive probability. A useful consequence: for continuous variables, P(X is less than c) and P(X is less than or equal to c) are equal, since the endpoint contributes nothing.

The uniform distribution

The uniform distribution on an interval from a to b spreads probability evenly, so its density is a flat, constant height. To make the total area (a rectangle) equal 1, the height must be

f(x) = 1 / (b - a) for x between a and b, and 0 otherwise.

Its mean sits at the midpoint and its variance follows a standard formula:

E[X] = (a + b) / 2 and Var(X) = (b - a)^2 / 12.

Worked example

A bus arrives at a uniformly random time between 2 and 10 minutes after you reach the stop, so a = 2 and b = 10. The density height is 1 / (10 - 2) = 1/8 = 0.125. Probabilities are just rectangle areas (width times height):

• P(X is less than 5) = (5 - 2) x (1/8) = 3/8 = 0.375.
• P(4 is less than X is less than 7) = (7 - 4) x (1/8) = 3/8 = 0.375.

The mean wait is (2 + 10)/2 = 6 minutes, and the variance is (10 - 2)^2 / 12 = 64/12 = 5.333, so the standard deviation is about 2.31 minutes.

The uniform distribution is the continuous cousin of "equally likely" outcomes: every equal-width slice of the interval is equally probable, and probability is always found the same way, as area.

The exponential distribution models the waiting time until an event when events occur at a constant average rate. It is the continuous partner of the Poisson: if events per hour follow a Poisson with rate lambda, the time between consecutive events is exponential with the same rate lambda.

Density and rate

For a rate lambda greater than 0, the exponential density for x greater than or equal to 0 is

f(x) = lambda e^(-lambda x).

The density starts at its highest value lambda at x = 0 and decays toward 0, so short waits are more likely than long ones. Its mean and variance are

E[X] = 1 / lambda and Var(X) = 1 / lambda^2.

The mean 1/lambda is the average time between events: if a bus comes at rate 2 per hour (lambda = 2), you wait 1/2 hour on average.

The survival function

Tail probabilities have a beautifully simple closed form. The probability of waiting longer than a time t is the survival function

P(X greater than t) = e^(-lambda t),

and therefore the probability of waiting at most t is P(X less than or equal to t) = 1 - e^(-lambda t), which is the cumulative distribution function.

Worked example

Suppose the time (in hours) until the next customer arrives is exponential with rate lambda = 0.5 per hour, so the mean wait is 1 / 0.5 = 2 hours. Then:

• P(wait more than 3 hours) = e^(-0.5 x 3) = e^(-1.5) = 0.2231.
• P(wait at most 2 hours) = 1 - e^(-0.5 x 2) = 1 - e^(-1) = 1 - 0.3679 = 0.6321.

The memoryless property

Like the geometric distribution, the exponential is memoryless: P(X greater than s + t given X greater than s) = P(X greater than t). If a machine has already run 5 hours without failing, the chance it lasts 3 more hours is the same as a brand-new machine lasting 3 hours. The exponential does not "age," which makes it the standard model for the lifetime of components with a constant failure rate and for the gaps between random arrivals.

The normal (Gaussian) distribution is the famous bell curve and the most important continuous distribution in all of statistics. It models heights, measurement errors, test scores, and, crucially, the behavior of sums and averages (as the next module shows).

Shape and parameters

A normal distribution is symmetric and bell-shaped, fully described by two numbers: its mean mu (the center, where the peak sits) and its standard deviation sigma (the spread). The mean, median, and mode all coincide at mu. Changing mu slides the curve left or right; changing sigma makes it wider and flatter or narrower and taller, but the total area always stays 1.

The standard normal and z-scores

The standard normal distribution Z has mean 0 and standard deviation 1. Any normal value x can be converted to a z-score that says how many standard deviations it lies from the mean:

z = (x - mu) / sigma.

Standardizing lets a single table (or the empirical rule) handle every normal distribution at once.

The 68-95-99.7 empirical rule

For any normal distribution:

• About 68% of values lie within 1 standard deviation of the mean.
• About 95% lie within 2 standard deviations.
• About 99.7% lie within 3 standard deviations.

Worked example: the empirical rule

IQ scores are modeled as normal with mu = 100 and sigma = 15. About 68% of people score between 100 plus or minus 15, that is 85 to 115. About 95% score between 100 plus or minus 30, that is 70 to 130. A score of 115 has z = (115 - 100)/15 = 1.0, exactly one standard deviation above the mean.

Worked example: a tail probability

What fraction of people score above 115 (z = 1)? Since 68% lie within one standard deviation, 32% lie outside it, split evenly between the two tails by symmetry, so about 16% lie above 115. The exact standard-normal value is P(Z greater than 1) = 0.1587, close to the empirical-rule estimate of 16%. To find the probability of scoring between 85 and 115 (within 1 SD), the empirical rule gives about 68%; the exact value is 0.6827. Standardizing plus symmetry lets you answer any normal probability question with a small set of known values.

Often two random variables are observed together, like a person's height and weight, and we want to know how they move as a pair. This calls for a joint distribution.

Joint and marginal distributions

For two discrete variables X and Y, the joint probability mass function p(x, y) = P(X = x and Y = y) gives the probability of each pair, and all the entries sum to 1. The marginal distribution of X is recovered by summing over Y (row totals), and the marginal of Y by summing over X (column totals). Consider this joint table:

The margins read off the edges: X takes values 0, 1, 2 with probabilities 0.30, 0.40, 0.30, and Y takes 0 or 1 with probabilities 0.40 and 0.60.

Covariance

Covariance measures whether X and Y tend to rise and fall together. It is defined as

Cov(X, Y) = E[XY] - E[X] E[Y].

A positive covariance means that above-average X tends to come with above-average Y; a negative covariance means they move in opposite directions; zero means no linear relationship.

Worked example. From the margins, E[X] = 0(0.30) + 1(0.40) + 2(0.30) = 1.0 and E[Y] = 0(0.40) + 1(0.60) = 0.6. For E[XY], only pairs with both X and Y nonzero contribute: the term X=1,Y=1 gives 1 x 1 x 0.30 = 0.30 and X=2,Y=1 gives 2 x 1 x 0.20 = 0.40, so E[XY] = 0.70. Therefore

Cov(X, Y) = 0.70 - (1.0)(0.6) = 0.70 - 0.60 = 0.10.

The positive value says X and Y tend to increase together.

Correlation

Covariance depends on the units of X and Y, so it is hard to interpret on its own. Dividing by both standard deviations gives the unitless correlation coefficient, always between -1 and 1:

rho = Cov(X, Y) / (sigma_X sigma_Y).

Values near +1 or -1 indicate a strong linear relationship; values near 0 indicate a weak one. For the table, Var(X) = E[X^2] - 1^2 = (0 + 0.40 + 1.20) - 1 = 0.60 and Var(Y) = 0.60 - 0.36 = 0.24, so sigma_X = 0.775 and sigma_Y = 0.490. Then rho = 0.10 / (0.775 x 0.490) = 0.264, a weak positive association.

Independence and covariance

If X and Y are independent, then E[XY] = E[X]E[Y], so their covariance (and correlation) is 0. The converse is not guaranteed: zero covariance means no linear relationship, but a nonlinear dependence can still exist. Independence implies zero covariance, not the other way around.

Before the Central Limit Theorem, we need the rules for the mean and variance of a sum of random variables. These rules are the engine behind averages, totals, and the CLT.

Expectation of a sum

Expectation is always linear, with no conditions at all:

E[X + Y] = E[X] + E[Y], and more generally E[X1 + X2 + ... + Xn] = E[X1] + ... + E[Xn].

This holds whether or not the variables are independent. For example, the expected total of two dice is 3.5 + 3.5 = 7.

Variance of a sum

Variance is trickier. In general,

Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X, Y).

When X and Y are independent, the covariance is 0 and the rule simplifies to

Var(X + Y) = Var(X) + Var(Y).

A subtle but important point: for independent variables, variance also adds for a difference, Var(X - Y) = Var(X) + Var(Y), because the negative sign squares away. Variances add; standard deviations do not.

Worked example. Let X and Y be independent with Var(X) = 4 and Var(Y) = 9. Then Var(X + Y) = 4 + 9 = 13, so the standard deviation of the sum is square root of 13 = 3.61, not 2 + 3 = 5.

The sample mean

Take n independent observations from a population with mean mu and standard deviation sigma. Their sample mean is X-bar = (X1 + X2 + ... + Xn) / n. Using the rules above:

• Mean of the sample mean: E[X-bar] = mu. The sample mean is centered on the true mean.
• Variance of the sample mean: Var(X-bar) = sigma^2 / n, so its standard deviation is sigma / (square root of n). This is the standard error.

The standard error shrinks as n grows: averaging cancels out noise, so larger samples give more precise estimates. Quadrupling n halves the standard error, because of the square root.

Worked example. A population has sigma = 15. For samples of size n = 25, the standard error of the mean is 15 / (square root of 25) = 15 / 5 = 3. For n = 100 it drops to 15 / 10 = 1.5. These facts about X-bar are exactly what the Central Limit Theorem builds on in the final lesson.

The Central Limit Theorem (CLT) is the crown jewel of probability theory and the reason the normal curve appears everywhere. It explains why so many real-world quantities are approximately normal even when the underlying process is not.

Statement of the theorem

Take a random sample of n independent observations from any population with mean mu and finite standard deviation sigma. The CLT says that as n grows, the distribution of the sample mean X-bar becomes approximately normal, with

mean = mu and standard deviation (standard error) = sigma / (square root of n),

regardless of the shape of the original population. The same is true of the sum of the observations. The astonishing phrase is "regardless of shape": even a skewed, bimodal, or otherwise lumpy population yields a bell-shaped distribution of averages once n is large enough.

How large is large enough?

A common rule of thumb is n at least 30. If the population is already fairly symmetric, smaller samples work well; if it is strongly skewed, larger samples are needed. When the population itself is normal, X-bar is exactly normal for every n.

Why it matters

Because X-bar is approximately normal, we can standardize it and use the normal model to compute probabilities about averages:

z = (X-bar - mu) / (sigma / square root of n).

This single idea underlies confidence intervals and hypothesis tests throughout statistics.

Worked example

A population has mean mu = 100 and standard deviation sigma = 15. You take a random sample of n = 25. What is the probability the sample mean exceeds 106?

Step 1, standard error: sigma / square root of n = 15 / 5 = 3.

Step 2, standardize: z = (106 - 100) / 3 = 6 / 3 = 2.0.

Step 3, use the normal tail: P(Z greater than 2) = 0.0228, about 2.3%. (The empirical-rule estimate is 2.5%, since about 95% lies within 2 SD.) Notice we divided by the standard error, not by sigma, because the question is about an average of 25 values, not a single observation. The CLT is what licenses using the bell curve here even without knowing the population's shape.