Introduction to Data Science

A team spends six weeks building a churn model. It scores 94% accuracy. It is deployed, and within a month everyone quietly stops looking at it. The model was fine. The problem was that nobody had asked what decision it was meant to inform, and the data it learned from was collected in a way that made the answer meaningless. Almost every failed data project fails somewhere other than the modeling step, which is why this course spends most of its time everywhere else.

The big picture

Data science is how a messy pile of numbers becomes a decision someone can act on. Before touching any tool, it helps to see the whole journey a project takes, because most of the value (and most of the mistakes) come from the early, unglamorous steps, not the modeling. Think of it like cooking a meal: the finished plate gets the applause, but the shopping, washing, and chopping are where the work really is.

Data science is the practice of turning data into understanding and better decisions. It sits at the meeting point of three things: statistics (reasoning under uncertainty), computing (getting a machine to do the work at scale), and domain knowledge (knowing what the numbers mean and which questions matter). A result that is statistically clean but answers the wrong question helps no one, which is why all three matter together.

Key idea: Data science is a team sport between math, code, and subject expertise, aimed at a real decision.

The pipeline

Almost every project moves through the same stages, and much of the effort lands early. A rough but honest saying in the field is that data scientists spend most of their time getting data into shape, not building fancy models. A handy way to remember the flow is ask, get, clean, explore, model, communicate, the same way a recipe runs from "decide what to cook" to "serve it."

1. Ask a question. State clearly what you want to learn or predict. "Do longer support calls get worse ratings?" is answerable; "make support better" is not yet.
2. Collect / obtain data. Pull from files, databases, sensors, surveys, or the web. Note where it came from and how it was measured.
3. Clean and prepare (wrangling). Fix wrong types, handle missing values, remove duplicates, and reshape the table so each row is one observation.
4. Explore (EDA). Summarize and plot the data to learn its shape, spot errors, and form hypotheses.
5. Model. Fit a statistical or machine-learning model to describe a relationship or make predictions.
6. Evaluate. Check honestly how well the model works on data it has not seen.
7. Communicate. Report the finding in plain language with an honest picture of its limits, so a decision can be made.

The arrows are not strictly one-way. Exploring often sends you back to clean more; a weak evaluation sends you back to model differently. Good practice is to loop, not march.

Key idea: The pipeline is a loop from a clear question to a clear message, and the early data-wrangling steps usually cost the most time.

Sharpening the question

Stage one is the one people skip, and it is the one that decides whether anything downstream is worth doing. A vague ask has to be turned into something a number could answer. Watch a real request get sharpened:

vague:       "Does tutoring help?"

sharper:     "Do students who attend tutoring pass at a higher rate?"
                still ambiguous: pass what? which students? over what period?

answerable:  "Among students enrolled in DATA 200 in spring 2026, is the
              pass rate higher for those who attended at least one tutoring
              session than for those who attended none?"
                population: DATA 200, spring 2026
                measure:    pass rate (fraction scoring 60 or above)
                comparison: attended at least once vs never
                decision:   whether to fund tutoring next term

Four things had to be pinned down: the population you are talking about, the measure you will compute, the comparison that makes it meaningful, and the decision it will inform. If you cannot name all four, you are not ready to touch data. Notice that the sharp version also tells you exactly which columns you need, which makes step two concrete instead of a fishing expedition.

Key idea: A usable question names the population, the measure, the comparison, and the decision it will inform.

The dataset we will use all course

To keep every idea concrete, one small dataset runs through the whole course. It is a record of a tutoring study in a single course, saved as study.csv:

student_id    identifier for one student
hours         hours spent studying for the exam
score         exam score out of 100
program       the student's degree program (Arts or Science)
attended      whether they attended tutoring (yes or no)
submitted_on  the date the exam was submitted

You will load it in Lesson 3, clean it in Lesson 4 (it arrives with a duplicate row, an impossible score, a number stored as text, and two inconsistent category spellings), explore it in Lesson 5, summarize it in Lesson 6, correlate it in Lesson 8, fit a regression to it in Lesson 9, and classify with it in Lesson 10. Following one dataset the whole way is the only honest way to see how much each earlier decision constrains what comes later.

Key idea: One small dataset, study.csv, carries through every stage of the course so you can see how each step depends on the last.

Making the work reproducible

One habit belongs at the very start rather than the end. An analysis is reproducible if someone else, given your files, can rerun it and get exactly your numbers. That someone is usually you, three months later, and it is a much harder standard than "it worked on my laptop". Four practices cover most of it:

• Never edit the raw data. Keep data/raw/study.csv read-only and write every cleaned version to data/processed/. If a cleaning step was wrong, you can go back.
• Put every step in code. A cleaning decision made by hand in a spreadsheet is invisible and unrepeatable. The same decision as one line of pandas is documented forever.
• Fix the random seed anywhere randomness enters, such as train_test_split(..., random_state=0), so the same split happens every run.
• Record where the data came from and when. A one-paragraph note on the source, the collection date, and any known limitations costs five minutes and saves whole projects.

None of this is glamorous, and all of it is the difference between a result you can defend and one you merely remember getting.

Key idea: Keep raw data untouched, express every step as code, fix random seeds, and record provenance, so your numbers can be reproduced later.

A quick example

Suppose a store wants to know which products to feature. The question is "which products drive the most repeat purchases?" You obtain the sales log, clean it by removing test orders and fixing dates stored as text, explore by plotting repeat-purchase rates per product, model the relationship between category and repeat rate, evaluate whether the pattern holds on last month's data, and communicate a short ranked list to the manager. Notice that only two of the seven steps are the "modeling" people imagine data science to be.

Here is the same journey in a compact table so you can match each stage to a concrete action.

Stage	What you actually do
Ask	Write the question so a yes/no or a number could answer it
Get	Download the sales log and record its source
Clean	Drop test orders, fix text dates into real dates
Explore	Plot repeat-purchase rate per product
Model	Relate category to repeat rate
Evaluate	Check the pattern on last month's untouched data
Communicate	Send a ranked shortlist to the manager

Key idea: Every project, however fancy, is just these seven concrete moves in order.

Descriptive versus predictive

Two broad goals run through the course. Descriptive work summarizes what happened (averages, trends, relationships in data you already have). Predictive work uses patterns to forecast new, unseen cases (will this email be spam?). Both rest on the same pipeline; they mainly differ at the modeling and evaluation stages. A useful test: if the answer is about the past you already recorded, it is descriptive; if it is a guess about a case you have not seen yet, it is predictive.

There is a third kind of question that people constantly mistake for the first two, and it is the hardest of all: the causal question. "Does tutoring cause higher scores?" is not answered by describing that tutees score higher, nor by predicting scores accurately. Prediction only needs a reliable association; causation needs to rule out every other explanation for it. The three sit in a strict order of difficulty:

question type   asks                                  what it takes
-------------   -----------------------------------   -------------------------------
descriptive     what happened?                        clean data, honest summaries
predictive      what will happen for a new case?      a model plus held-out testing
causal          what happens if we intervene?         a randomized experiment, or
                                                      very careful design and
                                                      assumptions you can defend

Most damage in applied data science comes from answering a descriptive question and reporting it as if it were causal. Knowing which of the three you are actually doing is the single most useful habit in this course.

Key idea: Describing the past and predicting the unseen use the same pipeline but part ways at the modeling step, and neither one answers a causal question.

Where people get stuck

• "Data science is mostly building models." In reality, obtaining and cleaning data usually take the most time; modeling is a small slice.
• "The pipeline runs once, top to bottom." It loops. Exploration sends you back to clean; a bad evaluation sends you back to model.
• "You can skip the question and let the data speak." Without a clear question you cannot tell a useful finding from a coincidence, and you will not know when you are done.
• "Descriptive and predictive are totally different fields." They share the same steps; they differ mainly in whether you forecast unseen cases.
• Starting from the data instead of the decision. "What can we learn from this file?" produces interesting trivia. "What decision needs making, and what would change it?" produces work someone uses.
• Editing the raw file. Once you overwrite the original, every cleaning decision becomes permanent and invisible. Write cleaned copies to a separate folder.
• Cleaning by hand in a spreadsheet. It feels faster and it destroys reproducibility, because nothing records what you changed or why.
• Treating provenance as paperwork. How and when data was collected determines what conclusions it can support. A survey of volunteers cannot tell you about people who did not volunteer, no matter how large it is.

Recap

• Data science blends statistics, computing, and domain knowledge to support a decision.
• The pipeline is ask, get, clean, explore, model, evaluate, communicate.
• A usable question names its population, measure, comparison, and decision.
• Cleaning and preparing data typically consume the most effort.
• The process is iterative: weak results loop you back to earlier stages.
• Descriptive work summarizes the past; predictive work forecasts the unseen.
• Reproducibility means untouched raw data, every step in code, fixed random seeds, and recorded provenance.

Sources

1. Wickham, H., Cetinkaya-Rundel, M., & Grolemund, G. (2023). Introduction and the data science workflow. In R for data science (2nd ed.). O'Reilly. r4ds.hadley.nz
2. Diez, D., Cetinkaya-Rundel, M., & Barr, C. D. (2019). Introduction to data. In OpenIntro statistics (4th ed.). OpenIntro. openintro.org
3. VanderPlas, J. (2016). Preface and introduction. In Python data science handbook. O'Reilly. jakevdp.github.io
4. Cetinkaya-Rundel, M., & Hardin, J. (2021). Data and study design. In Introduction to modern statistics. OpenIntro. openintro.org
5. Wickham, H. (2014). Tidy data. Journal of Statistical Software, 59(10), 1-23. vita.had.co.nz
6. pandas development team. (n.d.). 10 minutes to pandas. pandas documentation. pandas.pydata.org
7. scikit-learn developers. (n.d.). Common pitfalls and recommended practices: randomness and reproducibility. scikit-learn user guide. scikit-learn.org

Two Python libraries do almost all the heavy lifting in data science, and both of them are built on a single idea: stop writing loops. Once you stop thinking "for each row" and start thinking "the whole column at once", pandas code becomes short, fast, and much harder to get wrong. This lesson gets you to that switch, and shows the three places the switch surprises people.

The big picture

Almost all data work in Python leans on two libraries, and you only need to read and lightly tweak their code, not write it from scratch. Knowing what an array, a Series, and a DataFrame are makes every later lesson (cleaning, plotting, modeling) click into place. Think of these tools as the containers your data lives in: pick the right container and everything after is easier.

Data science in Python rests on two libraries. numpy provides fast numeric arrays, and pandas builds a labeled table on top of them. You do not need to be an expert programmer, but you do need to read and lightly edit code like the snippets below.

Key idea: numpy is the fast number engine; pandas is the labeled spreadsheet built on top of it.

numpy: the array

A numpy array (called ndarray) holds many numbers of the same type in one compact block of memory. Operations apply to the whole array at once, a style called vectorization. Picture a vending machine that serves a whole row of snacks in one pull instead of one snack per pull: that batch action is both shorter to write and much faster than looping in plain Python.

import numpy as np

a = np.array([2, 4, 6, 8])
a * 10          # array([20, 40, 60, 80])  - whole array at once
a.mean()        # 5.0
a[a > 4]        # array([6, 8])  - boolean filtering

The last line shows boolean indexing: a > 4 produces an array of True/False, and passing it back in keeps only the True positions. Think of it as a row of light switches, one per value; you keep only the values whose switch is on. This one pattern powers most data filtering in pandas too.

Worked example. For a = np.array([2, 4, 6, 8]), the mask a > 4 is [False, False, True, True], so a[a > 4] returns array([6, 8]). The mean is (2 + 4 + 6 + 8) / 4 = 20 / 4 = 5.0. Same idea, whole array at once.

Key idea: Vectorized math and boolean masks let you transform and filter a whole array in one line.

One type per array, and why that matters

The word "same type" in the definition is doing real work. Every array carries a single dtype, and that fixed type is exactly what makes the array fast:

a = np.array([1, 2, 3])
a.dtype        # dtype('int64')  - every element is an 8-byte integer
a.nbytes       # 24              - 3 values times 8 bytes, one contiguous block

a[0] = 3.9     # no error, no warning
a              # array([3, 2, 3])   <- 3.9 was truncated to fit the integer type

A Python list of the same three numbers is scattered pointers to three separate objects, each of which must be examined at runtime to find out what it is. The array is one solid block of identical, known-size values, so the loop can run in compiled C with no per-element type checking. That is the entire performance story, and it is why on a million values a vectorized operation typically runs tens of times faster than the equivalent Python loop.

The cost of that speed is the silent truncation above. Assigning 3.9 into an integer array does not promote the array to floats; it quietly drops the fraction. If a column might hold decimals, create it as a float array from the start.

Key idea: One fixed dtype per array is what makes numpy fast, and it is also why assigning a float into an integer array silently truncates.

Broadcasting

Broadcasting is the rule that lets arrays of different shapes combine without you writing a loop. The simplest case is a scalar, which is stretched to every element:

prices = np.array([10.0, 20.0, 30.0])
prices * 1.08          # array([10.8, 21.6, 32.4])  - the 1.08 reaches every element

The same idea works between arrays. A one-dimensional array is stretched across the rows of a two-dimensional one:

grid = np.array([[1, 2, 3],
                 [4, 5, 6]])

grid + np.array([10, 20, 30])
# array([[11, 22, 33],
#        [14, 25, 36]])

The rule is mechanical: line the shapes up from the right, and each pair of dimensions must be either equal or one. Here (2, 3) against (3,) matches on the last axis, so the small array is reused for both rows. When the shapes do not line up you get a clear error rather than a wrong answer:

grid + np.array([10, 20])
ValueError: operands could not be broadcast together with shapes (2,3) (2,)

Key idea: Broadcasting stretches smaller shapes to fit larger ones by matching dimensions from the right, each of which must be equal or one.

pandas: Series and DataFrame

A Series is a one-dimensional labeled array, like a single column with an index. A DataFrame is a two-dimensional table: a set of Series that share the same row index. Think of a DataFrame as a spreadsheet where every column has a name and a data type, and a Series as one column pulled out of that sheet.

Structure	Shape	Everyday analogy
Series	1 dimension	one labeled column
DataFrame	2 dimensions	a whole spreadsheet

Key idea: A Series is one labeled column; a DataFrame is many aligned columns sharing one row index.

Reading a DataFrame

import pandas as pd

df = pd.DataFrame({
    "name": ["Ana", "Ben", "Cara"],
    "age":  [25, 31, 29],
    "city": ["Reno", "Reno", "Provo"]
})

df["age"]              # one column (a Series)
df[["name", "age"]]    # two columns (a smaller DataFrame)
df[df["age"] > 28]     # rows where age exceeds 28 (Ben and Cara)

Two ways to grab rows and columns by label or position are worth knowing.

df.loc[rows, cols] selects by label, while df.iloc[rows, cols] selects by integer position. For example, df.loc[0, "name"] is "Ana", and df.iloc[0, 0] is also "Ana" because that cell is at row 0, column 0. A memory hook: the i in iloc stands for integer position.

The distinction stops being cosmetic the moment the index is not 0, 1, 2. Give the same rows the labels 10, 11, 12:

df = pd.DataFrame({"name": ["Ana", "Ben", "Cara"], "age": [25, 31, 29]},
                  index=[10, 11, 12])

df.loc[10, "name"]     # 'Ana'   - label 10
df.iloc[0]["name"]     # 'Ana'   - position 0
df.loc[0, "name"]      # KeyError: 0   - there is no label 0

df.loc[10:11]          # rows 10 AND 11   - loc slices are INCLUSIVE of the end
df.iloc[0:2]           # rows at positions 0 and 1 - iloc excludes the end, like a list

That inclusive-versus-exclusive difference is the one that bites hardest, because both forms run without error and return different numbers of rows.

Key idea: Selecting one column gives a Series; selecting several gives a smaller DataFrame; loc uses labels and slices inclusively, while iloc uses positions and slices exclusively.

Why labels help: automatic alignment

Because columns have names, pandas code reads almost like English: df[df["city"] == "Reno"]["age"].mean() reads as "the mean age of the Reno rows." Labels also let pandas line up data automatically when you combine tables, which prevents a whole class of off-by-one bugs that plague raw arrays.

Alignment means arithmetic between two Series matches on the index label, not on position. Watch it rescue you from a bug:

a = pd.Series([1, 2, 3],    index=["x", "y", "z"])
b = pd.Series([10, 20, 30], index=["z", "y", "x"])   # note the reversed order

a + b
# x    31      1 + 30, because both are labeled x
# y    22      2 + 20
# z    13      3 + 10

With raw numpy arrays that same expression would have added 1 to 10 and given nonsense. pandas matched the labels instead, which is exactly what you wanted.

The same mechanism produces the surprise, though, when the labels do not fully overlap. pandas takes the union of the two indexes and fills the gaps with NaN:

c = pd.Series([1, 2, 3], index=["x", "y", "w"])   # w instead of z

a + c
# w    NaN     w is missing from a
# x    2.0
# y    4.0
# z    NaN     z is missing from c

Two NaNs appear where you may have expected numbers. When an arithmetic result contains unexplained NaNs, mismatched indexes are almost always the reason, and a.index.equals(c.index) is the one-line check.

Key idea: pandas aligns on index labels rather than position, which prevents off-by-one bugs and produces NaN wherever the two indexes do not overlap.

The one warning everyone meets

Sooner or later pandas will print a SettingWithCopyWarning, and it is worth understanding rather than silencing. It appears when you index twice in a row and then assign, because the first index may have produced a temporary copy rather than a view of the original:

# risky: two separate indexing steps, then an assignment
df[df["age"] > 28]["age"] = 0
# SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

# correct: one indexing operation with .loc, selecting rows and column together
df.loc[df["age"] > 28, "age"] = 0

The first form may silently change nothing at all, which is far worse than an error. The rule that avoids it permanently: whenever you assign, use a single .loc[rows, columns] call rather than stacking two bracket operations.

Key idea: Chained indexing followed by assignment may write to a temporary copy, so always assign through one .loc[rows, columns] call.

Where people get stuck

• "A numpy array is just a Python list." An array stores one fixed type compactly and does math on the whole thing at once; a list can mix types and needs a loop.
• "loc and iloc are interchangeable." loc selects by label and iloc by integer position; they can return different cells when the index is not 0, 1, 2, and so on.
• "Selecting one column returns a DataFrame." One column (single brackets) returns a Series; you need double brackets to get a one-column DataFrame.
• "You must loop to filter rows." A boolean mask like df[df["age"] > 28] filters without any loop, exactly like numpy boolean indexing.
• Forgetting that loc slices include the endpoint. df.loc[10:11] returns both rows while df.iloc[0:2] returns two rows starting at zero. Both run; only one is what you meant.
• Unexplained NaNs after arithmetic. Almost always mismatched index labels rather than missing data. Compare the two indexes before blaming the file.
• Assigning through chained brackets. df[mask]["col"] = value may write to a copy and change nothing. Use df.loc[mask, "col"] = value.
• Writing a float into an integer array. numpy truncates without warning, so 3.9 becomes 3. Build the array as float if decimals are possible.
• Mixing and with element-wise masks. Python's and raises a ValueError on arrays; combine masks with & and |, and parenthesize each condition.

Recap

• numpy arrays are compact, same-type, and support fast vectorized math and boolean masks.
• One fixed dtype per array is what makes numpy fast and why float-into-int assignment truncates.
• Broadcasting matches shapes from the right, stretching dimensions that are equal or one.
• A pandas Series is one labeled column; a DataFrame is a table of aligned Series.
• Single brackets on a column give a Series; double brackets give a DataFrame; a boolean mask filters rows without a loop.
• loc selects by label and slices inclusively; iloc selects by position and slices exclusively.
• pandas aligns arithmetic on index labels, filling non-overlapping labels with NaN.

Sources

1. NumPy developers. (n.d.). NumPy: the absolute basics for beginners. NumPy manual. numpy.org
2. NumPy developers. (n.d.). Broadcasting. NumPy manual. numpy.org
3. NumPy developers. (n.d.). Indexing on ndarrays: boolean and fancy indexing. NumPy manual. numpy.org
4. pandas development team. (n.d.). Intro to data structures: Series and DataFrame. pandas user guide. pandas.pydata.org
5. pandas development team. (n.d.). Indexing and selecting data: loc, iloc, and returning a view versus a copy. pandas user guide. pandas.pydata.org
6. VanderPlas, J. (2016). Introduction to NumPy. In Python data science handbook (ch. 2). O'Reilly. jakevdp.github.io
7. VanderPlas, J. (2016). Data manipulation with pandas. In Python data science handbook (ch. 3). O'Reilly. jakevdp.github.io

Every dataset arrives claiming to be fine. The five commands in this lesson take about ten seconds to run and routinely find a wrong data type, an impossible value, or a column that is half empty, all before you have plotted a single point. Skipping them is how people spend a week analyzing a file that never parsed correctly in the first place.

The big picture

Before you trust a dataset, you load it and give it a quick physical exam, the way a mechanic checks a used car before quoting a repair. A handful of one-line commands tell you how big the data is, what types the columns are, and whether anything is obviously broken. Skipping this look is how people end up analyzing a file that never parsed correctly.

The first real step of any project is getting data into a DataFrame and looking at it before you trust it. The most common source is a CSV (comma-separated values) file, a plain-text table where columns are separated by commas and rows by line breaks. A CSV is like a spreadsheet saved as raw text: every comma is a wall between two cells.

Key idea: Always inspect a dataset's size, columns, and types before analyzing it.

Reading a file

import pandas as pd

df = pd.read_csv("sales.csv")   # load the file into a DataFrame
df.head()      # first 5 rows - a quick eyeball check
df.shape       # (rows, columns), e.g. (1000, 6)
df.columns     # the column names
df.info()      # column names, non-null counts, and data types
df.describe()  # count, mean, std, min, quartiles, max for numeric columns

Running these five lines before anything else is a habit worth building. Let us run them on the course dataset and read every number.

study.csv, exactly as it arrives

Here is the whole raw file. Twelve data rows, deliberately imperfect, because real files are:

student_id,hours,score,program,attended,submitted_on
101,2,52,Science,yes,2026-05-01
102,3,63,Science,Yes,2026-05-01
103,4,66,Arts,no,2026-05-02
104,5,78,Arts,YES,2026-05-02
105,6,74,Science,no,2026-05-03
106,7,88,Science,yes,2026-05-03
107,8,85,Arts,No,2026-05-04
108,9,96,arts,yes,2026-05-04
109,3,999,Science,no,2026-05-05
110,,71,Science,yes,2026-05-05
103,4,66,Arts,no,2026-05-02
111,10 hrs,98,Arts,YES,2026-05-06

Load it and look:

>>> df = pd.read_csv("data/raw/study.csv")
>>> df.shape
(12, 6)

>>> df.dtypes
student_id       int64
hours           object      <- should be a number
score            int64
program         object
attended        object
submitted_on    object      <- should be a date
dtype: object

>>> df.info()
RangeIndex: 12 entries, 0 to 11
 #   Column        Non-Null Count  Dtype
---  ------        --------------  -----
 0   student_id    12 non-null     int64
 1   hours         11 non-null     object      <- one value missing
 2   score         12 non-null     int64
 3   program       12 non-null     object
 4   attended      12 non-null     object
 5   submitted_on  12 non-null     object

Two problems are already visible and neither raised an error. hours came in as object, which means text: one row wrote "10 hrs" and that single value forced the whole column to be strings. And hours has 11 non-null entries out of 12, so one is blank.

Now the numeric summary, which is where the third problem surfaces:

>>> df.describe()
       student_id       score
count       12.00       12.00
mean       105.75      153.00
std          3.28      266.78
min        101.00       52.00
25%        103.00       66.00
50%        105.50       76.00
75%        108.25       90.00
max        111.00      999.00      <- a score of 999 out of 100

Read that score column carefully. The maximum is 999 on an exam marked out of 100, and that one value has dragged the mean to 153 and the standard deviation to 267, both of which are nonsense for exam scores. This is exactly what describe is for: a mean far outside the plausible range, or a standard deviation larger than the mean, is a flag before you have looked at anything else. Note also that describe summarized only two columns, because hours is currently text and the rest are categories.

One more check, which describe cannot show:

>>> df.duplicated().sum()
1

>>> df["program"].value_counts()
Science    6
Arts       5
arts       1        <- same category, different spelling

>>> df["attended"].value_counts()
yes    4
no     4
Yes    1
YES    2
No     1           <- five spellings of two categories

So a ten-second inspection has found five distinct problems: a text column that should be numeric, a missing value, an impossible score, one exact duplicate row, and inconsistent category spellings. None of them raised an error, and every one of them would have quietly corrupted a later result. Lesson 4 fixes all five.

Key idea: A five-command inspection of study.csv reveals a wrong dtype, a missing value, an impossible maximum, a duplicate row, and inconsistent categories, none of which produced an error.

read_csv arguments that save whole afternoons

The default read_csv is a guess. A handful of arguments turn the guess into a specification:

df = pd.read_csv(
    "data/raw/study.csv",
    na_values=["", "NA", "N/A", "unknown", "-", "999"],  # what counts as missing
    parse_dates=["submitted_on"],                        # real dates, not strings
    dtype={"student_id": "string"},                      # keep leading zeros in IDs
)

• na_values declares the sentinel values this particular file uses for "missing". Every organization has its own; some use -1, some use 9999, some use the word "none".
• parse_dates turns text into real datetimes so you can sort and subtract them. Without it, "2026-05-10" sorts before "2026-05-9" because that is alphabetical order.
• dtype pins down columns you do not want guessed. Reading an ID like "007" as a number destroys the leading zeros permanently.
• thousands="," handles values written as "1,234"; sep=";" handles the European CSV convention; encoding="latin-1" is the usual fix for a UnicodeDecodeError on an older export.

A note on the "999" above: declaring it as a missing value is only correct if you have confirmed that 999 is this file's code for "no score recorded". If it is actually a typo for 99, treating it as missing throws away recoverable information. Ask before you assume; that is the difference between cleaning and guessing.

Key idea: Specify na_values, parse_dates, and dtype rather than trusting read_csv to guess, and confirm what a sentinel value actually means before declaring it missing.

head shows whether the file parsed correctly, shape tells you how big it is, info reveals data types and how many values are missing per column, and describe gives a first numeric summary that often exposes impossible values (a maximum age of 999, say).

Worked reading. If df.shape returns (1000, 6), the table has 1000 rows and 6 columns. If df.info() then shows an "age" column with 992 non-null entries, you immediately know 1000 - 992 = 8 ages are missing, before you have plotted a single thing.

Key idea: head, shape, columns, info, and describe form a five-line first look that catches most obvious problems.

Data types matter

pandas assigns each column a dtype (its data type). Numbers become int64 or float64, text becomes object, and dates should become datetime64. A frequent surprise is a numeric column read as text because one stray value contained a letter or a currency symbol. If df.info() shows a price column as object, arithmetic on it will misbehave until you convert it. The dtype is like a label on a jar: if a jar of numbers is mislabeled "text," none of your number recipes will work until you relabel it.

Key idea: A wrong dtype (numbers stored as text) quietly breaks arithmetic until you fix it.

Tidy data

Data is tidy when each row is one observation, each column is one variable, and each cell holds one value. Tidy data is far easier to filter, group, and plot. A table that crams two facts into one cell (such as "Reno, NV" in a single column, or a "Q1/Q2/Q3/Q4" set of columns that are really one "quarter" variable) is untidy and usually needs reshaping first. The rule of thumb: one fact per cell, one meaning per column.

Untidy	Tidy fix
One "location" cell holding "Reno, NV"	Split into "city" and "state" columns
Columns Jan, Feb, Mar of the same measure	One "month" column and one "value" column

The second row of that table is the common one, and it has a name. Data in wide form spreads one variable across several columns; long form gives that variable its own column:

wide (untidy) - one column per month          long (tidy) - month is a variable
  student  Jan  Feb  Mar                        student  month  score
  101       52   61   66                        101      Jan     52
  102       63   65   70                        101      Feb     61
                                                101      Mar     66
                                                102      Jan     63
                                                102      Feb     65
                                                102      Mar     70

tidy = wide.melt(id_vars="student", var_name="month", value_name="score")

The wide form looks friendlier and is much worse to work with. "Average score per month" means naming Jan, Feb, and Mar explicitly, and adding April breaks every piece of that code. In the long form the same question is tidy.groupby("month")["score"].mean(), and April is simply more rows. That is the practical payoff of tidiness: your code stops depending on how many categories happen to exist.

You will not always receive tidy data, but knowing the target shape tells you what cleaning to aim for. The next lesson tackles the messiest part: missing and malformed values.

Key idea: Tidy data means one observation per row, one variable per column, and one value per cell.

Where people get stuck

• "If read_csv runs without error, the data is fine." It may still have wrong types, missing values, or impossible entries; that is what head, info, and describe are for.
• "describe() summarizes every column." By default it summarizes numeric columns; text columns need value_counts or describe(include='object').
• "A column of prices is automatically numeric." One stray symbol like a dollar sign makes the whole column an object (text) until you clean it.
• "Any table is ready to analyze." Untidy tables that pack two facts into a cell usually need reshaping before grouping or plotting.
• Reading head() and stopping. The first five rows of study.csv look perfect; every one of its five problems is further down. Always run info and describe as well.
• Skipping value_counts on category columns. describe ignores them, so five spellings of "yes" stay invisible until a groupby silently splits them into five groups.
• Reading IDs as numbers. Employee number 007 becomes 7 and never comes back. Pin identifier columns to a string dtype at load time.
• Trusting date sorting on text. Without parse_dates, "2026-05-10" sorts before "2026-05-9", and every time-based analysis is quietly wrong.

Recap

• Load a CSV with pd.read_csv, then inspect with head, shape, columns, info, and describe.
• shape gives (rows, columns); info reveals dtypes and missing counts per column.
• A numeric column shown as object usually contains a stray non-numeric character, as "10 hrs" does in study.csv.
• describe exposes impossible values: study.csv has a score of 999 out of 100, which inflates the mean to 153.
• Use duplicated() and value_counts() for the problems describe cannot see.
• Specify na_values, parse_dates, and dtype rather than letting read_csv guess.
• Tidy data is one observation per row, one variable per column, one value per cell.

Sources

1. pandas development team. (n.d.). IO tools (text, CSV, HDF5, ...): read_csv parameters. pandas user guide. pandas.pydata.org
2. pandas development team. (n.d.). Essential basic functionality: dtypes, head, and describe. pandas user guide. pandas.pydata.org
3. pandas development team. (n.d.). Time series / date functionality: parsing dates on import. pandas user guide. pandas.pydata.org
4. Wickham, H. (2014). Tidy data. Journal of Statistical Software, 59(10), 1-23. vita.had.co.nz
5. Wickham, H., Cetinkaya-Rundel, M., & Grolemund, G. (2023). Data tidying. In R for data science (2nd ed., ch. 5). O'Reilly. r4ds.hadley.nz
6. VanderPlas, J. (2016). Data manipulation with pandas. In Python data science handbook (ch. 3). O'Reilly. jakevdp.github.io
7. Diez, D., Cetinkaya-Rundel, M., & Barr, C. D. (2019). Data basics. In OpenIntro statistics (4th ed., ch. 1). OpenIntro. openintro.org

Deleting the rows with missing values is the most common data-cleaning decision in the world, and it is sometimes exactly right and sometimes the single worst thing you can do to a dataset. Which one it is does not depend on how many values are missing. It depends on why they are missing, and that is a question the data itself often cannot answer.

The big picture

Real data arrives dirty, and cleaning it is the quiet work that decides whether everything downstream is trustworthy. Three problems dominate: gaps where values are missing, columns stored as the wrong type, and duplicate rows that double-count. Cleaning is like washing and chopping ingredients before you cook: unglamorous, but skipping it ruins the dish.

Data cleaning is the work of making a table trustworthy by fixing errors, wrong types, and gaps, and skipping it quietly ruins every step downstream.

Key idea: Cleaning fixes missing values, wrong types, and duplicates so later analysis can be trusted.

Missing values

pandas marks a missing entry as NaN ("not a number"). A NaN is like a blank in a form: it is not a zero and not an empty word, just "no value recorded." Find them before deciding what to do.

df.isna().sum()     # count of missing values per column
df["age"].isna().mean()   # fraction missing in the age column

You then have two broad choices. Dropping removes rows (or columns) with missing data. Imputing fills the gaps with a reasonable stand-in, often the column's mean or median for numbers, or the most common value for categories. Imputing is like estimating a missing quiz score from a student's typical performance rather than throwing out their whole record.

df = df.dropna(subset=["age"])              # drop rows missing age
med = df["income"].median()
df["income"] = df["income"].fillna(med)     # fill missing income with the median

Which is right depends on how much is missing and why. Dropping a handful of rows from a large dataset is usually fine. Dropping half your rows throws away information and can bias the result if the missing rows differ systematically from the rest. Median imputation resists outliers, which is why it is often preferred over the mean for skewed columns like income.

Key idea: Decide between dropping and imputing based on how much data is missing and why it is missing.

Why the mechanism matters: MCAR, MAR, MNAR

"Why is it missing?" is not a philosophical question; it has three standard answers, and each one changes what you are allowed to do. Take the blank hours value for student 110 in study.csv and imagine three different explanations:

mechanism   the missingness depends on...        a story that would produce it
---------   ----------------------------------   ---------------------------------------
MCAR        nothing at all                       the sign-in tablet lost power one
            (missing completely at random)       arbitrary afternoon

MAR         other columns you DID observe        Arts students used a paper form that
            (missing at random)                  was often left blank, so missingness
                                                 depends on 'program', which you have

MNAR        the missing value itself             students who barely studied were
            (missing not at random)              embarrassed and left 'hours' blank,
                                                 so the blanks are systematically low

Now the consequences, which are completely different:

• MCAR. Dropping costs precision and nothing else; the remaining rows are still a fair sample. This is the only case where dropping is genuinely safe.
• MAR. Dropping biases the result, because you lose Arts students disproportionately - but the bias is fixable, since the column that predicts missingness (program) is sitting right there. Impute within program, or include program in the model.
• MNAR. Dropping is biased and imputing is biased, and no cleverness with the data at hand fixes it. If the blank hours are the low ones, mean-imputing pushes those students up to 6.0 and overstates average study time. The honest responses are to collect the missing information, or to report a sensitivity analysis showing how much the conclusion moves under two or three plausible assumptions.

Here is the part usually left out. You cannot determine the mechanism from the data alone. You can partly probe MCAR by checking whether missingness correlates with observed columns, but telling MAR from MNAR requires knowing the values you do not have. It is a judgment about data collection, which is exactly why Lesson 1 insisted on recording provenance. When you cannot rule out MNAR, say so.

Key idea: Dropping is safe only under MCAR, fixable under MAR, and biased under MNAR - and the mechanism is a fact about data collection that the data itself cannot fully reveal.

Fixing types

Convert columns to their true type so operations work. A price stored as text like "$12.50" must have the symbol stripped and then be converted to a number; a date stored as text should become a real datetime so you can sort and subtract dates.

df["price"] = df["price"].str.replace("$", "", regex=False).astype(float)
df["date"]  = pd.to_datetime(df["date"])

Key idea: Convert each column to its real type, or arithmetic and date math will silently misbehave.

Duplicates

Duplicate rows are rows identical to another, and they inflate counts and distort averages, like scanning the same receipt twice at checkout. Detect and drop them:

df.duplicated().sum()   # how many exact duplicate rows exist
df = df.drop_duplicates()

Two cautions. drop_duplicates() only catches exact matches, so "Ana Lopez" and "ana lopez " survive as two different people; normalize your text first. And sometimes two identical rows are legitimate - two separate purchases of the same item, at the same price, in the same second - so pass subset= the columns that genuinely identify a record rather than blindly deduplicating everything.

Key idea: Exact duplicate rows over-count observations, so detect and drop them, but normalize text first and check that identical rows are really duplicates.

Inconsistent categories

The quietest problem of all is a category spelled several ways. It never raises an error and it silently splits one group into three:

>>> df["attended"].value_counts()      # after the duplicate row was dropped, so 11 rows
yes    4
no     3
YES    2
Yes    1
No     1

Five labels for two categories. Every later groupby("attended") would produce five groups, each holding a fraction of the data, and every resulting average would rest on too few rows. The fix is one line per column:

df["attended"] = df["attended"].str.strip().str.lower()      # 'YES ' -> 'yes'
df["program"]  = df["program"].str.strip().str.title()       # 'arts' -> 'Arts'

>>> df["attended"].value_counts()
yes    7
no     4

.str.strip() removes invisible leading and trailing whitespace, which is the cause you will never spot by eye. After normalizing, always print value_counts() again and confirm the number of distinct labels is what you expect - if a "gender" column has 7 categories, you have found something.

Key idea: Inconsistent category spellings silently split one group into several, so strip and case-normalize every text column and check value_counts afterwards.

Outliers: find them, then judge them

An outlier is a value far from the rest of the data. The standard detector is the 1.5 IQR rule: flag anything below Q1 - 1.5 x IQR or above Q3 + 1.5 x IQR. On the raw study.csv scores:

Q1 = 66,  Q3 = 90,  IQR = 90 - 66 = 24

lower fence = 66 - 1.5 * 24 = 66 - 36 =  30
upper fence = 90 + 1.5 * 24 = 90 + 36 = 126

999 > 126   -> flagged

Q1, Q3 = df["score"].quantile([0.25, 0.75])
iqr = Q3 - Q1
df[(df["score"] < Q1 - 1.5 * iqr) | (df["score"] > Q3 + 1.5 * iqr)]

Now the crucial part the rule cannot do for you: an outlier is not automatically an error. The rule finds candidates; domain knowledge decides. A score of 999 out of 100 is impossible, so it becomes NaN. A genuine score of 12 would also be flagged and is perfectly real, and deleting it would quietly remove the struggling students - very often the population you most wanted to understand. Deleting inconvenient real values is not cleaning; it is fabrication.

Key idea: The 1.5 IQR rule finds candidate outliers, but only domain knowledge decides whether a value is an error to remove or a real observation to keep.

A worked cleaning decision

Suppose a 1,000-row survey has an "age" column with 8 missing values and 3 impossible entries of 200. The 3 impossible values are errors, so set them to NaN. Now 11 of 1,000 ages are missing, which is 11 / 1000 = 0.011, about 1%. Because that is a tiny fraction, dropping those 11 rows is defensible and simplest, leaving 989 clean rows. Had 400 values been missing, you would instead impute or investigate why so many are absent, because dropping 40% would distort the sample. The guiding rule: understand why data is missing before you decide how to handle it.

Key idea: Dropping a tiny fraction of rows is fine; losing a large fraction calls for imputation or investigation.

Cleaning study.csv end to end

Now do all of it, on the real file, counting rows at every step. Lesson 3 found five problems; here is each one fixed.

import pandas as pd, numpy as np

raw = pd.read_csv("data/raw/study.csv")
len(raw)                                   # 12

# 1. exact duplicate rows
raw.duplicated().sum()                     # 1   (student 103 appears twice)
df = raw.drop_duplicates().copy()
len(df)                                    # 11   <- one row gone

# 2. the type error: "10 hrs" made the whole column text
df["hours"] = pd.to_numeric(
    df["hours"].astype(str).str.replace(" hrs", "", regex=False),
    errors="coerce")
df["hours"].dtype                          # float64
len(df)                                    # 11   <- no rows lost, one value repaired

# 3. the impossible score
(df["score"] > 100).sum()                  # 1   (student 109 scored 999)
df.loc[df["score"] > 100, "score"] = np.nan
len(df)                                    # 11   <- no rows lost, one value flagged

# 4. inconsistent categories
df["program"]  = df["program"].str.strip().str.title()
df["attended"] = df["attended"].str.strip().str.lower()

# 5. the date column
df["submitted_on"] = pd.to_datetime(df["submitted_on"])

# 6. what is actually missing now?
df.isna().sum()
# hours    1        (student 110, blank in the file)
# score    1        (student 109, the 999 we just flagged)
# everything else 0

# 7. decide, then act
df = df.dropna(subset=["hours", "score"])
len(df)                                    # 9    <- two more rows gone

Keep the ledger, because "we cleaned the data" is not a reproducible statement and this is:

step                                       rows   what changed
----------------------------------------   ----   ----------------------------------
loaded                                       12
dropped 1 exact duplicate                    11    -1 row (student 103)
converted hours from text to number          11    1 value repaired ("10 hrs" -> 10.0)
set the impossible score to NaN              11    1 value flagged (999)
normalized program and attended spellings    11    5 values relabeled
parsed submitted_on as a date                11    11 values retyped
dropped rows missing hours or score           9    -2 rows (students 109, 110)
----------------------------------------   ----
final                                         9    75% of the original rows kept

Were those two rows safe to drop? Two of eleven is 18%, more than the "tiny fraction" rule comfortably allows, and this is where the mechanism discussion earns its place. Student 109's score went missing because it was impossible, which says nothing about how well that student did - effectively MCAR, so dropping is fine. Student 110's blank hours is the uncertain one: if low-effort students left it blank, that is MNAR and dropping nudges average study time upward. With one row it barely matters; with 200 rows out of 1100 it would matter enormously, and the write-up would have to say so.

The cleaned dataset, which every remaining lesson uses:

student_id  hours  score  program  attended  submitted_on
   101        2.0     52  Science    yes     2026-05-01
   102        3.0     63  Science    yes     2026-05-01
   103        4.0     66  Arts       no      2026-05-02
   104        5.0     78  Arts       yes     2026-05-02
   105        6.0     74  Science    no      2026-05-03
   106        7.0     88  Science    yes     2026-05-03
   107        8.0     85  Arts       no      2026-05-04
   108        9.0     96  Arts       yes     2026-05-04
   111       10.0     98  Arts       yes     2026-05-06

df.to_csv("data/processed/study_clean.csv", index=False)   # never overwrite the raw file

Key idea: Cleaning study.csv takes 12 rows to 9 through one duplicate drop, one type repair, one impossible value, five relabelings, and two dropped rows - and writing that ledger down is what makes the analysis defensible.

Where people get stuck

• "NaN is the same as 0 or an empty string." NaN means no value was recorded; treating it as zero can badly distort sums and averages.
• "Dropping missing rows is always safe." It is safe under MCAR, fixable under MAR, and biased under MNAR. The amount matters less than the reason.
• "The mean is always the best fill." For skewed columns the median resists outliers, so it is often the safer imputation.
• "Duplicates do not matter much." An identical row counted twice over-represents that case and skews counts and averages.
• Imputing and then reporting the result as if nothing were missing. Every imputed value is a guess. Say how many there were and what you filled them with, or your uncertainty is understated.
• Deleting outliers because they are inconvenient. The 1.5 IQR rule finds candidates; only impossibility makes a value an error. Removing real extreme values removes exactly the cases that usually matter most.
• Missing invisible whitespace. " yes" and "yes" are different categories to pandas and identical to your eye. Always .str.strip() before comparing or grouping.
• Cleaning before inspecting, or overwriting the raw file. Fix things in the order they occur to you and you never learn what the file actually contained; overwrite the original and no decision can be audited afterwards.

Recap

• Missing values show as NaN; count them with df.isna().sum() before acting.
• The mechanism decides what is allowed: MCAR makes dropping safe, MAR makes the bias fixable, MNAR makes both dropping and simple imputation biased.
• You cannot fully determine the mechanism from the data; it is a fact about how the data was collected.
• Median imputation resists outliers and suits skewed columns.
• Convert columns to their real types, strip and case-normalize categories, and drop true duplicates.
• The 1.5 IQR rule flags candidate outliers; domain knowledge decides whether each is an error or a real value.
• Cleaning study.csv went 12 rows to 9, and recording that ledger is what makes the result defensible.

Sources

1. Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009). Multiple imputation for missing data in epidemiological and clinical research: Potential and pitfalls. BMJ, 338, b2393. pmc.ncbi.nlm.nih.gov
2. pandas development team. (n.d.). Working with missing data. pandas user guide. pandas.pydata.org
3. pandas development team. (n.d.). Working with text data: str.strip, str.lower, and str.replace. pandas user guide. pandas.pydata.org
4. pandas development team. (n.d.). pandas.DataFrame.drop_duplicates. pandas API reference. pandas.pydata.org
5. scikit-learn developers. (n.d.). Imputation of missing values. scikit-learn user guide. scikit-learn.org
6. Wickham, H., Cetinkaya-Rundel, M., & Grolemund, G. (2023). Missing values. In R for data science (2nd ed., ch. 18). O'Reilly. r4ds.hadley.nz
7. Diez, D., Cetinkaya-Rundel, M., & Barr, C. D. (2019). Data collection and quality. In OpenIntro statistics (4th ed., ch. 1). OpenIntro. openintro.org

There is a dataset later in this lesson where tutoring makes students worse overall and better in every single program. Both statements come from the same 400 students and both are arithmetically correct. Learning to see how that happens - and to notice when it might be happening to you - is worth more than any chart type you will learn today.

The big picture

Exploratory data analysis is the getting-to-know-you phase with a cleaned dataset, where you look before you leap to conclusions. You summarize and plot variables one at a time, then in pairs, to learn their shape, catch leftover errors, and spot promising patterns worth testing later. It is detective work, not a verdict: you are gathering clues, not closing the case.

Exploratory data analysis (EDA) is the open-minded first look at a cleaned dataset. Its goals are to understand each variable's distribution, uncover relationships, spot errors and outliers the cleaning missed, and generate hypotheses worth testing. EDA is about questions, not conclusions; you are getting to know the data.

Key idea: EDA explores a cleaned dataset to understand it and raise hypotheses, not to prove them.

One variable at a time (univariate)

Univariate analysis examines one variable by itself to learn its distribution. The right tool depends on the variable type, exactly as in statistics.

• Quantitative variable: a histogram to see its shape (symmetric, skewed, bimodal) and a five-number summary or describe() for the numbers.
• Categorical variable: a bar chart of counts, and a frequency table via df["col"].value_counts().

Key idea: Use a histogram for one quantitative variable and a bar chart or value_counts for one categorical variable.

Two variables at a time (bivariate)

Bivariate analysis examines two variables together to see how they move. A scatterplot is the standard picture for two quantitative variables; a grouped summary compares a quantitative measure across the levels of a categorical variable.

df["category"].value_counts()          # counts per category
df.groupby("category")["sales"].mean() # average sales within each category
df[["age", "income"]].corr()           # correlation between two numbers

Key idea: A scatterplot pictures two quantitative variables; a grouped summary compares a measure across categories.

Grouping to compare

The groupby operation is the workhorse of EDA. It splits rows into groups by a categorical column, computes a summary within each group, and combines the results, a pattern often called split-apply-combine. Sorting laundry into piles by color, weighing each pile, then reading off the weights is the same idea. "Average sales per region" or "median age per plan" are one line each. Comparing group summaries is often where the first real insight appears.

Worked example. Suppose df.groupby("plan")["age"].mean() returns 24 for "basic" and 41 for "premium." That 17-year gap is a hypothesis worth exploring: perhaps older customers prefer the premium plan. EDA has surfaced the lead; confirming it comes later.

Key idea: groupby splits rows by a category, summarizes each group, and combines the results into a comparison.

EDA on the cleaned study.csv

Run the univariate pass first, on the nine clean rows from Lesson 4:

>>> df[["hours", "score"]].describe()
        hours   score
count    9.00    9.00
mean     6.00   77.78
std      2.74   15.55
min      2.00   52.00
25%      4.00   66.00
50%      6.00   78.00
75%      8.00   88.00
max     10.00   98.00

>>> df["program"].value_counts()
Arts       5
Science    4

>>> df["attended"].value_counts()
yes    6
no     3

Two things to notice immediately. The mean and median score are 77.78 and 78.00, almost identical, which says the scores are roughly symmetric with no dominant outlier - a good sign that Lesson 4's cleaning worked. And the group sizes are tiny, so every comparison below is a lead, never a finding.

Now the bivariate pass, one groupby per question:

>>> df.groupby("program")[["score", "hours"]].mean()
          score  hours
program
Arts      84.60   7.20
Science   69.25   4.50

>>> df.groupby("attended")[["score", "hours"]].mean()
           score  hours
attended
no         75.00   6.00
yes        79.17   6.00

Read the first table carefully, because it contains a trap. Arts students average 84.6 and Science students 69.25, a gap of over 15 points, which looks like a strong statement about programs. But the second column shows Arts students also studied 7.2 hours on average against Science's 4.5. The entire program gap could simply be the study-time gap wearing a different label. program and hours are confounded in this sample, and no amount of staring at the first column will separate them.

The second table is more interesting precisely because the confounder is absent: both attendance groups studied exactly 6.0 hours on average, so the 4.17-point gap in favour of tutoring is not explained by study time here. That is a genuine lead worth pursuing - with the enormous caveat that it rests on six students against three.

Key idea: Always summarize the other variables alongside the one you are comparing; a group difference that vanishes once you see a second column was never about the grouping variable.

Simpson's paradox

Confounding has an extreme form that reverses conclusions rather than merely weakening them. Take a full cohort of 400 students and compare pass rates by tutoring attendance:

All 400 students	Passed	Total	Pass rate
Attended tutoring	120	200	60.0%
Did not attend	130	200	65.0%

Tutoring appears to hurt: 60% against 65%. Now split the same 400 students by program:

Arts	Passed	Total	Pass rate
Attended	32	40	80.0%
Did not attend	120	160	75.0%

Science	Passed	Total	Pass rate
Attended	88	160	55.0%
Did not attend	10	40	25.0%

Tutoring wins in Arts (80% against 75%) and wins enormously in Science (55% against 25%). It wins in every group and loses overall. The totals check out exactly: attended is 40 + 160 = 200 students and 32 + 88 = 120 passes; not attended is 160 + 40 = 200 students and 120 + 10 = 130 passes. Nothing has been fudged. This reversal is Simpson's paradox.

The mechanism is the group sizes. Science is the harder course, with a much lower baseline pass rate, and tutoring is concentrated there: 160 of the 200 attendees are Science students, against only 40 of the 200 non-attendees. The overall "attended" number is therefore mostly a Science number and the overall "did not attend" number is mostly an Arts number. You are not comparing tutoring against no tutoring; you are comparing Science against Arts with a tutoring label on it.

who is in each overall group

                  Arts    Science    -> which course dominates the average
  attended          40        160       mostly Science (the hard course)
  did not attend   160         40       mostly Arts    (the easier course)

Which number is right? For the question "should we fund tutoring?", the within-program comparison is the honest one, because program is a cause of both attendance and passing rather than a consequence of tutoring. But that reasoning is causal, not statistical: the data cannot tell you which table to trust. You need to know how the world works. The practical habit is simple and worth adopting permanently: before reporting any aggregate comparison, break it down by the one or two variables most likely to differ between the groups. If the direction flips, you have just avoided publishing something backwards.

Key idea: Simpson's paradox is a comparison that reverses when you split by a lurking variable, and the fix is to break every aggregate down by the variables most likely to differ between the groups.

Producing those tables in pandas

Both views are one call each. pd.crosstab counts the combinations of two categorical columns, and normalize="index" turns each row into proportions:

>>> pd.crosstab(cohort["attended"], cohort["passed"])
passed      0    1
attended
no         70  130
yes        80  120

>>> pd.crosstab(cohort["attended"], cohort["passed"], normalize="index")
passed        0      1
attended
no         0.35   0.65        <- 65% pass rate without tutoring
yes        0.40   0.60        <- 60% pass rate with tutoring

Passing a list of columns as the row key produces the split view, and the direction flips:

>>> pd.crosstab([cohort["program"], cohort["attended"]],
...             cohort["passed"], normalize="index")
passed                    0      1
program  attended
Arts     no            0.25   0.75
         yes           0.20   0.80        <- tutoring better in Arts
Science  no            0.75   0.25
         yes           0.45   0.55        <- tutoring far better in Science

Note the argument that matters most: normalize="index" divides each row by its own total, giving "of the students who attended, what fraction passed?". Using normalize="columns" instead answers a completely different question ("of the students who passed, what fraction attended?"), and confusing the two is its own reliable source of wrong conclusions.

Key idea: pd.crosstab with a list of row keys produces the broken-down view in one line, and normalize="index" is what turns counts into the conditional rates you actually want to compare.

A short EDA story

Imagine a dataset of support tickets with columns for handling time and a 1-to-5 satisfaction rating. Univariate EDA shows handling time is right-skewed (most tickets are quick, a few drag on). A grouped summary of average rating by handling-time bucket shows ratings falling as handling time rises. That is a hypothesis, a tentative testable idea, not proof: maybe hard problems both take longer and frustrate people. EDA has done its job by pointing you toward a relationship worth testing carefully later. Always keep that humility: seeing a pattern is the start of an investigation, not the end.

Key idea: A pattern found in EDA is a lead to test, never a conclusion on its own.

Where people get stuck

• "A pattern found in EDA proves a conclusion." EDA suggests hypotheses; confirming them needs careful testing or an experiment.
• "One chart type fits every variable." Histograms suit quantitative variables and bar charts suit categorical ones; a scatterplot needs two quantitative variables.
• "Cleaning removes all errors, so EDA can skip error-checking." EDA regularly uncovers impossible values and outliers that cleaning missed.
• "groupby just counts rows." It can compute any summary per group, such as a mean, median, or sum, then combine them.
• Reporting a group mean without reporting the group size. "Arts averages 84.6" hides that Arts is five students. Always print count beside mean.
• Comparing groups on one column at a time. The Arts-versus-Science score gap looks like a program effect until the hours column shows the groups also differ in study time.
• Trusting an aggregate you have not broken down. Simpson's paradox is not exotic; it appears wherever group sizes are unequal, which is nearly everywhere.
• Testing a hypothesis on the same data that suggested it. If you look at twenty comparisons and report the most striking one, you have found the largest random fluctuation, not a finding.

Recap

• EDA is the open-minded first look that raises hypotheses, not conclusions.
• Univariate: histogram for quantitative, bar chart or value_counts for categorical.
• Bivariate: scatterplot for two quantitative variables; grouped summaries across categories.
• groupby follows split-apply-combine to compare a measure across groups; always report the group size too.
• Summarize the other variables alongside the comparison, or a confounder will masquerade as your effect.
• Simpson's paradox can reverse a comparison entirely when group sizes are unequal, and only knowledge of the world tells you which table to trust.
• Any pattern found is a lead to test carefully, not proof.

Sources

1. Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society, Series B, 13(2), 238-241. jstor.org
2. Kievit, R. A., Frankenhuis, W. E., Waldorp, L. J., & Borsboom, D. (2013). Simpson's paradox in psychological science: A practical guide. Frontiers in Psychology, 4, 513. pmc.ncbi.nlm.nih.gov
3. Wickham, H., Cetinkaya-Rundel, M., & Grolemund, G. (2023). Exploratory data analysis. In R for data science (2nd ed., ch. 10). O'Reilly. r4ds.hadley.nz
4. pandas development team. (n.d.). Group by: split-apply-combine. pandas user guide. pandas.pydata.org
5. VanderPlas, J. (2016). Aggregation and grouping. In Python data science handbook (ch. 3). O'Reilly. jakevdp.github.io
6. Tukey, J. W. (1977). Exploratory data analysis. Addison-Wesley. find source ↗
7. Cetinkaya-Rundel, M., & Hardin, J. (2021). Exploring categorical and numerical data. In Introduction to modern statistics. OpenIntro. openintro.org

"The average customer spends $84" is one of the most-repeated sentences in business and one of the least informative. It does not say whether most customers spend $80 or whether half spend $10 and a few spend $900. A center without a spread is not a summary; it is a single number pretending to be one. This lesson is about the small set of numbers that actually describe a column.

The big picture

Descriptive statistics squeeze a whole column of numbers into a few honest summaries so a person can grasp it at a glance. Two questions drive them: where is the center, and how spread out are the values. Getting fluent here pays off because these same numbers reappear in correlation, regression, and every model later in the course.

Descriptive statistics compress a column of numbers into a few honest summaries. Report a center and a spread together, and you have told most of the story.

Key idea: Summarize a variable by its center and its spread, and prefer resistant measures when outliers loom.

Measures of center

The mean is the sum divided by the count; picture it as the balance point of a seesaw where every value sits at its spot, so one very heavy value tips the balance toward itself. The median is the middle value of sorted data; picture a street of houses lined up by price, where the median is simply the middle house, unmoved by one mansion at the end. The mode is the most frequent value. The mean uses every value, so a single extreme value pulls it; the median barely moves, which makes it resistant to outliers.

Worked example. For the values 2, 4, 4, 5, 100: the mean is (2 + 4 + 4 + 5 + 100) / 5 = 115 / 5 = 23, but the median (middle of the sorted five) is 4. The lone value of 100 dragged the mean far above every typical value, while the median still describes the bulk of the data. When the mean and median disagree sharply, suspect skew or an outlier.

Key idea: The mean is a balance point pulled by outliers; the median is the middle house and resists them.

Measures of spread

The range is max minus min. The standard deviation is the typical distance of values from the mean; the variance is its square. The interquartile range (IQR = Q3 - Q1) is the spread of the middle 50% and, like the median, resists outliers.

Worked standard deviation. For 2, 4, 6, 8, the mean is 5. The deviations from the mean are -3, -1, 1, 3, and their squares are 9, 1, 1, 9, summing to 20. Dividing by n - 1 = 3 gives a sample variance of 20 / 3 = 6.67, so the standard deviation is square root of 6.67 = 2.58. Values sit about 2.58 units from the mean on average. (Sample formulas divide by n - 1 to correct bias; if the four numbers were the entire population you would divide by 4.)

Key idea: Standard deviation is the typical distance from the mean; the IQR is the resistant spread of the middle half.

Why n - 1, and the five-number summary

The n - 1 is not a typo and it is worth understanding once. You are estimating the spread around the true population mean, but you do not know it, so you use the sample mean instead. The sample mean is by construction the value the sample is closest to, so the squared deviations you compute are systematically a little too small. Dividing by n - 1 rather than n inflates the estimate just enough to cancel that bias. It is called Bessel's correction, and its effect shrinks as the sample grows: on the nine study.csv scores it changes the standard deviation from 14.66 to 15.55, a 6% difference, while on a thousand rows it changes almost nothing.

Practical consequence: pandas divides by n - 1 by default and numpy divides by n. The same column can give you two different standard deviations depending on which library you asked, which is a genuinely confusing afternoon if you do not know why.

df["score"].std()          # 15.55   pandas: sample, ddof=1
np.std(df["score"])        # 14.66   numpy: population, ddof=0
np.std(df["score"], ddof=1)  # 15.55  numpy, matching pandas

The five-number summary is the resistant alternative to mean-and-standard-deviation, and it is exactly what a boxplot draws:

study.csv scores:  52, 63, 66, 74, 78, 85, 88, 96, 98

  min    Q1    median    Q3    max
   52    66      78      88     98

  IQR   = Q3 - Q1 = 88 - 66 = 22        the middle half spans 22 points
  range = max - min = 98 - 52 = 46      the whole data spans 46

Notice that the IQR is less than half the range. That is normal: the range depends entirely on the two most extreme values and therefore grows with sample size and jumps with any single error, while the IQR describes where the bulk of the data actually lives.

Key idea: Dividing by n - 1 corrects a systematic underestimate of spread, pandas and numpy disagree on the default, and the five-number summary gives a resistant picture the boxplot draws directly.

Standardizing: the z-score

A raw value tells you nothing without its context. A score of 88 is excellent on one exam and mediocre on another. The z-score puts any value on a common scale by asking how many standard deviations it sits from the mean:

z = (value - mean) / standard deviation

study.csv scores, mean 77.78, sd 15.55

  value    z-score    reading
  -----    -------    ------------------------------------------
     52     -1.66     well below average
     66     -0.76     a bit below average
     78      0.01     essentially average
     88      0.66     somewhat above average
     98      1.30     clearly above, but not extraordinary

df["z_score"] = (df["score"] - df["score"].mean()) / df["score"].std()

Two facts make z-scores useful. First, a column of z-scores always has mean 0 and standard deviation 1, so two variables measured in completely different units become directly comparable - which is why several models in later lessons require standardized inputs. Second, for roughly bell-shaped data the empirical rule says about 68% of values fall within one standard deviation of the mean, about 95% within two, and about 99.7% within three. On study.csv, 6 of the 9 scores lie within one standard deviation, which is 67% - close to the rule, though with nine values that agreement is mostly luck.

A caution that matters: the empirical rule assumes an approximately normal shape. On a strongly skewed column such as income it fails badly, and |z| > 3 stops being a sensible outlier test. Check the histogram before leaning on it.

Key idea: A z-score expresses a value as standard deviations from the mean, making different variables comparable, and the 68-95-99.7 rule applies only to roughly bell-shaped data.

In pandas, one call each

df["income"].mean()      # average
df["income"].median()    # middle value, resists outliers
df["income"].std()       # standard deviation (sample, divides by n - 1)
df["income"].quantile([0.25, 0.5, 0.75])   # the quartiles

Key idea: pandas computes each summary in one call, so the thinking, not the arithmetic, is your job.

Reading the numbers together

No single number is enough. A mean far above the median signals right skew. A large standard deviation relative to the mean signals high variability. Reporting center and spread together, and preferring the median and IQR when outliers loom, keeps your summary honest. These same statistics reappear in every later module, so being fluent with them pays off.

The mean-versus-median gap is a diagnostic you can read in one glance:

mean vs median            shape             typical example
----------------------    --------------    ---------------------------
mean much > median        right-skewed      income, house prices, wait times
mean about = median       symmetric         exam scores, heights
mean much < median        left-skewed       exam scores with a hard ceiling,
                                            age at retirement

Apply it to study.csv: mean 77.78 against median 78.00, a gap of 0.22 points on a 15.55-point standard deviation. That is symmetric, and it confirms that no extreme value survived cleaning. Compare with the raw file before cleaning, where the mean was 153 and the median 76 - a gap of 77 points that screamed "something is wrong here" before anyone had opened a plot.

One more comparison worth knowing: the coefficient of variation, the standard deviation divided by the mean, expresses spread as a fraction of typical size. For study.csv scores it is 15.55 / 77.78 = 0.20, so the spread is about 20% of the average. It lets you say whether a standard deviation of 15 is large or small without knowing anything about the units, and it is meaningless for anything that can be zero or negative, such as temperature in Celsius.

Key idea: Read center and spread together, let a mean-versus-median gap flag skew, and use the coefficient of variation to judge whether a spread is large relative to the values themselves.

Where people get stuck

• "The mean is always the typical value." With an outlier or skew, the mean can sit far from every actual value; the median is often more representative.
• "The median ignores most of the data, so it is worse." The median deliberately resists extremes, which is a strength when outliers would mislead.
• "A big standard deviation means an error." It just means the values are widely spread; whether that is a problem depends on context.
• "Sample standard deviation divides by n." The sample formula divides by n - 1 to correct bias; only a full population divides by n.
• Getting two different standard deviations from the same column. pandas defaults to ddof=1 and numpy to ddof=0. Neither is wrong; you just have to know which one you asked for.
• Reporting a center with no spread. "The average is 78" is compatible with everyone scoring 78 and with half the class scoring 40. Report both, always.
• Using |z| > 3 as a universal outlier test. It assumes a roughly normal shape. On a skewed column the IQR rule is far more reliable.
• Averaging percentages or rates. The mean of three group pass rates is not the overall pass rate unless the groups are the same size - the same arithmetic that produces Simpson's paradox.

Recap

• Center: mean (balance point, outlier-sensitive), median (middle value, resistant), mode (most frequent).
• Spread: range, variance, standard deviation, and the resistant IQR.
• For 2, 4, 4, 5, 100 the mean is 23 but the median is 4, showing the outlier's pull.
• Sample standard deviation divides squared deviations by n - 1 to remove a systematic underestimate; pandas does this by default and numpy does not.
• The five-number summary (min, Q1, median, Q3, max) is what a boxplot draws; for study.csv it is 52, 66, 78, 88, 98.
• A z-score is (value - mean) / sd, giving a unit-free position, with the 68-95-99.7 rule holding only for roughly bell-shaped data.
• A mean much larger than the median signals right skew or a high outlier.

Sources

1. Diez, D., Cetinkaya-Rundel, M., & Barr, C. D. (2019). Summarizing data. In OpenIntro statistics (4th ed., ch. 2). OpenIntro. openintro.org
2. Cetinkaya-Rundel, M., & Hardin, J. (2021). Exploring numerical data. In Introduction to modern statistics. OpenIntro. openintro.org
3. pandas development team. (n.d.). Essential basic functionality: descriptive statistics and the ddof argument. pandas user guide. pandas.pydata.org
4. NumPy developers. (n.d.). NumPy: the absolute basics for beginners: mean, std, and array reductions. NumPy manual. numpy.org
5. Python Software Foundation. (n.d.). statistics - Mathematical statistics functions: stdev versus pstdev. Python 3 documentation. docs.python.org
6. VanderPlas, J. (2016). Aggregation and grouping. In Python data science handbook (ch. 3). O'Reilly. jakevdp.github.io
7. Wickham, H., Cetinkaya-Rundel, M., & Grolemund, G. (2023). Exploratory data analysis. In R for data science (2nd ed., ch. 10). O'Reilly. r4ds.hadley.nz

Give the same two hundred numbers to two analysts and ask each for a histogram. One comes back with a single smooth hump and says the group is homogeneous. The other comes back with two clear peaks and says there are two distinct populations. Neither has cheated; they chose different bin widths. Charts are not neutral windows onto data, and knowing where the choices live is what separates a chart that informs from one that persuades.

The big picture

A chart is a shortcut to a pattern: done well, a reader sees in a second what a table hides. Done badly, it invents a pattern that is not there, so choosing the right chart and drawing it honestly is a core skill, not decoration. The same numbers can whisper or shout depending only on how you draw them, which is exactly why the rules below matter.

A good chart lets the reader see a pattern faster than any table. A bad chart hides it, or worse, invents one that is not there. Choosing the right chart and drawing it honestly is a core data science skill, not decoration.

Key idea: Pick the chart that answers the question and draw it so the picture cannot mislead.

Match the chart to the job

You want to show...	Use
Distribution of one quantitative variable	Histogram or boxplot
Comparison across categories	Bar chart
Relationship between two quantitative variables	Scatterplot
Change over time	Line chart
Parts of a single whole	Bar chart (usually clearer than a pie)

That table is really an application of one underlying fact about human perception. People read some visual encodings far more accurately than others, and the ranking is remarkably stable:

most accurately read
      |    position along a common scale     scatterplot, dot plot
      |    length                            bar chart
      |    angle, slope                      pie chart, line slope
      |    area                              bubble chart
      v    color intensity, saturation       heatmap
least accurately read

Almost every piece of chart advice you will ever hear follows from that list. Bar charts beat pie charts because length beats angle. A scatterplot beats a bubble chart because position beats area. A heatmap is not forbidden, but it is the weakest encoding, so use it when you want an overall impression rather than precise comparisons.

Key idea: The chart type should follow from the question - distribution, comparison, relationship, or change over time - and from the fact that people read position most accurately and color intensity least.

Drawing study.csv honestly

The relationship question for our dataset is "does more study time go with a higher score?", so a scatterplot it is:

import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(6, 4))
ax.scatter(df["hours"], df["score"])
ax.set_xlabel("hours studied")
ax.set_ylabel("exam score (out of 100)")
ax.set_title("More study hours go with higher scores")   # the takeaway, not the variables
ax.set_ylim(0, 100)                                      # the full possible range
fig.tight_layout()

Four deliberate choices are in those six lines. The axes carry units, not just names. The title states the finding rather than restating the columns. The y-axis covers the whole 0-to-100 range the exam allows, so the reader can see how much of the scale the data actually occupies. And there is no gridline, legend, or color scheme, because with nine points none of them would add information.

A refinement of the zero rule is worth stating precisely here, because it is often taught as an absolute and it is not. The zero baseline is required for length encodings and optional for position encodings. A bar's meaning is its length, so cutting the axis breaks the correspondence between length and value. A dot's meaning is its position, and nothing is being compared by size, so a scatterplot or line chart of temperatures between 18 and 24 degrees is perfectly honest without dragging the axis to zero - and dragging it there would hide the pattern in a flat band at the top.

Key idea: Label axes with units, title the chart with its takeaway, and require a zero baseline for bars and areas but not for dots and lines.

The histogram bin problem

The histogram is the one chart whose appearance depends on a parameter you choose, and the effect is not subtle. The same 200 values, drawn three ways:

bins = 5     one broad hump          "scores cluster around 70"
bins = 12    two distinct peaks      "there appear to be two groups of students"
bins = 60    a jagged comb           "no shape at all, just noise"

All three are the same data. Too few bins smooths real structure away; too many turns random variation into apparent structure. Matplotlib's default of 10 bins is a guess that happens to suit medium-sized datasets and nothing else.

ax.hist(df["score"], bins=12, range=(0, 100), edgecolor="white")

The professional habit is to try several bin widths before choosing, pick the one that shows structure without inventing it, and mention the choice if it matters. If a claimed pattern appears at one bin width and vanishes at the next, it is not a pattern. Setting range explicitly is a second small honesty: it stops the axis from silently rescaling when you filter the data, so two histograms drawn side by side stay comparable.

Key idea: A histogram's shape depends on the bin width you chose, so try several and distrust any pattern that survives only one of them.

Log scales and color

Two more tools carry real risk of misleading if used without thought.

A log scale is the right answer when values span orders of magnitude. On a linear axis, plotting town populations from 500 to 5,000,000 crushes every small town into an indistinguishable line at the bottom. On a log axis, equal distances mean equal ratios, so 10 to 100 occupies the same space as 100 to 1,000, and the structure at every scale becomes visible. The obligation that comes with it is to label the axis unmistakably, because a reader who assumes a linear scale will badly misjudge every difference on the chart.

Color is the weakest encoding and also the one most likely to exclude readers. Roughly one man in twelve has some form of red-green color vision deficiency, so a chart that distinguishes "good" from "bad" by red against green is unreadable for a real fraction of any audience. Two rules cover it: use a colorblind-safe palette, and never let color be the only carrier of a distinction. Add a direct label, a different marker shape, or a position difference, so the chart still works in greyscale.

Key idea: Use a log scale when values span orders of magnitude and label it loudly, and never let color alone carry a distinction that some readers cannot see.

How charts mislead

Three traps are worth memorizing:

• Truncated axis. A truncated axis is a value axis that does not start at zero; on a bar chart it makes a rise from 92 to 95 look enormous. For bar charts, start the value axis at zero.
• Wrong chart type. A pie chart with ten thin slices is unreadable; a bar chart ranks them clearly. Line charts imply continuity, so do not use one for unordered categories.
• Overplotting and clutter. Overplotting is so many overlapping points that density is hidden; extra gridlines, 3-D effects, and loud colors distract from the data.

The two red bars and the two green bars show the same numbers. The truncated red chart screams "huge jump"; the honest green chart shows the small real difference. Same data, opposite impression.

Key idea: The most common way to mislead is a truncated axis that breaks the link between bar length and value.

Principles for honest graphics

1. Start bar axes at zero so bar length is proportional to value.
2. Label everything: title, both axes with units, and a legend when needed.
3. Keep it simple: remove clutter so the data stands out (an idea often called maximizing the data-to-ink ratio, the share of a chart's ink that shows real data rather than decoration).
4. Do not distort: avoid 3-D, dual axes chosen to fake a correlation, and misleading area scaling.
5. Show the right comparison: the chart should answer the actual question at a glance.

When in doubt, ask a colleague what they conclude from your chart before you add any words. If they read it wrong, the chart, not the reader, needs fixing.

Key idea: Start bar axes at zero, label clearly, cut clutter, avoid distortion, and show the comparison that answers the question.

A quick worked diagnosis

A bar chart of monthly revenue starts its y-axis at $48,000 and shows bars for $49,000 and $50,000 that look like one is nearly double the other. The real difference is (50000 - 49000) / 49000 = 0.02, about 2%, but the truncated axis makes it look huge. The fix is to start the y-axis at $0 so bar height is proportional to revenue, and the tiny real gap becomes visible.

Key idea: Compute the real percentage difference and make the axis reflect it honestly.

Where people get stuck

• "A fancier chart is a better chart." 3-D effects and heavy decoration usually hide the data; simple and clear wins.
• "Pie charts are best for comparisons." With more than a few slices a bar chart ranks categories far more clearly, because length is read more accurately than angle.
• "Any starting point for the axis is fine." For bar charts the value axis must start at zero, or bar length no longer matches value.
• "Every chart must start at zero." The opposite error. Position encodings such as scatter and line charts do not require it, and forcing zero can flatten the pattern into a useless band.
• "A line chart works for any categories." Lines imply order or continuity, so they suit time, not unordered categories.
• Accepting the default histogram bins. The shape you are interpreting is partly a consequence of a number you never chose. Try several.
• Using a log axis without saying so. A reader assuming a linear scale will misjudge every gap on the chart, usually by a factor of ten.
• Relying on red versus green. About one man in twelve cannot reliably distinguish them. Add labels or shapes so color is never the only signal.
• Plotting a summary when the raw points would fit. Two bars showing two group means hide the spread, the sample size, and any outlier. With small data, plot the points.

Recap

• Match the chart to the question: distribution, comparison, relationship, or time.
• People read position most accurately, then length, then angle and area, then color intensity, which explains most chart advice.
• Start bar-chart value axes at zero so length is proportional to value; position encodings such as scatter and line charts do not need it.
• A histogram's shape depends on the bin width, so try several before believing any structure.
• Watch for truncated axes, wrong chart types, and overplotting or clutter.
• Label the title with the takeaway, the axes with units, and any log scale unmistakably.
• Maximize the data-to-ink ratio, and never let color alone carry a distinction.

Sources

1. Wickham, H., Cetinkaya-Rundel, M., & Grolemund, G. (2023). Data visualization. In R for data science (2nd ed., ch. 1). O'Reilly. r4ds.hadley.nz
2. Wickham, H., Cetinkaya-Rundel, M., & Grolemund, G. (2023). Communication. In R for data science (2nd ed., ch. 11). O'Reilly. r4ds.hadley.nz
3. Matplotlib development team. (n.d.). Quick start guide: figures, axes, labels, and limits. Matplotlib documentation. matplotlib.org
4. Waskom, M. (n.d.). Visualizing distributions of data: histograms, bin width, and density estimates. seaborn documentation. seaborn.pydata.org
5. VanderPlas, J. (2016). Visualization with Matplotlib. In Python data science handbook (ch. 4). O'Reilly. jakevdp.github.io
6. Tufte, E. R. (2001). The visual display of quantitative information (2nd ed.). Graphics Press. find source ↗
7. Cairo, A. (2016). The truthful art: Data, charts, and maps for communication. New Riders. find source ↗

Four completely different datasets appear later in this lesson. One is a sensible cloud of points, one is a perfect curve, one is a straight line ruined by a single outlier, and one is eleven points stacked in a column. All four have the same mean, the same standard deviation, the same correlation of 0.816, and the same fitted line. Any of them reported as "r = 0.82" would be true and three of them would be a lie.

The big picture

Correlation puts a single number on how tightly two quantitative variables move together in a straight line. It is enormously useful and enormously easy to misread, because two things rising together does not mean one causes the other. Correlation is like noticing that two dancers move in sync: informative, but it does not tell you who is leading, or whether the music is moving both.

Correlation measures the strength and direction of a straight-line relationship between two quantitative variables. The most common measure is the Pearson correlation coefficient, written r, which always lies between -1 and +1.

Key idea: Correlation (r, from -1 to +1) summarizes how strongly and in which direction two variables move together in a line.

Reading r

• Sign: positive r means the variables rise together; negative r means one rises as the other falls.
• Size: r near +1 or -1 is a tight linear pattern; r near 0 means little or no linear relationship.
• Rough guide: |r| about 0.1 is weak, 0.5 is moderate, and 0.8 or more is strong. These are conventions, not laws.

Key idea: The sign of r gives the direction and the size gives the strength of the linear pattern.

Computing r on study.csv

The formula is less forbidding than it looks. For each variable, take the deviation of every value from its own mean. Then

            sum of (x deviation) times (y deviation)
    r = -------------------------------------------------
        sqrt( sum of x deviations squared  times  sum of y deviations squared )

Work it on the nine cleaned rows. Mean hours is 6.00 and mean score is 77.78:

hours  score   x-dev    y-dev     product    x-dev^2    y-dev^2
    2     52    -4.00   -25.78     103.11      16.00     664.50
    3     63    -3.00   -14.78      44.33       9.00     218.38
    4     66    -2.00   -11.78      23.56       4.00     138.72
    5     78    -1.00     0.22      -0.22       1.00       0.05
    6     74     0.00    -3.78       0.00       0.00      14.27
    7     88     1.00    10.22      10.22       1.00     104.49
    8     85     2.00     7.22      14.44       4.00      52.16
    9     96     3.00    18.22      54.67       9.00     332.05
   10     98     4.00    20.22      80.89      16.00     408.94
                                  --------    -------   --------
                          sums:    331.00      60.00    1933.56

    r = 331.00 / sqrt(60.00 * 1933.56)
      = 331.00 / sqrt(116013.3)
      = 331.00 / 340.61
      = 0.972

>>> df["hours"].corr(df["score"])
0.9718

An r of 0.97 is very tight, which should make you suspicious rather than pleased: with nine points from a teaching dataset, that is a designed relationship, not a discovered one. Real educational data on study time and grades typically shows something between 0.2 and 0.4, because study time is one of dozens of things that matter.

Key idea: r is the sum of paired deviations divided by the square root of the product of the summed squared deviations, and on study.csv it works out to 0.972.

Anscombe's quartet: why r is never enough

In 1973 the statistician Frank Anscombe built four small datasets that share almost every summary statistic:

                        set I     set II    set III   set IV
  mean of x              9.00      9.00      9.00      9.00
  mean of y              7.50      7.50      7.50      7.50
  correlation r          0.816     0.816     0.816     0.817
  fitted line       y = 3.00 + 0.500x   (identical for all four)
  R-squared              0.67      0.67      0.67      0.67

And here is what they actually look like:

  I    a plausible scatter around a rising line          the line is a fair summary
  II   a smooth arch (a parabola)                        the relationship is real
                                                         but a straight line is wrong
  III  ten points on a perfect straight line, plus       one outlier is tilting the
       one far-off point                                 entire fitted line
  IV   ten points stacked at x = 8, plus one point       a single observation is
       at x = 19                                         creating the whole result

Reporting "r = 0.82" would be arithmetically correct for all four and honest for exactly one. Set II needs a curve, set III needs its outlier investigated, and set IV has effectively one data point doing all the work. This is the single strongest argument for the rule that follows every correlation in this course: plot it before you believe it. A scatterplot takes four seconds and catches all three failures instantly.

Key idea: Anscombe's four datasets share their mean, standard deviation, correlation, and fitted line while looking completely different, so a summary statistic never substitutes for the scatterplot.

Correlation is not causation

This is the single most important warning in data science. A strong correlation between two variables does not prove that one causes the other. Ice cream sales and drowning deaths rise together, but neither causes the other; a hidden third variable, hot weather, drives both. Such a hidden cause is a confounder, a lurking variable that influences both things you measured. Establishing causation generally requires a controlled experiment (randomly assigning who gets the treatment), not just observed correlation.

It helps to have the full list in your head, because "correlation is not causation" is a slogan and this is a checklist. If A and B are correlated, exactly four kinds of explanation are available:

1. A causes B                     studying raises the score
2. B causes A (reverse)           students who find the material easy
                                  enjoy it, so they study more
3. C causes both (confounding)    conscientiousness raises both study
                                  time and score
4. selection or coincidence       the sample was chosen in a way that
                                  creates the pattern, or with n = 9 you
                                  simply got a striking draw

Run study.csv's r = 0.972 through that list and all four remain live. Nothing in the data distinguishes them, and no larger sample would distinguish them either; more data would only make the correlation more precisely estimated, not more causal. What distinguishes them is design: randomly assigning students to study more would rule out 2, 3, and 4 at once, which is exactly why randomized experiments are the gold standard.

Key idea: Correlation alone never proves causation, because reverse causation, a confounder, or selection can each produce it, and only the study design - not more data - can rule them out.

What correlation misses

Pearson's r only captures linear association. A perfect U-shaped relationship can have r near 0 even though the variables are tightly related, just not in a straight line. A single outlier can also inflate or hide a correlation. The lesson: always plot the scatterplot before trusting a single r value. The number summarizes; the picture protects you from being fooled.

df["hours"].corr(df["score"])   # Pearson r between two columns
df.corr(numeric_only=True)      # correlation matrix of all numeric columns

Worked reading. Suppose study hours and exam score have r = 0.78. That is a strong positive linear relationship: students who study more tend to score higher, and a straight line describes the trend well. It does not prove studying causes the gain (motivated students may both study more and score higher), and it says nothing about students far outside the observed range of hours.

When the relationship is clearly monotone but not straight, Spearman's rank correlation is the right tool. It simply replaces each value by its rank and computes Pearson r on the ranks, which makes it insensitive to curvature and to outliers:

hours:  2  3  4  5  6  7  8  9 10      ranks: 1 2 3 4 5 6 7 8 9
score: 52 63 66 78 74 88 85 96 98      ranks: 1 2 3 5 4 7 6 8 9

Spearman = Pearson on those two rank lists = 58 / 60 = 0.967

df["hours"].corr(df["score"], method="spearman")   # 0.9667
df["hours"].corr(df["score"])                       # 0.9718 (Pearson, the default)

The two are close here because the relationship really is close to a straight line. On a curved-but-always-rising relationship Spearman would be near 1 while Pearson dropped well below it, and a large gap between the two is itself a useful signal that the relationship is monotone but not linear.

One last distinction, constantly confused: r is not an effect size. It says how tightly the points hug a line, not how steep that line is. A correlation of 0.95 between hours studied and score is compatible with each hour adding 5 points or adding 0.05 points. Tightness and magnitude are different questions, and the second one is what regression answers in the next lesson.

Key idea: r measures only straight-line tightness, Spearman handles monotone curves, and neither tells you how much y changes per unit of x.

Where people get stuck

• "A strong correlation proves cause." It does not; reverse causation, a confounder, or selection can produce a strong r with no causal link from A to B.
• "r near 0 means the variables are unrelated." It means no straight-line relationship; a strong curve like a U can still have r near 0.
• "A big r means a big effect." r measures tightness of a linear pattern, not the size of the slope or the practical impact.
• "You can trust r without a plot." Anscombe's quartet settles this: four different pictures, one correlation.
• Collecting more data to establish causation. A larger sample gives a more precise correlation and no more causal warrant. Only the design changes that.
• Reading a correlation matrix as a list of findings. A matrix of ten variables holds 45 correlations, so a couple will look impressive by chance alone. Decide what you are testing before you look.
• Correlating on a restricted range. If you only observe students who studied between 5 and 6 hours, r collapses toward zero even when the underlying relationship is strong. Range restriction shrinks correlations.
• Computing r on categories coded as numbers. Program coded as 1, 2, 3 has no meaningful mean, so Pearson r on it is arithmetic without meaning.

Recap

• Pearson r ranges from -1 to +1 and measures linear association; on study.csv it is 0.972.
• The sign shows direction; the magnitude shows strength, and r is not an effect size.
• Anscombe's quartet shares mean, spread, r, and fitted line across four completely different shapes.
• Correlation is not causation: A causes B, B causes A, C causes both, or selection and chance.
• r captures only linear patterns, so a U-shape can give r near 0; Spearman handles monotone curves.
• Range restriction shrinks correlations, and a large correlation matrix guarantees some impressive-looking noise.
• Always plot the scatterplot before trusting a single r.

Sources

1. Diez, D., Cetinkaya-Rundel, M., & Barr, C. D. (2019). Linear regression with a single predictor: correlation. In OpenIntro statistics (4th ed., ch. 8). OpenIntro. openintro.org
2. Cetinkaya-Rundel, M., & Hardin, J. (2021). Exploring numerical data: scatterplots and correlation. In Introduction to modern statistics. OpenIntro. openintro.org
3. Kievit, R. A., Frankenhuis, W. E., Waldorp, L. J., & Borsboom, D. (2013). Simpson's paradox in psychological science: A practical guide. Frontiers in Psychology, 4, 513. pmc.ncbi.nlm.nih.gov
4. Wickham, H., Cetinkaya-Rundel, M., & Grolemund, G. (2023). Exploratory data analysis: covariation. In R for data science (2nd ed., ch. 10). O'Reilly. r4ds.hadley.nz
5. pandas development team. (n.d.). Essential basic functionality: Series.corr and DataFrame.corr, including the spearman method. pandas user guide. pandas.pydata.org
6. Anscombe, F. J. (1973). Graphs in statistical analysis. The American Statistician, 27(1), 17-21. find source ↗
7. VanderPlas, J. (2016). Visualization with Matplotlib: scatter plots. In Python data science handbook (ch. 4). O'Reilly. jakevdp.github.io

The line fitted to study.csv predicts that a student who studies 40 hours will score 265 out of 100. The model is not broken; the arithmetic is exactly right. It is simply being asked a question about a world it has never seen, and it answers with total confidence because a straight line has no way to say "I do not know". Knowing where a model stops being trustworthy is most of what it means to use one well.

The big picture

Correlation tells you two variables move together; regression draws the actual line so you can turn that relationship into a prediction. With one predictor, the whole model is a line you can read like a recipe: a starting value plus a fixed amount per unit. That simplicity is its power and its limit, so you also learn where a line can lead you astray.

Correlation says how tightly two variables track. Linear regression goes further: it fits an actual line so you can predict one variable from another. With one predictor it is called simple linear regression.

Key idea: Regression fits a line so you can predict a response from a predictor, not just measure their association.

The line

The fitted model has the form y = b0 + b1 * x, where y is the predicted response, x is the predictor, b0 is the intercept (predicted y when x = 0), and b1 is the slope (the change in y for a one-unit increase in x). Think of the intercept as your starting point and the slope as the steady step you take for each extra unit of x, like a taxi fare with a base charge plus a rate per mile. The slope is where the meaning usually lives.

Key idea: In y = b0 + b1 * x, the intercept is the starting value and the slope is the change in y per one-unit rise in x.

Worked example

A model predicting exam score from hours studied is score = 50 + 5 * hours. Read it directly: a student who studies 0 hours is predicted to score 50 (the intercept), and each additional hour adds 5 points (the slope). Predict the score for 6 hours: 50 + 5 * 6 = 50 + 30 = 80. Predict for 3 hours: 50 + 5 * 3 = 65. The line turns a relationship into concrete forecasts.

Key idea: Plug an x value into the fitted equation to get a concrete predicted y.

How the line is chosen: least squares

For each data point, the residual is the vertical gap between the actual y and the line's prediction (actual minus predicted). The least squares method picks the one line that makes the sum of the squared residuals as small as possible. Squaring keeps positive and negative errors from cancelling and penalizes big misses more, the way a test that docks extra for large mistakes discourages wild guesses. The result is the unique best-fitting straight line.

Key idea: Least squares chooses the line that minimizes the total of squared vertical gaps between points and the line.

Fitting the line to study.csv

The least-squares line has a closed-form answer, and it uses exactly the sums computed in the previous lesson:

slope      b1 = sum of (x-dev)(y-dev)  /  sum of (x-dev)^2   =  331.00 / 60.00  =  5.5167
intercept  b0 = mean of y - b1 * mean of x  =  77.78 - 5.5167 * 6  =  44.68

    score = 44.68 + 5.52 * hours

import numpy as np
b1, b0 = np.polyfit(df["hours"], df["score"], 1)
round(b1, 4), round(b0, 4)      # (5.5167, 44.6778)

Read the two numbers in context. The slope says each additional hour of study is associated with about 5.5 more points. The intercept says a student who studied zero hours is predicted to score about 44.7, which happens to be plausible here (a student who did no revision still knows something), though zero hours is outside the observed range of 2 to 10 and so is already a mild extrapolation.

Note the deliberate phrase "is associated with". The slope is a description of this dataset, not a promise about what would happen if you made a student study an extra hour. Everything Lesson 8 said about causation still applies; regression changes the arithmetic, not the epistemology.

Key idea: The least-squares slope is the paired-deviation sum divided by the squared x-deviation sum, giving score = 44.68 + 5.52 x hours, and the slope describes an association rather than an effect.

Every residual, in full

Now check the fit point by point. The residual is actual minus predicted:

hours   actual   predicted   residual
    2       52       55.71      -3.71
    3       63       61.23      +1.77
    4       66       66.74      -0.74
    5       78       72.26      +5.74     <- the largest miss
    6       74       77.78      -3.78
    7       88       83.29      +4.71
    8       85       88.81      -3.81
    9       96       94.33      +1.67
   10       98       99.84      -1.84
                              --------
                     sum of residuals:  0.00

The sum of residuals is exactly zero, and that is not a coincidence: it is a mathematical property of any least-squares line with an intercept. It is also why you can never assess a fit by adding the residuals up; the positives and negatives are guaranteed to cancel. That is precisely why the method squares them.

The squared residuals give the numbers that matter:

SSE  = sum of squared residuals               =  107.54     unexplained
SSR  = explained sum of squares               = 1826.02     explained
SST  = total variation in score (= Syy)       = 1933.56     total

     SSR + SSE = 1826.02 + 107.54 = 1933.56  = SST      the budget always balances

R-squared = SSR / SST = 1826.02 / 1933.56 = 0.944

residual standard error = sqrt(SSE / (n - 2)) = sqrt(107.54 / 7) = 3.92

That last number is the most useful and the least reported. R-squared is a unitless proportion; the residual standard error is in the units of the response, so it says the line's typical miss is about 3.9 points on the exam. When someone asks "how good is the model?", "it is usually within about 4 points" is a far more useful answer than "R-squared is 0.94".

Key idea: Residuals always sum to zero, the squared residuals split total variation into explained and unexplained, and the residual standard error reports the typical miss in the units people care about.

Reading the residual plot

Plotting residuals against the predicted values is the standard diagnostic, and the thing you are hoping to see is nothing: a shapeless band centred on zero. Any structure means the line is missing something.

pattern in the residual plot     what it means                  what to do
------------------------------   ----------------------------   ----------------------
shapeless band around zero       the line is adequate           nothing
a clear curve (U or arch)        the relationship is not        add a squared term or
                                 linear                          transform the variable
a widening funnel                the spread of y grows with x   transform y (often log)
one point far from the rest      an outlier or influential      investigate it; check
                                 observation                     the fit with and without

pred = b0 + b1 * df["hours"]
ax.scatter(pred, df["score"] - pred)
ax.axhline(0)

On study.csv the residuals run -3.71, +1.77, -0.74, +5.74, -3.78, +4.71, -3.81, +1.67, -1.84: no trend, no funnel, largest miss 5.74. With nine points that is about as much as can honestly be said.

Key idea: A good residual plot has no pattern at all; a curve means the model is wrong, a funnel means the spread changes, and a lone far point deserves investigation.

How good is the fit: R-squared

R-squared (the coefficient of determination) is the fraction of the variation in y that the line explains, from 0 to 1. An R-squared of 0.64 means the model explains 64% of the variability in the response; the remaining 36% is unexplained scatter. For simple regression, R-squared is exactly the square of the correlation r, so r = 0.8 gives R-squared = 0.64.

import numpy as np
b1, b0 = np.polyfit(df["hours"], df["score"], 1)  # slope, intercept
pred = b0 + b1 * 6                                 # predict for 6 hours

Key idea: R-squared is the share of variation the model explains, and for simple regression it equals r squared.

Cautions

Two warnings. First, do not extrapolate far beyond the data you fit; the line may be nonsense outside the observed range. Second, a line is only appropriate if the relationship is roughly linear, so plot the data and the residuals first. Regression is powerful precisely because it is simple, but that simplicity is also its limit.

Make the first warning concrete on our own fitted line. It was built from students who studied between 2 and 10 hours:

hours     prediction        verdict
    5          72.26        inside the observed range - fine
   10          99.84        at the very edge - already strained
   12         110.88        above the maximum possible score
   40         265.28        arithmetically correct, physically absurd

Nothing in the model objects. It has no concept of a 100-point ceiling, no concept of diminishing returns, and no way to represent uncertainty about a region it never saw. Every model does this: it will answer any question you ask in the same confident tone, whether or not it has grounds to. Guarding against it is entirely your job, and the simple guard is to record the range of every predictor you fitted on and refuse predictions outside it.

Key idea: Do not extrapolate beyond your data, use a line only when the relationship is roughly linear, and record the fitted range so out-of-range predictions can be refused rather than trusted.

Where people get stuck

• "The intercept is always meaningful." If x = 0 is far outside the data, the intercept is an extrapolation dressed up as a parameter.
• "A high R-squared proves the model is correct or causal." R-squared measures fit, not causation, and Anscombe's set II has R-squared 0.67 while being fitted by entirely the wrong shape.
• "Least squares minimizes the raw errors." It minimizes the sum of the squared residuals, which is why large misses are penalized more and why raw residuals always sum to zero.
• "You can predict any x with the line." Extrapolating gives 265 out of 100 for 40 hours of study, stated with complete confidence.
• Reporting R-squared and nothing else. The residual standard error, 3.92 points here, tells a reader far more about whether the model is useful.
• Skipping the residual plot. It is the one diagnostic that reveals a wrong functional form, and it takes two lines of code.
• Saying "the slope is the effect of studying". It is the association observed in this sample. Interventional language requires an interventional design.
• Chasing a higher R-squared by adding predictors. R-squared never decreases when you add a variable, even a column of random numbers, which is why multiple regression needs adjusted R-squared or held-out testing.

Recap

• Simple linear regression fits y = b0 + b1 * x to predict a response.
• The intercept b0 is y when x = 0; the slope b1 is the change in y per unit x; for study.csv the fit is score = 44.68 + 5.52 x hours.
• Least squares minimizes the sum of squared residuals, and those residuals always sum to exactly zero.
• SST splits into SSR plus SSE, so R-squared is the fraction of variation explained and equals r squared for simple regression: 0.944 here.
• The residual standard error, 3.92 points, states the typical miss in the response's own units.
• A residual plot should show no pattern; a curve, a funnel, or a lone far point each mean something specific.
• Avoid extrapolation, check that a line fits before trusting it, and describe the slope as an association.

Sources

1. Diez, D., Cetinkaya-Rundel, M., & Barr, C. D. (2019). Linear regression with a single predictor. In OpenIntro statistics (4th ed., ch. 8). OpenIntro. openintro.org
2. Cetinkaya-Rundel, M., & Hardin, J. (2021). Linear regression with a single predictor. In Introduction to modern statistics. OpenIntro. openintro.org
3. scikit-learn developers. (n.d.). Linear models: ordinary least squares. scikit-learn user guide. scikit-learn.org
4. NumPy developers. (n.d.). numpy.polyfit. NumPy manual. numpy.org
5. VanderPlas, J. (2016). In depth: linear regression. In Python data science handbook (ch. 5). O'Reilly. jakevdp.github.io
6. Anscombe, F. J. (1973). Graphs in statistical analysis. The American Statistician, 27(1), 17-21. find source ↗
7. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). Linear regression. In An introduction to statistical learning (2nd ed., ch. 3). Springer. find source ↗

Here is a model you can build in one line and that will be 97% accurate on real data: predict that no student ever fails. It will also never identify a single struggling student, which was the only reason anyone wanted it. Accuracy is the metric everyone reaches for first and the one most likely to hide a model that does nothing. This lesson gives you the arithmetic that exposes it.

The big picture

Sometimes the thing you want to predict is a label, not a number: spam or not, sick or healthy, will churn or stay. That is classification, and it needs its own way of measuring success, because plain accuracy can hide a model that fails at the one job that matters. This lesson gives you a simple classifier and the scorecard, called a confusion matrix, that keeps it honest.

So far the response we predicted (a score, a price) was a number, which is regression. When the response is a category instead (spam or not, disease or healthy, will churn or stay), the task is classification. The goal shifts from predicting a value to predicting a label.

Key idea: Classification predicts a category label, so it needs different tools and scores than numeric regression.

A simple classifier: k-nearest neighbors

One of the most intuitive classifiers is k-nearest neighbors (KNN). To classify a new point, it finds the k training points closest to it (by distance in the feature space) and takes a majority vote of their labels. It works on the everyday logic of "you resemble your neighbors": ask the five most similar known cases and go with the majority. If k = 5 and the five nearest known emails are 4 spam and 1 not-spam, KNN labels the new email spam. Choosing k matters: too small and the model chases noise; too large and it blurs real boundaries.

Key idea: KNN labels a new point by a majority vote of its k most similar known examples.

KNN's hidden requirement: scale your features

"Closest" means a distance, and a distance adds up differences across every feature. If one feature is measured in much larger numbers than another, it silently drowns the rest. Take two students from study.csv:

student A:  hours = 2,  score = 52
student B:  hours = 10, score = 98

raw distance = sqrt( (10 - 2)^2 + (98 - 52)^2 )
             = sqrt( 64 + 2116 )
             = sqrt( 2180 ) = 46.69

the score difference contributes 2116 of the 2180, which is 97% of the distance

The hours feature is contributing about 3% of the answer, not because it matters less but because it is measured in smaller numbers. Convert both to z-scores first, exactly as in Lesson 6, and the picture changes completely:

z(hours):  (2 - 6) / 2.74 = -1.46      (10 - 6) / 2.74 = 1.46      difference 2.92
z(score):  (52 - 77.78) / 15.55 = -1.66  (98 - 77.78) / 15.55 = 1.30  difference 2.96

scaled distance = sqrt( 2.92^2 + 2.96^2 ) = sqrt(17.28) = 4.16
now hours contributes 49% and score 51%

from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline

model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))

Using a pipeline rather than scaling by hand matters for a reason Lesson 11 will make precise: the scaler must learn its mean and standard deviation from the training data only. Every distance-based method - KNN, k-means, and anything with a regularization penalty - carries this requirement. Tree-based methods do not, because they only ever compare a feature to a threshold within itself.

Key idea: KNN measures distance across features, so an unscaled large-numbered feature dominates the vote; standardize first, and do it inside a pipeline.

Judging a classifier: the confusion matrix

For a two-class problem (call them positive and negative), every prediction falls into one of four boxes:

	Predicted positive	Predicted negative
Actually positive	True Positive (TP)	False Negative (FN)
Actually negative	False Positive (FP)	True Negative (TN)

This 2-by-2 table is the confusion matrix. Think of it as a report card that separates the two kinds of mistakes: crying wolf when there is none (a false positive) versus missing a real wolf (a false negative). From it come the core metrics:

• Accuracy = (TP + TN) / (all predictions): the fraction correct overall.
• Precision = TP / (TP + FP): of those predicted positive, how many really were.
• Recall = TP / (TP + FN): of the actual positives, how many were caught.

Key idea: The confusion matrix splits results into TP, FP, FN, and TN, and accuracy, precision, and recall summarize it.

Worked example

A spam filter is tested on 100 emails: TP = 40 (spam caught), TN = 50 (real mail kept), FP = 5 (real mail wrongly flagged), FN = 5 (spam that slipped through). Then accuracy = (40 + 50) / 100 = 90 / 100 = 0.90, or 90%. Precision = 40 / (40 + 5) = 40 / 45 = 0.89, and recall = 40 / (40 + 5) = 40 / 45 = 0.89. Different situations weigh these differently: a cancer screen prizes recall (do not miss a real case), while a spam filter that must never trash real mail prizes precision.

Precision and recall came out equal there only because FP and FN happened to both be 5; the two denominators became the same number by coincidence. Change one value and they separate immediately:

TP = 40   TN = 45   FP = 10   FN = 5        (still 100 emails)

accuracy  = (40 + 45) / 100      = 0.85
precision = 40 / (40 + 10) = 40/50 = 0.80    of the mail we flagged, 80% really was spam
recall    = 40 / (40 +  5) = 40/45 = 0.89    of the spam that existed, we caught 89%

Read the two sentences on the right, because they are what the numbers mean. Precision looks along the predicted-positive row and asks how many of your alarms were real. Recall looks along the actually-positive row and asks how much of the real thing you found. They answer different questions and they move in opposite directions.

Key idea: Which metric matters depends on which mistake is worse: missing positives (recall) or false alarms (precision), and they are equal only by coincidence.

Why accuracy alone can lie: the arithmetic

If only 1% of emails are spam, a lazy model that predicts "not spam" for everything scores 99% accuracy while catching zero spam. This is the imbalanced classes trap, where one label is far rarer than the other. Work it all the way through, because the numbers are worse than the slogan suggests.

A cohort of 1000 students, of whom 30 actually fail: a base rate of 3%. First the lazy model that predicts "passes" for everyone:

                     predicted fail   predicted pass
  actually fails            0               30          <- all 30 missed
  actually passes           0              970

  accuracy  = (0 + 970) / 1000 = 0.970      97% accurate
  recall    = 0 / (0 + 30)     = 0.000      catches nobody
  precision = 0 / (0 + 0)      = undefined  it never made a positive prediction

Ninety-seven percent accurate and completely worthless. Now a genuine model that actually tries:

                     predicted fail   predicted pass
  actually fails           18               12
  actually passes          60              910

  accuracy  = (18 + 910) / 1000  = 0.928
  precision = 18 / (18 + 60) = 18/78 = 0.231
  recall    = 18 / (18 + 12) = 18/30 = 0.600
  F1        = 2 * TP / (2*TP + FP + FN) = 36 / 108 = 0.333

The useful model has lower accuracy than the useless one: 92.8% against 97.0%. Anyone ranking these two models by accuracy would ship the one that helps nobody. Precision and recall tell the real story: this model finds 60% of the students who are heading for a fail, which is the entire point, at the cost of some false alarms.

Key idea: With a 3% base rate a do-nothing model scores 97% accuracy and a genuinely useful model scores 92.8%, so accuracy actively ranks them the wrong way round.

Base rates: why most of your alarms are wrong

Look again at that precision of 0.231. It means that when the model flags a student, there is only a 23% chance they will actually fail. More than three quarters of the flags are false alarms, from a model with respectable 60% recall. Nothing is broken; this is arithmetic.

The reason is the base rate. Failures are 3% of the cohort, so there are 970 chances to raise a false alarm and only 30 chances to raise a true one. Even a fairly picky model that wrongly flags just 6% of the passing students produces 60 false alarms, which swamps the 18 correct ones.

  30 actual failures     -> model catches 18       (true alarms)
 970 actual passes       -> model wrongly flags 60 (false alarms)
                            -------------------
                            78 alarms in total, of which 18 are right = 23%

This is the most under-appreciated fact about rare-event prediction, and it recurs in medical screening, fraud detection, and security alerts alike: a test can rarely be wrong about any individual case and still produce mostly false positives, purely because the thing it looks for is rare. Whenever someone quotes a detection rate, ask what fraction of the population actually has the condition. Without that number the rate cannot be interpreted at all.

Key idea: When positives are rare, the enormous pool of negatives generates most of the alarms, so a model with good recall can still be wrong about the large majority of the cases it flags.

Trading precision against recall

Most classifiers do not really output a label. They output a probability, and the label comes from comparing it to a threshold, which defaults to 0.5 for no deep reason. Moving the threshold moves both metrics, always in opposite directions:

threshold   behaviour                          precision   recall
   0.9      only flags near-certain cases        high        low
   0.5      the default                          middling    middling
   0.1      flags anything remotely suspicious   low         high

So "the model's precision is 0.23" is incomplete; it is the precision at that threshold. The threshold is a policy decision, not a statistical one, set by whichever error costs more: a screen feeding a cheap harmless follow-up should sit low and accept false alarms, while a model triggering an expensive intervention should sit high. The F1 score, 2 * TP / (2*TP + FP + FN), is a reasonable single number when both errors matter about equally (0.333 above), but it is no substitute for stating both.

Key idea: Precision and recall are set by a threshold you choose, so report both plus the threshold, and let the relative cost of the two errors decide where it sits.

Where people get stuck

• "High accuracy always means a good classifier." With imbalanced classes, always predicting the majority can score high accuracy while catching none of the rare cases.
• "Precision and recall are the same thing." Precision is about false alarms among predicted positives; recall is about missed positives among actual positives.
• "Bigger k in KNN is always better." Too large a k blurs real boundaries, just as too small a k chases noise; k must be tuned.
• "Classification and regression use the same success measures." Regression uses errors like residuals; classification uses the confusion matrix and its metrics.
• Comparing models by accuracy on imbalanced data. The useless model scored 97% and the useful one 92.8%. Accuracy ranked them backwards.
• Forgetting the base rate when reading a detection rate. A model with 60% recall can still be wrong about 77% of the cases it flags. Without the prevalence, no detection rate can be interpreted.
• Running KNN on unscaled features. The feature with the biggest numbers silently decides every vote, whether or not it is the informative one.
• Treating the 0.5 threshold as given. It is a default, not a finding. Set it from the relative cost of a false alarm against a miss.
• Reporting one number for a classifier. Give the confusion matrix, or at minimum precision, recall, and the base rate; any single metric hides a specific failure mode.

Recap

• Classification predicts a category; regression predicts a number.
• KNN classifies a point by a majority vote of its k nearest labeled neighbors, and requires standardized features because it uses distance.
• The confusion matrix holds TP, FP, FN, and TN, and every metric is a ratio drawn from it.
• Accuracy = (TP + TN) / total; precision = TP / (TP + FP); recall = TP / (TP + FN); F1 = 2TP / (2TP + FP + FN).
• With a 3% base rate a do-nothing model scores 97% accuracy while a genuinely useful one scores 92.8%, so accuracy ranks them wrongly.
• When positives are rare, most alarms are false even from a good model: precision 0.231 at recall 0.600.
• Precision and recall trade off through a threshold you choose, so report both together with the base rate.

Sources

1. scikit-learn developers. (n.d.). Metrics and scoring: quantifying the quality of predictions: accuracy, precision, recall, and F1. scikit-learn user guide. scikit-learn.org
2. scikit-learn developers. (n.d.). Nearest neighbors: KNeighborsClassifier and distance metrics. scikit-learn user guide. scikit-learn.org
3. scikit-learn developers. (n.d.). Evaluate the performance of a classifier with a confusion matrix. scikit-learn examples. scikit-learn.org
4. VanderPlas, J. (2016). Introducing scikit-learn. In Python data science handbook (ch. 5). O'Reilly. jakevdp.github.io
5. Cetinkaya-Rundel, M., & Hardin, J. (2021). Probability and conditional probability: two-way tables and base rates. In Introduction to modern statistics. OpenIntro. openintro.org
6. Diez, D., Cetinkaya-Rundel, M., & Barr, C. D. (2019). Conditional probability and Bayes' theorem. In OpenIntro statistics (4th ed., ch. 3). OpenIntro. openintro.org
7. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). Classification. In An introduction to statistical learning (2nd ed., ch. 4). Springer. find source ↗

A 1-nearest-neighbor model gets 100% of its training data right, every single time, on any dataset you give it. Not because it is good, but because each training point's closest neighbour is itself. That perfect score is a mathematical certainty and it carries exactly zero information about how the model will behave on anything new. Everything in this lesson exists to stop numbers like that from fooling you.

The big picture

A model that has already seen the answers can look brilliant and still be useless on anything new. The core discipline of prediction is to measure performance on data the model never learned from, because that is the only honest estimate of real-world behavior. This is why we hold data back, and why we watch for a model that memorizes instead of learning.

A model that has already seen the answers can look brilliant and still be useless. The core discipline of predictive modeling is to measure performance on data the model did not learn from, because that is the only honest estimate of how it will do in the real world.

Key idea: Judge a model on data it has never seen, or its score is not honest.

The train/test split

Split your data into two parts, typically about 80% for training and 20% for testing. Fit the model on the training set only, then measure its accuracy on the untouched test set. Because the test rows were hidden during learning, their score estimates generalization, the model's performance on new cases. Grading a model on its training data is like grading students on the exact homework questions they were handed with the answers attached: a great score that proves nothing about a real exam.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)   # 80% train, 20% test
model.fit(X_train, y_train)                # learn only from training data
model.score(X_test, y_test)                # honest score on unseen data

Key idea: Fit on the training set and score on the untouched test set to estimate generalization.

Overfitting and underfitting

Overfitting happens when a model learns the training data too well, memorizing its noise and quirks instead of the real pattern, like a student who memorizes the answer key rather than understanding the material and then fails a reworded question. The telltale sign is high training accuracy but poor test accuracy. Underfitting is the opposite: the model is too simple to capture the real structure, so it does poorly on both training and test data. The goal is the middle ground, a model complex enough to learn the signal but not so complex it chases noise.

The blue training-error curve always falls as the model grows more flexible. The red test-error curve falls at first, then rises once the model starts memorizing noise. The green line marks the sweet spot: the complexity where test error is lowest.

Key idea: Overfitting shows a large train-test gap; underfitting does poorly on both; the best model sits between.

The curve, with actual numbers

Fit a KNN classifier to 1000 students, 800 for training and 200 held out, and sweep the one knob it has. In KNN, small k means high complexity, because the decision boundary can wiggle around individual points:

   k     train accuracy   test accuracy   gap     diagnosis
   1          1.000            0.71       0.29    severe overfitting
   5          0.880            0.79       0.09    still overfitting
  15          0.820            0.81       0.01    about right  <- best test score
  51          0.750            0.74       0.01    starting to underfit
 201          0.680            0.68       0.00    badly underfitting

Three things in that table are worth pausing on.

The k = 1 row is exactly 1.000 and always will be. Each training point's nearest neighbour is itself, so it votes for its own label and is never wrong. A perfect training score here proves nothing at all; it is a property of the algorithm, not evidence about the model.

The gap column is the overfitting detector. It shrinks steadily as complexity falls, from 0.29 to essentially nothing. A large gap always means the model learned things about the training rows specifically.

And the bottom rows have a tiny gap and a bad score, which is the signature of underfitting. That is why the gap alone is not enough: k = 201 is as well-calibrated as k = 15 and 13 points worse. You want a small gap and a high test score, which happens at k = 15.

Key idea: Track the train score, the test score, and the gap between them - a large gap means overfitting, and a small gap with a low score means underfitting.

Data leakage: the mistake that looks like success

A held-out test set only measures generalization if the model genuinely never saw those rows, and "saw" is subtler than it sounds. Data leakage is any way information from the test set, or from the future, sneaks into training. Its signature is a test score that is suspiciously good.

The most common form is preprocessing before splitting:

# WRONG - the scaler computed its mean and sd from every row, test rows included
scaler = StandardScaler().fit(X)
X_scaled = scaler.transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)

# RIGHT - the scaler only ever learns from the training rows
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
scaler  = StandardScaler().fit(X_train)
X_train = scaler.transform(X_train)
X_test  = scaler.transform(X_test)

# BEST - a pipeline makes the mistake structurally impossible
model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=15))
model.fit(X_train, y_train)

The wrong version leaks only a mean and a standard deviation, which sounds harmless and typically inflates the test score by a little. Other forms are far worse:

• A feature caused by the target. Predicting loan default using "number of collection letters sent" gives a spectacular model that cannot be used, because the letters are sent after the default. Ask of every feature: would this value actually be available at the moment I need the prediction?
• Duplicate rows split across both sides. If the same student appears twice and one copy lands in train and one in test, the model has literally memorized a test answer. Deduplicate before splitting.
• Ignoring time. A random split of time-ordered data trains on the future to predict the past. For anything temporal, split by date: train on everything before a cutoff, test on everything after.
• Imputing with the full dataset. Filling missing values with the overall mean uses the test rows to compute that mean. Impute inside the pipeline, from training data only.

The practical rule: if a model performs far better than a domain expert believes is possible, look for leakage before celebrating. A test accuracy that seems too good almost always is.

Key idea: Leakage is any information from the test set or the future reaching the model, it makes results look better than they are, and pipelines plus a time-aware split prevent most of it.

Three sets, not two

The table above chose k = 15 by comparing test scores - and in doing so, quietly spoiled the test set. Once you use a set to make a choice, its score is no longer an unbiased estimate, because you selected the value that happened to look best on those particular rows. The test set has silently become a validation set.

The honest arrangement uses three:

  training set    (about 60%)   fit the model's parameters
  validation set  (about 20%)   choose hyperparameters such as k
  test set        (about 20%)   measure once, at the very end, and then stop

The discipline that matters is the last clause. The test set is looked at once. If you check it, adjust the model, and check again, you have begun fitting to it by hand, and its number stops meaning what you will report it to mean.

Key idea: Use a validation set to choose hyperparameters and reserve the test set for a single final measurement, because any set you tune against stops being an honest estimate.

Cross-validation

A single train/test split can be lucky or unlucky depending on which rows landed where. k-fold cross-validation reduces that luck: split the data into k equal parts (folds), then train k times, each time holding out a different fold as the test set and averaging the k scores.

It is like grading a student on several different quizzes instead of one, so a single easy or hard draw does not decide the outcome. With k = 5, every row is used for testing exactly once, giving a more stable estimate of true performance than any single split. This is the standard way to compare models fairly before choosing one.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X, y, cv=5)
scores          # array([0.79, 0.83, 0.76, 0.81, 0.78])
scores.mean()   # 0.794
scores.std()    # 0.024

Look at the spread before the mean. The five folds range from 0.76 to 0.83, a seven-point swing produced by nothing but which rows happened to be held out. Had you run a single split and drawn the 0.83 fold, you would have reported a model four points better than it is; had you drawn the 0.76 fold you would have understated it by three. Reporting "0.79, with a fold-to-fold standard deviation of 0.02" is both more honest and more useful than any single number.

Key idea: k-fold cross-validation averages several train/test splits to give a more stable estimate, and the spread across folds is itself worth reporting.

Where people get stuck

• "A model scoring 99% on its training data is excellent." That score can be memorization; only the test score reveals whether it generalizes. With k = 1 the training score is a guaranteed 100% and means nothing.
• "Overfitting and underfitting are the same failure." Overfitting is too complex (great train, poor test); underfitting is too simple (poor on both).
• "A more complex model is always better." Past the sweet spot, extra complexity chases noise and test error rises.
• "One train/test split is always enough." A single split can be lucky or unlucky; the five folds above spanned seven points of accuracy.
• Scaling or imputing before splitting. The transformer then learns from the test rows, and the reported score is optimistic. Put every preprocessing step inside a pipeline.
• Using a feature that will not exist at prediction time. Anything recorded after the outcome leaks the answer and produces a model that is spectacular in testing and useless in production.
• Splitting time-ordered data randomly. That trains on the future to predict the past. Split by date instead.
• Tuning on the test set. Every peek costs you some of its honesty. Choose hyperparameters on a validation set and touch the test set once.
• Celebrating a suspiciously high score. If the model beats what a domain expert thinks is possible, the first hypothesis should be leakage, not brilliance.

Recap

• Test on held-out data to estimate performance on new cases; a common split is about 80% training and 20% testing.
• Overfitting: high train, low test accuracy; underfitting: low on both. Track the gap and the test score together.
• In KNN, small k means high complexity, and k = 1 always scores a meaningless 100% on training data.
• Aim for the complexity where test error is lowest.
• Data leakage - preprocessing before splitting, post-outcome features, duplicate rows, random splits of time series - makes results look better than they are.
• Use train, validation, and test sets, and measure on the test set exactly once.
• k-fold cross-validation averages k splits for a stable estimate, and the fold-to-fold spread belongs in the report.

Sources

1. scikit-learn developers. (n.d.). Cross-validation: evaluating estimator performance: train/validation/test and k-fold. scikit-learn user guide. scikit-learn.org
2. scikit-learn developers. (n.d.). Common pitfalls and recommended practices: data leakage and inconsistent preprocessing. scikit-learn user guide. scikit-learn.org
3. scikit-learn developers. (n.d.). Underfitting vs. overfitting. scikit-learn examples. scikit-learn.org
4. scikit-learn developers. (n.d.). train_test_split: random_state, stratify, and shuffle. scikit-learn API reference. scikit-learn.org
5. VanderPlas, J. (2016). Hyperparameters and model validation. In Python data science handbook (ch. 5). O'Reilly. jakevdp.github.io
6. Cetinkaya-Rundel, M., & Hardin, J. (2021). Foundations for inference: sampling variability. In Introduction to modern statistics. OpenIntro. openintro.org
7. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). Resampling methods. In An introduction to statistical learning (2nd ed., ch. 5). Springer. find source ↗

An algorithm used on tens of millions of patients in the United States decided who received extra medical care. It contained no race variable, no discriminatory rule, and no bug. It was still systematically directing care away from Black patients, because of a single modeling decision that looked entirely sensible at the time. That case, examined below, is the clearest demonstration there is that bias usually enters through the data and the target rather than through the code.

The big picture

Data science shapes real decisions about loans, hiring, medical care, and policing, so doing it responsibly is part of doing it well. Two themes matter most: models can quietly inherit human bias from the data, and data about people carries genuine ethical duties. A model is a mirror of its data, and a mirror will faithfully reflect a crooked room.

Data science shapes decisions about loans, hiring, medical care, and policing, so doing it responsibly is not optional. Two themes matter most: models can inherit human bias, and data about people carries real ethical duties.

Key idea: Models can absorb bias from their data, and working with people's data brings real ethical obligations.

Where bias comes from

Bias here means a systematic error that makes data or a model unfair or unrepresentative. A model is only as fair as the data and choices behind it, and bias creeps in at several points:

• Sampling bias: the data does not represent the population. A model trained mostly on one group may perform badly on others.
• Historical bias: the data faithfully records a biased past. If past hiring favored one group, a model that learns "who was hired before" will repeat that pattern even with no protected attribute in the file.
• Measurement bias: the thing you measured is a flawed stand-in for what you care about. Arrests are not the same as crime; clicks are not the same as satisfaction.
• Labeling bias: human-applied labels carry the labelers' assumptions.

A famous failure mode: a model given no race or gender field can still discriminate, because other features (ZIP code, name, purchase history) act as proxies for them. A proxy is a stand-in feature that quietly carries the information you thought you removed, so dropping the sensitive column does not remove the bias.

Key idea: Bias enters through sampling, history, measurement, and labels, and proxies let it persist even after a sensitive field is removed.

A documented case: bias in the label, not the algorithm

In 2019 Obermeyer and colleagues published an analysis in Science of a commercial risk-prediction algorithm used by health systems covering roughly 200 million people in the United States. Its job was to identify patients sick enough to enrol in extra care management. It worked like this:

what the designers wanted to predict:   how much care will this patient need?
what the algorithm was trained on:      how much will be spent on this patient?

the assumption:  spending is a good stand-in for need
the reality:     for equally sick patients, less had historically been
                 spent on Black patients - because of reduced access,
                 mistrust, and unequal treatment

Because the target variable was cost, the algorithm learned that these patients would cost less, and therefore scored them as healthier. The researchers found that at any given risk score, Black patients were substantially sicker than White patients with the same score. Re-training the model on a measure of active chronic conditions rather than cost raised the share of Black patients automatically enrolled in the extra-care programme from 17.7% to 46.5%.

Every part of that failure is instructive. There was no race variable and no discriminatory rule. The code was correct. The choice of label - a reasonable-sounding, easily measured proxy - carried a historical inequity into every prediction, and it did so at enormous scale and completely silently. The lesson generalizes: when your target variable is a proxy, audit the proxy, because whatever unfairness sits in the gap between the proxy and the real thing will be learned and amplified.

Key idea: A widely deployed health algorithm discriminated with no race field and no bug, because it predicted spending as a proxy for need, which shows that bias enters through the training data and the chosen target rather than the code.

Provenance: knowing what your data can support

Data provenance is the record of where a dataset came from and what happened to it since. It is not paperwork; it is what determines which conclusions the data can bear. Five questions belong in the notes of every project:

• Who collected it, when, and why? Data gathered for billing answers billing questions well and clinical questions badly.
• Who is in the sample, and who is systematically absent? A survey of app users cannot describe people who never downloaded it, however many respondents it has.
• How was each variable measured? Arrests are not crimes, clicks are not interest, and spending is not need. Any measure that stands in for something else is a proxy, and every proxy has a gap.
• What use did the subjects agree to? Consent given for treatment is not consent to train a commercial model.
• What has been changed since collection? Cleaning decisions, merges, and filters all shape what the file can say.

The rule that follows is worth internalizing: the strongest claim a dataset can support is bounded by how it was collected, not by how large it is. A million rows of self-selected volunteers is still a study of volunteers.

Key idea: Provenance - who collected the data, from whom, how, and under what consent - sets a ceiling on the claims it can support, and no sample size raises that ceiling.

Privacy, consent, and re-identification

Three principles carry most of the weight when the rows are people. Collection limitation: gather only what the question needs, because data you never collected cannot leak. Purpose limitation: use it for what it was collected for, and treat a new purpose as needing new permission. Retention limitation: delete it when the purpose is served.

Then there is the assumption that quietly undermines all three: that removing names makes data anonymous. It does not. Latanya Sweeney showed that roughly 87% of the United States population could be uniquely identified by just three fields that no one thinks of as identifying: five-digit ZIP code, gender, and full date of birth. She demonstrated it concretely by taking a public release of "de-identified" state employee hospital records, cross-referencing a purchased voter registration list, and picking out the medical record of the governor of Massachusetts.

the linkage attack

  "anonymized" medical data          public voter roll
  ZIP, birth date, sex, diagnosis    ZIP, birth date, sex, name, address
        |                                  |
        +----------- join on -------------+
                ZIP + birth date + sex

  result: name attached to diagnosis, with no name ever released

The defence is to think in terms of quasi-identifiers: fields that identify nobody alone and almost everybody in combination. Standard mitigations coarsen them, for instance releasing the birth year rather than the date and the first three ZIP digits rather than five, so that every record is indistinguishable from at least k others - a property called k-anonymity. That helps and is not a guarantee, since a group of k people who all share the same diagnosis reveals it regardless. Modern releases increasingly use differential privacy, which adds calibrated noise and offers a mathematical bound on what any release can disclose.

The practical takeaway for a working analyst is smaller and more useful than any of that machinery: treat "we removed the names" as the beginning of an anonymization argument, never the end of one.

Key idea: Names are not what identifies people; ZIP, birth date, and sex alone uniquely identify about 87% of the US population, so anonymization means coarsening quasi-identifiers, not deleting names.

The feedback loop

Deployed models can make their own bias worse. If a policing model sends more patrols to a neighborhood, more incidents are recorded there, which "confirms" the model and sends still more patrols. This feedback loop is a cycle where a model's outputs shape the future inputs that reinforce it, and it can entrench inequity under a veneer of objectivity. Watching for such loops is part of responsible deployment.

Key idea: A model's outputs can shape future data in a feedback loop that entrenches its own bias.

Ethical duties

1. Privacy and consent: collect only what you need, protect it, and respect how people agreed it could be used. Data that can identify individuals demands special care.
2. Fairness: check whether error rates differ across groups, not just overall accuracy. A model that is 95% accurate overall but far worse for one group is not fair.
3. Transparency and accountability: be able to explain what a model does and who is responsible when it is wrong, especially for high-stakes decisions.
4. Anonymization limits: stripping names is often not enough. Combining a few "harmless" fields (ZIP, birth date, sex) can re-identify people, so anonymization must be done carefully.

Key idea: Responsible practice protects privacy, checks fairness across groups, stays accountable, and treats anonymization as hard.

A practical habit

Before shipping any model that affects people, ask three questions: Who is in the data, and who is missing? What could go wrong, and for whom? Would I accept this decision if it were made about me? These questions do not have tidy formulas, but asking them is what separates competent data science from harmful data science.

Key idea: Ask who is missing, what could go wrong and for whom, and whether you would accept the decision yourself.

Where people get stuck

• "Removing race or gender makes a model fair." Proxy features like ZIP code or name can carry the same information, so bias can remain.
• "An algorithm is objective by nature." A model learns from human-made data and choices, so it can reflect and even amplify existing bias.
• "High overall accuracy means the model is fair." A model can be accurate overall yet much worse for a specific group; fairness needs per-group checks.
• "Deleting names fully anonymizes data." ZIP, birth date, and sex together identify most people. Coarsen quasi-identifiers instead.
• Choosing an easily measured target without auditing it. The health algorithm predicted cost because cost was in the database. Every proxy target deserves the question "what does the gap between this and the real thing contain?"
• Treating the model as the only thing to check. The label, the sampling frame, the measurement, and the deployment loop are all upstream of the model and all more likely to be the problem.
• Assuming consent transfers between purposes. Agreement to share data for one use is not agreement for another, however technically convenient the second use is.
• Reporting one fairness number. Different fairness definitions - equal error rates, equal positive rates, equal calibration - are mathematically incompatible in general. Say which one you used and why.
• Believing more data fixes bias. If the bias is in the labels or the sampling frame, more of the same data makes the biased pattern more precisely learned, not less.

Recap

• Bias enters via sampling, historical, measurement, and labeling problems.
• Proxy features let bias survive the removal of a sensitive attribute, and a proxy target can carry inequity into every prediction with no discriminatory rule anywhere in the code.
• Feedback loops can make a deployed model's bias worse over time.
• Provenance - who collected the data, from whom, how, and under what consent - bounds what it can support, regardless of size.
• Duties include collection, purpose, and retention limitation, plus fairness and transparency.
• Deleting names does not anonymize: ZIP, birth date, and sex uniquely identify about 87% of the US population.
• Check error rates across groups, not just overall accuracy, and say which definition of fairness you used.

Sources

1. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453. science.org
2. Sweeney, L. (2000). Simple demographics often identify people uniquely (Data Privacy Working Paper 3). Carnegie Mellon University. dataprivacylab.org
3. Barocas, S., Hardt, M., & Narayanan, A. (2023). Fairness and machine learning: Limitations and opportunities. MIT Press. fairmlbook.org
4. Diez, D., Cetinkaya-Rundel, M., & Barr, C. D. (2019). Data collection and study design. In OpenIntro statistics (4th ed., ch. 1). OpenIntro. openintro.org
5. Cetinkaya-Rundel, M., & Hardin, J. (2021). Study design, sampling, and generalizability. In Introduction to modern statistics. OpenIntro. openintro.org
6. scikit-learn developers. (n.d.). Metrics and scoring: per-group evaluation and the classification report. scikit-learn user guide. scikit-learn.org
7. O'Neil, C. (2016). Weapons of math destruction: How big data increases inequality and threatens democracy. Crown. find source ↗

Everything in this course now has to survive contact with a person who has four minutes and no statistics training. That person will act on one sentence, remember one number, and never see your notebook. Getting that sentence and that number right, with the caveats attached rather than buried, is the last skill of the pipeline and the one that determines whether any of the earlier work mattered.

The big picture

A brilliant analysis that nobody understands or trusts changes nothing, so communication is the final, decisive stage of the pipeline. Your job is to help a decision-maker act wisely, which means being clear, honest, and tuned to your audience. The best analysis is only as good as the sentence a busy manager remembers from it.

Communicating results is the final stage of the pipeline: turning a finding into a clear, honest message a decision-maker can act on.

Key idea: Communication turns analysis into action by being clear, honest, and audience-aware.

Lead with the answer

Non-technical audiences want the conclusion first, not a tour of your methods. Use the inverted pyramid: state the finding and its recommended action up front, then support it, then put technical detail last for those who want it. It is the reverse of a mystery novel, where you save the reveal; here you give away the ending first. "Customers who contact support in week one are twice as likely to renew, so we recommend a proactive week-one check-in" is a lead. The regression coefficients belong in an appendix.

Key idea: Put the conclusion and recommendation first, with methods and detail last.

Translate, do not dumb down

Replace jargon with plain meaning while keeping the substance. Say "the model is right about 9 times in 10" instead of quoting a raw accuracy of 0.902; say "these two tend to rise together" before, or instead of, "r = 0.78." Use a concrete unit ("about $40 more per order") rather than a standardized coefficient. Respect your audience's intelligence, not their jargon.

When you must mention statistical significance, translate the p-value too: it measures how surprised you would be by your result if nothing were really going on, so a very small p-value means "this pattern would be a big surprise by chance alone." Say "this is very unlikely to be a fluke" rather than quoting "p < 0.01" and leaving the reader to guess.

Key idea: Convert numbers and jargon into plain, concrete meaning without losing the substance.

What a p-value is, and the four things it is not

Since you will be translating p-values, you had better be translating them correctly, and the standard translations are mostly wrong. Precisely, a p-value is the probability of observing data at least as extreme as yours assuming the null hypothesis is true and every modeling assumption holds. In 2016 the American Statistical Association took the unusual step of issuing a formal statement about it, because the misreadings were doing real damage. Four of them:

NOT: "there is a 3% chance the null hypothesis is true"
     the p-value is computed assuming the null IS true; it cannot
     also be a probability about it

NOT: "there is a 3% chance the result is due to chance"
     same error in different clothing

NOT: "p = 0.03 means the effect is important"
     a tiny, useless effect measured on a huge sample gives a tiny p-value

NOT: "p = 0.049 found something and p = 0.051 did not"
     0.05 is a convention, not a boundary in nature, and the two
     results are practically identical

The ASA's principles also include one that is easy to skip and matters most: proper inference requires full reporting. If you tried twenty comparisons and reported the one that reached significance, the p-value attached to it no longer means anything, because you selected on it. That practice is called p-hacking, and the defence is to decide what you are testing before you look, and to disclose everything you tried.

The practical recommendation for a report: lead with the estimate and its uncertainty - "about 5.5 points per hour, plausibly between 4 and 7" - and let the p-value be one piece of supporting context rather than the verdict. A reader can act on an interval; nobody can act on a threshold.

Key idea: A p-value assumes the null is true, so it cannot be the probability that the null is true, it says nothing about effect size, and it means nothing at all if you selectively reported the comparison that produced it.

Be honest about uncertainty

Every data conclusion has limits, and hiding them destroys trust the moment reality diverges. Good communication states:

• Uncertainty: a range or margin, not a single false-precision number. "Between 8% and 12%," not "10.37%."
• Assumptions: what had to be true for the result to hold.
• Scope: who and what the finding applies to, and where it does not. A result from one city may not travel.
• Correlation, not proven cause: if you only observed an association, say so plainly rather than implying you proved causation.

Key idea: State uncertainty, assumptions, scope, and whether you found association or proven cause.

Let the chart carry the message

Apply the visualization principles from Module 3: pick the chart that answers the question, start bar axes at zero, label clearly, and cut clutter. One clean chart that a manager reads correctly in five seconds beats a dense dashboard nobody parses. Title the chart with the takeaway ("Repeat buyers spend 40% more"), not just the variables ("Spend by customer type").

Key idea: Use one clear, honest chart with a takeaway title that states the conclusion.

A short template

1. Bottom line: the finding and recommended action, in one or two sentences.
2. Evidence: the key numbers and one clear chart.
3. Caveats: uncertainty, assumptions, and scope, stated plainly.
4. Next step: what you would do to confirm or act on it.

Done well, communication closes the loop the course opened with: a real question, answered with data, delivered so a human can act on it responsibly. That is the whole of data science in one sentence.

Key idea: A tight report gives the bottom line, the evidence, the caveats, and the next step.

The study.csv analysis, written up

Here is the entire course applied to the dataset we have carried since Lesson 1, in the four-part template, at the length a manager will actually read.

Bottom line. In this sample, each extra hour of study went with about 5.5 more exam points, and students who attended tutoring averaged 4 points higher than those who did not. Before committing budget to tutoring, we recommend running a small randomized trial next term.

Evidence. The file arrived with 12 rows and one duplicate, one impossible score of 999, one study time recorded as text, and inconsistent category spellings; after cleaning, 9 rows remained. Study hours and score were strongly related (r = 0.97), with a fitted line of score = 44.7 + 5.5 x hours explaining 94% of the variation and missing by about 4 points on average. Tutoring attendees averaged 79 points against 75 for non-attendees, and both groups studied 6.0 hours on average, so study time does not explain that gap.

Caveats.

• Nine students. Every figure above would move substantially with a different nine. Treat all of them as indicative, not measured.
• Observational, not experimental. Students chose whether to attend tutoring, so the 4-point gap cannot be separated from whatever made them choose it.
• One row was dropped for a missing study time. If students who studied least are likelier to leave it blank, our average study time is overstated.
• The line is fitted between 2 and 10 hours and says nothing outside that range; it predicts 265 points for 40 hours of study.

Next step. Randomly assign 60 volunteers to a tutoring offer and 60 to a waitlist, then compare exam scores. That design answers the causal question these data cannot, and the cost is one term.

Notice what the caveats section does. It is not hedging or false modesty; every bullet names a specific thing that would change the conclusion, and the last line turns the biggest limitation into a concrete proposal. A reader finishes it knowing exactly how much to believe and exactly what to do next, which is the whole job.

Key idea: A complete report states the finding, shows the evidence including what cleaning cost, names the specific limitations that could change the answer, and proposes the design that would settle it.

What you hand over

A report is not the only deliverable. Reproducibility, introduced in Lesson 1, is finally paid off here: what you hand over should let someone else obtain your numbers.

project/
  data/raw/study.csv              never modified, ever
  data/processed/study_clean.csv  the output of the cleaning script
  src/clean.py                    the cleaning ledger, expressed as code
  src/model.py                    fitting and evaluation, with a fixed seed
  notebooks/01-explore.ipynb      restarted and run top to bottom before sharing
  README.md                       the question, the data source, the caveats
  requirements.txt                pinned library versions

Two habits do most of the work. Restart and run every notebook from the top before you share it, because a notebook executed out of order can display numbers that no linear run would ever produce, and that includes runs by you next month. And write the README first: if you cannot state the question, the source, and the main caveat in a paragraph before starting, the analysis is not ready to start.

Key idea: Hand over untouched raw data, the cleaning and modeling code, a notebook that runs top to bottom, pinned versions, and a README stating the question, source, and caveats.

Where people get stuck

• "Start with the methods and build to the conclusion." Busy readers want the answer first; the inverted pyramid leads with the finding.
• "More precise numbers sound more credible." False precision like 10.37% overstates certainty; an honest range builds more trust.
• "Plain language means dumbing down." Good translation keeps the substance while dropping needless jargon.
• "A chart title should name the variables." A takeaway title states the conclusion so the point lands at a glance.
• "p < 0.05 means the finding is real." The p-value assumes the null is true and says nothing about effect size, importance, or how many comparisons you tried first.
• Burying the caveats in an appendix. If a limitation could change the decision, it belongs beside the finding. Caveats readers never reach are the same as caveats you never wrote.
• Using causal verbs for observational results. "Tutoring raises scores" and "tutoring attendees scored higher" differ by an entire research design. Choose the verb your data earned.
• Hiding what cleaning cost. "We analyzed the data" conceals that a quarter of the rows were dropped. Say how many and why; it is part of the evidence.
• Sharing a notebook you have not restarted. Out-of-order execution can display numbers that no clean run reproduces, including on your own machine.

Recap

• Lead with the conclusion and recommendation, detail last.
• Translate numbers and jargon into concrete, plain meaning.
• State uncertainty as a range, plus assumptions and scope.
• A p-value is computed assuming the null is true, so it is not the probability that the null is true, not a measure of importance, and meaningless under selective reporting.
• Distinguish observed association from proven cause, in the verbs as well as the caveats.
• Use one clear chart with a takeaway title; follow the bottom-line, evidence, caveats, next-step template.
• Hand over untouched raw data, the code for every step, a notebook that runs top to bottom, pinned versions, and a README.

Sources

1. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. tandfonline.com
2. Wickham, H., Cetinkaya-Rundel, M., & Grolemund, G. (2023). Communication. In R for data science (2nd ed., ch. 11). O'Reilly. r4ds.hadley.nz
3. Diez, D., Cetinkaya-Rundel, M., & Barr, C. D. (2019). Foundations for inference: p-values and interpretation. In OpenIntro statistics (4th ed., ch. 5). OpenIntro. openintro.org
4. Cetinkaya-Rundel, M., & Hardin, J. (2021). Inference and the pitfalls of multiple comparisons. In Introduction to modern statistics. OpenIntro. openintro.org
5. scikit-learn developers. (n.d.). Common pitfalls and recommended practices: controlling randomness and reporting reproducible results. scikit-learn user guide. scikit-learn.org
6. Matplotlib development team. (n.d.). Quick start guide: titles, labels, and figure export. Matplotlib documentation. matplotlib.org
7. Cairo, A. (2016). The truthful art: Data, charts, and maps for communication. New Riders. find source ↗