Introduction to Data Science

Data science is the practice of turning data into understanding and better decisions. It sits at the meeting point of three things: statistics (reasoning under uncertainty), computing (getting a machine to do the work at scale), and domain knowledge (knowing what the numbers mean and which questions matter). A result that is statistically clean but answers the wrong question helps no one, which is why all three matter together.

The pipeline

Almost every project moves through the same stages, and much of the effort lands early. A rough but honest saying in the field is that data scientists spend most of their time getting data into shape, not building fancy models.

1. Ask a question. State clearly what you want to learn or predict. "Do longer support calls get worse ratings?" is answerable; "make support better" is not yet.
2. Collect / obtain data. Pull from files, databases, sensors, surveys, or the web. Note where it came from and how it was measured.
3. Clean and prepare (wrangling). Fix wrong types, handle missing values, remove duplicates, and reshape the table so each row is one observation.
4. Explore (EDA). Summarize and plot the data to learn its shape, spot errors, and form hypotheses.
5. Model. Fit a statistical or machine-learning model to describe a relationship or make predictions.
6. Evaluate. Check honestly how well the model works on data it has not seen.
7. Communicate. Report the finding in plain language with an honest picture of its limits, so a decision can be made.

The arrows are not strictly one-way. Exploring often sends you back to clean more; a weak evaluation sends you back to model differently. Good practice is to loop, not march.

A quick example

Suppose a store wants to know which products to feature. The question is "which products drive the most repeat purchases?" You obtain the sales log, clean it by removing test orders and fixing dates stored as text, explore by plotting repeat-purchase rates per product, model the relationship between category and repeat rate, evaluate whether the pattern holds on last month's data, and communicate a short ranked list to the manager. Notice that only two of the seven steps are the "modeling" people imagine data science to be.

Descriptive versus predictive

Two broad goals run through the course. Descriptive work summarizes what happened (averages, trends, relationships in data you already have). Predictive work uses patterns to forecast new, unseen cases (will this email be spam?). Both rest on the same pipeline; they mainly differ at the modeling and evaluation stages.

Data science in Python rests on two libraries. numpy provides fast numeric arrays, and pandas builds a labeled table on top of them. You do not need to be an expert programmer, but you do need to read and lightly edit code like the snippets below.

numpy: the array

A numpy array (called ndarray) holds many numbers of the same type in one compact block of memory. Operations apply to the whole array at once, a style called vectorization. This is both shorter to write and much faster than looping in plain Python.

import numpy as np

a = np.array([2, 4, 6, 8])
a * 10          # array([20, 40, 60, 80])  - whole array at once
a.mean()        # 5.0
a[a > 4]        # array([6, 8])  - boolean filtering

The last line shows boolean indexing: a > 4 produces an array of True/False, and passing it back in keeps only the True positions. This one pattern powers most data filtering in pandas too.

pandas: Series and DataFrame

A Series is a one-dimensional labeled array, like a single column with an index. A DataFrame is a two-dimensional table: a set of Series that share the same row index. Think of a DataFrame as a spreadsheet where every column has a name and a data type.

Reading a DataFrame

import pandas as pd

df = pd.DataFrame({
    "name": ["Ana", "Ben", "Cara"],
    "age":  [25, 31, 29],
    "city": ["Reno", "Reno", "Provo"]
})

df["age"]              # one column (a Series)
df[["name", "age"]]    # two columns (a smaller DataFrame)
df[df["age"] > 28]     # rows where age exceeds 28 (Ben and Cara)

Two ways to grab rows and columns by label or position are worth knowing. df.loc[rows, cols] selects by label, while df.iloc[rows, cols] selects by integer position. For example, df.loc[0, "name"] is "Ana", and df.iloc[0, 0] is also "Ana" because that cell is at row 0, column 0.

Why labels help

Because columns have names, pandas code reads almost like English: df[df["city"] == "Reno"]["age"].mean() reads as "the mean age of the Reno rows." Labels also let pandas line up data automatically when you combine tables, which prevents a whole class of off-by-one bugs that plague raw arrays.

The first real step of any project is getting data into a DataFrame and looking at it before you trust it. The most common source is a CSV (comma-separated values) file, a plain-text table where columns are separated by commas and rows by line breaks.

Reading a file

import pandas as pd

df = pd.read_csv("sales.csv")   # load the file into a DataFrame
df.head()      # first 5 rows - a quick eyeball check
df.shape       # (rows, columns), e.g. (1000, 6)
df.columns     # the column names
df.info()      # column names, non-null counts, and data types
df.describe()  # count, mean, std, min, quartiles, max for numeric columns

Running these five lines before anything else is a habit worth building. head shows whether the file parsed correctly, shape tells you how big it is, info reveals data types and how many values are missing per column, and describe gives a first numeric summary that often exposes impossible values (a maximum age of 999, say).

Data types matter

pandas assigns each column a dtype. Numbers become int64 or float64, text becomes object, and dates should become datetime64. A frequent surprise is a numeric column read as text because one stray value contained a letter or a currency symbol. If df.info() shows a price column as object, arithmetic on it will misbehave until you convert it.

Tidy data

Data is tidy when each row is one observation, each column is one variable, and each cell holds one value. Tidy data is far easier to filter, group, and plot. A table that crams two facts into one cell (such as "Reno, NV" in a single column, or a "Q1/Q2/Q3/Q4" set of columns that are really one "quarter" variable) is untidy and usually needs reshaping first.

You will not always receive tidy data, but knowing the target shape tells you what cleaning to aim for. The next lesson tackles the messiest part: missing and malformed values.

Real datasets arrive dirty. Data cleaning is the unglamorous work of making the table trustworthy, and skipping it quietly ruins every step downstream. Three problems dominate: missing values, wrong types, and duplicates.

Missing values

pandas marks a missing entry as NaN ("not a number"). Find them before deciding what to do.

df.isna().sum()     # count of missing values per column
df["age"].isna().mean()   # fraction missing in the age column

You then have two broad choices. Dropping removes rows (or columns) with missing data. Imputing fills the gaps with a reasonable stand-in, often the column's mean or median for numbers, or the most common value for categories.

df = df.dropna(subset=["age"])              # drop rows missing age
med = df["income"].median()
df["income"] = df["income"].fillna(med)     # fill missing income with the median

Which is right depends on how much is missing and why. Dropping a handful of rows from a large dataset is usually fine. Dropping half your rows throws away information and can bias the result if the missing rows differ systematically from the rest. Median imputation resists outliers, which is why it is often preferred over the mean for skewed columns like income.

Fixing types

Convert columns to their true type so operations work. A price stored as text like "$12.50" must have the symbol stripped and then be converted to a number; a date stored as text should become a real datetime.

df["price"] = df["price"].str.replace("$", "", regex=False).astype(float)
df["date"]  = pd.to_datetime(df["date"])

Duplicates

Duplicate rows inflate counts and distort averages. Detect and drop them:

df.duplicated().sum()   # how many exact duplicate rows exist
df = df.drop_duplicates()

A worked cleaning decision

Suppose a 1,000-row survey has an "age" column with 8 missing values and 3 impossible entries of 200. The 3 impossible values are errors, so set them to NaN. Now 11 of 1,000 ages are missing, about 1%. Because that is a tiny fraction, dropping those 11 rows is defensible and simplest, leaving 989 clean rows. Had 400 values been missing, you would instead impute or investigate why so many are absent, because dropping 40% would distort the sample. The guiding rule: understand why data is missing before you decide how to handle it.

Exploratory data analysis (EDA) is the open-minded first look at a cleaned dataset. Its goals are to understand each variable's distribution, uncover relationships, spot errors and outliers the cleaning missed, and generate hypotheses worth testing. EDA is about questions, not conclusions; you are getting to know the data.

One variable at a time (univariate)

Start by looking at variables individually. The right tool depends on the variable type, exactly as in statistics.

• Quantitative variable: a histogram to see its shape (symmetric, skewed, bimodal) and a five-number summary or describe() for the numbers.
• Categorical variable: a bar chart of counts, and a frequency table via df["col"].value_counts().

Two variables at a time (bivariate)

Then look at how variables move together. A scatterplot is the standard picture for two quantitative variables; a grouped summary compares a quantitative measure across the levels of a categorical variable.

df["category"].value_counts()          # counts per category
df.groupby("category")["sales"].mean() # average sales within each category
df[["age", "income"]].corr()           # correlation between two numbers

Grouping to compare

The groupby operation is the workhorse of EDA. It splits rows into groups by a categorical column, computes a summary within each group, and combines the results. "Average sales per region" or "median age per plan" are one line each. Comparing group summaries is often where the first real insight appears.

A short EDA story

Imagine a dataset of support tickets with columns for handling time and a 1-to-5 satisfaction rating. Univariate EDA shows handling time is right-skewed (most tickets are quick, a few drag on). A grouped summary of average rating by handling-time bucket shows ratings falling as handling time rises. That is a hypothesis, not proof: maybe hard problems both take longer and frustrate people. EDA has done its job by pointing you toward a relationship worth testing carefully later. Always keep that humility: seeing a pattern is the start of an investigation, not the end.

Descriptive statistics compress a column of numbers into a few honest summaries. Two questions drive them: where is the center, and how spread out are the values?

Measures of center

The mean is the sum divided by the count. The median is the middle value of sorted data. The mode is the most frequent value. The mean uses every value, so a single extreme value pulls it; the median barely moves, which makes it resistant to outliers.

Worked example. For the values 2, 4, 4, 5, 100: the mean is (2 + 4 + 4 + 5 + 100) / 5 = 115 / 5 = 23, but the median (middle of the sorted five) is 4. The lone value of 100 dragged the mean far above every typical value, while the median still describes the bulk of the data. When the mean and median disagree sharply, suspect skew or an outlier.

Measures of spread

The range is max minus min. The standard deviation is the typical distance of values from the mean; the variance is its square. The interquartile range (IQR = Q3 - Q1) is the spread of the middle 50% and, like the median, resists outliers.

Worked standard deviation. For 2, 4, 6, 8, the mean is 5. Squared deviations are 9, 1, 1, 9, summing to 20. Dividing by n - 1 = 3 gives a sample variance of 20 / 3 = 6.67, so the standard deviation is square root of 6.67 = 2.58. Values sit about 2.58 units from the mean on average. (Sample formulas divide by n - 1 to correct bias; if the four numbers were the entire population you would divide by 4.)

In pandas, one call each

df["income"].mean()      # average
df["income"].median()    # middle value, resists outliers
df["income"].std()       # standard deviation (sample, divides by n - 1)
df["income"].quantile([0.25, 0.5, 0.75])   # the quartiles

Reading the numbers together

No single number is enough. A mean far above the median signals right skew. A large standard deviation relative to the mean signals high variability. Reporting center and spread together, and preferring the median and IQR when outliers loom, keeps your summary honest. These same statistics reappear in every later module, so being fluent with them pays off.

A good chart lets the reader see a pattern faster than any table. A bad chart hides it, or worse, invents one that is not there. Choosing the right chart and drawing it honestly is a core data science skill, not decoration.

Match the chart to the job

How charts mislead

Three traps are worth memorizing:

• Truncated axis. A bar chart whose y-axis starts at 90 instead of 0 makes a rise from 92 to 95 look enormous. For bar charts, start the value axis at zero.
• Wrong chart type. A pie chart with ten thin slices is unreadable; a bar chart ranks them clearly. Line charts imply continuity, so do not use one for unordered categories.
• Overplotting and clutter. Thousands of overlapping points hide density; extra gridlines, 3-D effects, and loud colors distract from the data.

The two red bars and the two green bars show the same numbers. The truncated red chart screams "huge jump"; the honest green chart shows the small real difference. Same data, opposite impression.

Principles for honest graphics

1. Start bar axes at zero so bar length is proportional to value.
2. Label everything: title, both axes with units, and a legend when needed.
3. Keep it simple: remove clutter so the data stands out (a idea often called maximizing the data-to-ink ratio).
4. Do not distort: avoid 3-D, dual axes chosen to fake a correlation, and misleading area scaling.
5. Show the right comparison: the chart should answer the actual question at a glance.

When in doubt, ask a colleague what they conclude from your chart before you add any words. If they read it wrong, the chart, not the reader, needs fixing.

Correlation measures the strength and direction of a straight-line relationship between two quantitative variables. The most common measure is the Pearson correlation coefficient, written r, which always lies between -1 and +1.

Reading r

• Sign: positive r means the variables rise together; negative r means one rises as the other falls.
• Size: r near +1 or -1 is a tight linear pattern; r near 0 means little or no linear relationship.
• Rough guide: |r| about 0.1 is weak, 0.5 is moderate, and 0.8 or more is strong. These are conventions, not laws.

Correlation is not causation

This is the single most important warning in data science. A strong correlation between two variables does not prove that one causes the other. Ice cream sales and drowning deaths rise together, but neither causes the other; a hidden third variable, hot weather, drives both. Such a hidden cause is a confounder. Establishing causation generally requires a controlled experiment (randomly assigning who gets the treatment), not just observed correlation.

What correlation misses

Pearson's r only captures linear association. A perfect U-shaped relationship can have r near 0 even though the variables are tightly related, just not in a straight line. A single outlier can also inflate or hide a correlation. The lesson: always plot the scatterplot before trusting a single r value. The number summarizes; the picture protects you from being fooled.

df["hours"].corr(df["score"])   # Pearson r between two columns
df.corr(numeric_only=True)      # correlation matrix of all numeric columns

Worked reading. Suppose study hours and exam score have r = 0.78. That is a strong positive linear relationship: students who study more tend to score higher, and a straight line describes the trend well. It does not prove studying causes the gain (motivated students may both study more and score higher), and it says nothing about students far outside the observed range of hours.

Correlation says how tightly two variables track. Linear regression goes further: it fits an actual line so you can predict one variable from another. With one predictor it is called simple linear regression.

The line

The fitted model has the form y = b0 + b1 * x, where y is the predicted response, x is the predictor, b0 is the intercept (predicted y when x = 0), and b1 is the slope (the change in y for a one-unit increase in x). The slope is where the meaning usually lives.

Worked example

A model predicting exam score from hours studied is score = 50 + 5 * hours. Read it directly: a student who studies 0 hours is predicted to score 50 (the intercept), and each additional hour adds 5 points (the slope). Predict the score for 6 hours: 50 + 5 * 6 = 50 + 30 = 80. Predict for 3 hours: 50 + 5 * 3 = 65. The line turns a relationship into concrete forecasts.

How the line is chosen: least squares

For each data point, the residual is the vertical gap between the actual y and the line's prediction (actual minus predicted). The least squares method picks the one line that makes the sum of the squared residuals as small as possible. Squaring keeps positive and negative errors from cancelling and penalizes big misses more. The result is the unique best-fitting straight line.

How good is the fit: R-squared

R-squared (the coefficient of determination) is the fraction of the variation in y that the line explains, from 0 to 1. An R-squared of 0.64 means the model explains 64% of the variability in the response; the remaining 36% is unexplained scatter. For simple regression, R-squared is exactly the square of the correlation r, so r = 0.8 gives R-squared = 0.64.

import numpy as np
b1, b0 = np.polyfit(df["hours"], df["score"], 1)  # slope, intercept
pred = b0 + b1 * 6                                 # predict for 6 hours

Cautions

Two warnings. First, do not extrapolate far beyond the data you fit; the line may be nonsense outside the observed range (studying 100 hours will not give a score of 550). Second, a line is only appropriate if the relationship is roughly linear, so plot the data and the residuals first. Regression is powerful precisely because it is simple, but that simplicity is also its limit.

So far the response we predicted (a score, a price) was a number, which is regression. When the response is a category instead (spam or not, disease or healthy, will churn or stay), the task is classification. The goal shifts from predicting a value to predicting a label.

A simple classifier: k-nearest neighbors

One of the most intuitive classifiers is k-nearest neighbors (KNN). To classify a new point, it finds the k training points closest to it (by distance in the feature space) and takes a majority vote of their labels. If k = 5 and the five nearest known emails are 4 spam and 1 not-spam, KNN labels the new email spam. The idea is that similar inputs tend to have similar labels. Choosing k matters: too small and the model chases noise; too large and it blurs real boundaries.

Judging a classifier: the confusion matrix

For a two-class problem (call them positive and negative), every prediction falls into one of four boxes:

This 2-by-2 table is the confusion matrix. From it come the core metrics:

• Accuracy = (TP + TN) / (all predictions): the fraction correct overall.
• Precision = TP / (TP + FP): of those predicted positive, how many really were.
• Recall = TP / (TP + FN): of the actual positives, how many were caught.

Worked example

A spam filter is tested on 100 emails: TP = 40 (spam caught), TN = 50 (real mail kept), FP = 5 (real mail wrongly flagged), FN = 5 (spam that slipped through). Then accuracy = (40 + 50) / 100 = 0.90, or 90%. Precision = 40 / (40 + 5) = 0.89, and recall = 40 / (40 + 5) = 0.89. Different situations weigh these differently: a cancer screen prizes recall (do not miss a real case), while a spam filter that must never trash real mail prizes precision.

Why accuracy alone can lie

If only 1% of emails are spam, a lazy model that predicts "not spam" for everything scores 99% accuracy while catching zero spam. This is the imbalanced classes trap. That is exactly why precision and recall exist: they expose a model that looks accurate but fails at the job that matters.

A model that has already seen the answers can look brilliant and still be useless. The core discipline of predictive modeling is to measure performance on data the model did not learn from, because that is the only honest estimate of how it will do in the real world.

The train/test split

Split your data into two parts, typically about 80% for training and 20% for testing. Fit the model on the training set only, then measure its accuracy on the untouched test set. Because the test rows were hidden during learning, their score estimates generalization, the model's performance on new cases. Grading a model on its training data is like grading students on the exact questions they were given as homework with answers attached.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)   # 80% train, 20% test
model.fit(X_train, y_train)                # learn only from training data
model.score(X_test, y_test)                # honest score on unseen data

Overfitting and underfitting

Overfitting happens when a model learns the training data too well, memorizing its noise and quirks instead of the real pattern. The telltale sign is high training accuracy but poor test accuracy. Underfitting is the opposite: the model is too simple to capture the real structure, so it does poorly on both training and test data. The goal is the middle ground, a model complex enough to learn the signal but not so complex it chases noise.

The blue training-error curve always falls as the model grows more flexible. The red test-error curve falls at first, then rises once the model starts memorizing noise. The green line marks the sweet spot: the complexity where test error is lowest.

Cross-validation

A single train/test split can be lucky or unlucky depending on which rows landed where. k-fold cross-validation reduces that luck: split the data into k equal parts (folds), then train k times, each time holding out a different fold as the test set and averaging the k scores. With k = 5, every row is used for testing exactly once, giving a more stable estimate of true performance than any single split. This is the standard way to compare models fairly before choosing one.

Data science shapes decisions about loans, hiring, medical care, and policing, so doing it responsibly is not optional. Two themes matter most: models can inherit human bias, and data about people carries real ethical duties.

Where bias comes from

A model is only as fair as the data and choices behind it. Bias creeps in at several points:

• Sampling bias: the data does not represent the population. A model trained mostly on one group may perform badly on others.
• Historical bias: the data faithfully records a biased past. If past hiring favored one group, a model that learns "who was hired before" will repeat that pattern even with no protected attribute in the file.
• Measurement bias: the thing you measured is a flawed stand-in for what you care about. Arrests are not the same as crime; clicks are not the same as satisfaction.
• Labeling bias: human-applied labels carry the labelers' assumptions.

A famous failure mode: a model given no race or gender field can still discriminate, because other features (ZIP code, name, purchase history) act as proxies for them. Removing the sensitive column does not remove the bias.

The feedback loop

Deployed models can make their own bias worse. If a policing model sends more patrols to a neighborhood, more incidents are recorded there, which "confirms" the model and sends still more patrols. This feedback loop can entrench inequity under a veneer of objectivity. Watching for such loops is part of responsible deployment.

Ethical duties

1. Privacy and consent: collect only what you need, protect it, and respect how people agreed it could be used. Data that can identify individuals demands special care.
2. Fairness: check whether error rates differ across groups, not just overall accuracy. A model that is 95% accurate overall but far worse for one group is not fair.
3. Transparency and accountability: be able to explain what a model does and who is responsible when it is wrong, especially for high-stakes decisions.
4. Anonymization limits: stripping names is often not enough. Combining a few "harmless" fields (ZIP, birth date, sex) can re-identify people, so anonymization must be done carefully.

A practical habit

Before shipping any model that affects people, ask three questions: Who is in the data, and who is missing? What could go wrong, and for whom? Would I accept this decision if it were made about me? These questions do not have tidy formulas, but asking them is what separates competent data science from harmful data science.

A brilliant analysis that no one understands or trusts changes nothing. Communicating results is the final, decisive stage of the pipeline. Your job is to help a decision-maker act wisely, which means being clear, honest, and audience-aware.

Lead with the answer

Non-technical audiences want the conclusion first, not a tour of your methods. Use the inverted pyramid: state the finding and its recommended action up front, then support it, then put technical detail last for those who want it. "Customers who contact support in week one are twice as likely to renew, so we recommend a proactive week-one check-in" is a lead. The regression coefficients belong in an appendix.

Translate, do not dumb down

Replace jargon with plain meaning while keeping the substance. Say "the model is right about 9 times in 10" instead of quoting a raw accuracy of 0.902; say "these two tend to rise together" before, or instead of, "r = 0.78." Use a concrete unit ("about $40 more per order") rather than a standardized coefficient. Respect your audience's intelligence, not their jargon.

Be honest about uncertainty

Every data conclusion has limits, and hiding them destroys trust the moment reality diverges. Good communication states:

• Uncertainty: a range or margin, not a single false-precision number. "Between 8% and 12%," not "10.37%."
• Assumptions: what had to be true for the result to hold.
• Scope: who and what the finding applies to, and where it does not. A result from one city may not travel.
• Correlation, not proven cause: if you only observed an association, say so plainly rather than implying you proved causation.

Let the chart carry the message

Apply the visualization principles from Module 3: pick the chart that answers the question, start bar axes at zero, label clearly, and cut clutter. One clean chart that a manager reads correctly in five seconds beats a dense dashboard nobody parses. Title the chart with the takeaway ("Repeat buyers spend 40% more"), not just the variables ("Spend by customer type").

A short template

1. Bottom line: the finding and recommended action, in one or two sentences.
2. Evidence: the key numbers and one clear chart.
3. Caveats: uncertainty, assumptions, and scope, stated plainly.
4. Next step: what you would do to confirm or act on it.

Done well, communication closes the loop the course opened with: a real question, answered with data, delivered so a human can act on it responsibly. That is the whole of data science in one sentence.