Linear Algebra

Linear algebra begins with the humblest object in mathematics: the straight-line equation. A linear equation in the variables x1, x2, ..., xn is any equation that can be written as a1 x1 + a2 x2 + ... + an xn = b, where the coefficients a1, ..., an and the constant b are known numbers. The defining feature is that every variable appears to the first power only: there are no squares, no products of variables, no roots, and no trig or exponential functions. So 2x - 3y = 5 is linear, but x^2 + y = 1 and xy = 4 are not.

Systems of equations

A system of linear equations is a collection of one or more linear equations in the same variables. A solution is a list of values, one for each variable, that satisfies every equation at once. For two variables, each equation is a line in the plane, and a solution is a point where the lines cross. Consider:

x + y = 5 and x - y = 1

Adding the two equations eliminates y: 2x = 6, so x = 3, and then y = 2. The single solution is (3, 2), the point where the two lines meet.

The augmented matrix

Writing the variables over and over is wasteful. We instead record only the numbers in a rectangular array called a matrix. The coefficient matrix holds the coefficients, and the augmented matrix attaches the right-hand-side constants as an extra column, marked off conceptually by a bar. For the system above:

Coefficient matrix	Augmented matrix
[ 1  1 ; 1 -1 ]	[ 1  1 | 5 ; 1 -1 | 1 ]

Each row is one equation, each column is one variable, and the last column is the constants. All of the information lives in these numbers, so from now on we solve systems by manipulating their augmented matrices.

Three possible outcomes

No matter how many equations or variables, a linear system has exactly one of three fates:

• Exactly one solution - the equations pin down a single point. The system is consistent and independent. Example: the crossing lines above.
• No solution - the equations contradict each other. The system is inconsistent. Example: x + y = 2 and x + y = 5 describe parallel lines that never meet.
• Infinitely many solutions - the equations are secretly the same constraint. The system is consistent and dependent. Example: x + y = 2 and 2x + 2y = 4 are the same line.

A system can never have, say, exactly two solutions. This three-way trichotomy is a foundational fact, and the elimination method of the next lesson tells us, mechanically, which case we are in.

Gaussian elimination is a systematic procedure that solves any linear system by simplifying its augmented matrix. It uses three elementary row operations, each of which changes the matrix without changing its solution set:

1. Swap two rows.
2. Scale a row by a nonzero constant.
3. Replace a row by itself plus a multiple of another row.

The goal is row echelon form (REF): all zero rows sit at the bottom, and the first nonzero entry of each row, its pivot, lies to the right of the pivot above it, creating a staircase of zeros in the lower-left. Once in this form, the last equation involves one variable, which we solve and substitute upward, a process called back-substitution.

Worked example, a 3x3 system

Solve the system whose augmented matrix is shown, using the rows R1, R2, R3:

R1: [1 1 1 | 6]   R2: [0 2 5 | -4]   R3: [2 5 -1 | 27]

This came from x + y + z = 6, 2y + 5z = -4, and 2x + 5y - z = 27.

1. Clear the 2 below the first pivot. Replace R3 with R3 - 2*R1: the row becomes [0 3 -3 | 15].
2. Now use the pivot in column 2. Replace R3 with R3 - (3/2)*R2: 3 - (3/2)(2) = 0, -3 - (3/2)(5) = -10.5, and 15 - (3/2)(-4) = 21, giving [0 0 -10.5 | 21].
3. The matrix is now in echelon form:
   [1 1 1 | 6]
[0 2 5 | -4]
[0 0 -10.5 | 21]

Back-substitute from the bottom. Row 3 says -10.5 z = 21, so z = -2. Row 2 says 2y + 5(-2) = -4, so 2y = 6 and y = 3. Row 1 says x + 3 + (-2) = 6, so x = 5. The unique solution is (5, 3, -2). You can verify it in the original third equation: 2(5) + 5(3) - (-2) = 10 + 15 + 2 = 27. Correct.

Reduced row echelon form and free variables

If we keep going, scaling each pivot to 1 and clearing the entries above each pivot too, we reach reduced row echelon form (RREF), where the solution can be read off with no back-substitution. RREF also exposes systems with infinitely many solutions. A column without a pivot corresponds to a free variable, one we may set to any value. For instance, if reduction yields

[1 0 2 | 3]   [0 1 -1 | 4]   [0 0 0 | 0]

then column 3 has no pivot, so z is free. Writing z = t, the first row gives x = 3 - 2t and the second gives y = 4 + t. The solution set is the infinite family (3 - 2t, 4 + t, t). Finally, a row like [0 0 0 | 5] would say 0 = 5, a contradiction signaling an inconsistent system with no solution. Gaussian elimination thus decides all three cases automatically.

A matrix is a rectangular array of numbers arranged in rows and columns. A matrix with m rows and n columns is called an m x n matrix, and the number in row i, column j is its (i, j) entry. Matrices are not just bookkeeping; they are objects with their own arithmetic.

Addition and scalar multiplication

Two matrices of the same size are added entry by entry. A matrix is multiplied by a scalar (a single number) by multiplying every entry. For example, with

A = [1 2 ; 3 4] and B = [5 6 ; 7 8]

we get A + B = [6 8 ; 10 12] and 3A = [3 6 ; 9 12]. Addition of differently sized matrices is undefined.

Matrix multiplication

The product AB is defined only when the number of columns of A equals the number of rows of B. If A is m x n and B is n x p, then AB is m x p. The entry in row i, column j of AB is the dot product of row i of A with column j of B: multiply matching entries and add.

Worked example. Multiply the 2x2 matrices A = [1 2 ; 3 4] and B = [5 6 ; 7 8].

• Top-left: row 1 of A dotted with column 1 of B: 1*5 + 2*7 = 5 + 14 = 19.
• Top-right: row 1 with column 2: 1*6 + 2*8 = 6 + 16 = 22.
• Bottom-left: row 2 with column 1: 3*5 + 4*7 = 15 + 28 = 43.
• Bottom-right: row 2 with column 2: 3*6 + 4*8 = 18 + 32 = 50.

So AB = [19 22 ; 43 50].

Order matters

Unlike ordinary numbers, matrix multiplication is not commutative: in general AB is not equal to BA. Reversing the product above gives BA = [23 34 ; 31 46], a completely different matrix. The identity matrix I, with 1s on the main diagonal and 0s elsewhere, is the one exception that behaves like the number 1: AI = IA = A for any square A. Multiplication also distributes over addition, A(B + C) = AB + AC, and is associative, (AB)C = A(BC), so those familiar rules still hold. Only the freedom to swap factors is lost.

For an ordinary nonzero number a, the reciprocal 1/a undoes multiplication: a * (1/a) = 1. The matrix version is the inverse. A square matrix A is invertible (or nonsingular) if there is a matrix A^(-1) with

A * A^(-1) = A^(-1) * A = I,

where I is the identity. Not every matrix has an inverse. A matrix that lacks one is called singular. Only square matrices can be invertible, and even then it depends on the entries.

The 2x2 inverse formula

For a 2x2 matrix there is a clean shortcut. If A = [a b ; c d], define the number det(A) = ad - bc, called the determinant. Then:

A^(-1) = (1 / (ad - bc)) * [d -b ; -c a]

In words: swap the diagonal entries, negate the off-diagonal entries, and divide everything by ad - bc. The formula works if and only if ad - bc is not zero. If ad - bc = 0, the matrix is singular and has no inverse.

Worked example. Invert A = [4 7 ; 2 6].

1. Compute the determinant: ad - bc = 4*6 - 7*2 = 24 - 14 = 10. Nonzero, so the inverse exists.
2. Swap and negate: [d -b ; -c a] = [6 -7 ; -2 4].
3. Divide by 10: A^(-1) = [0.6 -0.7 ; -0.2 0.4].

Check: A * A^(-1) should be the identity. Top-left entry: 4*0.6 + 7*(-0.2) = 2.4 - 1.4 = 1. Top-right: 4*(-0.7) + 7*0.4 = -2.8 + 2.8 = 0. The full product is [1 0 ; 0 1], as required.

Solving Ax = b with the inverse

Any linear system can be written as a single matrix equation Ax = b, where A is the coefficient matrix, x is the column of unknowns, and b is the column of constants. If A is invertible, multiply both sides on the left by A^(-1):

A^(-1) A x = A^(-1) b, so x = A^(-1) b.

For example, to solve 4x + 7y = 1, 2x + 6y = 3, write b = [1 ; 3] and multiply: x = A^(-1) b = [0.6 -0.7 ; -0.2 0.4][1 ; 3]. The first entry is 0.6*1 + (-0.7)*3 = 0.6 - 2.1 = -1.5, and the second is -0.2*1 + 0.4*3 = -0.2 + 1.2 = 1. So x = -1.5, y = 1. The inverse gives a single clean formula for the solution whenever it exists. For larger matrices we compute the inverse by row-reducing [A | I] until the left block becomes I, at which point the right block is A^(-1).

The determinant is a single number attached to a square matrix that captures crucial information: whether the matrix is invertible, and geometrically, how it scales area or volume. We write it det(A) or with vertical bars around the array.

The 2x2 determinant

For A = [a b ; c d], the determinant is det(A) = ad - bc: the product of the main diagonal minus the product of the anti-diagonal. For instance det[3 1 ; 5 2] = 3*2 - 1*5 = 6 - 5 = 1. When the determinant is zero the matrix is singular; when it is nonzero the matrix is invertible.

The 3x3 determinant by cofactor expansion

For a 3x3 matrix we expand along the first row. Each entry is multiplied by the 2x2 determinant of the matrix left after deleting that entry's row and column (its minor), with a checkerboard of signs + - +. For

A = [a b c ; d e f ; g h i],

det(A) = a(ei - fh) - b(di - fg) + c(dh - eg).

Worked example. Compute the determinant of

A = [2 1 0 ; 1 3 1 ; 0 2 1].

1. First term: 2 * det[3 1 ; 2 1] = 2 * (3*1 - 1*2) = 2 * 1 = 2.
2. Second term: -1 * det[1 1 ; 0 1] = -1 * (1*1 - 1*0) = -1 * 1 = -1.
3. Third term: +0 * det[1 3 ; 0 2] = 0.
4. Add: 2 - 1 + 0 = 1.

So det(A) = 1. Because it is nonzero, this matrix is invertible.

The diagonal shortcut for 3x3

A quick alternative (which works only for 3x3) sums the three full down-right diagonals and subtracts the three full down-left diagonals: det = aei + bfg + cdh - ceg - bdi - afh. Applying it to the same matrix: down-right gives 2*3*1 + 1*1*0 + 0*1*2 = 6 + 0 + 0 = 6; down-left gives 0*3*0 + 1*1*1 + 2*1*2 = 0 + 1 + 4 = 5; the determinant is 6 - 5 = 1, matching the cofactor result.

How row operations change a determinant

• Swapping two rows multiplies the determinant by -1.
• Scaling a row by k multiplies the determinant by k.
• Adding a multiple of one row to another leaves the determinant unchanged.

These rules make large determinants tractable: reduce to triangular form, where the determinant is simply the product of the diagonal entries, keeping track of any swaps and scalings.

The determinant is more than a computation; it is a diagnostic that ties together nearly every idea in this course. Here are its most important properties and what they mean.

Invertibility test

The single most useful fact: a square matrix A is invertible if and only if det(A) is not zero. A zero determinant signals a singular matrix, a system that is either inconsistent or has infinitely many solutions, and columns that are linearly dependent (a connection you will meet in Module 5). So the determinant is a one-number test for a whole cluster of properties.

The multiplicative property

Determinants respect multiplication: for square matrices of the same size,

det(AB) = det(A) * det(B).

For example, if det(A) = 3 and det(B) = 5, then det(AB) = 15 without computing the product at all. A direct consequence is the inverse rule det(A^(-1)) = 1 / det(A), since det(A) det(A^(-1)) = det(I) = 1. Note that determinants do not add: in general det(A + B) is not det(A) + det(B).

Geometric meaning: area and volume

Place the two columns of a 2x2 matrix as arrows from the origin. They span a parallelogram, and the absolute value of the determinant is that parallelogram's area. For A = [3 0 ; 0 2], the columns are along the axes with lengths 3 and 2, spanning a 3-by-2 rectangle of area 6, and indeed det(A) = 6. In three dimensions the absolute value of a 3x3 determinant gives the volume of the parallelepiped spanned by its three columns. A determinant of zero then means the shape has collapsed flat (zero area or volume), which is exactly why zero-determinant matrices are singular: they squash space into a lower dimension and cannot be undone. The sign of the determinant records orientation, whether the transformation flips space (negative) or preserves its handedness (positive).

These properties recur constantly. When you later ask whether a transformation is reversible, whether a basis is valid, or whether eigenvalues make a matrix singular, the determinant is the number you reach for.

A vector in R^n is an ordered list of n real numbers, written as a column such as v = [2 ; 3] or w = [1 ; -1 ; 4]. Geometrically a vector is an arrow from the origin to a point; algebraically it is just its coordinates. Vectors are the raw material of linear algebra.

Addition, scaling, and linear combinations

Vectors of the same length add entry by entry, and a scalar multiplies every entry, exactly like a one-column matrix. Geometrically, addition follows the tip-to-tail rule and scaling stretches or reverses the arrow. Combining these, a linear combination of vectors v1, ..., vk is any expression c1 v1 + c2 v2 + ... + ck vk with scalar weights ci. For example, 2[1 ; 0] + 3[0 ; 1] = [2 ; 3]. Linear combinations are the central operation of the whole subject.

The dot product, length, and angle

The dot product of two equal-length vectors multiplies matching entries and adds: v . w = v1 w1 + v2 w2 + ... + vn wn, producing a single number. For v = [3 ; 4] and w = [2 ; 1], v . w = 3*2 + 4*1 = 10. The dot product measures both size and alignment:

• The length (or norm) of v is ||v|| = sqrt(v . v). For v = [3 ; 4], ||v|| = sqrt(9 + 16) = sqrt(25) = 5.
• The angle between vectors satisfies v . w = ||v|| ||w|| cos(theta).
• Two vectors are orthogonal (perpendicular) exactly when v . w = 0.

For instance [1 ; 0] and [0 ; 1] have dot product 0, confirming the axes are perpendicular.

Span

The span of a set of vectors is the collection of all their linear combinations, that is, every vector you can build from them by scaling and adding. The span of a single nonzero vector in R^2 is the line through the origin along that vector. The span of two vectors that point in different directions is the entire plane R^2. Asking whether a particular vector lies in a given span is the same as asking whether a certain linear system is consistent, which is why Gaussian elimination from Module 1 quietly underlies this geometric language. Span is how we describe, precisely, the reach of a set of vectors.

So far vectors have been columns of numbers. The power of linear algebra comes from noticing that many other objects, such as polynomials, functions, and matrices, obey the very same arithmetic rules. We capture this with the abstract idea of a vector space.

What a vector space is

A vector space is a set V of objects called vectors, together with an addition and a scalar multiplication, that obey the familiar algebraic rules. Concretely, addition must be commutative and associative, there must be a zero vector that adds to leave things unchanged, every vector must have a negative, and scalar multiplication must distribute over addition in both ways, among a total of eight axioms. The point is not to memorize all eight but to recognize the pattern: whenever a set lets you add and scale in the ordinary way, calculus done there behaves like R^n. The space R^n itself is the model example.

Subspaces

Usually we care about a piece of a vector space that is itself a vector space. A subspace of V is a nonempty subset that is closed under the vector-space operations. To check that a subset W is a subspace, you only need three things:

1. The zero vector is in W.
2. W is closed under addition: if u and v are in W, so is u + v.
3. W is closed under scalar multiplication: if v is in W and c is any scalar, then cv is in W.

If all three hold, W is automatically a vector space in its own right; if any fails, it is not a subspace.

Worked example

Is the line W = { [t ; 2t] : t any real } a subspace of R^2?

• Zero: setting t = 0 gives [0 ; 0], so the zero vector is in W. Good.
• Addition: [a ; 2a] + [b ; 2b] = [a+b ; 2(a+b)], again of the form [t ; 2t]. Closed.
• Scaling: c[a ; 2a] = [ca ; 2(ca)], still of the same form. Closed.

All three pass, so this line through the origin is a subspace. By contrast, the line { [t ; 2t + 1] } is not a subspace, because it misses the origin: no value of t gives [0 ; 0]. The subspaces of R^2 are exactly the zero vector alone, every line through the origin, and all of R^2; in R^3 they are the origin, lines through the origin, planes through the origin, and all of R^3. Passing through the origin is the non-negotiable requirement: any subspace must contain the zero vector.

Some vectors in a set carry new information; others are redundant, expressible from the rest. Linear independence makes this precise, and it leads to the single most important idea in the theory of vector spaces: the basis.

Linear independence

Vectors v1, ..., vk are linearly independent if the only way to write the zero vector as a linear combination c1 v1 + ... + ck vk = 0 is with every weight zero (c1 = ... = ck = 0). If some nonzero weights also produce zero, the vectors are linearly dependent, meaning at least one is a combination of the others and adds nothing new.

Worked example. Are v1 = [1 ; 2] and v2 = [2 ; 4] independent? Notice v2 = 2 v1, so 2 v1 - v2 = 0 is a zero combination with nonzero weights. They are dependent. By contrast [1 ; 0] and [0 ; 1] are independent: c1[1 ; 0] + c2[0 ; 1] = [c1 ; c2] = [0 ; 0] forces c1 = c2 = 0.

A reliable test: form a matrix with the vectors as columns and row-reduce. The vectors are independent exactly when every column has a pivot (no free variables). For square matrices, this is the same as a nonzero determinant.

Basis

A basis of a vector space (or subspace) is a set of vectors that is both

• linearly independent (no redundancy), and
• spanning (their linear combinations produce the whole space).

A basis is thus a minimal, complete set of building blocks: enough to reach everything, with nothing wasted. Every vector in the space can be written as a linear combination of basis vectors in exactly one way, and those unique weights are the vector's coordinates in that basis.

The standard basis

The most familiar basis of R^n is the standard basis of unit vectors along the axes. In R^2 it is e1 = [1 ; 0] and e2 = [0 ; 1]; in R^3 it adds e3 = [0 ; 0 ; 1]. Any vector [a ; b] is a*e1 + b*e2, so its coordinates in the standard basis are just its ordinary entries. But other bases exist too: [1 ; 1] and [1 ; -1] also form a basis of R^2, since they are independent and span. Choosing a clever basis is often the key to simplifying a problem, a theme that returns powerfully with eigenvectors.

How big is a vector space? Not how many vectors it holds (usually infinitely many), but how many independent directions it has. That count is its dimension, and for matrices the closely related number is the rank.

Dimension

A remarkable theorem guarantees that every basis of a given vector space has the same number of vectors. That common number is the dimension of the space. So R^2 has dimension 2 (its bases have two vectors), R^3 has dimension 3, and R^n has dimension n. A line through the origin has dimension 1; a plane through the origin has dimension 2. Dimension makes the intuitive notion of "how many degrees of freedom" exact.

Rank

The rank of a matrix is the number of pivots in its row echelon form, equivalently the number of linearly independent rows, which always equals the number of linearly independent columns. It measures how much genuinely independent information the matrix carries.

Worked example. Find the rank of

A = [1 2 1 ; 2 4 3 ; 1 2 2].

1. Replace R2 with R2 - 2*R1: [0 0 1]. Replace R3 with R3 - R1: [0 0 1].
2. Replace R3 with R3 - R2: [0 0 0]. The echelon form is [1 2 1 ; 0 0 1 ; 0 0 0].
3. There are pivots in columns 1 and 3, so rank(A) = 2.

The rank is 2, not 3, because the columns are dependent (the second column is twice the first). A 3x3 matrix of rank less than 3 is singular, with determinant zero, tying rank back to invertibility.

The rank-nullity theorem

The columns without pivots correspond to free variables, and the space of solutions to Ax = 0 (the null space) has dimension equal to the number of free columns, called the nullity. These two counts must add up to the total number of columns:

rank(A) + nullity(A) = number of columns of A.

For the matrix above, rank = 2 and there is 1 free column, so nullity = 1, and indeed 2 + 1 = 3 columns. This rank-nullity theorem is a precise conservation law: every column is either a pivot column (contributing to rank) or a free column (contributing to nullity), never both and never neither. It links the size of a matrix's range to the size of the space it collapses to zero, and it will reappear as we study linear transformations next.

A matrix is not only a static array; it is a machine that transforms vectors. Multiplying a vector x by a matrix A produces a new vector Ax, so the matrix defines a function from one space to another. The functions that arise this way are exactly the linear transformations.

The definition

A function T from R^n to R^m is a linear transformation if it respects the two vector-space operations, for all vectors u, v and every scalar c:

1. Additivity: T(u + v) = T(u) + T(v).
2. Homogeneity: T(c v) = c T(v).

Together these say T preserves linear combinations: T(a u + b v) = a T(u) + b T(v). A consequence is that T must send the zero vector to the zero vector. Multiplication by a matrix always satisfies both rules, and conversely every linear transformation between coordinate spaces is multiplication by some matrix.

The standard matrix

Because a linear transformation preserves combinations, it is completely determined by what it does to the standard basis vectors. The standard matrix of T is the matrix whose columns are the images T(e1), T(e2), ..., T(en). Then T(x) = Ax for every x.

Worked example. Let T rotate the plane 90 degrees counterclockwise. It sends e1 = [1 ; 0] to [0 ; 1] and e2 = [0 ; 1] to [-1 ; 0]. Placing these images as columns gives the standard matrix

A = [0 -1 ; 1 0].

Check on [3 ; 0]: A[3 ; 0] = [0*3 + (-1)*0 ; 1*3 + 0*0] = [0 ; 3], correctly rotating the arrow a quarter turn.

A gallery of 2D transformations

Transformation	Standard matrix
Scale by factor k	[k 0 ; 0 k]
Reflect across the x-axis	[1 0 ; 0 -1]
Rotate 90 degrees counterclockwise	[0 -1 ; 1 0]
Horizontal shear by k	[1 k ; 0 1]
Project onto the x-axis	[1 0 ; 0 0]

Composing two transformations corresponds to multiplying their matrices: doing S after T is the matrix product (matrix of S)(matrix of T). This is the deep reason matrix multiplication is defined by the row-by-column rule and why its order matters: applying transformations in a different order generally gives a different result.

Two subspaces reveal everything about a linear transformation T(x) = Ax: what it sends to zero, and what it can produce. These are the kernel and the range.

The kernel

The kernel (or null space) of T is the set of all inputs that T crushes to the zero vector: ker(T) = { x : Ax = 0 }. It is a subspace of the input space, and finding it means solving the homogeneous system Ax = 0. The kernel measures how much information the transformation destroys.

Worked example. For A = [1 2 ; 2 4], solve Ax = 0. Both rows give x1 + 2 x2 = 0, so x1 = -2 x2. Setting x2 = t, the kernel is { t[-2 ; 1] : t real }, a line through the origin. Because the kernel contains nonzero vectors, this transformation is not reversible; different inputs collapse to the same output.

The range

The range (or column space) of T is the set of all outputs it can actually produce: range(T) = { Ax : x in R^n }. Since Ax is a linear combination of the columns of A with weights from x, the range is exactly the span of the columns of A. Its dimension is the rank of A. For the matrix above, both columns are multiples of [1 ; 2], so the range is the single line spanned by [1 ; 2], and the rank is 1.

One-to-one and onto

The kernel and range answer the two basic questions about any function.

• T is one-to-one (injective, distinct inputs give distinct outputs) exactly when its kernel is just the zero vector. Any larger kernel means collisions.
• T is onto (surjective, it reaches every vector in the target space) exactly when its range is the whole target space, that is, when the rank equals the dimension of the codomain.

The matrix A = [1 2 ; 2 4] is neither: its kernel is a line (so not one-to-one) and its range is a line inside R^2 (so not onto). Notice how the rank-nullity theorem ties this together: for this 2-column matrix, rank + nullity = 1 + 1 = 2. Growing the range shrinks the kernel and vice versa. For a square matrix, one-to-one and onto happen together, and both are equivalent to the matrix being invertible with nonzero determinant, the grand unifying statement of the first half of this course.

Most vectors change direction when a matrix acts on them. A precious few do not: they are only stretched or shrunk, keeping their line. These special directions are the eigenvectors, and the stretch factors are the eigenvalues. They expose the deepest structure of a matrix.

The eigenvalue equation

A nonzero vector v is an eigenvector of a square matrix A, with eigenvalue lambda, if

A v = lambda v.

In words, multiplying by A has the same effect on v as simply scaling it by the number lambda. The direction is preserved; only the length (and possibly the sign) changes. We require v to be nonzero, since A * 0 = 0 always holds trivially.

The characteristic equation

Rewrite A v = lambda v as A v - lambda v = 0, or (A - lambda I) v = 0. For a nonzero v to solve this homogeneous system, the matrix A - lambda I must be singular, which means its determinant is zero. That gives the characteristic equation:

det(A - lambda I) = 0.

Its solutions are the eigenvalues. For a 2x2 matrix this is a quadratic in lambda, so there are at most two eigenvalues.

Worked example

Find the eigenvalues and eigenvectors of A = [2 1 ; 1 2].

1. Form A - lambda I = [2 - lambda, 1 ; 1, 2 - lambda].
2. Its determinant is (2 - lambda)^2 - (1)(1) = lambda^2 - 4 lambda + 4 - 1 = lambda^2 - 4 lambda + 3.
3. Set it to zero and factor: lambda^2 - 4 lambda + 3 = (lambda - 1)(lambda - 3) = 0. So lambda = 1 and lambda = 3.

Now find an eigenvector for each.

• For lambda = 3: solve (A - 3I)v = 0, that is [-1 1 ; 1 -1] v = 0. The equation is -v1 + v2 = 0, so v1 = v2. An eigenvector is [1 ; 1]. Check: A[1 ; 1] = [2*1 + 1*1 ; 1*1 + 2*1] = [3 ; 3] = 3[1 ; 1]. Correct.
• For lambda = 1: solve (A - I)v = 0, that is [1 1 ; 1 1] v = 0. The equation is v1 + v2 = 0, so v1 = -v2. An eigenvector is [1 ; -1]. Check: A[1 ; -1] = [2 - 1 ; 1 - 2] = [1 ; -1] = 1[1 ; -1]. Correct.

Each eigenvector spans a line that the matrix leaves in place, stretching it by its eigenvalue. Eigenvectors are only determined up to a scalar, so any nonzero multiple of [1 ; 1] is also an eigenvector for lambda = 3.

Eigenvectors are useful because, in the basis they provide, a complicated matrix becomes a simple diagonal one. This process, diagonalization, turns hard matrix computations into easy scalar ones.

What diagonalization means

A square matrix A is diagonalizable if it can be written as

A = P D P^(-1),

where D is a diagonal matrix and P is invertible. The recipe is built directly from eigen-data: the columns of P are eigenvectors of A, and D is the diagonal matrix of the corresponding eigenvalues in the same order. An n x n matrix is diagonalizable exactly when it has n linearly independent eigenvectors, enough to form a basis.

Worked example

Diagonalize A = [2 1 ; 1 2], using the eigen-data from the previous lesson: eigenvalue 3 with eigenvector [1 ; 1], and eigenvalue 1 with eigenvector [1 ; -1].

1. Put the eigenvectors as columns of P: P = [1 1 ; 1 -1].
2. Put the matching eigenvalues on the diagonal of D in the same order: D = [3 0 ; 0 1].
3. Then A = P D P^(-1). Here det(P) = 1*(-1) - 1*1 = -2, so P^(-1) = (1/-2)[-1 -1 ; -1 1] = [0.5 0.5 ; 0.5 -0.5].

The two independent eigenvectors guarantee this works. In the eigenvector basis, A simply scales one axis by 3 and the other by 1, which is exactly what D encodes.

Why it matters: matrix powers

Diagonalization makes powers effortless. Because the inner P^(-1) P factors cancel,

A^k = P D^k P^(-1),

and raising a diagonal matrix to a power just raises each diagonal entry to that power: D^k = [3^k 0 ; 0 1^k]. So instead of multiplying A by itself k times, you compute 3^k and 1^k and reassemble. This is the engine behind long-run predictions: in a Markov chain that models how a population moves between states each step, the eigenvalues govern the steady state, and in models of population growth or discrete dynamical systems, the largest eigenvalue sets the long-term growth rate. The eigenvalue with the biggest absolute value dominates as k grows, so the system lines up with its leading eigenvector. Eigenvalues thus predict long-term behavior that would be invisible in the raw matrix, which is why they matter far beyond pure mathematics.

Orthogonality, the linear-algebra word for perpendicularity, brings geometry into full force. It makes bases easy to use and underlies the most important application of the whole subject, least-squares fitting.

Orthogonal and orthonormal sets

Recall that two vectors are orthogonal when their dot product is zero. A set of vectors is an orthogonal set if every pair in it is orthogonal. If, in addition, every vector has length 1, the set is orthonormal. The standard basis {[1 ; 0], [0 ; 1]} is orthonormal: the two vectors are perpendicular and each has length 1. Orthogonal bases are prized because the coordinates of any vector in such a basis come straight from dot products, with no system to solve.

Projecting one vector onto another

Given a vector b and a direction a, the projection of b onto a is the shadow b casts along the line through a. It is the point on that line closest to b, and the formula is

proj_a(b) = ( (a . b) / (a . a) ) a.

Worked example. Project b = [3 ; 4] onto a = [1 ; 0].

1. Numerator: a . b = 1*3 + 0*4 = 3.
2. Denominator: a . a = 1*1 + 0*0 = 1.
3. Projection: (3/1)[1 ; 0] = [3 ; 0].

The shadow of [3 ; 4] onto the x-axis is [3 ; 0], exactly as geometry predicts. The leftover piece, b - proj_a(b) = [0 ; 4], is perpendicular to a; every vector splits uniquely into a part along a and a part orthogonal to it.

Why projections matter

Projection answers the question "what is the closest vector in a subspace to a given target?" The gap between the target and its projection is as small as possible and is orthogonal to the subspace. That optimality is the key to fitting models to imperfect data, the subject of the next lesson. A related procedure, Gram-Schmidt, uses repeated projections to convert any basis into an orthonormal one by subtracting off overlaps, turning a tilted coordinate system into a clean perpendicular one.

Real data rarely fits a model exactly, so the system Ax = b is often inconsistent, with no exact solution. Linear algebra offers the next best thing: the vector x that comes as close as possible. This is the least-squares method, and it is projection in action.

The least-squares idea

If Ax = b has no solution, then b lies outside the column space of A. The best we can do is replace b by its projection onto the column space, the closest reachable vector, and solve exactly for that. The resulting x-hat minimizes the length of the error ||Ax - b||, which is the same as minimizing the sum of squared errors, hence the name. The condition that the error be orthogonal to the column space gives the normal equations:

A^T A x-hat = A^T b,

where A^T is the transpose of A (rows and columns swapped). These always have a solution, and it is the least-squares answer.

Worked example: fitting a line

Suppose we want the best-fit line y = m x + c through the three points (0, 1), (1, 2), and (2, 2). Each point demands c + m x = y, giving an overdetermined system with

A = [1 0 ; 1 1 ; 1 2], unknowns [c ; m], and b = [1 ; 2 ; 2].

Build the normal equations. The transpose product A^T A collects the sums: with three points, sum x = 3 and sum x^2 = 0 + 1 + 4 = 5, so

A^T A = [3 3 ; 3 5] and A^T b = [5 ; 6]

(since sum y = 1 + 2 + 2 = 5 and sum x y = 0 + 2 + 4 = 6). Solve [3 3 ; 3 5][c ; m] = [5 ; 6]. Subtracting the equations, 2m = 1, so m = 0.5, and then 3c + 3(0.5) = 5 gives 3c = 3.5, so c = 7/6. The best-fit line is y = 0.5 x + 7/6. No straight line hits all three points, but this one makes the total squared miss as small as possible.

Applications across fields

The tools of this course power an enormous range of technology:

• Computer graphics and games move and rotate objects with transformation matrices, exactly the ones from Module 6.
• Search engines historically ranked web pages using the dominant eigenvector of a huge link matrix, a direct use of eigenvalues.
• Data science and statistics compress and analyze data by projecting onto the directions of greatest variance, an eigenvalue technique.
• Engineering and physics model vibrations, circuits, and quantum states as eigenvalue problems, where eigenvalues are natural frequencies or energy levels.
• Machine learning trains many models by solving least-squares and related linear systems at massive scale.

Every one of these rests on the same handful of ideas you have now learned: systems, matrices, vector spaces, eigenvalues, and orthogonality. Linear algebra is the quiet engine beneath modern computation, and you have just seen how it runs.