🚧 Lesson 3 of 25 in Level 05
Level 05 • Lesson 3

Eigenvalues & Eigenvectors

Understanding transformations. Principal components.

Eigenvalues and Eigenvectors

For a matrix A, find v and λ such that:

A @ v = λ * v # v = eigenvector (direction) # λ = eigenvalue (scaling factor) # The eigenvector direction doesn't change, # only its magnitude (scaled by λ)

Intuition

Geometric view: Most vectors rotate when multiplied by A. Eigenvectors are special directions that only stretch or shrink.

In Machine Learning

📝 Practice Exercises

Exercise 1: Find Eigenvalues

Given matrix A:

A = [[4, 2], [1, 3]]

Task: Find the eigenvalues by solving det(A - λI) = 0

Hint: Expand (4-λ)(3-λ) - 2×1 = 0

Show Solution

Characteristic equation: (4-λ)(3-λ) - 2 = 0

12 - 7λ + λ² - 2 = 0 → λ² - 7λ + 10 = 0

(λ - 5)(λ - 2) = 0

Eigenvalues: λ₁ = 5, λ₂ = 2

Exercise 2: Verify Eigenvector

Given matrix A and vector v:

A = [[2, 1], [1, 2]] v = [1, 1]

Task: Show that v is an eigenvector and find its eigenvalue.

Show Solution

A @ v = [2×1 + 1×1, 1×1 + 2×1] = [3, 3]

[3, 3] = 3 × [1, 1] = 3v ✓

Eigenvalue: λ = 3

Exercise 3: PCA Application (Python)

Complete the code to find principal components using eigendecomposition:

import numpy as np # Sample data (2D points) X = np.array([[1, 2], [2, 3], [3, 4], [4, 5]]) # Step 1: Center the data X_centered = X - np.mean(X, axis=0) # Step 2: Compute covariance matrix cov_matrix = np.cov(X_centered.T) # Step 3: Find eigenvalues and eigenvectors eigenvalues, eigenvectors = np.linalg.eig(_______) # Step 4: Principal component is eigenvector with largest eigenvalue print("Eigenvalues:", eigenvalues) print("Principal component:", eigenvectors[:, np.argmax(_______)])
Show Solution
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix) print("Principal component:", eigenvectors[:, np.argmax(eigenvalues)])

🧠 Quick Quiz

Question 1

If Av = λv, what does λ represent?

  • A) The rotation angle applied to v
  • B) The scaling factor (eigenvalue)
  • C) The determinant of A
  • D) The inverse of v
Show Answer

B) The scaling factor (eigenvalue) — λ tells us how much the eigenvector v is stretched or shrunk.

Question 2

In PCA, which eigenvectors do we select as principal components?

  • A) Those with the smallest eigenvalues
  • B) Those with eigenvalues equal to 1
  • C) Those with the largest eigenvalues
  • D) All eigenvectors equally
Show Answer

C) Those with the largest eigenvalues — Larger eigenvalues correspond to directions of maximum variance in the data.

Question 3

What does it mean if a matrix has an eigenvalue of 0?

  • A) The matrix is the identity matrix
  • B) The matrix is singular (non-invertible)
  • C) All eigenvectors are orthogonal
  • D) The matrix is symmetric
Show Answer

B) The matrix is singular (non-invertible) — det(A) = product of eigenvalues, so if any λ=0, det(A)=0.

Question 4

For a symmetric matrix, eigenvectors corresponding to different eigenvalues are always:

  • A) Parallel
  • B) Orthogonal
  • C) Zero vectors
  • D) Unit vectors
Show Answer

B) Orthogonal — Symmetric matrices have orthogonal eigenvectors, which is why PCA works so well.

Question 5

In neural networks, eigenvalues of the weight matrix help determine:

  • A) The learning rate schedule
  • B) Whether gradients will explode or vanish
  • C) The number of layers needed
  • D) The activation function choice
Show Answer

B) Whether gradients will explode or vanish — Eigenvalues > 1 can cause exploding gradients; < 1 can cause vanishing gradients.