The Math Behind Machine Learning

Source: Concepts adapted from Ace the Data Science Interview — Chapter 7 (“Machine Learning”) by Kevin Huo & Nick Singh. Rewritten in original wording for this course.

Probability and stats show up constantly in DS interviews. Full linear-algebra and multivariable-calculus proofs are rarer — unless you are interviewing for research scientist or quant finance, where whiteboarding least squares, PCA, or related derivations is fair game. Even then, interviewers sometimes hide the math inside a product story (e.g. “what drives portfolio factor returns?” → PCA).

For everyone else: you still want the basics. They unlock why PCA, regression, and neural-net training work the way they do. This lesson covers that foundation — linear algebra, gradient descent, evaluation / selection, bias–variance, overfitting, regularization, interpretability — then how models are actually trained: cross-validation, bootstrapping / bagging, hyperparameter search, and learning curves. Concept quizzes, slider explorers (same FormulaExplorer pattern as Statistics), numeric drills, labs, and coding after each core topic.

Linear algebra

The linear-algebra topic that comes up most often is eigenvalues and eigenvectors. For an $n \times n$ matrix $A$, a nonzero vector $x$ is an eigenvector with eigenvalue $\lambda$ when:

$$Ax = \lambda x$$

Geometrically, $A$ is a linear map. On an eigenvector it only scales $x$ (same direction, factor $\lambda$).

Writing a square matrix in terms of its eigenvectors is an eigendecomposition. Not every matrix is square; non-square matrices use singular value decomposition (SVD):

$$A = U \Sigma V^{\top}$$

with $U$ ($m \times m$), $\Sigma$ ($m \times n$), and $V$ ($n \times n$).

ML uses this toolkit constantly: matrix multiplies in backprop, eigendecomposition of a covariance matrix in PCA, projections, inverses, determinants, orthonormal bases, and diagonalization. Research / quant loops may ask you to whiteboard follow-ups on PCA or linear regression from these building blocks.

Quiz

Linear algebra for ML

1 / 5

x is an eigenvector of A when…

Pick an answer to see if you are right.

Compute

Compute: eigenvalues

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

Let A = [[5, 0], [0, 2]] and x = [1, 0]. If Ax = λx, what is the eigenvalue λ for this eigenvector? Round to 2 decimals.

Enter a number (at most 2 decimal places).

Gradient descent

Training minimizes an objective (loss / cost): how badly the model fits the data. Lower is better. Optimization methods search for good parameters.

Gradient descent repeatedly steps opposite the gradient — the direction of steepest increase — so each step goes “downhill”:

$$x_{t+1} = x_t - \alpha\, \nabla f(x_t)$$

Here $\alpha > 0$ is the learning rate. For convex losses, iterating until successive $x$ values barely change means convergence near a minimum.

Many losses are sums over examples, so the full gradient is a sum of per-example gradients. Computing that every step is expensive, and non-convex landscapes have local minima and saddles. Variants:

Method	Data per step	Notes
Batch GD	Full training set	Stable gradient; costly
Stochastic GD (SGD)	One example (classic)	Noisy; escapes flats; unbiased gradient estimate
Mini-batch GD	Small fixed batch	Common practical middle ground

Play with a 1-D bowl (landscape + iteration path) — same FormulaExplorer pattern as in Statistics:

Interviews like GD/SGD because almost every modern trainer uses them. Expect “when would you use which?” and occasionally a tiny SGD coding sketch.

Quiz

Gradient descent

1 / 5

The learning rate α in x ← x − α ∇f(x) controls…

Pick an answer to see if you are right.

Compute

Compute: GD updates

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

Point (x,y)=(3,6), learning rate α=0.1, gradient ∇f=(4,1). After one GD step, what is the new x-coordinate? Round to 2 decimals.

Enter a number (at most 2 decimal places).

Model evaluation and selection

Evaluation: after fitting on a train set, score the model on held-out test data. A common split is roughly 80% train / 20% test. Generalization — performance on unseen data — is what matters.

Selection: after evaluating candidates, pick the model to ship, under product and business constraints. Tiny metric lifts can be huge in production (e.g. ad CTR), so interview case studies often turn into “compare models and choose.”

When data is scarce, a single 80/20 cut is noisy — that is where cross-validation (next section) earns its keep. Until then, treat “held-out performance” as the north star.

Quiz

Evaluation & selection

1 / 4

Why hold out a test set?

Pick an answer to see if you are right.

Compute

Compute: train/test sizes

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

You run 10-fold cross-validation on 200 examples (equal folds). How many examples are in each validation fold?

Enter a number (at most 0 decimal places).

Bias–variance trade-off

We try to learn a function $f(x)$ that predicts $y$ from $x$:

$$y = f(x) + \varepsilon$$

with noise $\varepsilon$ (often modeled as zero-mean Gaussian in regression). Prediction error decomposes into:

1. Bias — how far average predictions sit from the true $f$ (lower is better)
2. Variance — how much predictions swing when the training set changes (lower is better)
3. Irreducible error — noise you cannot remove

High bias / low variance (e.g. a too-simple linear model): systematically off, but stable.
Low bias / high variance (e.g. an over-flexible net): can track $f$ closely but thrash with different samples.

Interviews rarely demand the full algebra; they want situational reasoning: more data / regularization for high variance; richer features / more flexible models for high bias — always tied to product needs.

Quiz

Bias–variance

1 / 5

High bias usually means the model…

Pick an answer to see if you are right.

Overfitting, underfitting, and complexity

George Box: “All models are wrong, but some are useful.” We want models that generalize. Occam’s razor in ML: simpler, more parsimonious models often travel better out of sample.

Overfitting: train fit looks great because the model absorbed noise; test performance suffers.
Underfitting: the model never captured the real structure (often high bias).

Interviewers ask how you detect overfitting (train≪test gap, CV scores, learning curves) and how you fight it — which leads to regularization.

Quiz

Overfitting & complexity

1 / 4

Overfitting means the model…

Pick an answer to see if you are right.

Regularization

Regularization shrinks effective complexity: usually lower variance, slight higher bias. Penalties added to the objective are also called shrinkage methods.

Method	Penalty idea	Typical effect
L1 (lasso)	$\lambda \sum \|w_i\|$	Many weights → exactly 0 (sparse / feature selection)
L2 (ridge)	$\lambda \sum w_i^2$	Shrinks weights; rarely exact zeros
Elastic net	Mix of L1 + L2	Sparse + grouped-shrinkage blend

Use L1 when you want sparsity; L2 when you want smooth shrinkage of correlated features; elastic net when you want both flavors.

Quiz

Regularization

1 / 5

Relative to bias–variance, regularization usually…

Pick an answer to see if you are right.

Compute

Compute: L1 / L2 penalties

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

L1 (lasso) penalty is λ∑|wᵢ|. Weights w=[3, -1, 0] and λ=0.5. What is the penalty value? Round to 2 decimals.

Enter a number (at most 2 decimal places).

Practice: ridge, lasso, elastic net

Pure Python / NumPy (no sklearn) — easy → hard.

Interpretability and explainability

Kaggle may optimize a single metric. Production often also needs why — loan denials, clinical decisions, bias audits, compliance. More complex models can raise accuracy while lowering interpretability.

Built-in hooks:

• Linear models — coefficients
• Random forests — feature-importance scores

Model-agnostic tools:

• SHAP — Shapley-style average marginal contribution of a feature across coalitions
• LIME — local sparse linear surrogate around one prediction

You rarely derive SHAP in an interview; bringing up the performance ↔ interpretability trade-off in case questions is the win.

Quiz

Interpretability

1 / 5

Why might a bank care about interpretability beyond AUC?

Pick an answer to see if you are right.

Model training

Evaluation tells you which model won. Training is how you teach candidates in the first place. The 80/20 split is only the start: cross-validation, resampling, hyperparameter search, and learning curves decide whether that win was luck or signal.

Cross-validation

Cross-validation (CV) estimates performance by rotating which slice of the training data is held out. For each subsample you train on most of the data, score on the left-out block, then combine the scores. That avoids “always testing the same lucky 20%” and is especially valuable when more labeled data is expensive.

$k$-fold CV (classic interview ask):

1. Shuffle into $k$ equal folds.
2. For each fold $i$: train on all folds except $i$; validate on fold $i$.
3. Average the $k$ validation errors → estimate of true error.

Leave-one-out CV (LOOCV) is $k$-fold with $k = n$ (one point held out each round). Thorough, but $n$ full retrainings — painful on large $n$.

On big datasets, full CV can be too expensive. Prefer a three-way split: train / validation (dev) / test, with the val set often ~10–20%. Use val for model / hyperparameter choices; keep test sealed for a final report.

Time series: do not shuffle into random $k$-folds — that leaks the future into the past. Train only on history up to a cutoff, validate the next window, then slide / expand the cutoff forward.

Quiz

Cross-validation

1 / 5

k-fold CV mainly helps you…

Pick an answer to see if you are right.

Compute

Compute: CV averages & splits

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

n=1000 rows with a 60/15/25 train/val/test split. How many rows are in the validation (dev) set?

Enter a number (at most 0 decimal places).

Bootstrapping and bagging

Bootstrapping: sample with replacement from your data, estimate a quantity on each draw, then average (or otherwise aggregate) those estimates. Useful when $n$ is small, and for rare classes (oversample the minority via bootstrap draws).

Bagging (bootstrap aggregation) builds many models on bootstrap samples and averages their predictions — the engine behind random forests. Ensembles (RF, AdaBoost, XGBoost) are interview favorites; a classic compare/contrast is “random forest vs XGBoost” (bagged parallel trees vs boosted sequential trees — details in Random Forests & Boosting).

Quiz

Bootstrap & bagging

1 / 4

Bootstrapping draws samples…

Pick an answer to see if you are right.

Hyperparameter tuning

Hyperparameters (learning rate, tree depth, regularization strength, …) shape training time, cost, and generalization. You set them outside the core parameter updates.

Method	Idea	Trade-off
Grid search	Cartesian product of candidate values; try all	Exhaustive but cost grows exponentially in #hparams
Random search	Sample each hparam from a distribution	Explores large spaces cheaper; no optimality guarantee
Bayesian optimization	Model which regions look promising; sample there	Smarter sequential search (common for NNs / boosting)

Interview bar: name a few hyperparameters for a model you like and what each does to fit vs generalize. Deep Bayes-opt math is rarer outside research / MLE loops.

Quiz

Hyperparameter tuning

1 / 4

Grid search’s main downside is…

Pick an answer to see if you are right.

Compute

Compute: grid size & bootstrap uniques

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

You draw a bootstrap sample of size 200 with replacement from 200 rows. About how many unique rows do you expect? Use n(1−(1−1/n)ⁿ). Round to 2 decimals.

Enter a number (at most 2 decimal places).

Training time and learning curves

Training cost scales with $n$ (rows) and $d$ (features / parameters). Big-O sketches (coding chapter) help compare algorithms; in production you also weigh wall-clock and infra cost against a tiny metric lift.

Learning curves plot a metric (accuracy, loss, …) against experience (iterations or dataset size). Overlay train and validation:

• Train ↑ while val stalls / worsens → overfitting (early stop, regularize, simplify).
• Both stuck low → underfitting.
• Wide gap that never shrinks → data may be non-representative (or leakage / mismatch).

Classic interview prompt: “How do you tell the model is overfitting?” — answer with learning curves + train/val gap, not only vibes.

Quiz

Learning curves & training cost

1 / 4

Train metric keeps improving but validation gets worse late in training. Likely…

Pick an answer to see if you are right.

Quick synthesis

Idea	Interview sound bite
Eigen / SVD	Scaling directions of a linear map; PCA / low-rank structure
Gradient descent	Walk opposite ∇f with step α; SGD adds noise / speed
Eval / selection	Hold out data; pick under product constraints
Bias–variance	Simple vs flexible models; fix with data / capacity / regularization
Regularization	L1 sparsifies; L2 shrinks; elastic net mixes
Explainability	Coefficients, importances, SHAP / LIME when the box is dark
Cross-validation	Rotate held-out folds; LOOCV = k=n; no shuffle for time series
Bootstrap / bagging	Resample with replacement; aggregate many models (e.g. RF)
Hparam search	Grid (exhaustive) vs random / Bayesian (scalable exploration)
Learning curves	Train vs val over time — diagnose under/overfit and data gaps
Coding drills	After each topic: lin-alg, GD, ridge/lasso/elastic-net, tuning

Next: Linear Regression · Classification fundamentals. Later model-family lessons can reuse the same Piston sklearn stack.