Linear Regression

Source: Concepts adapted from Ace the Data Science Interview — Chapter 7 (“Machine Learning”) by Kevin Huo & Nick Singh. Rewritten in original wording for this course. Coding labs use scikit-learn / NumPy on Piston.

Linear regression is the workhorse continuous-target model in interviews: clear math, readable coefficients, and a gateway to regularization, GLMs, and “why did this coefficient flip sign?” debugging. This lesson covers the model, OLS, assumptions, metrics, multicollinearity, and regularized cousins — plus quizzes, slider explorers (same FormulaExplorer pattern as Statistics / Math Behind ML), and Python coding drills after each topic (OLS, metrics, sklearn).

Piston ML stack

The learn-programming Piston runtime installs scikit-learn (import as sklearn) via api/src/learn_runtimes.json, along with NumPy / SciPy / pandas. That same package covers other classical algos later (logistic regression, trees, k-NN, SVM, clustering, pipelines). Restart / re-ensure the API after pulling so pip libs refresh.

The model

We model a continuous response as a linear function of features plus noise:

$$y = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p + \varepsilon$$

In matrix form $y = X\beta + \varepsilon$ (with a column of ones in $X$ for the intercept). Simple regression has one $x$; multiple regression has many.

Interpretation: $\beta_j$ is the change in expected $y$ for a one-unit increase in $x_j$, holding other features fixed (ceteris paribus). That “holding fixed” story breaks down under multicollinearity or omitted-variable bias — interview gold.

Quiz

Linear model basics

1 / 4

In ŷ = β₀ + β₁x, β₁ is…

Pick an answer to see if you are right.

Compute

Compute: predictions & slopes

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

A line through (1, 0) and (5, 4) has slope (y₂−y₁)/(x₂−x₁). What is the slope? Round to 2 decimals.

Enter a number (at most 2 decimal places).

Ordinary least squares (OLS)

OLS chooses $\beta$ to minimize squared residuals:

$$\hat\beta = \arg\min_\beta \| y - X\beta \|_2^2$$

When $X^{\top}X$ is invertible, the closed form is:

$$\hat\beta = (X^{\top}X)^{-1} X^{\top} y$$

(Quant / research loops sometimes ask you to derive this via $\nabla_\beta = 0$.) Numerically, libraries use QR / SVD rather than forming $(X^{\top}X)^{-1}$ naively.

Geometrically, $\hat y = X\hat\beta$ is the projection of $y$ onto the column space of $X$. Residuals $e = y - \hat y$ are orthogonal to those columns.

Quiz

OLS

1 / 4

OLS minimizes…

Pick an answer to see if you are right.

Practice: OLS

NumPy / pure Python — easy → hard. After Check, compare Expected vs Received in the results table.

Assumptions (interview checklist)

Classic “LINE”-style assumptions for inference / BLUE-style stories:

1. Linearity — $E[y \mid X]$ is linear in the parameters (or transform features)
2. Independence — errors independent (watch time series / clustered data)
3. Homoscedasticity — constant error variance (else WLS / robust SEs)
4. Normality of errors — mainly for exact $t$/$F$ inference; prediction can still work without it
5. No perfect multicollinearity — full column rank
6. Exogeneity — errors uncorrelated with predictors (causal claims need more)

Interviewers love: name the assumptions, say what breaks, and what you do (plots, transforms, robust methods, different model).

Quiz

Assumptions

1 / 4

Homoscedasticity means…

Pick an answer to see if you are right.

Fit metrics

Metric	Formula idea	Notes
MSE	$\frac{1}{n}\sum e_i^2$	Same units² as $y$
RMSE	$\sqrt{\mathrm{MSE}}$	Same units as $y$
MAE	$\frac{1}{n}\sum \|e_i\|$	More robust to outliers than MSE
R²	$1 - \mathrm{SS_{res}}/\mathrm{SS_{tot}}$	Fraction of variance “explained”; can mislead

Always compare metrics on held-out data (or CV). High train R² + weak test R² → overfit.

Quiz

Regression metrics

1 / 4

RMSE relates to MSE by…

Pick an answer to see if you are right.

Compute

Compute: MSE, RMSE, R²

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

Residuals e = [1, -1, 2, 0]. MSE = (1/n)∑eᵢ². What is MSE? Round to 2 decimals.

Enter a number (at most 2 decimal places).

Practice: fit metrics

Implement MAE, MSE/RMSE, R², then the full pack — easy → hard.

Multicollinearity and regularization

Multicollinearity: predictors move together → unstable, hard-to-interpret $\beta$s (large SEs, sign flips). Checks: correlation matrix, VIFs, condition number. Fixes: drop / combine features, domain constraints, ridge.

Ridge (L2) and lasso (L1) add penalties (see Math Behind ML): shrink coefficients; lasso can zero some out. Use when $p$ is large, features correlated, or you want sparsity.

Quiz

Multicollinearity & regularized regression

1 / 4

Strong multicollinearity typically…

Pick an answer to see if you are right.

Practice: scikit-learn

Use the Piston sklearn stack for fit/predict and a ridge shrink check.

Expected vs received

Click Check on any exercise: the results panel lists every test in a table with Expected and Received side by side so mismatches are obvious.

Quick synthesis

Topic	Sound bite
Model	$y = X\beta + \varepsilon$; read $\beta_j$ as a partial effect
OLS	Minimize $\|y - X\beta\|^2$; $\hat\beta = (X^{\top}X)^{-1}X^{\top}y$ when invertible
Assumptions	LINE + rank + exogeneity for clean inference
Metrics	MAE / MSE / RMSE / R² — always check held-out
Collinearity	Unstable β; ridge / drop features / combine
Code	Drills after OLS, after metrics, and after regularization (sklearn)

Next model-family lessons: Classification fundamentals and Classification models. The same Piston sklearn stack covers logistic regression, trees, and more.