Random Forests & Boosting

Source: Concepts adapted from Ace the Data Science Interview — Chapter 7 (“Machine Learning”) by Kevin Huo & Nick Singh. Rewritten in original wording for this course. Coding uses scikit-learn on Piston.

A single decision tree can overfit: grow deep enough and every training row can sit in its own leaf. Ensembles average (or sequentially correct) many trees so out-of-sample predictions stabilize. This lesson covers random forests and boosting (AdaBoost → gradient boosting / XGBoost) — the compare/contrast pair interviewers love — with slider explorers, quizzes, and coding after each block.

Prereq: Classification models (trees, entropy / IG) and bagging intuition from Math Behind ML.

Random forests

A random forest fits many decision trees and aggregates their votes (classification) or averages their predictions (regression). Two design choices cut overfitting and tree-to-tree correlation:

1. Bagging — each tree trains on a bootstrap sample of rows; predictions are averaged. Averaging uncorrelated trees shrinks variance roughly like $\sigma^2/B$.
2. Feature subsample (mtry) — at each split, only a random subset of features is eligible. Strong features cannot dominate the top of every tree, so trees diversify.

Why teams use RF: solid default accuracy, feature importance, trees train in parallel, and the story is easy to tell. Interview bar: RF vs one tree, and RF vs boosting.

Quiz

Random forests

1 / 4

A single deep tree often overfits because…

Pick an answer to see if you are right.

Compute

Compute: bagging / mtry

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

Bagging average of B uncorrelated trees each with variance σ². σ²=4, B=4. What is Var(mean) = σ²/B? Round to 2 decimals.

Enter a number (at most 2 decimal places).

Practice: bagging & forests

Boosting

Boosting trains a sequence of weak models (often shallow trees). Each new model focuses on where the current ensemble is still wrong.

• AdaBoost: assign a weight to each training point; misclassified points get higher weight next round. Final prediction is a weighted vote of the weak learners.
• Gradient boosting: identify shortcomings via the gradient of the loss (fit the next tree to residuals / pseudo-residuals). Weak learners typically get equal additive weight, scaled by a shrinkage $\nu$ (learning rate).
• XGBoost (and similar libraries) is industry-standard gradient boosting: fast, regularized, strong tabular performance.

Noisy labels + aggressive boosting → easy overfit. Prefer small trees, shrinkage, early stopping / CV.

Quiz

Boosting

1 / 4

Boosting’s core idea is to…

Pick an answer to see if you are right.

Compute

Compute: boosting step

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

Gradient boosting update: F ← F + ν·h. Current F=0, weak learner predicts h=1 on the residual, ν=0.1. What is the new F? Round to 2 decimals.

Enter a number (at most 2 decimal places).

Practice: boosting

RF vs boosting (interview contrast)

	Random forest	Boosting (e.g. XGBoost)
Tree order	Parallel / independent (bagged)	Sequential (each fixes leftovers)
Error focus	Variance ↓ via averaging	Bias ↓ by chasing residuals / hard points
Typical trees	Deeper OK	Prefer weak / shallow + shrinkage
Overfit risk	Lower if B large + mtry diversifies	Higher if noisy data + too many rounds
Speed story	Embarrassingly parallel	Fast libs (XGBoost) but inherently sequential rounds
Classic Q	“RF vs one tree?”	“XGBoost vs random forest?”

Quick synthesis

Topic	Sound bite
RF	Bagging + random features → average many trees
AdaBoost	Reweight hard points; weighted vote of stumps
GBM / XGBoost	Fit trees to loss gradients; shrink with $\nu$
Interview	Parallel average vs sequential residual correction

Next: Dimensionality reduction — curse of dimensionality and PCA.