Neural Networks

Source: Concepts adapted from Ace the Data Science Interview — Chapter 7 (“Machine Learning”) by Kevin Huo & Nick Singh. Rewritten in original wording for this course. Coding uses NumPy-style math and scikit-learn MLPClassifier on Piston.

Neural nets have existed since the 1950s; the last ~15 years made them practical — more data plus cheap compute. If your résumé shows deep-learning projects, interviewers treat NN details as fair game. For many product DS or noisy finance roles, classical ML usually suffices — know the vocabulary, not every architecture paper.

Questions span qualitative (NN vs trees / linear models) and quantitative (GD, backprop). This lesson covers perceptrons / MLPs, activations, backprop, training pitfalls, regularization, and a short tour of CNNs / RNNs / LSTMs — with explorers, quizzes, and coding after each block. Gradient-descent basics also live in Math Behind ML.

Perceptron & multilayer nets

A perceptron (neuron) takes inputs, applies weights, forms a linear combination $z = w^\top x + b$, then passes $z$ through an activation $\sigma$. Stack neurons into layers → multilayer perceptron (MLP) / feed-forward net.

• Input layer — features
• Hidden layers — intermediate transforms (“deep” = many of these)
• Output layer — prediction (regression number, class logits / probabilities)

Inputs combine linearly; nonlinear activations make the overall map nonlinear — that is why NNs shine on complex patterns that linear models miss. Pure stacks of linear layers without nonlinearities collapse to one linear map.

Quiz

Perceptron & MLPs

1 / 3

A perceptron computes…

Pick an answer to see if you are right.

Activation functions

Common activations (know formulas + when they show up):

Name	Formula (sketch)	Notes
Unit / sign	step or $\mathrm{sign}(z)$	Classic perceptron variants
Linear	$\sigma(z)=z$	Like linear / Adaline
Sigmoid	$\sigma(z)=1/(1+e^{-z})$	$(0,1)$; saturates
tanh	$\tanh(z)$	$(-1,1)$; saturates
ReLU	$\max(0,z)$	Default hidden unit; sparse, helps vanishing grads
softplus	$\ln(1+e^{z})$	Smooth ReLU-like

Interview depth: trade-offs (saturation → vanishing gradients; ReLU can “die” for $z<0$).

Compute

Compute: activations

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

ReLU(z)=max(0,z). What is ReLU(3)? Round to 2 decimals.

Enter a number (at most 2 decimal places).

Practice: activations

Forward pass & backpropagation

Feed-forward: data flows input → hidden → output. Training compares prediction to target with a loss — often MSE for regression, cross-entropy for classification — then backpropagation sends error signals backward.

Chain rule (scalar sketch): if $z$ is the network output and $L(z,y)$ the loss,

$$\frac{\partial L}{\partial w}=\frac{\partial L}{\partial z}\frac{\partial z}{\partial w}, \qquad w \leftarrow w - \alpha\frac{\partial L}{\partial w}.$$

$\alpha$ is the learning rate. Updates use SGD / mini-batches. ML-heavy interviews may whiteboard this for tiny models (linear / logistic).

Hyperparameters that come up constantly: learning rate, # hidden layers / width, activations, batch size, epochs, regularization.

$\alpha$ too small	Training crawls / stalls
$\alpha$ too large	Unstable / overshoots, may “converge” poorly

Quiz

Backprop & hyperparameters

1 / 4

Backpropagation…

Pick an answer to see if you are right.

Compute

Compute: ∂L/∂w and SGD

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

L=½(wx−y)² with w=1.5, x=2, y=0. What is ∂L/∂w=(wx−y)·x? Round to 2 decimals.

Enter a number (at most 2 decimal places).

Practice: backprop arithmetic

Training problems & optimizers

Deep nets train millions of weights and many hyperparameters — things go wrong.

Vanishing gradients: chain-rule products of small derivatives (classic sigmoid / tanh tails) → early layers barely update. Exploding gradients: products blow up. Mitigations: ReLU-family activations, residual / skip connections (ResNets), gated memory (LSTMs), gradient clipping, careful init.

Vanilla SGD issues: noisy steps, bad $\alpha$, saddle points / poor local minima, features with different scales/frequencies.

Momentum: keep a velocity that exponentially averages past gradients so updates keep a consistent direction/speed:

$$v \leftarrow \beta v + g, \qquad w \leftarrow w - \alpha v.$$

(Modern practice also uses Adam-family adaptive methods — know the problem they address even if you skip every formula.)

Quiz

Vanishing grads & momentum

1 / 3

Vanishing gradients happen when…

Pick an answer to see if you are right.

Compute

Compute: momentum & vanish product

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

Momentum: v ← βv + g. β=0.8, current v=1, g=0.5. New v? Round to 2 decimals.

Enter a number (at most 2 decimal places).

Practice: momentum

Transfer learning

Transfer learning reuses weights from a model trained on a large related dataset (ImageNet, BERT corpora), then adapts / fine-tunes on your smaller task. Ideal when you lack data but a strong pretrained pool exists — e.g. hot-dog detector: start from ImageNet features, then fine-tune on your 100 hot-dog photos.

Addressing overfitting

Deep nets overfit easily (huge capacity). Interview toolkit:

Technique	Idea
More data	Best variance fix if you can collect / augment
Input standardization	0 mean / unit variance → stabler, faster learning (avoids wild weight scales)
Batch normalization	Normalize activations within a mini-batch so hidden reps stay well-scaled; layers train more independently; also regularizes
Dropout	Randomly deactivate neurons each step → train an ensemble of thinner nets; noise ≈ regularizer

Batch norm + dropout both inject a kind of training noise that curbs co-adaptation.

Quiz

Regularization for NNs

1 / 3

Dropout…

Pick an answer to see if you are right.

Architectures: CNN, RNN, LSTM

Family	Core idea	Typical use
CNN	Convolution filters on local neighborhoods + pooling + dense head	Vision — edges/textures → objects; translation-ish invariance from pooling
RNN	Directed recurrence + hidden state (memory) along a sequence	Audio / video / text where context depends on history; variable lengths
LSTM	Cell + input / output / forget gates regulating memory flow	Long-range dependencies — preferred over vanilla RNNs in practice

CNN pipeline sketch: conv (+ ReLU) → pool → … → flatten → fully connected → softmax.

RNNs are not plain feed-forward: they reuse state across time. LSTMs add gates so useful signal can persist longer than a vanilla RNN’s short memory.

Quiz

CNN / RNN / LSTM

1 / 4

CNNs are popular in vision because…

Pick an answer to see if you are right.

Practice: sklearn MLP

Quick synthesis

Topic	Sound bite
Neuron	$z=w^\top x+b$ → nonlinear $\sigma$
Depth	Hidden layers compose hierarchical features
Train	Forward loss → backprop chain rule → SGD / momentum
Failures	Vanish / explode; fix with ReLU, residuals, gates, clipping
Regularize	Data, standardize, batch norm, dropout
Transfer	Pretrain big → fine-tune small
CNN / RNN / LSTM	Spatial filters vs sequential memory vs gated long memory

Interview mode	Expect
Product DS / noisy tabular	Classical ML first; NN high-level OK
ML-heavy / DL on résumé	Activations, backprop sketch, hparams, overfit tricks, CNN vs RNN

That wraps the Ace Ch.7 neural-net tour after clustering and the classical supervised toolkit. Next: Reinforcement Learning and the End-to-end ML workflow.