11.3 Training and backpropagation

Source: Ideas from Computational Physics (Landau, Páez, Bordeianu), §11.2.3–11.2.5. Rewritten in original wording — no textbook SimpleNet.py dump. Continues 11.2b. Next: 11.3b.

Teach with known answers

Feed training pairs with known correct outputs $y^{(c)}$. Compare to predictions $y^{(p)}$. Define Loss, nudge weights with derivatives (backpropagation), then repeat. Pick MSE, the SGD step, and the logistic $f'=f(1-f)$ identity among lookalikes.

Finally evaluate on unseen data. Trained nets are efficient pattern engines but poor explainers: the parameter soup rarely narrates why a decision was made (hence the Blueprint quote in §11.1).

Meson-track toy

Classify $\pi$ ($1$) vs $\mu$ ($0$) from track length/width. Mean-center features (length mean $14$, width mean $8$):

ID	Length$-14$	Width$-8$	Label
A	$-1$	$-2$	$1$ ($\pi$)
B	$2$	$2$	$0$ ($\mu$)
C	$1$	$1$	$0$
D	$-2$	$-1$	$1$

If the net always predicts $0$, then $\mathcal{L}=0.5$ on these four labels — right half the time by chance.

Gradients via the chain rule

Extremize $\mathcal{L}$ in the weights/biases (six $w$'s and three $b$'s even in the tiny 2–2–1 net): seek $\partial\mathcal{L}/\partial w_i=0$ and $\partial\mathcal{L}/\partial b_j=0$. In practice, compute gradients and take the SGD steps you picked above.

Example (book walk-through style): one sample with $x_1=-2$, $x_2=-1$, $y^{(c)}=1$, all weights $1$, biases $0$:

$$h_1=h_2=f(-3)\approx 0.0474,\qquad y^{(p)}=f(2h_1)\approx 0.524.$$

Then

$$\frac{\partial\mathcal{L}}{\partial w_1} =\frac{\partial\mathcal{L}}{\partial y^{(p)}} \frac{\partial y^{(p)}}{\partial h_1} \frac{\partial h_1}{\partial w_1} \approx 0.0215$$

(positive ⇒ decrease $w_1$ to lower $\mathcal{L}$). Sweep parameters one-at-a-time or in batches until $\mathcal{L}$ stalls.

Practice checklist

1. Implement MSE Loss; check $\mathcal{L}=0.5$ for all-zero predictions on the four labels.
2. Forward the all-ones-weight net on $(x_1,x_2)=(-2,-1)$; lock $y^{(p)}$.
3. Compute $f'(-3)$ and $\partial\mathcal{L}/\partial w_1$; apply one SGD step and re-forward.
4. (Open) Train a 2–2–1 net on all four tracks; try 3 hidden nodes or two hidden layers and compare final $\mathcal{L}$.

Takeaway

Step	Action
Forward	Predict $y^{(p)}$
Loss	MSE vs $y^{(c)}$
Backward	Chain-rule $\partial\mathcal{L}/\partial w$
Update	$w\leftarrow w-\eta\,\partial\mathcal{L}/\partial w$
Test	Unseen tracks / holdout

Quiz

Training / backprop

1 / 4

MSE Loss averages…

Pick an answer to see if you are right.

Next: 11.3b Training labs.