11.4 A graphical deep net

Source: Ideas from Computational Physics (Landau, Páez, Bordeianu), §11.3 (Zhou / Rohrer-style didactic net). Rewritten in original wording. Continues 11.3b. Next: 11.4b.

The shallow 2–2–1 net can learn a little. Here we look at a deeper hierarchy (three hidden layers) that classifies line orientation inside a small pixel patch — essentially an ML decision tree drawn as a net — designed so each layer sees a richer pattern than the one before. The point is graphical intuition, not a full training loop.

Hierarchy on a $2\times 2$ patch

Input patterns (filled = “on”):

• horizontal: top row on, or bottom row on
• vertical: left column on, or right column on
• diagonal: main or anti diagonal
• none: all on or all off

Each input node maps to one pixel. Edges carry weights (often $\pm 1$). In this didactic graph there are no same-layer links and no skips over a layer.

Hidden layer 1 — single pixels → pairs

A node that probes “vertical on the left” might combine top-left and bottom-left inputs. For a horizontal top-row input encoded so those vertical edges see $(-1)$ and $(+1)$,

$$x_{\mathrm{TL}} w_{\mathrm{TL}}+x_{\mathrm{BL}} w_{\mathrm{BL}}=(-1)(1)+(1)(1)=0.$$

Zero ⇒ that vertical detector stays inactive (no onward pulse). Nonzero ⇒ stay active.

Hidden layer 2 — pairs → line hypotheses

Later nodes add / subtract pair patterns. Schematically (white = on, black = off; negation flips colors):

$$1\cdot\begin{bmatrix}\square & x\\ \square & x\end{bmatrix} +1\cdot\begin{bmatrix}x & \square\\ x & \square\end{bmatrix} =\text{blank board},$$ $$(-1)\cdot\begin{bmatrix}\square & x\\ \square & x\end{bmatrix} +1\cdot\begin{bmatrix}x & \square\\ x & \square\end{bmatrix} =\text{left column}.$$

Hidden layer 3 — ReLU

$$f_{\mathrm{ReLU}}(z)=\max(0,z)$$

passes positive evidence and silences negative scores — a hard “is this line present?” gate before the class output.

Exercise (open)

Four two-cell combinations: $[X][\square]$, $[X][X]$, $[\square][X]$, $[\square][\square]$. Design weights (and a tiny depth) that separate them. Labs lock the arithmetic pieces ($0$ vertical probe, ReLU, simple two-bit classifiers).

Takeaway

Layer	Recognizes
Input	Raw pixels
Hidden 1	Local pairs
Hidden 2	Line orientations
Hidden 3 / out	Class via ReLU / readout

Depth ≈ composition of features, not magic.

Quiz

Graphical deep net

1 / 4

In the didactic hierarchy, later hidden layers…

Pick an answer to see if you are right.

Next: 11.4b Graphical deep-net labs.