11.2 A simple neural network

Source: Ideas from Computational Physics (Landau, Páez, Bordeianu), §11.2–11.2.2. Rewritten in original wording — no textbook NeuralNet.py dump. Continues 11.1b. Next: 11.2b.

One AI neuron

Inputs arrive with weights; the body forms an affine map $\Sigma$, then an activation $f$ decides the outgoing signal. Soft activations (logistic, tanh, ReLU) train more easily than hard $0/1$ thresholds. Pick $\Sigma$/$y=f(\ldots)$, the logistic sigmoid, and the 2–2–1 nesting among lookalikes.

Weights and biases are the learnable parameters. Example: $w_1=-1$, $w_2=1$, $b=0$, $x_1=12$, $x_2=8$ ⇒ $\Sigma=-4$. With $f(x)=x$ the output is $-4$. Logistic sigmoid ∈ $(0,1)$; tanh ∈ $(-1,1)$; ReLU passes positives and zeros negatives. Our shallow demos use the logistic sigmoid.

Shallow network (2–2–1)

Two inputs, two hidden nodes, one output (“shallow” — more than one hidden layer would be deep). The output neuron’s inputs are the hidden activations $h_1,h_2$ (not the raw $x$’s) — as in the formulas you picked.

Hand check: if every neuron uses the same weights $(w_a,w_b)=(0,1)$ and bias $0$, with $x_1=2$, $x_2=3$, then $h_1=h_2=f(3)$ and $y=f(h_2)\approx 0.7216$.

Coding sketch (structure only)

# neuron_forward_sketch.py — original structure
import math

def sigmoid(z):
    return 1.0 / (1.0 + math.exp(-z))

def neuron(xs, ws, b=0.0, f=sigmoid):
    return f(sum(w * x for w, x in zip(ws, xs)) + b)

# network: h1 = neuron([x1,x2], [w1,w2], b1); ...; y = neuron([h1,h2], [w5,w6], b3)

Libraries (NumPy / PyTorch) vectorize the same math; labs stay pure Python so every float is inspectable.

Takeaway

1. Node = affine map + activation.
2. Stack nodes into layers; forward pass is nested activations.
3. Soft activations train more easily than hard thresholds.
4. Even a tiny net has many weights — training comes next.

Quiz

Simple NN

1 / 4

With w₁=-1, w₂=1, b=0, x=(12,8) and f=identity, y is…

Pick an answer to see if you are right.

Next: 11.2b Simple NN labs.