Everything here — the network, back-propagation, optimisers and visualisations — is ~400 lines of plain JavaScript,
with gradients verified against finite differences.
Model
a⁽⁰⁾ = features(x₁, x₂)
z⁽ˡ⁾ = W⁽ˡ⁾ a⁽ˡ⁻¹⁾ + b⁽ˡ⁾, a⁽ˡ⁾ = φ(z⁽ˡ⁾)
ŷ = σ(z⁽ᴸ⁾), L = −[y log ŷ + (1 − y) log(1 − ŷ)] + (λ/2)‖W‖²
Hidden activations φ: ReLU, Leaky ReLU, tanh, sigmoid or GELU, with He / Xavier initialisation to match.
Back-propagation
δ⁽ᴸ⁾ = ŷ − y
δ⁽ˡ⁾ = (W⁽ˡ⁺¹⁾ᵀ δ⁽ˡ⁺¹⁾) ⊙ φ′(z⁽ˡ⁾)
∂L/∂W⁽ˡ⁾ = δ⁽ˡ⁾ a⁽ˡ⁻¹⁾ᵀ
Optimisers
- SGD: w ← w − η g
- Momentum: v ← 0.9 v − η g, w ← w + v
- Adam: bias-corrected first/second moment estimates, w ← w − η m̂ / (√v̂ + ε)
Visualisations
- Every neuron shows its activation over the input square; edges are coloured by weight sign and thickness by magnitude.
- The decision boundary plots ŷ(x₁, x₂); filled dots are training points, ringed dots are held-out test points.
- Gradient flow plots the RMS gradient of each weight layer per epoch on a log scale. With deep sigmoid
networks the early layers’ gradients collapse by orders of magnitude — the vanishing-gradient problem that ReLU,
careful initialisation and adaptive optimisers mitigate.
Things to try
- Solve the spiral with only x₁, x₂ — then add sin features and remove layers.
- Pick the “Draw” dataset and click to create your own problem (Shift+click for orange).
- Compare SGD and Adam on the same network and seed with Reset.