NumMLP

NumMLP is a multilayer perceptron implemented from first principles with NumPy. The implementation is written entirely in Python, keeping the codebase simple and readable while focusing on the neural-network math itself. It covers manual forward propagation, backpropagation, mini-batch SGD, MNIST training and evaluation, checkpointing, handwritten-digit inference, numerical gradient checking, and optional validation against PyTorch.

Technical walkthrough

For a step-by-step explanation of the network architecture, forward pass, backpropagation, training process, and correctness checks, see How NumMLP Works.

What "from scratch" means

The neural-network mathematics are written directly against NumPy arrays rather than delegating to a deep-learning framework. NumMLP implements its own:

dense layers
ReLU
softmax
categorical cross-entropy
backpropagation and parameter gradients
mini-batch stochastic gradient descent

NumPy provides array arithmetic, and Pillow decodes image files. PyTorch is never used to train NumMLP — it appears only as an optional, independent reference used to check the hand-derived gradients.

Architecture

The MNIST model is a fully connected network:

784 -> 128 -> 128 -> 10

28 x 28 grayscale image
        |  flatten
        v
     784 inputs
        |  dense + ReLU
        v
   128 hidden units
        |  dense + ReLU
        v
   128 hidden units
        |  dense
        v
      10 logits
        |  softmax
        v
  10 class probabilities

Forward pass

Each dense layer computes

Z = X W + b

where X is (batch, in_features), W is (in_features, out_features), and b is (1, out_features) broadcast across the batch. Hidden layers apply

A = ReLU(Z) = max(0, Z)

and the output layer converts logits to probabilities with a max-shifted softmax:

P = softmax(Z) = exp(Z - max(Z)) / sum(exp(Z - max(Z)))

Subtracting the row maximum keeps large logits from overflowing without changing the result. The loss is mean categorical cross-entropy over the batch:

L = -mean(sum(Y * log(P)))

Predicted probabilities are clipped to [1e-10, 1 - 1e-10] so a zero probability on the true class cannot produce log(0).

Backward pass

Gradients are derived and implemented by hand. Because softmax and cross-entropy are composed, their combined derivative simplifies to

dZ = (P - Y) / batch_size

for the output layer. From there the backward pass walks the layers in reverse, at each step computing the weight gradient from the cached layer input, summing the bias gradient over the batch, propagating the gradient into the previous layer through the transposed weights, and masking it wherever the ReLU input was not positive. Each call replaces the stored gradients, so weight_gradients and bias_gradients always describe the most recent batch.

Parameters are then updated with plain SGD and an iteration-decayed learning rate:

lr_effective = lr / (1 + decay * iterations)
parameter    = parameter - lr_effective * gradient

These analytical gradients are checked two independent ways: against central finite differences, and against PyTorch autograd. See Correctness validation.

Preprocessing

MNIST images arrive as uint8 pixels in [0, 255]. They are scaled to roughly [-1, 1] and flattened:

(images.astype(np.float32) - 127.5) / 127.5   # -> (N, 784) float32

Handwritten images use exactly the same transformation after conversion to 8-bit grayscale.

Canonical result

The default configuration is:

Setting	Value
architecture	784 → 128 → 128 → 10
epochs	2
batch size	32
learning rate	0.1
decay	1e-6
seed	0

which produces:

test accuracy: 95.72%
test loss:     0.139268

on the 10,000-image MNIST test split. This is a small fully connected network written for learning and verification; convolutional models reach higher accuracy on MNIST.

Correctness validation

The implementation is checked at three levels.

Finite differences. Analytical gradients are compared against central differences, (L(θ+ε) - L(θ-ε)) / 2ε, for every weight and bias of a small deterministic network whose pre-activations sit well away from the ReLU kink. Agreement is on the order of 1e-11, far inside the 1e-6 tolerance the test enforces. This runs as part of the normal test suite.

PyTorch autograd. On the deterministic reference fixture, the hand-written gradients agree with PyTorch autograd to approximately machine precision (about 1e-16), with the losses matching exactly. The comparison runs on CPU in float64 so agreement is judged at the limits of double precision.

Full MNIST forward comparison. The canonical checkpoint is mapped into an equivalent PyTorch model and run over all 10,000 test images:

NumMLP accuracy:          95.72%
PyTorch accuracy:         95.72%
prediction disagreements: 0
probabilities agree to floating-point precision

Project structure

NumMLP/
├── src/nummlp/
│   ├── __init__.py
│   ├── activations.py      ReLU and softmax
│   ├── checkpoint.py       versioned .npz save/load
│   ├── handwritten.py      single-image preprocessing
│   ├── losses.py           categorical cross-entropy
│   ├── mnist.py            IDX parsing, download, preprocessing
│   └── network.py          the network, forward and backward passes
├── scripts/
│   ├── train.py            train on MNIST and write a checkpoint
│   ├── evaluate.py         score a checkpoint on the test split
│   └── predict_image.py    classify a single 28x28 image
├── reference/
│   └── pytorch_reference.py   optional PyTorch cross-check
├── tests/
├── examples/handwritten/   four hand-drawn sample digits
├── pyproject.toml
└── README.md

data/ and artifacts/ are not part of the repository. They are created locally on first use — data/ for the downloaded MNIST files, artifacts/ for trained checkpoints — and both are ignored by Git.

Setup

Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[test]"

Training

python scripts\train.py

Any missing MNIST files are downloaded automatically into data/MNIST/raw/, and the trained model is written to artifacts/mnist.npz. To reproduce the canonical result explicitly:

python scripts\train.py --seed 0

Other options include --epochs, --batch-size, --learning-rate, --decay, --data-dir, --checkpoint, --train-limit, --test-limit, and --no-download.

Evaluation

python scripts\evaluate.py

This loads artifacts/mnist.npz by default, so run the training step first. It reports test loss and accuracy without retraining.

Handwritten digit prediction

python scripts\predict_image.py examples\handwritten\digit_5.png

python scripts\predict_image.py examples\handwritten\digit_5.png --top-k 3

The input must already be a 28 × 28 image. It is converted to grayscale and normalized exactly like MNIST — nothing is cropped, centered, resized, or inverted, and an image of any other size is rejected rather than reshaped.

The bundled examples are hand-drawn rather than drawn from MNIST, so they sit outside the distribution the model was trained on. The canonical model classifies two of the four correctly; the other two are confidently wrong. They are included as an honest demonstration of how a small MLP generalizes beyond its training data.

PyTorch reference

The cross-check is optional and needs the extra dependency:

python -m pip install -e ".[reference]"
python reference\pytorch_reference.py

This builds an equivalent PyTorch model and compares parameter mapping, forward probabilities, loss, and hand-written gradients against autograd on a deterministic fixture. It exits non-zero if any comparison falls outside tolerance.

With a trained checkpoint it also compares both implementations across the full test split:

python reference\pytorch_reference.py `
  --checkpoint artifacts\mnist.npz `
  --data-dir data\MNIST\raw

Tests

python -m pytest -q

The suite is offline and deterministic: MNIST fixtures are synthesized in temporary directories, so no test downloads data or depends on a trained model.

Checkpoint format

Checkpoints are compressed NumPy .npz files holding the architecture metadata, every weight and bias array, the training iteration count, and a format version. Loading validates the version and checks that the stored shapes agree with each other and with the declared architecture. Files are read with allow_pickle=False, so restoring a model never deserializes arbitrary Python objects.

Dependencies

Purpose	Packages
Runtime	NumPy, Pillow
Tests	pytest
Optional reference validation	PyTorch

PyTorch is not required to train, evaluate, or use NumMLP.

Reproducibility

NumMLP seeds NumPy explicitly for both parameter initialization and batch shuffling, so --seed 0 with the canonical defaults reproduces the documented result exactly in the validated environment. Results may differ across NumPy or BLAS builds that reorder floating-point accumulation.

Limitations

A fully connected MLP, not a convolutional network; accuracy is limited accordingly.
Preprocessing is MNIST-specific.
Input images must already be 28 × 28; there is no automatic cropping, centering, or deskewing.
Training uses plain SGD with learning-rate decay — no momentum, adaptive optimizers, regularization, or batch normalization.
Written as a readable reference implementation rather than a performance-oriented framework.