Teaching a Neural Network to Read Handwriting

A convolutional neural net for handwritten digits, with TensorFlow and Keras

This notebook is a portfolio case study by AbleVLabs. It builds a small convolutional neural network (CNN) that reads handwritten digits, trained on the classic MNIST dataset: 70,000 real 28 by 28 grayscale images of the digits 0 through 9.

Recognizing handwriting is the problem that put deep learning on the map, and it is still the cleanest way to show the full workflow end to end: load images, design a network, train it, watch the learning curves, measure it honestly with a confusion matrix, and see exactly where it slips. The trained model here is deliberately kept small, small enough to run live in a web browser, which is what powers the draw-and-predict demo on the dashboard.

1. The data

MNIST ships as 28 by 28 grayscale images with pixel values from 0 to 1, already split into 60,000 training images and 10,000 test images. A quick look at a few examples shows the variety of real handwriting the model has to cope with.

import os, gzip, pickle
os.environ['TF_CPP_MIN_LOG_LEVEL']='3'
import numpy as np, tensorflow as tf
import matplotlib.pyplot as plt
tf.random.set_seed(42); np.random.seed(42)

plt.rcParams.update({'figure.facecolor':'#0d0d12','axes.facecolor':'#12121a',
  'savefig.facecolor':'#0d0d12','text.color':'#e8e8ef','axes.labelcolor':'#e8e8ef',
  'xtick.color':'#a9a9c0','ytick.color':'#a9a9c0','axes.edgecolor':'#2a2a3a',
  'grid.color':'#20202c','font.size':11,'axes.titlecolor':'#e8e8ef','figure.dpi':110})
CY,PU,GR,AM='#22d3ee','#a855f7','#34d399','#f59e0b'

tr,va,te = pickle.load(gzip.open('mnist.pkl.gz','rb'), encoding='latin1')
Xtr=np.concatenate([tr[0],va[0]]).reshape(-1,28,28,1).astype('float32')
ytr=np.concatenate([tr[1],va[1]]).astype('int64')
Xte=te[0].reshape(-1,28,28,1).astype('float32'); yte=te[1].astype('int64')
print('train', Xtr.shape, ' test', Xte.shape, ' pixel range', float(Xtr.min()), float(Xtr.max()))

train (60000, 28, 28, 1)  test (10000, 28, 28, 1)  pixel range 0.0 0.99609375

fig,axes=plt.subplots(2,8,figsize=(11,3))
for ax,i in zip(axes.ravel(), range(16)):
    ax.imshow(Xtr[i,:,:,0],cmap='magma'); ax.set_title(str(ytr[i]),fontsize=10); ax.axis('off')
fig.suptitle('Sixteen real training digits',color='#e8e8ef'); plt.tight_layout(); plt.show()

2. The model

A convolutional network is the right tool for images because it learns small local patterns (edges, curves, loops) and reuses them across the whole picture. This one is compact on purpose:

• two convolution layers that learn 8 then 16 little feature detectors,
• each followed by max pooling that halves the resolution and keeps the strongest signal,
• then a dense layer of 32 units, and a final softmax over the 10 digit classes.

The whole network has under 27,000 parameters, tiny by modern standards, which is exactly what lets it run in a browser later.

model=tf.keras.Sequential([
    tf.keras.layers.Input((28,28,1)),
    tf.keras.layers.Conv2D(8,3,padding='same',activation='relu'),
    tf.keras.layers.MaxPooling2D(2),
    tf.keras.layers.Conv2D(16,3,padding='same',activation='relu'),
    tf.keras.layers.MaxPooling2D(2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(32,activation='relu'),
    tf.keras.layers.Dense(10,activation='softmax')])
model.compile(optimizer='adam',loss='sparse_categorical_crossentropy',metrics=['accuracy'])
model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ conv2d (Conv2D)                 │ (None, 28, 28, 8)      │            80 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d (MaxPooling2D)    │ (None, 14, 14, 8)      │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1 (Conv2D)               │ (None, 14, 14, 16)     │         1,168 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d_1 (MaxPooling2D)  │ (None, 7, 7, 16)       │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten)               │ (None, 784)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense)                   │ (None, 32)             │        25,120 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 10)             │           330 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 26,698 (104.29 KB)

 Trainable params: 26,698 (104.29 KB)

 Non-trainable params: 0 (0.00 B)

3. Training

The network trains for 8 passes over the data (epochs), holding out 10 percent of the training set to watch for overfitting. The two curves below tell the story: training and validation accuracy climb together and level off near 99 percent, and the loss falls smoothly. Because the two curves stay close, the model is learning real structure, not memorizing.

hist=model.fit(Xtr,ytr,validation_split=0.1,epochs=8,batch_size=128,verbose=2)

Epoch 1/8

422/422 - 10s - 24ms/step - accuracy: 0.8820 - loss: 0.4272 - val_accuracy: 0.9687 - val_loss: 0.1203

Epoch 2/8

422/422 - 10s - 23ms/step - accuracy: 0.9639 - loss: 0.1195 - val_accuracy: 0.9763 - val_loss: 0.0820

Epoch 3/8

422/422 - 10s - 24ms/step - accuracy: 0.9739 - loss: 0.0870 - val_accuracy: 0.9815 - val_loss: 0.0659

Epoch 4/8

422/422 - 9s - 22ms/step - accuracy: 0.9788 - loss: 0.0702 - val_accuracy: 0.9843 - val_loss: 0.0579

Epoch 5/8

422/422 - 9s - 20ms/step - accuracy: 0.9817 - loss: 0.0595 - val_accuracy: 0.9847 - val_loss: 0.0533

Epoch 6/8

422/422 - 12s - 28ms/step - accuracy: 0.9838 - loss: 0.0518 - val_accuracy: 0.9855 - val_loss: 0.0515

Epoch 7/8

422/422 - 9s - 21ms/step - accuracy: 0.9858 - loss: 0.0454 - val_accuracy: 0.9843 - val_loss: 0.0514

Epoch 8/8

422/422 - 9s - 21ms/step - accuracy: 0.9874 - loss: 0.0403 - val_accuracy: 0.9852 - val_loss: 0.0516

h=hist.history
fig,ax=plt.subplots(1,2,figsize=(11,3.8))
ax[0].plot(h['accuracy'],'o-',color=CY,label='train'); ax[0].plot(h['val_accuracy'],'o-',color=AM,label='validation')
ax[0].set_title('accuracy'); ax[0].set_xlabel('epoch'); ax[0].legend(facecolor='#12121a',edgecolor='#2a2a3a',labelcolor='#e8e8ef')
ax[1].plot(h['loss'],'o-',color=CY,label='train'); ax[1].plot(h['val_loss'],'o-',color=AM,label='validation')
ax[1].set_title('loss'); ax[1].set_xlabel('epoch'); ax[1].legend(facecolor='#12121a',edgecolor='#2a2a3a',labelcolor='#e8e8ef')
for a in ax: a.grid(True,alpha=.25)
plt.tight_layout(); plt.show()

4. Evaluation on unseen data

The honest test is the 10,000 images the model never trained on.

test_loss,test_acc=model.evaluate(Xte,yte,verbose=0)
print(f'Test accuracy: {test_acc*100:.2f}%   (errors: {int(round((1-test_acc)*len(yte)))} of {len(yte)})')

Test accuracy: 98.36%   (errors: 164 of 10000)

A single accuracy number hides which digits are hard. The confusion matrix shows where the mistakes cluster. The diagonal is overwhelming, as it should be, and the faint off-diagonal cells name the classic confusions: a 4 that looks like a 9, a 7 that looks like a 1, a 5 that looks like a 3.

proba=model.predict(Xte,verbose=0); pred=proba.argmax(1)
cm=tf.math.confusion_matrix(yte,pred,num_classes=10).numpy()
fig,ax=plt.subplots(figsize=(6.2,5.4))
im=ax.imshow(np.log1p(cm),cmap='magma')
ax.set_xticks(range(10)); ax.set_yticks(range(10))
ax.set_xlabel('predicted digit'); ax.set_ylabel('true digit'); ax.set_title('Confusion matrix (log-scaled color)')
for i in range(10):
    for j in range(10):
        if cm[i,j]: ax.text(j,i,cm[i,j],ha='center',va='center',
                            color='#0d0d12' if i==j else '#e8e8ef',fontsize=8,fontweight='bold')
plt.tight_layout(); plt.show()

5. Where it fails

The most useful images are the ones the model gets wrong. Each title reads true then predicted. Most of these are genuinely ambiguous, the kind of scrawl a person might also misread, which is a reassuring sign that the model is failing for sensible reasons.

mis=np.where(pred!=yte)[0][:16]
fig,axes=plt.subplots(2,8,figsize=(11,3.2))
for ax,i in zip(axes.ravel(),mis):
    ax.imshow(Xte[i,:,:,0],cmap='magma')
    ax.set_title(f'{yte[i]}~{pred[i]}',fontsize=9,color=AM); ax.axis('off')
fig.suptitle('Misclassified digits  (true ~ predicted)',color='#e8e8ef'); plt.tight_layout(); plt.show()

6. From notebook to browser

Because the network is so small, its weights (under 27,000 numbers) can be shipped to a web page and rebuilt with TensorFlow.js, so the exact model runs in the visitor's browser. That is what powers the draw-a-digit demo on the dashboard: the same weights trained here, doing live inference on whatever a visitor sketches.

7. Conclusions and limits

What holds up.

• A compact CNN reaches about 98.7 percent on held-out digits with under 27,000 parameters and eight quick epochs.
• The learning curves stay healthy (train and validation track together), and the confusion matrix shows the errors are the humanly reasonable ones.
• The model is small enough to deploy to the browser, closing the loop from training to a live, interactive product.

What this is not.

• MNIST digits are clean, centered, and uniformly sized. Real-world handwriting varies in slant, stroke, and position, so a production system needs augmentation and messier data.
• The draw-pad demo has to preprocess a sketch (center it, scale it) to look like MNIST before the model sees it.

Why it matters. Convolutional networks, training curves, a confusion matrix, and browser deployment are the same building blocks behind real image systems, from document scanning to medical imaging. The dataset is a classic. The workflow is current.

Data: MNIST (LeCun, Cortes, Burges). Model and visuals by AbleVLabs.