Every LLM is a neural network trained with backpropagation, so this is the floor under every other lesson. Many AI engineer interview loops include a round on ML and deep learning basics: bias and variance, regularisation, optimisers, backpropagation, loss functions and classification metrics. The questions are short ("why cross-entropy?", "why does Adam work?", "validation loss is rising, what now?") and the interviewer expects a precise answer in under a minute. You also need this to read a training curve, to debug a fine-tune that does not converge, and to write the from-scratch coding problems on this site.

A neuron: weights, bias, activation

A neuron takes a list of numbers, multiplies each by a weight, adds them up with a bias, and passes the result through an activation function.

With inputs x = [1, 2], weights w = [0.5, -0.25] and bias b = 0.1, z = 0.5 - 0.5 + 0.1 = 0.1. The weights say how much each input matters. The bias shifts the point where the neuron switches on. Weights and biases are the parameters, the numbers that training changes.

Activation functions and why non-linearity is needed

Without an activation function, two layers collapse into one: W2(W1 x) = (W2 W1) x, which is still a single matrix. A stack of 100 linear layers can only draw straight-line boundaries. The non-linear function between layers is what lets a network fit curves, and XOR is the smallest example a linear model cannot fit.

Softmax turns logits (raw scores) into probabilities. Logits [2, 1, 0] become [0.665, 0.245, 0.090]: every gap of 1 in the logits is a factor of e, about 2.72, in probability.

A ReLU unit whose input is always negative outputs 0 and gets zero gradient, so it never recovers. This is the "dead ReLU" problem. Leaky ReLU and GELU reduce it by keeping a small non-zero slope for negative inputs.

Layers and the forward pass

A layer is many neurons that share the same inputs, written as one matrix multiply. A two-layer network is:

Computing the output from the input is the forward pass. Count parameters layer by layer. A network with 784 inputs, 128 hidden units and 10 outputs has 784 x 128 + 128 = 100,480 parameters in the first layer and 128 x 10 + 10 = 1,290 in the second: 101,770 in total. The raw outputs before softmax are called logits.

Loss functions

The loss is one number that says how wrong the predictions are. Training minimises it.

• Mean squared error (MSE) is the average of (prediction - target)^2. Use it for regression.
• Cross-entropy is for classification. With a one-hot label (1 for the correct class, 0 for the others) it is the negative log of the probability given to the correct class.

If the model gives the correct class probability 0.7, the loss is -ln 0.7 = 0.357. At 0.01 it is 4.605. Confident wrong answers are punished hard. With softmax or sigmoid outputs, the gradient of cross-entropy with respect to the logits is simply p - y. MSE on a sigmoid output multiplies the gradient by the sigmoid slope, which is near zero when the model is confidently wrong, so learning stalls exactly when it should be fastest.

Gradient descent, learning rate and mini-batches

The gradient of the loss with respect to a weight says which way the loss rises. Step the other way:

The learning rate (eta) is the step size. Too small and training takes forever. Too large and the loss jumps around or diverges. It is the first hyperparameter (a setting you choose, not one the model learns) to tune. Most training runs warm it up from near zero for the first steps and then decay it.

• Batch gradient descent uses the whole dataset for each step: exact but slow.
• Stochastic gradient descent (SGD) uses one example: fast but noisy.
• Mini-batch SGD uses 32 to 512 examples: a good estimate of the gradient, and one GPU matrix multiply per batch. This is what everyone uses.

One pass over the training set is an epoch. The noise from small batches also helps a little against overfitting.

Backpropagation

Backpropagation is the chain rule applied to the computation graph, from the loss back to the inputs. The forward pass stores every intermediate value. The backward pass starts with the gradient 1 at the loss and visits the nodes in reverse order. Each node multiplies the gradient it receives by its local slope and passes the result on.

Computation graph of a one-hidden-unit network with forward values above each arrow and gradients below

Three local rules cover most networks. An add node passes the gradient through unchanged. A multiply node gives each input the gradient times the other input. An activation multiplies by its slope at the stored value. For a two-layer network with a batch X this becomes four lines:

One backward pass costs about the same as one forward pass and gives the gradient for every weight at once. That is why training networks with billions of weights is possible. Frameworks do this automatically (autograd), but interviewers still ask you to derive it for a small network.

Initialisation, vanishing and exploding gradients

If all weights start at zero, every neuron in a layer computes the same output and gets the same gradient, so they stay identical for ever. Start with small random values, scaled so the signal keeps roughly the same variance from layer to layer:

• Xavier initialisation: variance 2 / (fan_in + fan_out). Use with tanh and sigmoid.
• He initialisation: variance 2 / fan_in. Use with ReLU, which zeroes half its inputs.

Here fan_in is the number of inputs to a unit and fan_out is the number of units its output feeds.

The backward pass multiplies one factor per layer. If the factors are below 1 the gradient shrinks: the slopes of ten sigmoid layers multiply to at most 0.25^10, about 0.000001, so early layers stop learning. This is the vanishing gradient problem. Fixes: ReLU-family activations, correct initialisation, normalisation layers, and residual connections (output = x + F(x)), which give the gradient a direct path around each block. If the factors are above 1 the gradient explodes and the loss becomes NaN (not a number). Fix it with gradient clipping (rescale the gradient when its norm passes a limit such as 1.0) and a lower learning rate.

Overfitting, bias and variance

Bias is the error from a model too simple to capture the pattern. Variance is the error from a model that fits the noise in its particular training set. You diagnose them from two numbers.

Regularisation is anything that reduces overfitting:

• L2 (weight decay): add lambda times the sum of squared weights to the loss. Large weights are pulled toward zero and the function becomes smoother. L1 uses absolute values and pushes weights to exactly zero.
• Dropout: during training, set each activation to zero with probability p (0.1 to 0.5) and divide the rest by 1 - p. The network cannot rely on any single unit. Turn it off at inference.
• Early stopping: track the validation loss and keep the checkpoint (a saved copy of the weights) where it was lowest.
• More data, real or augmented, is the most reliable fix.

Optimisers

• SGD with momentum keeps a running average of past gradients: v = beta x v + g, then w = w - eta x v, with beta = 0.9. It speeds up along directions where gradients agree and damps zig-zags.
• Adam keeps a running average of the gradient (m) and of the squared gradient (v), corrects both for starting at zero, and divides one by the square root of the other. Each weight gets its own step size: a weight with small gradients takes relatively larger steps. Defaults: beta1 = 0.9, beta2 = 0.999, learning rate 0.001.
• AdamW applies weight decay directly to the weights and not through the gradient. It is the usual choice for transformers.

Here the hats mean bias-corrected: m divided by (1 - beta1^t) and v divided by (1 - beta2^t). Epsilon is a tiny constant, 1e-8 by default, that prevents division by zero. Adam works with little tuning. SGD with momentum needs a tuned learning-rate schedule but is still common for vision models.

Batch norm vs layer norm

Normalisation layers rescale activations to mean 0 and variance 1, then apply a learned scale and shift. They keep signals in a stable range, so deeper networks train with larger learning rates.

Transformers use layer norm because sequences have different lengths and batch statistics over padded tokens are unreliable.

Train, validation and test splits

Split the data once, for example 80 / 10 / 10. Fit weights on the training set. Choose hyperparameters and the early-stopping point on the validation set. Touch the test set once, at the end, to report the final number. If you tune on the test set, its score no longer tells you how the model does on new data. Watch for leakage: split by user or by time when rows are related, and compute scaling statistics on the training set only.

Classification metrics

Count four outcomes: true positives (TP), false positives (FP), false negatives (FN), true negatives (TN).

Take 1,000 transactions with 10 frauds. A model flags 20 and 8 of them are fraud. Precision = 8 / 20 = 0.40, recall = 8 / 10 = 0.80, F1 = 0.53, accuracy = 986 / 1,000 = 0.986. A model that never flags anything scores 0.99 accuracy and finds no fraud. On imbalanced data, report precision and recall, not accuracy. In the ROC curve, the true positive rate is recall and the false positive rate is FP / (FP + TN). ROC-AUC equals the chance that a random positive gets a higher score than a random negative: 0.5 is random, 1.0 is perfect. Moving the decision threshold trades precision for recall.

One full training step by hand. The network has one input, one hidden ReLU unit and one linear output. Start with x = 2, target y = 1, w1 = 0.5, b1 = 0.5, w2 = 2.0, b2 = -1.0. The loss is half the squared error.

Forward pass. Store every value.

• z = w1 x + b1 = 0.5 x 2 + 0.5 = 1.5
• h = ReLU(1.5) = 1.5
• pred = w2 h + b2 = 2.0 x 1.5 - 1.0 = 2.0
• L = 0.5 x (2.0 - 1)^2 = 0.5

Backward pass. Start at the loss and move left. Each line is "gradient arriving times local slope".

• dL/dpred = pred - y = 1.0
• dL/db2 = 1.0 (an add passes the gradient through)
• dL/dw2 = 1.0 x h = 1.5
• dL/dh = 1.0 x w2 = 2.0
• dL/dz = 2.0 x 1 = 2.0 (ReLU slope is 1 because z is positive)
• dL/db1 = 2.0
• dL/dw1 = 2.0 x x = 4.0

Update with learning rate 0.05. w1 = 0.5 - 0.05 x 4.0 = 0.3. b1 = 0.5 - 0.05 x 2.0 = 0.4. w2 = 2.0 - 0.05 x 1.5 = 1.925. b2 = -1.0 - 0.05 x 1.0 = -1.05.

Check that the loss fell. z = 0.3 x 2 + 0.4 = 1.0, h = 1.0, pred = 1.925 - 1.05 = 0.875, L = 0.5 x 0.125^2 = 0.0078. The loss dropped from 0.5 to 0.0078 in one step.

Same gradients, learning rate 0.1. w1 = 0.1, b1 = 0.3, w2 = 1.85, b2 = -1.1. Then z = 0.5, h = 0.5, pred = 0.925 - 1.1 = -0.175 and L = 0.5 x 1.175^2 = 0.69. The step was in the right direction but too long, and the loss is now higher than where it started. The gradient only describes the slope at the current point, which is why the learning rate matters so much.

Sanity check. Nudge w1 to 0.501 and recompute: z = 1.502, pred = 2.004, L = 0.5 x 1.004^2 = 0.504008. The loss moved by 0.004008 for a nudge of 0.001, a slope of about 4.0, which matches dL/dw1.

"""A two-layer network that learns XOR, with backprop written by hand.

Shapes: X (4, 2) -> hidden (4, 4) with tanh -> output (4, 1) with sigmoid.
Loss: binary cross-entropy. A numerical gradient check proves the backward pass.
"""
import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=np.float64)
Y = np.array([[0], [1], [1], [0]], dtype=np.float64)


def init_params(hidden=4, seed=0):
    rng = np.random.default_rng(seed)
    return {
        # Xavier initialisation: std = sqrt(2 / (fan_in + fan_out))
        "W1": rng.normal(0, np.sqrt(2 / (2 + hidden)), (2, hidden)),
        "b1": np.zeros(hidden),
        "W2": rng.normal(0, np.sqrt(2 / (hidden + 1)), (hidden, 1)),
        "b2": np.zeros(1),
    }


def forward(p, X):
    z1 = X @ p["W1"] + p["b1"]          # (4, hidden)
    h = np.tanh(z1)                     # hidden activations
    z2 = h @ p["W2"] + p["b2"]          # (4, 1) logits
    y_hat = 1 / (1 + np.exp(-z2))       # sigmoid: probability of class 1
    return y_hat, h


def bce(y_hat, Y):
    return float(-np.mean(Y * np.log(y_hat) + (1 - Y) * np.log(1 - y_hat)))


def backward(p, X, Y):
    """Return the loss and dLoss/dparam for every parameter (chain rule, back to front)."""
    y_hat, h = forward(p, X)
    n = len(X)
    d_z2 = (y_hat - Y) / n              # sigmoid + cross-entropy simplify to (p - y)
    d_h = d_z2 @ p["W2"].T              # send the gradient back through W2
    d_z1 = d_h * (1 - h ** 2)           # tanh slope is 1 - tanh(z)^2
    grads = {
        "W2": h.T @ d_z2, "b2": d_z2.sum(axis=0),
        "W1": X.T @ d_z1, "b1": d_z1.sum(axis=0),
    }
    return bce(y_hat, Y), grads


def gradient_check(p, X, Y, eps=1e-5):
    """Compare backprop with (L(w + eps) - L(w - eps)) / (2 eps) for every weight."""
    _, grads = backward(p, X, Y)
    worst = 0.0
    for name, arr in p.items():
        for idx in np.ndindex(arr.shape):
            old = arr[idx]
            arr[idx] = old + eps
            plus = bce(forward(p, X)[0], Y)
            arr[idx] = old - eps
            minus = bce(forward(p, X)[0], Y)
            arr[idx] = old
            numeric = (plus - minus) / (2 * eps)
            analytic = grads[name][idx]
            rel = abs(numeric - analytic) / max(1e-12, abs(numeric) + abs(analytic))
            worst = max(worst, rel)
    return worst


if __name__ == "__main__":
    params = init_params()
    worst = gradient_check(params, X, Y)
    print(f"gradient check: worst relative error = {worst:.2e}")
    assert worst < 1e-6, "backward pass is wrong"
    print("gradient check passed")

    lr = 1.0
    for step in range(2001):
        loss, grads = backward(params, X, Y)
        if step in (0, 50, 100, 200, 500, 1000, 2000):
            print(f"step {step:4d}  loss {loss:.4f}")
        for name in params:
            params[name] -= lr * grads[name]      # plain gradient descent on the full batch

    y_hat, _ = forward(params, X)
    print("predictions:", np.round(y_hat.ravel(), 3), "targets:", Y.ravel().astype(int))
    assert np.array_equal((y_hat > 0.5).astype(int), Y.astype(int))
    print("XOR learned: all 4 points classified correctly")