Interviewers rarely ask for dates. They ask "why": why did transformers replace RNNs, why did deep learning take off only in 2012, why do we pretrain. Those are history questions in disguise, and the strong answer is always "the older method had this limit, and the new one removed it". The same habit helps in design rounds. RAG, agents and reasoning models each exist because a plain LLM has a specific limit, and you choose between them by naming that limit. Skim this lesson once and keep the table.

Rules first: symbolic AI and the perceptron

The field got its name at a summer workshop in 1956, six years after the Turing test was proposed. Early systems were symbolic: a programmer wrote facts and if-then rules, and the program searched through them. This worked on closed problems such as logic puzzles. It failed on real input, because nobody can write a rule for every way a cat can appear in a photo.

The perceptron (1958) was the first well-known alternative: one artificial neuron that learns its weights from labelled examples. In 1969 its limit was shown clearly. A single layer can only separate classes with a straight line, so it cannot learn even XOR. More layers would fix that, but nobody had a practical way to train them.

Timeline of AI from 1950 to today with the years since 2012 enlarged into eight steps

AI winters and expert systems

Promises ran ahead of results, and funding collapsed twice: in the mid-1970s and again in the late 1980s. These periods are called AI winters. Between them, the 1980s belonged to expert systems: thousands of if-then rules collected from human specialists, for narrow jobs such as diagnosing faults. They earned money, but every rule was written and maintained by hand, and they broke on any case their authors had not foreseen.

Backpropagation and statistical machine learning

In 1986 backpropagation was shown to train networks with hidden layers: pass the error backward with the chain rule and adjust every weight. That removed the XOR limit. The networks stayed small, though. Datasets were tiny, computers were slow, and gradients faded in deep stacks.

So the 1990s and 2000s went to statistical machine learning: support vector machines (SVMs), decision trees, random forests and boosting. They learn the decision from data, and boosted trees are still a top choice for tables of numbers. Their limit was the input. People had to design the features for each task, such as edge detectors for images or word counts for text.

2012: deep learning works at scale

In 2012 a deep convolutional neural network (CNN) won the main image classification contest with 15.3% top-5 error. The runner-up, built on hand-designed features, scored 26.2%. Three things came together: 1.2 million labelled images, GPUs fast enough to train on them, and training tricks such as ReLU and dropout. The network learned its own features from raw pixels, so the hand-design step disappeared. Two limits remained: it needed very large labelled datasets, and language was still unsolved.

Words, sequences and attention

Word embeddings (2013) gave every word a dense vector learned from raw, unlabelled text. Similar words sit close together, and king - man + woman lands near queen. The limit: one fixed vector per word, so "bank" means the same thing next to "river" and next to "loan".

Sequence-to-sequence models (2014) used one recurrent neural network (RNN) to read a sentence into a single vector and a second one to write the translation. Long sentences did not fit through that one vector. Attention, added the same year, let the decoder look back at every input word at each step. The RNN underneath still read one token at a time, which is slow to train and loses long-range context.

The transformer and pretraining

The transformer (2017) dropped recurrence and kept only attention. Any two tokens connect in one step, and all positions train in parallel on GPUs. Its costs: attention grows with the square of the sequence length, and each task still needed its own labelled data.

Pretraining (2018 onward) fixed the data problem. Train a transformer on raw text to predict hidden or next tokens, then fine-tune it on a small labelled set. Encoder models (BERT style) became the standard for classification and search, decoder models (GPT style) for generation. The limit: one fine-tune per task.

In 2020 a decoder with 175 billion parameters showed that a large enough model can do a new task from a few examples in the prompt, with no weight updates. This is few-shot learning. It was still hard to steer, it made things up, and it knew nothing after its training data ended.

Chat assistants, retrieval, agents and reasoning

Chat assistants (2022) added instruction tuning and alignment from human feedback, so the model follows a plain request. What a chat model still cannot do explains most of today's stack:

• Stale or missing knowledge, invented facts: retrieval-augmented generation (RAG) fetches documents at query time.
• It can only write text: agents let the model call tools in a loop and act on the results.
• Mistakes on multi-step problems: reasoning models think before they answer, trained with reinforcement learning on rewards that can be checked automatically.

Still open: cost, latency, reliability over long tasks, and security.

The chain in one table

Why one perceptron cannot learn XOR, and how a hidden layer fixes it. This small proof is the first "limit, then idea" step in the chain, and you can do it on paper.

A perceptron outputs 1 when w1 x1 + w2 x2 + b is greater than 0, and 0 otherwise. XOR outputs 1 only when exactly one input is 1. For a perceptron to compute XOR, four conditions must hold:

1. Input (0, 0) gives 0, so b is at most 0.
2. Input (0, 1) gives 1, so w2 + b is greater than 0.
3. Input (1, 0) gives 1, so w1 + b is greater than 0.
4. Input (1, 1) gives 0, so w1 + w2 + b is at most 0.

Add conditions 2 and 3: w1 + w2 + 2b is greater than 0, so w1 + w2 + b is greater than -b. Condition 1 says -b is at least 0. So w1 + w2 + b is greater than 0, which contradicts condition 4. No weights exist. In a picture: no single straight line puts (0, 1) and (1, 0) on one side and (0, 0) and (1, 1) on the other.

Now add two hidden units with the same step rule:

• h1 = step(x1 + x2 - 0.5), which is OR
• h2 = step(x1 + x2 - 1.5), which is AND
• output = step(h1 - h2 - 0.5), which is "OR and not AND"

All four rows are correct. So a hidden layer has enough capacity. The open problem was how to learn those hidden weights, because the step function has zero slope and gives no signal about which way to move. Smooth activations plus backpropagation (1986) answered that, which is the next link in the chain.

"""Why the field needed hidden layers: a perceptron learns AND but never XOR.

Part 1 trains a single perceptron with the classic learning rule.
Part 2 shows that two hidden units with hand-set weights solve XOR.
"""
import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
AND = np.array([0, 0, 0, 1])
XOR = np.array([0, 1, 1, 0])


def step(z):
    return (z > 0).astype(int)


def train_perceptron(X, y, epochs=100, lr=0.1):
    """Perceptron rule: on each mistake, move the weights toward the example."""
    w, b = np.zeros(2), 0.0
    for epoch in range(1, epochs + 1):
        mistakes = 0
        for xi, target in zip(X, y):
            pred = int(w @ xi + b > 0)
            if pred != target:
                w += lr * (target - pred) * xi
                b += lr * (target - pred)
                mistakes += 1
        if mistakes == 0:                      # a full pass with no mistake: converged
            return w, b, epoch
    return w, b, None


def two_layer_xor(X):
    """Hidden unit 1 computes OR, hidden unit 2 computes AND, output = OR and not AND."""
    h1 = step(X[:, 0] + X[:, 1] - 0.5)         # OR
    h2 = step(X[:, 0] + X[:, 1] - 1.5)         # AND
    return step(h1 - h2 - 0.5)


if __name__ == "__main__":
    for name, y in [("AND", AND), ("XOR", XOR)]:
        w, b, epoch = train_perceptron(X, y)
        acc = (step(X @ w + b) == y).mean()
        status = f"converged in {epoch} epochs" if epoch else "never converged in 100 epochs"
        print(f"perceptron on {name}: {status}, accuracy {acc:.2f}")
        assert (acc == 1.0) == (name == "AND")

    out = two_layer_xor(X)
    print("two-layer network on XOR:", out, "target:", XOR)
    assert (out == XOR).all()
    print("one hidden layer solves XOR; a single perceptron cannot")