Backpropagation

Efficient gradient computation that lets multilayer networks learn internal representations; the training algorithm behind virtually all deep learning.

AI & roboticsSoftwareLandmark
  1. 1986

    First version

    David Rumelhart, Geoffrey Hinton, Ronald Williams (popularised); Seppo Linnainmaa, Paul Werbos (earlier)

    Image: Carnegie Mellon University · © Carnegie Mellon University; university seal (trademark), no licence stated · Carnegie Mellon University brand guidelines, 'University Seal' page (one-colour black seal sample)
  2. 1989
    Backprop-trained CNNs

    Backprop-trained CNNs

    LeCun applies backprop to convolutional nets for real-world digit recognition.

    Image: Petri Kutvonen (University of Helsinki), photo c. 1988 · Unclear; no licence stated (© Petri Kut… · Petri Kutvonen's photo pages, University of Hel…
  3. 1997

    LSTM (Hochreiter & Schmidhuber)

    Backprop through time made workable for long sequences via gated memory cells.

    Image: IDSIA (Istituto Dalle Molle di Studi sull'Intelligenza Artificiale) · Unclear; no licence stated (institute l… · Wayback Machine capture of www.idsia.ch, 16 Jul…
  4. 2015

    Automatic-differentiation frameworks

    TensorFlow (2015) and PyTorch (2016) make backprop automatic for any model.

    Image: TensorFlow - vectors combined, edited - Begoon · Apache License 2.0 · Wikimedia Commons

Contested: Linnainmaa published reverse-mode differentiation in 1970 and Werbos proposed it for neural nets in 1974/1982; the 1986 Nature paper popularised it.

See it on the timeline