Statistical machine learning (decision trees, SVMs, ensembles)

Algorithms that learn prediction rules from labelled examples; they ran most practical machine learning before deep learning and still lead on table-shaped business data.

AI & roboticsSoftware
  1. 1984

    First version

    Leo Breiman, Jerome Friedman, Richard Olshen and Charles Stone (CART); Ross Quinlan (ID3); Vladimir Vapnik and colleagues (SVM, Bell Labs)

    Image: Retro-Computing Society of Rhode Island · CC BY-SA 3.0 · Wikimedia Commons
  2. 1992

    Support-vector machines with kernels

    Boser, Guyon and Vapnik combine maximum-margin classifiers with the kernel trick to learn non-linear boundaries with strong guarantees.

    Image: Bell Labs · Public domain · Wikimedia Commons
  3. 1997

    Boosting (AdaBoost)

    Freund and Schapire show many weak learners can be combined into one highly accurate model.

    Image: AT&T Corp. (AT&T Laboratories) · Unclear; no licence stated (company log… · Wayback Machine capture (27 Jul 1997) of www.re…
  4. 2001

    Random forests

    Breiman averages many randomised decision trees into a robust, near tuning-free model.

    Image: University of California, Berkeley · Unclear; institutional logo, no licence… · Wayback Machine capture of www.berkeley.edu, 6 …
  5. 2016

    XGBoost (scalable gradient boosting)

    A fast, open-source gradient-boosted tree system that dominates competitions and industry work on tabular data.

    Image: Hcho3wiki · CC BY-SA 4.0 · Wikimedia Commons

Contested: Tree learners predate the 1984 CART book: the THAID program (1972) built classification trees, CART circulated as reports from 1977, and Quinlan's ID3 work ran in parallel (published 1986).

See it on the timeline