What Is Machine Learning?

Learning from data

A program learns from experience $E$ with respect to a task $T$ and performance measure $P$ if its performance on $T$, measured by $P$, improves with $E$ (Mitchell, 1997).

Types of learning

Supervised learning

Every training example has a label. If the label is a number we call it regression; if it is a category we call it classification.

Unsupervised learning

There are no labels. We look for structure instead: clusters, low-dimensional representations, or density estimates.

The training workflow

1. Split the data into training, validation and test sets.
2. Choose a model family and a loss function.
3. Fit the parameters on the training set.
4. Tune hyperparameters on the validation set.
5. Report the final score once on the test set.

Never tune anything on the test set — otherwise it stops being an unbiased estimate of real-world performance.