The “learning” in machine learning involves the acquisition of a model that reflects the behavior or the functionality behind some dataset. Since there are so many techniques (called classifiers) to process data and extract some behavior or functionality from it, model selection involves criteria for selecting one of those techniques, and its associated model, based on the difference between the expected behavior or functionality and the one learned by the technique. The availability of a technique to estimate such an error is crucial, since it provides a measure for the future, that is, how far is the learned model from the expected result for future data. This information becomes interesting in many critical areas, such as medicine, defense, security, and economics, where machine learning models are behind many important applications.

The book starts by motivating how statistical learning theory (SLT) is becoming very important in machine learning by providing some measures for predictions on future data. Also, it clarifies the differences between deduction, induction, and abduction learning, and the two main approaches to statistical inference: Bayesian and frequentist. The book puts together six different approaches to error estimation depending on the underlying techniques:

(1) Resampling methods, which are useful when there are large datasets that can be split into smaller sets used for training and error estimation;

(2) Complexity-based methods, which are useful when the model is described as a set of rules that can be analyzed independently in order to identify relevant and nonrelevant rules;

(3) Compression methods, which are useful when the dataset incorporates some smaller critical dataset to be identified;

(4) Algorithmic stability, which is useful for datasets where training from a bigger or smaller dataset is not so important because the learned model is very similar;

(5) Probably approximately correct Bayes (PAC-Bayes), when different classifiers are combined to obtain a sharper model; and

(6) Differential privacy, when it is important that the model becomes independent of each one of the single observations in the dataset.

This book collects many interesting approaches and criteria for both selecting a learning technique and estimating the upper-bound of an error introduced in the learned model. It is written for machine learning experts, as it is very technical and requires a strong statistical background. However, each chapter is short (five to ten pages) and includes its own references (around 50), which gives the impression of a very condensed and handy piece of work. Unfortunately, there are not many examples that could help readers identify the differences between the approaches or their applicability in real scenarios.