The evaluation of machine learning algorithms has always been controversial. What datasets, experimental setting, and statistical tests must be chosen? Dataset repositories such as UCI have been enormously handy for machine learning research. However, the selection of datasets is commonly careless (when not cherry-picked).

This paper is not the first criticism about the way these datasets are used, but it is the most comprehensive, insightful, and constructive so far. The use of complexity measures and a characterization of the UCI datasets are most welcome. Nonetheless, the analysis could have been more complete with complexity measures derived from (approximations of) Kolmogorov complexity and with other performance metrics (only accuracy is used).

Despite the implausibility of the assumptions of the no-free-lunch theorem, it pervades the authors’ notion of diversity. More diversity does not mean that problems should cover all ranges of error and complexity measures in a uniform way. More effort should be done to clarify what a “representative” sample of real problems is if we want to assess whether the UCI repository is diverse and “challenging” enough.

The authors also present a basic dataset generator based on injecting distortion (a pattern-based generator would possibly be a better option). Anyway, the use of dataset generators jointly with a more regulated and automated evaluation procedure is the way to go.

This paper should not only contribute to a debate in the community, but it should also become a must-read for everyone using the UCI datasets for the evaluation of machine learning algorithms.