Li et al. present a novel approach for optimizing neural network implementations, that is, “a new architecture of stochastic neural networks” with a hidden approximate activation function and a hybrid stochastic multiplier that substantially reduce the hardware costs of the implementation.

The paper thoroughly outlines stochastic computing and describes stochastic neural networks that provide low hardware costs and low recognition error rates. The proposed approach defines “a hardware-oriented approximate activation function,” that is, the approximate sigmoid function, and explains it in a detailed way. To reduce hardware costs, a hybrid stochastic multiplier replaces the matrix multiplier; it decreases the number of inputs required for the binary parallel counters of the network. Furthermore, the authors propose “a new stochastic neuron with matrix multiplications and an approximate activation function”; they explain its activation along with its output--“a bit stream going through a comparator.” The paper also discusses stochastic implementations of the multilayer perceptron, the restricted Boltzmann machine, and convolutional neural networks.

The experimentation section shows the hybrid stochastic multiplier’s influence on error rates and hardware costs by comparing the results of the different implementations. This section, like the whole paper, is very orderly and well motivated.

Very well written and argued, with clear didactic introductions to each topic, this paper is not only a good read for scholars interested in neural network optimization, but also for students and professionals new to the field of neural networks and their implementation.