Differentiable programming, or automatic differentiation, is a powerful technique in many fields, including dynamic systems, machine learning, and computer vision, mainly for solving nonlinear problems. Forward (versus reverse) differentiation based on chain rule is already implemented in software, both by operator overloading and source-to-source transformation. In this paper, the authors propose a functional programming framework, based on array processing and optimization rules, to generate efficient forward differentiable programs.

The reader will enjoy this well-structured, comprehensive paper’s many technicalities, some of which are explained via inspired examples. The presentation uses fundamental techniques from functional programming (mainly λ-calculus, Standard ML, F#) and functional linear algebra (similar to matrix-oriented processors like MATLAB, R, or NumPy).

In seven sections, the authors present the required background on automatic differentiation (section 2), the proposed functional array-processing language (section 3), the differentiation process (section 4), the transformation rules and code generation (section 5), experimental results (section 6), and connections to related works and possible future developments (section 7).

The most important developments relate to the differentiation process: the deriv construct; the high-level differentiation application programming interface for (diff, grad, jacob); and the automatic differentiation rules for expressions in order to support “source-to-source translation for implementing forward-mode automatic differentiation.” Many classes of transformation rules are considered for efficient differentiation (rules based on λ-calculus, rules according to the algebraic structure, fusion rules for pull-arrays, rewriting rules for conditional expressions, normalization rules for loops, partial evaluation rules for tuples, and loop fission rules). This technical part closes with C code generated for an optimized differentiated program, and a final discussion on key challenges for applying rewriting rules, heuristics, cost models, and destination-passing style techniques.

The proposed approach is compared against DiffSharp, Tapenade, Theano, TensorFlow, and Futhark. The optimized differentiated approach with improved memory management for low-level code generation outperformed competitors. However, there is no online reference to a compiler implementation demonstrating such performance.