Computing Reviews
Today's Issue Hot Topics Search Browse Recommended My Account Log In
Review Help
Search
Efficient differentiable programming in a functional array-processing language
Shaikhha A., Fitzgibbon A., Vytiniotis D., Peyton Jones S.  Proceedings of the ACM on Programming Languages 3 (ICFP): 1-30, 2019. Type: Article
Date Reviewed: Jul 20 2021

Differentiable programming, or automatic differentiation, is a powerful technique in many fields, including dynamic systems, machine learning, and computer vision, mainly for solving nonlinear problems. Forward (versus reverse) differentiation based on chain rule is already implemented in software, both by operator overloading and source-to-source transformation. In this paper, the authors propose a functional programming framework, based on array processing and optimization rules, to generate efficient forward differentiable programs.

The reader will enjoy this well-structured, comprehensive paper’s many technicalities, some of which are explained via inspired examples. The presentation uses fundamental techniques from functional programming (mainly λ-calculus, Standard ML, F#) and functional linear algebra (similar to matrix-oriented processors like MATLAB, R, or NumPy).

In seven sections, the authors present the required background on automatic differentiation (section 2), the proposed functional array-processing language (section 3), the differentiation process (section 4), the transformation rules and code generation (section 5), experimental results (section 6), and connections to related works and possible future developments (section 7).

The most important developments relate to the differentiation process: the deriv construct; the high-level differentiation application programming interface for (diff, grad, jacob); and the automatic differentiation rules for expressions in order to support “source-to-source translation for implementing forward-mode automatic differentiation.” Many classes of transformation rules are considered for efficient differentiation (rules based on λ-calculus, rules according to the algebraic structure, fusion rules for pull-arrays, rewriting rules for conditional expressions, normalization rules for loops, partial evaluation rules for tuples, and loop fission rules). This technical part closes with C code generated for an optimized differentiated program, and a final discussion on key challenges for applying rewriting rules, heuristics, cost models, and destination-passing style techniques.

The proposed approach is compared against DiffSharp, Tapenade, Theano, TensorFlow, and Futhark. The optimized differentiated approach with improved memory management for low-level code generation outperformed competitors. However, there is no online reference to a compiler implementation demonstrating such performance.

Reviewer:  G. Albeanu Review #: CR147312
Bookmark and Share
  Featured Reviewer  
 
General (D.3.0 )
 
 
Automatic Differentiation (G.1.4 ... )
 
 
General (G.0 )
 
Would you recommend this review?
yes
no
Other reviews under "General": Date
Constructing quotient inductive-inductive types
Kaposi A., Kovács A., Altenkirch T.  Proceedings of the ACM on Programming Languages 3(POPL): 1-24, 2019. Type: Article
Jul 9 2021
 Understanding programming languages
Jones C.,  Springer, Cham, Switzerland, 2020. 240 pp. Type: Book (978-3-030592-56-1)
Jun 29 2021
 The kollected Kode Vicious: opinionated advice for programmers
Neville-Neil G.,  Addison-Wesley, Boston, MA, 2020. 311 pp. Type: Book (978-1-367882-46-1)
Apr 30 2021
more...

E-Mail This Printer-Friendly
Send Your Comments
Contact Us
Reproduction in whole or in part without permission is prohibited.   Copyright © 2000-2021 ThinkLoud, Inc.
Terms of Use
| Privacy Policy