MURAL - Maynooth University Research Archive Library



    Reverse-Mode AD in a Functional Framework: Lambda the Ultimate Backpropagator


    Pearlmutter, Barak A. and Siskind, Jeffrey Mark (2008) Reverse-Mode AD in a Functional Framework: Lambda the Ultimate Backpropagator. ACM Transactions on Programming Languages and Systems, 30 (2). 7.1-7.36. ISSN 0164-0925

    [thumbnail of BP_Reverse-Mode.pdf] PDF
    BP_Reverse-Mode.pdf

    Download (354kB)

    Abstract

    We show that reverse-mode AD (Automatic Differentiation)—a generalized gradient-calculation operator—can be incorporated as a first-class function in an augmented lambda calculus, and therefore into a functional-programming language. Closure is achieved, in that the new operator can be applied to any expression in the augmented language, yielding an expression in that language. This requires the resolution of two major technical issues: (a) how to transform nested lambda expressions, including those with free-variable references, and (b) how to support self application of the AD machinery. AD transformations preserve certain complexity properties, among them that the reverse phase of the reverse-mode AD transformation of a function have the same temporal complexity as the original untransformed function. First-class unrestricted AD operators increase the expressive power available to the numeric programmer, and may have significant practical implications for the construction of numeric software that is robust, modular, concise, correct, and efficient.
    Item Type: Article
    Additional Information: The work of B. A. Pearlmutter was supported, in part, by Science Foundation Ireland grant 00/PI.1/C067 and the Higher Education Authority of Ireland. The work of J. M. Siskind was supported, in part, by National Science Foundation (NSF) grant CCF-0438806. © ACM, 2008. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Transactions on Programming Languages and Systems (TOPLAS) Volume 30 Issue 2, March 2008, http://doi.acm.org/10.1145/1330017.1330018
    Keywords: closures; derivatives; forward-mode AD; higher-order AD; higher-order functional languages; Jacobian; program transformation; reflection;
    Academic Unit: Faculty of Science and Engineering > Computer Science
    Faculty of Science and Engineering > Research Institutes > Hamilton Institute
    Item ID: 3504
    Depositing User: Barak Pearlmutter
    Date Deposited: 29 Feb 2012 14:25
    Journal or Publication Title: ACM Transactions on Programming Languages and Systems
    Publisher: Association for Computing Machinery (ACM)
    Refereed: Yes
    Funders: Higher Education Authority of Ireland, National Science Foundation (NSF), Science Foundation Ireland
    Related URLs:
    URI: https://mu.eprints-hosting.org/id/eprint/3504
    Use Licence: This item is available under a Creative Commons Attribution Non Commercial Share Alike Licence (CC BY-NC-SA). Details of this licence are available here

    Repository Staff Only (login required)

    Item control page
    Item control page

    Downloads

    Downloads per month over past year

    Origin of downloads