Automatic Differentiation
Related Products
Related Services
Learn more about Automatic Differentiation
Overview
Automatic Differentiation is an algorithmic means to numerically evaluate the derivatives of functions given by a computer program. Partial derivatives of all specified dependent variables are calculated with respect to all specified independent variables based on the repeated application of the chain rule given by derivative calculus.
Automatic Differentiation overcomes critical drawbacks that are encountered in differentiation by hand, symbolic differentiation, or finite differentiation:
- No more time consuming and error-prone tasks leading to excessive use of human resources
- No more low speed in symbolic differentiation
- No more difficulties in converting computer programs into expressions meaningful to computer algebra
- No more round-off errors due to discretization
- No more cancellation errors due to finite number precision
Accuracy
Why is the accuracy of your derivatives so important, and why should you care about the round-off and cancellation errors of finite differences? The answer is simple: these errors, although small, destroy the convergence of numerical solvers for non-linear systems of equations. For example, Newton-Raphson solvers require derivatives that are more accurate than obtained from finite differencing in order to achieve quadratic convergence rates (see Press et al., in Numerical Recipes, Cambridge University Press, 2nd ed.) Many methods in non-linear optimization show similar properties and requirements. In addition, stronger non-linearities or more equations generally demand very high accuracy.
By applying derivative calculus, Automatic Differentiation does not suffer from the round-off and cancellation errors seen in finite differencing: machine-precision, accurate derivatives are directly obtained. Thus, Automatic Differentiation combines the advantages of differentiation by hand with respect to accuracy while simultaneously eliminating the time consuming, error-prone and complex tasks inherent to most other solutions.
Performance
Despite the drawbacks of differentiation by hand, many R&D teams still adhere to this method, because traditional Automatic Differentiation software tools are lacking in efficiency and performance.
High Performance is therefore one of the key issues in Automatic Differentiation. Only if automatic derivative calculation can compete in speed with hard-wired solutions is it acceptable to be incorporated in realistic applications. The performance issue has driven many source-to-source code transformation tools, where efficient derivative code is generated. However, these tools are most often
- Unstructured in the specification of dependent and independent variables,
- Have difficulties with modern language features, e.g., as in the array operations of FORTRAN 90/95,
- Are not fully compliant with the latest FORTRAN or C/C++ standards, or
- Do not produce warnings in case of mathematically non-differentiable situations.
In summary, source-to-source transformation tools usually lack broad usability. R&D teams spend more time getting it to work and running than they can and should afford.
The competing method to source-to-source technique is the operator overloading approach, which is now possible with modern FORTRAN and C/C++ compilers. Usability and compatibility of Automatic Differentiation software based on operator overloading is undoubtedly superior to the old source-to-source transformation tools. But can it compete with performance?
Vivlabs Automatic Differentiation software for FORTRAN and C/C++ demonstrates that winning performance is possible with the operator overloading approach. It is achieved with vivlabs built-in sparsity exploits and complexity reductions.
Usability
Accuracy and performance are two key requirements an Automatic Differentiation tool must meet before it should be considered for use in your real-world R&D applications. However, overall success also depends on the usability of the Automatic Differentiation software. Sooner or later you need to address the following questions:
- How long does it take to add Automatic Differentiation capabilities to your software, what is the impact on your human resources ?
- What changes are necessary to the existing code? Is complexity being added that is more costly to maintain?
- Does your Automatic Differentiation solution make it more complicated to make changes to your existing code?
- Do you need to port your code because your tool cannot cope with the full FORTRAN and C/C++ standards?
- What do you do if your code is written in both FORTRAN and C/C++?
Vivlabs provides answers and the best solutions to all of these important questions. Our software solutions leap ahead in usability, thereby giving you an invaluable edge over the competition.
Why choose vivlabs?
Vivlabs strategy is to deliver winning Automatic Differentiation software tools that combine at the same time Accuracy, Performance and Usability. In addition, our premier Support Services will help you get most out of vivlabs software products.
Look and Feel
A simple example demonstrates the look and feel of the vivlabs ADF03 software for FORTRAN. Suppose you want to calculate first-order derivatives with respect to the independent vector variable 'x':
01 subroutine example(f, x) 02 03 use mod_ADF 04 05 type(ADF_dpr) :: f(2), x(2) 06 07 ! ... some computations (no active variables) 08 09 ! x becomes independent 10 call ADF_independent((/1,2/), x, (/1.0,5.0/)) 11 12 f = sin(x**2) 13 14 ! ... more computations including function calls, 15 subroutine calls and temporary variables 16 16 ! retrieve derivatives 18 write(*,*) "df/dx(1) = ", ADF_deriv(f,1) 19 20 end subroutine example
Changes to the original code are highlighted in blue color. The differentiation process is performed completely in the background and the code containing the mathematical evaluation (lines 12-15) is not changed, even though it contains array arithmetic.