Project : tropics
Section: New Results
Keywords : reverse mode of AD, recompute-all, store-all, storage-recomputation tradeoff.
The Recompute-or-Store alternative
In the reverse mode of Automatic Differentiation, values must be made available to the computation of the derivatives in the reverse of their original computation order. This can be achieved either by storing intermediate values as they are computed, or by recomputing them when needed. This work is an effort to study the relative merits of these two strategies, known as store-all, implemented in tapenade and adifor, and recompute-all, implemented in taf. Neither strategy is optimal in terms of execution time and memory space. The optimal is probably a combination of these two strategies.
This year, we designed a framework which, for each intermediate value, offers the choice between storing or recomputing. We call it the Recompute-Or-Store Alternative (ROSA). This framework is based on the data dependence graph of the considered program. We investigated how this framework captures the previous strategies, and we proposed heuristics to find an efficient combination of recomputing and storing. We study how the ROSA framework blends with the classical checkpointing strategy which trades execution time for memory space at a coarser grain level in the program. Experiments are under way in tapenade
This ROSA framework was presented at the AD2004 conference.