Abstract

Obtaining good performance for tensor problems on accelerators (both mapping workloads to existing accelerators, and performing design-space exploration to select hardware parameters) requires optimizing an objective function over a large, nonconvex space. This objective function represents a performance metric which may be modeled, simulated, or (if possible) measured. Each such objective function incurs different tradeoffs in terms of speed, accuracy, and the strength of results that can be formally proven, and as a result requires its own optimization methods.

In this talk, I will describe optimization approaches for several such models. In a simple communication model, we derive an unconditional communication lower bound for convolutions and “projective” tensor operations, and show that this can always be attained (up to a constant factor) by solving a mathematical optimization problem. We then add additional hardware constraints to this optimization program, resulting in a fast one-shot mapper that encompasses loop tiling, permutation, and spatio-temporal ordering. Furthermore, we describe ways to incorporate measured or simulated performance results into the objective for this mapping, and discuss recent progress on solving this problem when these results may be expensive to collect.

I’ll also describe ongoing work on extending this approach to model performance, compute lower bounds, and determine optimizations for sparse tensor operations.