Abstract

Distributed data-flow frameworks, such as MapReduce and Dryad, have made it simpler to program a large number of computers at once. However, as many researchers have observed, existing frameworks provide a restricted programming model that inhibits the implementation of many naturally data-parallel algorithms. To address this problem, we have developed CIEL , which is a distributed execution engine that extends previous systems by allowing data-dependent control flow. This enables CIEL to natively execute iterative and recursive algorithms, including k-means clustering, PageRank and many sparse linear algebra methods.

In this talk, I will introduce CIEL , and describe the techniques that we use to make execution reliable. I will also describe Skywriting, which is a Turing-complete scripting language that runs directly on a CIEL cluster, and simplifies the construction of iterative jobs. We have deployed CIEL on a cloud computing platform, and I will present performance results that compare CIEL with Hadoop, the open-source implementation of MapReduce.