Naiad: a system for incremental, iterative and interactive parallel computation

Add to your list(s) Download to your calendar using vCal

• Frank McSherry (MSR Silicon Valley)
• Wednesday 07 November 2012, 16:15-17:00
• FW26, Computer Laboratory, William Gates Builiding.

If you have a question about this talk, please contact Eiko Yoneki.

We are developing a new system for large-scale data analysis—called “Naiad”—which has the goal of supporting complex iterative queries over dynamic inputs at interactive timescales. Like many existing systems, Naiad supports high-level declarative queries, data-parallel execution, and transparent distribution. Unlike these systems, Naiad can efficiently execute queries with multiple (possibly nested) iterative loops, while simultaneously supporting low-latency incremental changes to the query inputs. As a highlight of its characteristics, Naiad can not only efficiently compute the strongly connected component structure of a 24 hour sliding window of the Twitter @mention graph (using a doubly nested fixed-point computation), but also maintains the computation with sub-second latencies in the face of Twitter’s full volume of continuously arriving tweets.

I will describe the computational model underlying Naiad, a generalization of traditional incremental dataflow to partially ordered logical times, and work through some of the (very friendly, picture oriented) mathematical details. I will also highlight several new distributed systems challenges faced in order to fully realize the multiple orders-of-magnitude performance improvements Naiad presents.

This is joint work with Derek Murray, Rebecca Isaacs, Michael Isard, and Martìn Abadi.

Bio: Frank McSherry is a Senior Researcher at Microsoft’s Silicon Valley Lab, where he focuses on issues related to large-scale data analysis. He has previously worked on machine learning and privacy issues, and is currently hard at work on large-scale low-latency computational infrastructure.

This talk is part of the Computer Laboratory Systems Research Group Seminar series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• CL's SRG seminar
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge talks
• Chris Davis' list
• Computer Laboratory Systems Research Group Seminar
• Department of Computer Science and Technology talks and seminars
• FW26, Computer Laboratory, William Gates Builiding
• Interested Talks
• School of Technology
• Trust & Technology Initiative - interesting events
• bld31
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.