Fernandez, R. C., Migliavacca, M., Kalyvianaki, E. & Pietzuch, P. (2014). Making State Explicit for Imperative Big Data Processing. Paper presented at the USENIX ATC ’14: 2014 USENIX Annual Technical Conference, 19-06-2015 - 20-06-2015, Philadelphia, USA.
- Published Version
Download (410kB) | Preview
Data scientists often implement machine learning algorithms in imperative languages such as Java, Matlab and R. Yet such implementations fail to achieve the performance and scalability of specialised data-parallel processing frameworks. Our goal is to execute imperative Java programs in a data-parallel fashion with high throughput and low latency. This raises two challenges: how to support the arbitrary mutable state of Java programs without compromising scalability, and how to recover that state after failure with low overhead.
Our idea is to infer the dataflow and the types of state accesses from a Java program and use this information to generate a stateful dataflow graph (SDG). By explicitly separating data from mutable state, SDGs have specific features to enable this translation: to ensure scalability, distributed state can be partitioned across nodes if computation can occur entirely in parallel; if this is not possible, partial state gives nodes local instances for independent computation, which are reconciled according to application semantics. For fault tolerance, large inmemory state is checkpointed asynchronously without global coordination. We show that the performance of SDGs for several imperative online applications matches that of existing data-parallel processing frameworks.
|Item Type:||Conference or Workshop Item (Paper)|
|Additional Information:||This paper was first published in the Proceedings of USENIX ATC ’14: 2014 USENIX Annual Technical Conference.Copyright to this work is retained by the author[s]. Permission is granted for the noncommercial reproduction of the complete work for educational or research purposes.|
|Subjects:||Q Science > QA Mathematics > QA75 Electronic computers. Computer science|
|Divisions:||School of Informatics > Department of Computing|
Actions (login required)
Downloads per month over past year