Thomas Plagemann
Lecturer Thomas Plagemann
University of Oslo, Norway
Title “Semantic and utility aware operator migration for Distributed Stream Processing”
Date Tuesday, 24th October 2023; 11:00 am
Location S3|20 (Rundeturmstraße 10); room 111 and via Zoom
Abstract
Operator migration is an essential adaptation mechanism in distributed stream processing systems. Its main challenge is to minimize the experienced latency for the data consumers and to avoid data loss. To improve on the state-of-the-art, we introduce an operator semantic aware migration mechanism, called Lazy Migration. The semantics of operators determines how their state is managed and partitioned, e.g., window extents for aggregation and tuple lists for join. There are two modes in Lazy Migration to migrate such state partitions: (1) for the latency mode we exploit, for example, the window semantics to migrate the parts exactly when the operators need them to minimize the output latency of tuples, and (2) in the utility mode each partial state is assigned a utility value based on its anticipated future demand. The utility mode is designed for situations in which it might be impossible to migrate the entire state in time to ensure that always the state parts with the highest utility are at the new host. Finally, we demonstrate join and aggregation scenarios where Lazy Migration excels, in addition to pros and cons of the other migration mechanisms.