Thomas Plagemann
Vortragender Thomas Plagemann
University of Oslo, Norway
Titel „Semantic and utility aware operator migration for Distributed Stream Processing“
Zeit Dienstag, 24. Oktober 2023; 11:00 Uhr
Ort S3|20 (Rundeturmstraße 10); Raum 111 und via Zoom
Abstract
Operator migration is an essential adaptation mechanism in distributed stream processing systems. Its main challenge is to minimize the experienced latency for the data consumers and to avoid data loss. To improve on the state-of-the-art, we introduce an operator semantic aware migration mechanism, called Lazy Migration. The semantics of operators determines how their state is managed and partitioned, e.g., window extents for aggregation and tuple lists for join. There are two modes in Lazy Migration to migrate such state partitions: (1) for the latency mode we exploit, for example, the window semantics to migrate the parts exactly when the operators need them to minimize the output latency of tuples, and (2) in the utility mode each partial state is assigned a utility value based on its anticipated future demand. The utility mode is designed for situations in which it might be impossible to migrate the entire state in time to ensure that always the state parts with the highest utility are at the new host. Finally, we demonstrate join and aggregation scenarios where Lazy Migration excels, in addition to pros and cons of the other migration mechanisms.