Downloads: 0 | Views: 8
Informative Article | Data & Knowledge Engineering | India | Volume 11 Issue 2, February 2022 | Rating: 2.9 / 10
Optimizing Data Stream Processing Pipelines: Using In-Memory DB and Change Data Capture for Low - Latency Enrichment
Abstract: This paper presents a new way of optimization of data stream processing pipelines using Redis, an in - memory data store, and Change Data Capture for real - time data synchronization. We will detail how this combination reduces latency during the data enrichment process-for instance, one critical building block for nearly all stream processing architectures today. Our experiments show massive improvements in the speed and efficiency of processing as against traditional approaches, peaking at 80% latency reduction and 3x increase in throughput. On - demand scalable solutions to large volume and real - time data streams are enabled in application domains such as financial analytics, IoT, and social media analytics.
Keywords: Data stream processing, Pipeline optimization, In - memory databases, Change data capture (CDC), Low - latency, Data enrichment, Real - time processing, Data integration, ETL, Stream analytics
Edition: Volume 11 Issue 2, February 2022,
Pages: 1355 - 1357