Downloads: 1 | Views: 151 | Weekly Hits: ⮙1 | Monthly Hits: ⮙1

Informative Article | Data & Knowledge Engineering | India | Volume 11 Issue 2, February 2022 | Popularity: 4.6 / 10

Rate This Article!      

Optimizing Data Stream Processing Pipelines: Using In-Memory DB and Change Data Capture for Low - Latency Enrichment

Purshotam S Yadav

Abstract: This paper presents a new way of optimization of data stream processing pipelines using Redis, an in - memory data store, and Change Data Capture for real - time data synchronization. We will detail how this combination reduces latency during the data enrichment process-for instance, one critical building block for nearly all stream processing architectures today. Our experiments show massive improvements in the speed and efficiency of processing as against traditional approaches, peaking at 80% latency reduction and 3x increase in throughput. On - demand scalable solutions to large volume and real - time data streams are enabled in application domains such as financial analytics, IoT, and social media analytics.

Keywords: Data stream processing, Pipeline optimization, In - memory databases, Change data capture (CDC), Low - latency, Data enrichment, Real - time processing, Data integration, ETL, Stream analytics

Edition: Volume 11 Issue 2, February 2022

Pages: 1355 - 1357

DOI: https://www.doi.org/10.21275/SR24708103903

Make Sure to Disable the Pop-Up Blocker of Web Browser

Click to Cite this Article

Similar Articles