Downloads: 0 | Views: 16

Research Paper | Computer Science & Engineering | United States of America | Volume 12 Issue 11, November 2023 | Rating: 3.9 / 10

Real-Time Data Processing with Streaming ETL

Dhamotharan Seenivasan

Abstract: Real-time ETL processing using streaming ETL is critical for organizations desiring to utilize up-to-date information and make decisions based on that data. This paper discusses the building blocks, approaches, and advantages of real-time streaming ETL techniques, as well as focuses on structures, tools, and methods by means of which real-time data management could be well-organized and efficient in data processing. The discussion also encompasses real-world examples from different industries globally, with a focus on illustrating the use case and benefits of embracing streaming ETL. Some of the critical issues include data consistency, latency, and fault tolerance, which are discussed with the available solutions and future trends in Real-time data processing. Continuous assessments of data quality and rapid change in data requirements are some ways that make the traditional batch processing methods inadequate to cater for the organizations? need for real-time information, which has led to the integration of streaming ETL systems. These systems involve a constant stream of data processing, whereby information that has not been altered can be converted to tangible results in real-time. Therefore, the goal of this paper shall be to provide a starting point for understanding the architecture, tools, methodologies, and issues of real-time streaming ETL systems.

Keywords: Real-Time Data Processing, Streaming ETL, Low Latency, Fault Tolerance, Data Ingestion

Edition: Volume 12 Issue 11, November 2023,

Pages: 2185 - 2192