Downloads: 19 | Views: 612 | Weekly Hits: ⮙1 | Monthly Hits: ⮙2
Informative Article | Data & Knowledge Engineering | India | Volume 7 Issue 11, November 2018 | Rating: 5.8 / 10
Design and Implementation of High-Throughput Data Streams using Apache Kafka for Real-Time Data Pipelines
Preyaa Atri
Abstract: In an era dominated by the need for real-time data processing, Apache Kafka emerges as a crucial technology for constructing high-throughput data pipelines capable of handling extensive data streams with minimal latency. This paper provides an in-depth exploration into the design and implementation of Kafka-based data pipelines, discussing their architectural patterns, performance optimization techniques, and practical applications across various domains. Through detailed analysis and expert recommendations, this study addresses current challenges and maps out future research directions, underlining Kafka's pivotal role in advancing real-time data processing systems. The insights presented aim to guide professionals in enhancing the efficiency and reliability of their real-time data solutions.
Keywords: Apache Kafka, data streams, real-time processing, data pipelines, high- throughput, distributed systems, stream processing, big data
Edition: Volume 7 Issue 11, November 2018,
Pages: 1988 - 1991