Downloads: 5 | Views: 190 | Weekly Hits: ⮙1 | Monthly Hits: ⮙2

Informative Article | Engineering Science | India | Volume 12 Issue 10, October 2023 | Popularity: 4.7 / 10

Rate This Article!      

Evaluating and Selecting Appropriate Technologies and Tools for Data Ingestion: Apache Kafka and Custom - Built Solutions

Fasihuddin Mirza

Abstract: The landscape of data ingestion technologies is rapidly evolving to meet the growing demands of big data analytics. As organizations strive to gain real - time insights from their data, the need for robust, scalable, and efficient data ingestion platforms has never been more critical. This paper presents a comprehensive evaluation of Apache Kafka, a prominent open - source streaming platform, against custom - built data ingestion solutions. We examine the scalability, performance, fault tolerance, and ease of integration of these technologies. A systematic approach is employed to evaluate their suitability for various use cases, including high - throughput systems, IoT environments, and real - time analytics.

Keywords: Apache Kafka, Data ingestion technologies, Stream processing, Big data analytics, Real - time data processing, Scalability, Fault tolerance, Performance benchmarking, System integration, Custom - built data solutions, Messaging systems, Distributed systems, Internet of Things (IoT), High - throughput systems, Data pipeline architecture, Publish - subscribe model, Data integrity, Data volume, Cloud computing, Data engineering.

Edition: Volume 12 Issue 10, October 2023

Pages: 2140 - 2143

DOI: https://www.doi.org/10.21275/SR24418102327

Make Sure to Disable the Pop-Up Blocker of Web Browser

Click to Cite this Article