Downloads: 4 | Views: 89 | Weekly Hits: ⮙2 | Monthly Hits: ⮙2
Informative Article | Engineering Science | India | Volume 12 Issue 10, October 2023 | Rating: 4.8 / 10 | Rate this Article
Evaluating and Selecting Appropriate Technologies and Tools for Data Ingestion: Apache Kafka and Custom - Built Solutions
Abstract: The landscape of data ingestion technologies is rapidly evolving to meet the growing demands of big data analytics. As organizations strive to gain real - time insights from their data, the need for robust, scalable, and efficient data ingestion platforms has never been more critical. This paper presents a comprehensive evaluation of Apache Kafka, a prominent open - source streaming platform, against custom - built data ingestion solutions. We examine the scalability, performance, fault tolerance, and ease of integration of these technologies. A systematic approach is employed to evaluate their suitability for various use cases, including high - throughput systems, IoT environments, and real - time analytics.
Keywords: Apache Kafka, Data ingestion technologies, Stream processing, Big data analytics, Real - time data processing, Scalability, Fault tolerance, Performance benchmarking, System integration, Custom - built data solutions, Messaging systems, Distributed systems, Internet of Things (IoT), High - throughput systems, Data pipeline architecture, Publish - subscribe model, Data integrity, Data volume, Cloud computing, Data engineering.
Edition: Volume 12 Issue 10, October 2023,
Pages: 2140 - 2143