Downloads: 0 | Views: 75
Informative Article | Software Engineering | India | Volume 11 Issue 12, December 2022 | Rating: 4.5 / 10
Efficiently Managing Billions of Data Points with Configurable and Extensible Functionality
Abstract: The solution outlines a scalable and adaptable data management system comprising three main components: Config, Gate, and Store. The Config component centrally defines data operation parameters such as table structures, field definitions, and sharding criteria, ensuring consistency and flexibility across different database types. It integrates a user - friendly UI for configuring indexes, optimizing query performance, and maintaining database integrity through automated backend enforcement. The Gate component acts as an intermediary layer, directing data operations to appropriate tables based on Config - defined rules. Utilizing a key - value database for efficient shard key mapping, it simplifies data distribution complexities for application logic, enhancing focus on higher - level functionalities without compromising performance or scalability. The Store component serves as the physical data storage layer, abstracting database - specific details to offer a unified interface for CRUD operations, table management, and indexing. It supports database agnosticism through technologies like SQLAlchemy, facilitating seamless integration across relational and NoSQL databases while ensuring consistent data management practices. Technical considerations include robust security measures against SQL injection and other threats, deployed on a web server infrastructure to manage frontend UI and backend services cohesively. By leveraging these components, the system aims to achieve modular architecture, scalability, and adaptability to diverse database environments, promising efficient data handling and optimized performance across various use cases.
Keywords: Data, Large volume of Data, IoT Data, E - Commerce Applications
Edition: Volume 11 Issue 12, December 2022,
Pages: 1335 - 1338