Comparative Analysis of Hadoop and Spark Performance for Real-time Big Data Smart Platforms Utilizing IoT Technology in Electrical Facilities
Gabdullin M.T. Suinullayev Y. Kabi Y. Kang J.W. Mukasheva A.
September 2024Korean Institute of Electrical Engineers
Journal of Electrical Engineering and Technology
2024#19Issue 74595 - 4606 pp.
As the adoption of IoT technology in power systems accelerates and the need for improved methods to handle large volumes of data emerges, real-time big data smart platforms must address the growing data processing demands in IoT integrated power systems. Therefore, in this study, we assess the performance of Hadoop and Spark for iterative computing and real-time data processing applications. Our evaluation is based on metrics such as execution time, resource utilization, and scalability, particularly with increasing data volume. The comparison aims to provide guidance to researchers, practitioners and entrepreneurs on platform selection depending on their specific requirements. The study identified the strengths and weaknesses of both platforms and provided valuable insights into optimizing the performance of big data applications. Text documents and charts for Word Count and PageRank tasks were used for comparison, and performance testing was performed on datasets of different sizes. The results showed that Spark outperforms Hadoop in most applications, especially in iterative computation and real-time data processing, due to its use of in-memory computation. However, Hadoop is best suited for batch processing operations that require multiple steps. It can perform these operations in parallel across multiple cluster nodes, enabling fast processing of large amounts of data. This comprehensive performance comparison of Hadoop and Spark in iterative computing and real-time data processing applications provides valuable information for researchers, practitioners, and enterprises on the trade-offs and benefits of using these big data platforms.
Hadoop , IoT , Performance evaluation , Real-time data processing , Spark
Text of the article Перейти на текст статьи
School of Materials Science and Green Technologies, Kazakh-British Technical University, Almaty, Kazakhstan
Almaty University of Power Engineering and Telecommunications, Almaty, Kazakhstan
Transportation System Engineering, Korea National University of Transportation, Gyeonggi-Do, Uiwang-Si, 16106, South Korea
School of Information Technology and Engineering, Kazakh-British Technical University, Almaty, Kazakhstan
School of Materials Science and Green Technologies
Almaty University of Power Engineering and Telecommunications
Transportation System Engineering
School of Information Technology and Engineering
10 лет помогаем публиковать статьи Международный издатель
Книга Публикация научной статьи Волощук 2026 Book Publication of a scientific article 2026