Paper Info
Title
Large Scale Distributed Optimization using Apache Spark: Distributed Scalable Shade-Bat (DistSSB)
Abstract
Large Scale Global Optimization (LSGO) is interesting for its applications in machine learning, e.g. in deep learning or knowledge graph embeddings. Evolutionary algorithms (EA) have been found efficient in solving complex optimization problems. However, the performance of conventional EAs degrades with the increasing number of decision variables due to the lack of scalability. This paper proposes a scalable, parallel, distributed and hybrid EA named Distributed Scalable Shade-Bat-Scalable Shade Bat (DistSSB) to solve LSGO problems. DistSSB is inspired by the exploration capability of the SHADE algorithm and exploitation feature of the Bat algorithm (BA). To achieve scalability, DistSSB is implemented using the popular distributed in-memory framework, Apache Spark. DistSSB distributes its population into multiple sub-populations using the island model. Each sub-population is independently evolved using SHADE or BA. After the migration interval, the best solutions are broadcasted employing the mesh topology. We have compared the scalability, efficiency, and efficacy of DistSSB with SHADE-ILS (CEC-2018 Winner) and GL-SHADE algorithm on the CEC-2013 LSGO benchmark function suite. For most functions, DistSSB has obtained better optimization results in lesser execution time as compared to SHADE-ILS and GL-SHADE. We have tested and shown the scalability of DistSSB for up to "one million" dimensions, whereas SHADE-ILS and GLSHADE fail to scale up for larger problems.
Year
DOI
Venue
2021
10.1109/CEC45853.2021.9504853
2021 IEEE CONGRESS ON EVOLUTIONARY COMPUTATION (CEC 2021)
Keywords
DocType
Citations 
LSGO, Apache Spark, Differential Evolution, SHADE-ILS, GL-SHADE, Evolutionary Computation, Distributed, Parallel, Scalable
Conference
0
PageRank 
References 
Authors
0.34
0
4
Name
Order
Citations
PageRank
Fahad Maqbool100.34
Saad Razzaq211.74
Asif Yar300.34
Hajira Jabeen403.04