1. Home
  2. Vol 7, No 4 (2023)
  3. Frank

Advanced Genetic Programming vs. State-of-the-Art AutoML in Imbalanced Binary Classification

Franz Frank, Fernando Bacao

Abstract


The objective of this article is to provide a comparative analysis of two novel genetic programming (GP) techniques, differentiable Cartesian genetic programming for artificial neural networks (DCGPANN) and geometric semantic genetic programming (GSGP), with state-of-the-art automated machine learning (AutoML) tools, namely Auto-Keras, Auto-PyTorch and Auto-Sklearn. While all these techniques are compared to several baseline algorithms upon their introduction, research still lacks direct comparisons between them, especially of the GP approaches with state-of-the-art AutoML. This study intends to fill this gap in order to analyze the true potential of GP for AutoML. The performances of the different tools are assessed by applying them to 20 benchmark datasets of the imbalanced binary classification field, thus an area that is a frequent and challenging problem. The tools are compared across the four categories average performance, maximum performance, standard deviation within performance, and generalization ability, whereby the metrics F1-score, G-mean, and AUC are used for evaluation. The analysis finds that the GP techniques, while unable to completely outperform state-of-the-art AutoML, are indeed already a very competitive alternative. Therefore, these advanced GP tools prove that they are able to provide a new and promising approach for practitioners developing machine learning (ML) models.

 

Doi: 10.28991/ESJ-2023-07-04-021

Full Text: PDF


Keywords


Genetic Programming; Automated Machine Learning; AutoML; Imbalanced Binary Classification.

References


He, X., Zhao, K., & Chu, X. (2021). AutoML: A survey of the state-of-the-art. Knowledge-Based Systems, 212, 106622. doi:10.1016/j.knosys.2020.106622.

Wever, M., Tornede, A., Mohr, F., & Hullermeier, E. (2021). AutoML for Multi-Label Classification: Overview and Empirical Evaluation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9), 3037–3054. doi:10.1109/TPAMI.2021.3051276.

Le, T. T., Fu, W., & Moore, J. H. (2020). Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics, 36(1), 250–256. doi:10.1093/bioinformatics/btz470.

Castelli, M., Pinto, D. C., Shuqair, S., Montali, D., & Vanneschi, L. (2022). The Benefits of Automated Machine Learning in Hospitality: A Step-By-Step Guide and AutoML Tool. Emerging Science Journal, 6(6), 1237–1254. doi:10.28991/ESJ-2022-06-06-02.

Märtens, M., & Izzo, D. (2019). Neural network architecture search with differentiable cartesian genetic programming for regression. Proceedings of the Genetic and Evolutionary Computation Conference Companion. doi:10.1145/3319619.3322003.

Bakurov, I., Castelli, M., Fontanella, F., & Vanneschi, L. (2019). A Regression-like Classification System for Geometric Semantic Genetic Programming. Proceedings of the 11th International Joint Conference on Computational Intelligence, Vienna, Austria. doi:10.5220/0008052900400048.

Bakurov, I., Castelli, M., Fontanella, F., Scotto di Freca, A., & Vanneschi, L. (2022). A novel binary classification approach based on geometric semantic genetic programming. Swarm and Evolutionary Computation, 69, 101028. doi:10.1016/j.swevo.2021.101028.

Conrad, F., Mälzer, M., Schwarzenberger, M., Wiemer, H., & Ihlenfeldt, S. (2022). Benchmarking AutoML for regression tasks on small tabular data in materials design. Scientific Reports, 12(1), 19350. doi:10.1038/s41598-022-23327-1.

Alsharef, A., Aggarwal, K., Sonia, Kumar, M., & Mishra, A. (2022). Review of ML and AutoML Solutions to Forecast Time-Series Data. Archives of Computational Methods in Engineering, 29(7), 5297–5311. doi:10.1007/s11831-022-09765-0.

Jin, H., Song, Q., & Hu, X. (2019). Auto-Keras: An Efficient Neural Architecture Search System. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. doi:10.1145/3292500.3330648.

Zimmer, L., Lindauer, M., & Hutter, F. (2021). Auto-Pytorch: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9), 3079–3090. doi:10.1109/TPAMI.2021.3067763.

Feurer, M., Klein, A., Eggensperger, K., Springenberg, J. T., Blum, M., & Hutter, F. (2019). Auto-Sklearn: Efficient and Robust Automated Machine Learning. The Springer Series on Challenges in Machine Learning, 113–134. doi:10.1007/978-3-030-05318-5_6.

Douzas, G., & Bacao, F. (2019). Geometric SMOTE a geometrically enhanced drop-in replacement for SMOTE. Information Sciences, 501, 118–135. doi:10.1016/j.ins.2019.06.007.

Douzas, G., & Bacao, F. (2018). Effective data generation for imbalanced learning using conditional generative adversarial networks. Expert Systems with Applications, 91, 464–471. doi:10.1016/j.eswa.2017.09.030.

He, H., & Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9), 1263–1284. doi:10.1109/TKDE.2008.239.

Santu, S. K. K., Hassan, M. M., Smith, M. J., Xu, L., Zhai, C., & Veeramachaneni, K. (2022). AutoML to Date and Beyond: Challenges and Opportunities. ACM Computing Surveys, 54(8), 1–36. doi:10.1145/3470918.

Bahri, M., Salutari, F., Putina, A., & Sozio, M. (2022). AutoML: state of the art with a focus on anomaly detection, challenges, and research directions. International Journal of Data Science and Analytics, 14(2), 113–126. doi:10.1007/s41060-022-00309-0.

Cerrada, M., Trujillo, L., Hernández, D. E., Correa Zevallos, H. A., Macancela, J. C., Cabrera, D., & Vinicio Sánchez, R. (2022). AutoML for Feature Selection and Model Tuning Applied to Fault Severity Diagnosis in Spur Gearboxes. Mathematical and Computational Applications, 27(1), 6. doi:10.3390/mca27010006.

Celik, B., Singh, P., & Vanschoren, J. (2022). Online AutoML: an adaptive AutoML framework for online learning. Machine Learning. doi:10.1007/s10994-022-06262-0.

Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., & Hutter, F. (2022). Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning. Journal of Machine Learning Research, 23. doi:10.48550/arXiv.2007.04074.

Koza, J. R. (1994). Genetic programming as a means for programming computers by natural selection. Statistics and Computing, 4(2), 87–112. doi:10.1007/BF00175355.

Tran, B., Xue, B., & Zhang, M. (2019). Genetic programming for multiple-feature construction on high-dimensional classification. Pattern Recognition, 93, 404–417. doi:10.1016/j.patcog.2019.05.006.

Vanneschi, L., Castelli, M., Scott, K., & Trujillo, L. (2019). Alignment-based genetic programming for real life applications. Swarm and Evolutionary Computation, 44, 840–851. doi:10.1016/j.swevo.2018.09.006.

Moraglio, A., Krawiec, K., & Johnson, C.G. (2012). Geometric Semantic Genetic Programming. Parallel Problem Solving from Nature - PPSN XII, Lecture Notes in Computer Science, volume 7491, Springer, Berlin, Germany. doi:10.1007/978-3-642-32937-1_3.

Vanneschi, L., Castelli, M., & Silva, S. (2014). A survey of semantic methods in genetic programming. Genetic Programming and Evolvable Machines, 15(2), 195–214. doi:10.1007/s10710-013-9210-0.

Bakurov, I., Castelli, M., Gau, O., Fontanella, F., & Vanneschi, L. (2021). Genetic programming for stacked generalization. Swarm and Evolutionary Computation, 65, 100913. doi:10.1016/j.swevo.2021.100913.

Vanneschi, L., Castelli, M., Manzoni, L., & Silva, S. (2013). A New Implementation of Geometric Semantic GP and Its Application to Problems in Pharmacokinetics. Genetic Programming. EuroGP 2013. Lecture Notes in Computer Science, volume 7831, Springer, Berlin, Germany. doi:10.1007/978-3-642-37207-0_18.

Krawiec, K., & Pawlak, T. (2013). Locally geometric semantic crossover: A study on the roles of semantics and homology in recombination operators. Genetic Programming and Evolvable Machines, 14(1), 31–63. doi:10.1007/s10710-012-9172-7.

Turner, A. J., & Miller, J. F. (2015). Introducing a cross platform open source Cartesian Genetic Programming library. Genetic Programming and Evolvable Machines, 16(1), 83–91. doi:10.1007/s10710-014-9233-1.

Turner, A. J., & Miller, J. F. (2017). Recurrent Cartesian Genetic Programming of Artificial Neural Networks. Genetic Programming and Evolvable Machines, 18(2), 185–212. doi:10.1007/s10710-016-9276-6.

Miller, J. F., & Smith, S. L. (2006). Redundancy and computational efficiency in Cartesian genetic programming. IEEE Transactions on Evolutionary Computation, 10(2), 167–174. doi:10.1109/TEVC.2006.871253.

Izzo, D., Biscani, F., & Mereta, A. (2017). Differentiable Genetic Programming. Genetic Programming. EuroGP 2017, Lecture Notes in Computer Science, vol 10196. Springer, Cham, Switzerland. doi:10.1007/978-3-319-55696-3_3.

Guyon, I. (2003). Design of experiments of the NIPS 2003 variable selection benchmark. NIPS 2003 workshop on feature extraction and feature selection, 11-13 December, 2003, Whistler, United States.

Haynes, W. (2013). Wilcoxon Rank Sum Test. Encyclopedia of Systems Biology, 2354–2355, Springer, New York, United States. doi:10.1007/978-1-4419-9863-7_1185.

Fonseca, J., Douzas, G., & Bacao, F. (2021). Increasing the effectiveness of active learning: Introducing artificial data generation in active learning for land use/land cover classification. Remote Sensing, 13(13), 2619. doi:10.3390/rs13132619.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, E. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12, 2825–2830.


Full Text: PDF

DOI: 10.28991/ESJ-2023-07-04-021

Refbacks

  • There are currently no refbacks.