Mutual information based feature selection and ensemble learning for enhanced air quality index classification

Abstract

Features are an important resource in many classification protocols. However, working with all features in a given dataset requires a considerable amount of effort due to the computational complexity and increased training time. Often, one ends up with a set of features with a significant amount of dependence amidst one another. Hence it is important to develop protocols that extract the features with the least dependence between one another. This thesis delves into the application of information theoretic feature selection methods, specifically focusing on mutual information, to enhance the accuracy of predictive models for air quality classification. Specifically, three feature selection methods—Mutual Information Based Feature Selection (MIFS), Normalized Mutual Information Based Feature Selection (NMIFS), and Mutual Information Feature Selection Based on Distance-Based Elimination Strategies (DMIFS) were employed to select subsets of 6 and 8 features each from the original dataset. To benchmark the efficacy of these feature selection methods, we compared them against other state of the art methods, namely, Pearson Correlation Coefficient, Grey Wolf Optimization (GWO), and Principal Component Analysis (PCA). We utilized three classifiers: Extreme Gradient Boosting (XGBoost), K-Nearest Neighbors (KNN), and Support Vector Machine (SVM), each subjected to Optuna hyperparameter tuning and evaluated via 10-fold cross-validation. The performance of the models was evaluated using various metrics such as Accuracy, Precision, Recall, F1 Score, Mean Absolute Error (MAE), and the ROC-AUC curve. Additionally, confusion matrices were presented to provide deeper insights into model performance. The highest accuracy for XGBoost classifier was 93.8%, achieved with NMIFS using 8 features. KNN also achieved a commendable accuracy of 93.7% with DMIFS using 8 features, while SVM achieved an accuracy of 89.6% with NMIFS using 8 features. To further optimize the feature selection process, union and intersection operations were performed on the subsets of the obtained features, resulting in four distinct feature sets. In order to enhance model performance, we leveraged ensemble learning techniques. For the sake of building a robust ensemble model, we employed three base learners including XGBoost, KNN, and SVM. These models were integrated using stacking and majority voting techniques. The ensemble model achieved a peak accuracy of 94.2% on the union of features while the lowest accuracy achieved on the intersection of features was 93.8%. The results demonstrate that feature selection significantly reduces computational complexity while maintaining high prediction accuracy. The NMIFS method, in particular, achieved the highest accuracy scores. These findings underscore the importance of feature selection in improving the efficiency and performance of predictive models in air quality monitoring. By focusing on the information theoretic aspects of feature selection, we demonstrate the superior accuracy and efficiency of mutual information-based methods compared to other advanced feature selection strategies. The findings highlight the critical role of selecting a subset of features with minimal interdependence, which not only enhances the predictive power but also simplifies the modeling process. This research contributes to the field of environmental informatics by offering a robust model that can aid in better understanding and managing air pollution, ultimately leading to improved public health and environmental sustainability.