Abstract

Deep learning models often struggle with datasets exhibiting long-tailed distributions, where the majority of data is concentrated in a few categories, leaving many with very few samples. This imbalance results in models favouring well-represented categories, leading to poorer performance for those with fewer instances. Existing methodologies focus on addressing class-wise imbalance but disregard the attribute-wise disparities. By assigning equal weight to each instance within a class, these approaches overlook the long-tailed distribution of attributes, thus underrepresenting information from infrequent attributes. The reduction in feature diversity consequently diminishes model performance. To address this challenge, we introduce an innovative methodology, namely Key Attribute Learning (KAL). It emphasises the importance of less common attributes by utilising the Instance Diversity Index (IDI) to assess and prioritise attribute diversity for each instance. KAL effectively expands feature margins among categories and addresses the overfitting problem. Our results demonstrate that KAL is non-invasive in both single-model and Mixture of Experts (MoE) settings. Implementing our method on BalPoE, we attained state-of-the-art (SOTA) performance on CIFAR -100-im100, ImageNet-LT, and iNaturalist datasets, showcasing its broad applicability and significant improvements across both balanced and diverse test distributions. Co-Author: Jungong Han*, Xiang Chang, Changrui Chen, Changjing Shang, Qiang Shen