An Introduction to Deep Long-Tailed Learning

Class Re-balancing

  1. Re-sampling: Re-sampling is one of the most widely used methods over the last decade for tackling class imbalance issues. The most common types of sampling done are random over-sampling (ROS) and random under-sampling (RUS). The former randomly repeats the samples from the tail classes, while the latter randomly discards samples from the head classes. Recently however, various other forms of sampling techniques with less randomness have been developed, such as class-balanced resampling and scheme-oriented sampling.
  2. Cost-sensitive Learning: Cost-sensitive learning seeks to re-balance classes by adjusting loss values for different classes during training. Recent studies have developed various cost-sensitive long-tailed learning methods to handle class imbalance, including class-level re-weighting and class-level re-margining. Class-level re-weighting directly uses label frequencies of training samples for loss re-weighting. Class-level re-margining seeks to handle class imbalance by adjusting the minimal margin (i.e., distance), between the learned features and the model classifier, for different classes [1].
  3. Logit Adjustment: Logit adjustment is a technique that shifts the model logits based on label frequencies. Recently, Menon et al. at Google published a study called “Long-Tail Learning via Logit Adjustment” which proved that logit adjustment is Fisher consistent to minimize the average per-class error.

Information Augmentation

  1. Transfer Learning: Transfer learning seeks to transfer the knowledge from a source domain (e.g., datasets, tasks or classes) to enhance model training on a target domain. In deep long-tailed learning, there are four main transfer learning schemes: head-to-tail knowledge transfer, model pre-training, knowledge distillation, and self-training [1].
  2. Data Augmentation: Data augmentation is essentially a set of techniques used to create more instances of training data from the existing training data itself, which enhances the size and quality of the datasets for model training. Two groups of data augmentation methods exist in the field of Deep Long-Tailed Learning including transfer-based augmentation and conventional (non-transfer) augmentation.

Module Improvement

  1. Representation Learning: Existing representation learning methods for long-tailed learning are based on four main paradigms, i.e., metric learning, sequential training, prototype learning, and transfer learning. Metric learning aims at designing task-specific distance metrics for establishing similarity or dissimilarity between objects. An example of sequential learning is Hierarchical feature learning (HFL), which hierarchically clusters objects into visually similar groups, forming a hierarchical cluster tree. In this study, Ouyang et al. create such a cluster tree in which the model in the original node is pre-trained on ImageNet-1K; the model in each child node inherits the model parameters from its parent node and is then fine-tuned based on samples in the cluster node. In this way, the knowledge from the groups with massive classes is gradually transferred to their sub-groups with fewer classes. Prototype learning based methods seek to learn class-specific feature prototypes to enhance long-tailed learning performance. Transfer-learning based long-tailed methods that improve representation learning include SSP, LEAP, and unsupervised discovery (UD)[1][3].
  2. Classifier Design: Different designs of classifiers can also be helpful in addressing long-tailed problems. Various different designs have been explored in many studies, depending on their use-case. A few examples are RTC (Realistic Taxonomic Classifier), Casual classifier, and GIST classifier.
  3. Decoupled Training: Decoupled training decouples the learning procedure into representation learning and classifier training. In decoupled training, the main observations are twofold: (1) instance-balanced sampling is surprisingly the best strategy for representation learning; (2) the devil is in classification: re-adjusting the classifier leads to significant performance improvement in long-tailed recognition [1].
  4. Ensemble Learning: Ensemble learning based methods strategically generate and combine multiple network modules (namely, multiple experts) to solve long-tailed visual learning problems [1]. An illustration of existing ensemble-based long-tailed learning methods is shown below:

--

--

--

Data Science & Machine Learning

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Getting started with MONK

Matrix multiplication and inversion

An In-Depth Tutorial on the F-Score For NER

A picture of some text with named entities encircled

Facebook Research just published an awesome paper on learning hierarchical representations

Adam: The Birthchild of AdaGrad and RMSProp

Main Challenges in Image Classification

Unsupervised NLP task in Python with doc2vec

Cybersecurity problem solving with explainable machine learning

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Samer Baslan

Samer Baslan

Data Science & Machine Learning

More from Medium

The SwAV method

Paper Review: Denoising Diffusion Probabilistic Models

A survey on Meta Learning in Neural Networks

Paper Review: Grokking-generalization and over-fitting